# ZenML changelog > An MLOps framework for building portable, production-ready machine learning pipelines. - Vendor: ZenML - Category: AI - Official site: https://www.zenml.io - Tracked by: What's New (https://whatsnew.fyi/product/zenml) - Harvested from: GitHub (zenml-io/zenml) - Entries below: 10 (newest first) What's New is an index, not a publisher: every entry below links to the vendor's own release notes, which are the authoritative source. Entries are labelled where they are hand-curated sample data, pre-releases, or drawn from a secondary source such as a developer blog. Reuse: the summaries, labels and curation here are © What's New. Quote freely with attribution and a link back; wholesale republication of the corpus is not permitted — terms: https://whatsnew.fyi/terms. The vendors' own release notes remain their publishers'. ## Releases ### 0.96.3 - Date: 2026-08-07 - Version: 0.96.3 - Original notes: https://github.com/zenml-io/zenml/releases/tag/0.96.3 - Permalink: https://whatsnew.fyi/product/zenml/releases/0.96.3 - **added** — Command steps can now run across multiple Kubernetes pods with the Kubernetes step operator by setting pod_count in KubernetesStepOperatorSettings to launch the step as an indexed job - **added** — ZenML now includes a local Docker sandbox with a unified settings model for containerized sandboxes, supporting file upload and download for testing containerized ZenML behavior locally - **added** — The Helm chart now supports server.pro.enrollmentKeySecretRef to inject ZenML Pro enrollment keys from an existing Kubernetes Secret - **added** — zenml login now accepts --stack flag to connect to a server and immediately set the active stack in one command - **added** — The dashboard timeline now shows steps that were not started because a run was cancelled and supports filtering for Not Started - **changed** — Cloudpickle-materialized artifacts now store a SHA-256 hash when written and validate that hash before loading - **changed** — Secret values in the dashboard no longer expand indefinitely in the UI and are truncated for readability with a direct copy action available - **changed** — Local SQL stores backed by SQLite now wait up to 60 seconds for write locks instead of Python's 5-second default - **fixed** — Artifact version deletion now correctly applies project scope when checking whether an artifact is unused on remote deployed servers - **fixed** — Exception reporting now includes the full traceback lineage instead of only the final exception traceback - **fixed** — Kubernetes orchestrator pods are no longer retried when the pipeline run is already in a finished state that cannot be retried - **fixed** — Prefix-based lookups now keep scoped and default filters properly constrained when matching by ID or name prefix - **fixed** — Generic and stack-deployment tokens now preserve the original service account or device provenance used to create them - **fixed** — ZenML's MLflow experiment tracker now behaves more reliably when managed runtimes inject MLflow environment variables such as MLFLOW_RUN_ID - **fixed** — Existing workspace-level API keys remain valid when a workspace service account is adopted by an organization-level service account with the same name ###### Runtime and orchestration - **Multi-pod Kubernetes step operator jobs**: Command steps can now run across multiple Kubernetes pods with the Kubernetes step operator. Set `pod_count` in `KubernetesStepOperatorSettings` to launch the step as an indexed job, making it easier to distribute command-style workloads across pods. [PR #5104](https://github.com/zenml-io/zenml/pull/5104) - **Local Docker sandbox**: ZenML now includes a local Docker sandbox, plus a unified settings model for containerized sandboxes. Local sandbox workflows also support file upload and download, making it easier to test containerized ZenML behavior locally before moving to remote infrastructure. [PR #5102](https://github.com/zenml-io/zenml/pull/5102) ###### Deployment, security, and artifact integrity - **Enrollment keys from Kubernetes Secrets**: The Helm chart now supports `server.pro.enrollmentKeySecretRef`, so ZenML Pro enrollment keys can be injected from an existing Kubernetes Secret instead of being stored inline in Helm release values. The secret reference is applied consistently to the server, migration, and worker containers. [PR #5123](https://github.com/zenml-io/zenml/pull/5123) - **Cloudpickle artifact hash validation**: Cloudpickle-materialized artifacts now store a SHA-256 hash when written and validate that hash before loading. This helps detect corrupted or unexpectedly modified artifact files earlier and fail with a clearer integrity signal. [PR #5103](https://github.com/zenml-io/zenml/pull/5103) ###### CLI and dashboard UX - **Select a stack during login**: `zenml login` now accepts `--stack`, allowing you to connect to a server and immediately set the active stack in one command. When used together with `--project`, ZenML applies the project first so the stack is resolved in the intended project context. [PR #5125](https://github.com/zenml-io/zenml/pull/5125) - **Improved secret value display**: Secret values in the dashboard no longer expand indefinitely in the UI and are truncated for readability. A direct copy action is now available, making it easier to work with long secret values without disrupting the page layout. [PR #1104](https://github.com/zenml-io/zenml-dashboard/pull/1104) - **Timeline updates for cancelled runs**: The dashboard timeline now shows steps that were not started because a run was cancelled. The timeline filter also supports filtering for `Not Started`, making cancelled or partially executed runs easier to inspect. [PR #1108](https://github.com/zenml-io/zenml-dashboard/pull/1108)
Fixed - **Fewer SQLite lock failures for local stores**: Local SQL stores backed by SQLite now wait up to 60 seconds for write locks instead of using Python’s 5-second default. This reduces `sqlite3.OperationalError: database is locked` failures when concurrent steps, such as mapped dynamic pipeline steps, finish and publish artifacts at the same time. [PR #5096](https://github.com/zenml-io/zenml/pull/5096) - **Artifact deletion on deployed servers**: Artifact version deletion now correctly applies project scope when checking whether an artifact is unused. This fixes deletion through the API on remote deployed servers where the default project is disabled. [PR #5100](https://github.com/zenml-io/zenml/pull/5100) - **Complete exception tracebacks**: Exception reporting now includes the full traceback lineage instead of only the final exception traceback. This makes chained failures easier to debug because the original cause is preserved alongside the final error. [PR #5098](https://github.com/zenml-io/zenml/pull/5098) - **Smarter Kubernetes dynamic pipeline retries**: Kubernetes orchestrator pods are no longer retried when the pipeline run is already in a finished state that cannot be retried. This avoids unnecessary pod restarts that would immediately exit with `Run is already finished.` [PR #5107](https://github.com/zenml-io/zenml/pull/5107) _[Truncated at 4000 characters — full notes: https://github.com/zenml-io/zenml/releases/tag/0.96.3]_ ### 0.96.2 - Date: 2026-07-17 - Version: 0.96.2 - Original notes: https://github.com/zenml-io/zenml/releases/tag/0.96.2 - Permalink: https://whatsnew.fyi/product/zenml/releases/0.96.2 - **added** — Dynamic pipelines now support start_after parameter when calling steps to control which concurrently launched steps should wait for others before starting - **added** — Configure whether JSON-serializable raw values passed to steps in dynamic pipelines should be treated as parameters instead of artifacts through environment variables and explicit APIs - **added** — Dynamic pipelines now support CONTINUE_ON_FAILURE execution mode to allow queued or asynchronous work to continue when an async step fails - **added** — First-class DigitalOcean integration with support for DigitalOcean Spaces artifact stores and DigitalOcean Container Registry stack components - **added** — Helm chart now exposes server logging options and OpenTelemetry settings in values for console or JSON logging, service names, OTEL endpoints, and trace/metric/log configuration - **added** — Delete artifact version data through the server API from the UI or thin clients without direct stack access - **added** — Projects now support arbitrary project_metadata on create, update, and hydrated response models - **changed** — start_after is now a reserved step keyword in dynamic pipelines - **changed** — fileio.copy() now streams cross-filesystem copies in bounded chunks instead of reading entire files into memory - **changed** — ZenML caches commonly reused project, store, stack, pipeline run, and completed step run responses in process to reduce server requests - **changed** — DAG endpoint now does less unnecessary parsing and object construction when serving large pipeline graphs - **changed** — FastAPI updated to 0.138.0, Starlette lower bound raised to 0.46.0, and OpenTelemetry packages updated for compatibility - **changed** — Removed unused fastapi_utils dependency and cleaned up deprecated FastAPI response and lifespan usage - **changed** — Dynamic pipelines now model implicit dependencies from newly launched steps to the last completed sync step - **fixed** — User tokens issued before a password change are now rejected - **fixed** — Tokens derived from API keys are tied to the key generation so rotated keys no longer leave stale sessions usable - **fixed** — Dashboard now correctly rotates API keys when a non-zero retention period is configured - **fixed** — ZenML now properly handles in-memory Docker credentials when a local Docker credential store already has entries for the same registry ###### Known issues Artifact version deletion via the API is broken in this release and will be fixed in `0.96.3`. Artifact versions can still be deleted using the python SDK: ```python from zenml.client import Client Client().delete_artifact_version(..., server_side=False) ``` ###### Dynamic pipelines - **Explicit start ordering for dynamic steps**: Dynamic pipelines now support `start_after=...` when calling steps, letting you control which concurrently launched steps should wait for others before starting. This makes it easier to model ordering constraints without turning concurrent parts of a dynamic pipeline into fully synchronous execution. Note that `start_after` is now a reserved step keyword, so steps that previously used a parameter with this name will need to be updated. [PR #4995](https://github.com/zenml-io/zenml/pull/4995) - **More flexible dynamic step inputs**: You can now configure whether JSON-serializable raw values passed to steps in dynamic pipelines should be treated as parameters instead of artifacts. The new environment-variable threshold defaults to `0` to preserve existing behavior, while explicit APIs such as `with_options(parameters=...)` and `ExternalArtifact(...)` remain available when you want to force either behavior. [PR #5079](https://github.com/zenml-io/zenml/pull/5079) - **Improved dynamic execution semantics**: Dynamic pipelines now support `CONTINUE_ON_FAILURE` execution mode, allowing already queued or asynchronous work to continue when an async step fails. ZenML also models implicit dependencies from newly launched steps to the last completed sync step, making mixed sync/async dynamic pipelines execute in a more predictable order. [PR #5052](https://github.com/zenml-io/zenml/pull/5052) ###### Integrations and deployment - **DigitalOcean integration**: ZenML now includes a first-class `digitalocean` integration with support for DigitalOcean Spaces artifact stores and DigitalOcean Container Registry stack components. Spaces support builds on the existing S3-compatible implementation while handling DigitalOcean regions and endpoint generation for you. [PR #5054](https://github.com/zenml-io/zenml/pull/5054) - **Helm chart logging and OpenTelemetry configuration**: The ZenML Helm chart now exposes server logging options and OpenTelemetry settings directly in values. You can configure console or JSON logging, service names, OTEL endpoints, and enable or disable traces, metrics, and logs without custom chart modifications. [PR #5048](https://github.com/zenml-io/zenml/pull/5048) ###### Data and metadata management - **Server-side artifact data deletion**: ZenML can now delete artifact version data through the server API, not only from a full client with direct stack access. This enables artifact metadata and backing data to be deleted from the UI or from thin clients that do not have the artifact store stack component locally available. [PR #5034](https://github.com/zenml-io/zenml/pull/5034) - **Project metadata**: Projects now support arbitrary `project_metadata` on create, update, and hydrated response models. Metadata is stored as portable JSON, preserved when omitted, replaced when explicitly supplied, and can be cleared by sending an empty object. [PR #5086](https://github.com/zenml-io/zenml/pull/5086) ###### Performance and scalability - **Lower-memory cross-filesystem copies**: `fileio.copy()` now streams cross-filesystem copies in bounded chunks instead of reading the entire file into memory. This significantly reduces peak memory usage for local-to-remote and remote-to-local artifact operations, including `PathMaterializer`, directory copies, integration materializers, and code archive upload/download flows. [PR #5031](https://github.com/zenml-io/zenml/pull/5031) - **Fewer server requests during runs**: ZenML now caches commonly reused project, store, stack, pipeline run, and completed step run _[Truncated at 4000 characters — full notes: https://github.com/zenml-io/zenml/releases/tag/0.96.2]_ ### 0.96.1 - Date: 2026-07-02 - Version: 0.96.1 - Original notes: https://github.com/zenml-io/zenml/releases/tag/0.96.1 - Permalink: https://whatsnew.fyi/product/zenml/releases/0.96.1 - **added** — SSH orchestrator and SSH step operator for executing workloads on remote machines accessible via SSH - **Run pipelines and steps over SSH**: ZenML now includes an SSH orchestrator and SSH step operator for executing workloads on remote machines accessible via SSH ([PR #4953](https://github.com/zenml-io/zenml/pull/4953)). This makes it easier to use existing servers or on-prem infrastructure as execution targets without adopting a full cluster-based backend, while still managing runs through ZenML. ##### What's Changed * Add version 0.95.1 to legacy docs by @github-actions[bot] in https://github.com/zenml-io/zenml/pull/5014 * Remove unused release notes generation by @schustmi in https://github.com/zenml-io/zenml/pull/5013 * Add missing group info import by @schustmi in https://github.com/zenml-io/zenml/pull/5012 * Add 0.96.0 to the migration tests by @github-actions[bot] in https://github.com/zenml-io/zenml/pull/5016 * Always log in to the local docker SDK by @stefannica in https://github.com/zenml-io/zenml/pull/5018 * Add OAuth service connector logo by @schustmi in https://github.com/zenml-io/zenml/pull/5019 * SSH Orchestrator + Step Operator by @htahir1 in https://github.com/zenml-io/zenml/pull/4953 * Prepare release 0.96.1 by @github-actions[bot] in https://github.com/zenml-io/zenml/pull/5020 **Full Changelog**: https://github.com/zenml-io/zenml/compare/0.96.0...0.96.1 ### 0.96.0 - Date: 2026-07-02 - Version: 0.96.0 - Original notes: https://github.com/zenml-io/zenml/releases/tag/0.96.0 - Permalink: https://whatsnew.fyi/product/zenml/releases/0.96.0 - **added** — Add Trackio experiment tracking integration for logging experiment data through Trackio's public API - **added** — Add Backblaze B2 as an S3-compatible artifact store option - **added** — Add baseten step operator flavor for running GPU workloads as Baseten Training jobs - **added** — Add generic OAuth2 service connector for authenticating external services using static tokens, client credentials, or client ID with refresh token - **added** — Add step_default_input_overrides parameter to override step inputs by name during pipeline replay - **added** — Add trigger cycle protection to detect and skip cyclic execution loops in Platform Event Trigger chains - **added** — Add optional sandbox cleanup configuration to destroy sandboxes automatically when sessions exit - **added** — Add configurable retry behavior for Kubernetes API calls in Kubernetes-based deployments - **changed** — Raise minimum supported transformers version for Hugging Face compatibility - **changed** — Update Azure integration to require newer Azure dependency versions and remove support for deprecated azureml-core library - **changed** — Restrict service account and API key management to admins only in open-source ZenML server deployments without RBAC - **changed** — Improve dashboard Timeline View with additional filtering options and more efficient connector selector - **changed** — Optimize pipeline sorting by latest run query to avoid scanning all runs on large deployments - **changed** — Improve efficiency of run, artifact, DAG, and model version queries to load only necessary data - **changed** — Change expired API transaction cleanup to use bounded catch-up passes instead of fixed single deletes ###### Known Issues If the Docker credential helper contains invalid credentials for a registry that ZenML is trying to use, those will have precendence over the credentials set by the service connector and will cause the following error: `RuntimeError: Docker error: denied: Your authorization token has expired. Reauthenticate and try again..`. This will be fixed in the next release, in the meantime `docker logout ` should fix it. ###### Breaking Changes * The minimum supported `transformers` version has been raised. If you use ZenML with Hugging Face/`transformers`, update your environment and dependency pins to a newer compatible `transformers` release before upgrading ZenML. [PR #4976](https://github.com/zenml-io/zenml/pull/4976) * The Azure integration now requires newer Azure dependency versions, and support for the deprecated `azureml-core` library has been fully removed. If you use ZenML on Azure, update your Azure-related dependency pins and migrate any remaining `azureml-core` usage to the currently supported Azure SDK packages before upgrading. [PR #4987](https://github.com/zenml-io/zenml/pull/4987) * In open-source ZenML server deployments without RBAC enabled, service account and API key management is now restricted to admins only. Non-admin users will no longer be able to manage service accounts or API keys they previously created, so move any required credentials and automation to admin-managed accounts as part of your upgrade. [PR #5007](https://github.com/zenml-io/zenml/pull/5007) ###### New integrations and execution backends - **Trackio experiment tracking**: ZenML now includes a Trackio experiment tracker integration, allowing pipelines to log experiment data through Trackio’s public API. This makes it easier to manage trial results and connect ZenML runs with Hugging Face-backed Trackio workflows such as datasets, spaces, and buckets. [PR #4841](https://github.com/zenml-io/zenml/pull/4841) - **Backblaze B2 artifact store**: You can now configure Backblaze B2 as a ZenML artifact store. This adds another S3-compatible storage option for teams that want to store pipeline artifacts in Backblaze infrastructure. [PR #4791](https://github.com/zenml-io/zenml/pull/4791) - **Baseten step operator**: ZenML now supports a `baseten` step operator flavor for running GPU workloads as Baseten Training jobs. It supports regular single-node steps with ZenML artifacts and logs, as well as multi-node distributed training through command steps that can consume Baseten’s distributed training environment variables. [PR #4973](https://github.com/zenml-io/zenml/pull/4973) - **Generic OAuth2 service connector**: A new OAuth2 service connector lets you authenticate external services using a static token, client credentials, or a client ID with refresh token. This provides a reusable connector option for services that expose OAuth2-based authentication. [PR #4992](https://github.com/zenml-io/zenml/pull/4992) ###### Workflow controls and platform operations - **Replay input overrides by step name**: When replaying a run, you can now use `step_default_input_overrides` to override a step input for every invocation of a step with the same name. Per-invocation `step_input_overrides` still take precedence, giving you both broad and targeted control during replay. [PR #4978](https://github.com/zenml-io/zenml/pull/4978) - **Trigger cycle protection**: ZenML now detects execution loops in Platform Event Trigger chains at the pipeline level. Cyclic trigger dispatches are skipped with the new `SKIPPED_TRIGGER_CYCLE` status, while unrelated downstream dispatches can continue normally and the affected cycle can be inspected through the SDK. [PR #4971](https://github.com/zenml-io/zenml/pull/4971) - **Optional sandbox cleanup on exit**: Sandbox sessions can now be configured to destroy the sandbox automatically when the session exits. The option defaults to _[Truncated at 4000 characters — full notes: https://github.com/zenml-io/zenml/releases/tag/0.96.0]_ ### 0.95.1 - Date: 2026-06-18 - Version: 0.95.1 - Original notes: https://github.com/zenml-io/zenml/releases/tag/0.95.1 - Permalink: https://whatsnew.fyi/product/zenml/releases/0.95.1 - **fixed** — Fixed an issue where running a step with a step operator in a dynamic pipeline could fail unless that step was explicitly listed in pipeline.depends_on - **changed** — ZenML now falls back to the orchestrator image when steps use step operators in dynamic pipelines, matching the behavior used for isolated steps without step operators - **changed** — Improved database query performance by adjusting how related data is loaded for common pipeline run and step run queries - **fixed** — Fixed a deadlock that could happen during logging context shutdown when using fsspec-based artifact log stores with debug logs enabled - **changed** — ZenML now avoids writing back into the log store during shutdown to improve reliability for pipeline and step log collection ###### Dynamic pipelines with step operators Dynamic pipeline execution is more reliable when steps use step operators.
Fixed - Fixed an issue where running a step with a step operator in a dynamic pipeline could fail unless that step was explicitly listed in `pipeline.depends_on`. - ZenML now falls back to the orchestrator image in this case, matching the behavior already used for isolated steps without step operators. [PR #4960](https://github.com/zenml-io/zenml/pull/4960)
###### Faster pipeline and step run queries Common pipeline run and step run views should now load more efficiently, especially on larger deployments. - Improved database query performance by adjusting how related data is loaded for common pipeline run and step run queries. - This helps avoid expensive query plans in MySQL for paginated run listings, making these queries more scalable. [PR #4965](https://github.com/zenml-io/zenml/pull/4965) ###### Logging stability Logging shutdown is now safer when using artifact-backed log stores.
Fixed - Fixed a deadlock that could happen during logging context shutdown when using `fsspec`-based artifact log stores with debug logs enabled. - ZenML now avoids writing back into the log store during shutdown, improving reliability for pipeline and step log collection. [PR #4964](https://github.com/zenml-io/zenml/pull/4964)
##### What's Changed * Add version 0.94.6 to legacy docs by @github-actions[bot] in https://github.com/zenml-io/zenml/pull/4957 * Add 0.95.0 to the migration tests by @github-actions[bot] in https://github.com/zenml-io/zenml/pull/4959 * Exclude `zenml` from its own dependency audit by @strickvl in https://github.com/zenml-io/zenml/pull/4961 * Fallback to orchestrator image by @schustmi in https://github.com/zenml-io/zenml/pull/4960 * Fix logging shutdown deadlock by @schustmi in https://github.com/zenml-io/zenml/pull/4964 * Improve DB query plans by using selectinload by @schustmi in https://github.com/zenml-io/zenml/pull/4965 * Prepare release 0.95.1 by @github-actions[bot] in https://github.com/zenml-io/zenml/pull/4966 **Full Changelog**: https://github.com/zenml-io/zenml/compare/0.95.0...0.95.1 ### 0.95.0 - Date: 2026-06-17 - Version: 0.95.0 - Original notes: https://github.com/zenml-io/zenml/releases/tag/0.95.0 - Permalink: https://whatsnew.fyi/product/zenml/releases/0.95.0 - **added** — Define steps and hooks with async def; ZenML now runs async functions on a fresh event loop for both normal and dynamic pipeline usage - **added** — Run arbitrary commands as pipeline steps with CommandStep(...), including non-Python commands and Python callables that do not require ZenML in the execution environment - **added** — Invoke deployments asynchronously: a new deployment endpoint can submit a pipeline run and return immediately instead of waiting for completion - **added** — Added the core Sandbox stack component abstraction for running untrusted or generated code in isolated sessions, including a built-in local flavor for subprocess-based execution - **added** — Added a kubernetes sandbox flavor where each sandbox session runs in a dedicated Kubernetes pod, with streamed command execution and support for re-attaching to running sessions - **added** — Added a Modal orchestrator flavor so complete ZenML pipelines can run on Modal, using Modal sandboxes for orchestration and step execution - **changed** — ZenML now supports Python 3.14, and environments using the local or server extras must accommodate the SQLModel upgrade from 0.18.0 to 0.38.0 - **changed** — Local MLflow tracking now uses a SQLite backend by default when no tracking_uri is configured - **changed** — ZenML now requires opentelemetry-sdk==1.40.0 instead of 1.38.0 - **changed** — Step and pipeline hooks have been reworked into a new lifecycle-based hook system with persisted hook invocation records - **changed** — ZenML server rate limiting no longer trusts raw X-Forwarded-For headers by default - **changed** — CLI list commands now return the newest items first by default instead of the oldest first - **changed** — Kubernetes deployments now merge pod_settings.resources into the deployment template context - **changed** — Databricks-managed MLflow deployments now support machine-to-machine OAuth authentication via service principals - **changed** — Improved list endpoint ordering so descending sorts can use matching index scans instead of forcing expensive mixed-direction database sorts - **changed** — ZenML now requires modal>=1.4.0,<2.0.0 when using the Modal integration - **removed** — The deprecated singular tag field has been removed from TaggableFilters - **fixed** — Pipeline execution may now raise different exception types depending on how step futures are awaited - **added** — Added targeted database indexes for common pagination and hydration query patterns across pipeline runs, snapshots, step configurations, step runs, and artifact versions - **changed** — Adjusted request timeout behavior so only deduplicated/cacheable requests may return a timeout or backpressure response while work continues in the background ###### Known Issues * Enabling debug logs while having a GCP, Azure or S3 artifact log store will cause a deadlock at the end of a pipeline or step run. Fixed in `0.95.1`. ###### Breaking Changes * [PR #4844](https://github.com/zenml-io/zenml/pull/4844): ZenML now supports Python 3.14, and environments using the `local` or `server` extras must also accommodate the SQLModel upgrade from 0.18.0 to 0.38.0. If you depend on those extras, review and update any pinned SQLModel-related dependencies before upgrading. * [PR #4900](https://github.com/zenml-io/zenml/pull/4900): Local MLflow tracking now uses a SQLite backend by default when no `tracking_uri` is configured. New tracking metadata is stored in `/mlflow.db` and artifacts under the local artifact store, so users relying on the previous default local MLflow layout or behavior should update their local setup and migration expectations. * [PR #4790](https://github.com/zenml-io/zenml/pull/4790): ZenML now requires `opentelemetry-sdk==1.40.0` instead of 1.38.0. If your environment pins OpenTelemetry packages, update them to compatible versions before upgrading ZenML. * [PR #4875](https://github.com/zenml-io/zenml/pull/4875): Step and pipeline hooks have been reworked into a new lifecycle-based hook system with persisted hook invocation records. If you use hooks or related internal APIs, review your existing integrations and update them to the new hook semantics and lifecycle events. * [PR #4919](https://github.com/zenml-io/zenml/pull/4919): ZenML server rate limiting no longer trusts raw `X-Forwarded-For` headers by default. If you run ZenML behind an ingress or reverse proxy, make sure proxy header handling is explicitly configured so login rate limiting continues to use the correct client IPs. * [PR #4459](https://github.com/zenml-io/zenml/pull/4459): CLI `list` commands now return the newest items first by default instead of the oldest first. If you have scripts or workflows that assumed the previous ordering, update them to explicitly sort or handle the new default order. * [PR #4566](https://github.com/zenml-io/zenml/pull/4566): The deprecated singular `tag` field has been removed from `TaggableFilter`s. Update any API or client code to use the supported tag filtering format instead of passing a single `tag` value. * [PR #4950](https://github.com/zenml-io/zenml/pull/4950): Pipeline execution may now raise different exception types depending on how step futures are awaited. If you catch exceptions around pipeline execution, review and update your error-handling logic to account for `StepExecutionException` being raised in implicit await scenarios. * [PR #4867](https://github.com/zenml-io/zenml/pull/4867): ZenML now requires `modal>=1.4.0,<2.0.0` when using the Modal integration. ###### New ways to run code and pipelines This release expands how you can execute work in ZenML, from async Python to arbitrary commands and new remote execution backends. - Define steps and hooks with `async def`; ZenML now runs async functions on a fresh event loop for both normal and dynamic pipeline usage. [PR #4913](https://github.com/zenml-io/zenml/pull/4913) - Run arbitrary commands as pipeline steps with `CommandStep(...)`, including non-Python commands and Python callables that do not require ZenML in the execution environment. [PR #4904](https://github.com/zenml-io/zenml/pull/4904) - Invoke deployments asynchronously: a new deployment endpoint can submit a pipeline run and return immediately instead of waiting for completion. [PR #4906](https://github.com/zenml-io/zenml/pull/4906) ###### Sandboxes and Modal execution ZenML now includes the core sandbox abstraction for isolated execution, plus new backend support for Kubernetes and Modal-based workloads. - Added the core `Sandbox` stack component abstraction for running untrusted or generated code in isolated sessions, including a _[Truncated at 4000 characters — full notes: https://github.com/zenml-io/zenml/releases/tag/0.95.0]_ ### 0.94.6 - Date: 2026-06-02 - Version: 0.94.6 - Original notes: https://github.com/zenml-io/zenml/releases/tag/0.94.6 - Permalink: https://whatsnew.fyi/product/zenml/releases/0.94.6 - **fixed** — Fixed GCP service connector failures when connecting to private GKE clusters that use Google's DNS-based control plane endpoint by connecting using the same method as gcloud container clusters get-credentials --dns-endpoint - **fixed** — Resolved an issue where ZenML would fail when no container engine was available even when users explicitly set skip_build=True in their DockerSettings - **fixed** — Include hook sources as string in snapshot config template ###### Infrastructure & Deployment Improvements - **Enhanced GKE Private Cluster Support**: Fixed GCP service connector failures when connecting to private GKE clusters that use Google's DNS-based control plane endpoint. ZenML now connects using the same method as `gcloud container clusters get-credentials --dns-endpoint`, ensuring reliable access to private clusters. [PR #4856](https://github.com/zenml-io/zenml/pull/4856)
Fixed - **Docker Build Requirements**: Resolved an issue where ZenML would fail when no container engine was available, even when users explicitly set `skip_build=True` in their DockerSettings. The build checksum computation now correctly respects the skip_build flag. [PR #4879](https://github.com/zenml-io/zenml/pull/4879)
##### What's Changed * Add version 0.94.4 to legacy docs by @github-actions[bot] in https://github.com/zenml-io/zenml/pull/4872 * Add 0.94.5 to the migration tests by @github-actions[bot] in https://github.com/zenml-io/zenml/pull/4874 * Don't require container engine when specifying skip_build by @schustmi in https://github.com/zenml-io/zenml/pull/4879 * Fix GKE private cluster access via DNS control plane endpoint in GCP service connector by @stefannica in https://github.com/zenml-io/zenml/pull/4856 * Include hook sources as string in snapshot config template by @schustmi in https://github.com/zenml-io/zenml/pull/4880 * Prepare release 0.94.6 by @github-actions[bot] in https://github.com/zenml-io/zenml/pull/4883 **Full Changelog**: https://github.com/zenml-io/zenml/compare/0.94.5...0.94.6 ### 0.94.5 - Date: 2026-05-29 - Version: 0.94.5 - Original notes: https://github.com/zenml-io/zenml/releases/tag/0.94.5 - Permalink: https://whatsnew.fyi/product/zenml/releases/0.94.5 - **added** — Stream custom events in real-time from running pipelines via zenml.streaming.publish() with Server-Sent Events (SSE) support and Redis-based broker with automatic catch-up, gap signaling, and idle cleanup - **added** — Add POST /api/v1/runs/statistics endpoint to query aggregated metrics across pipeline runs with grouping by status, pipeline, stack, user, time buckets, metadata values, and tags - **added** — Weights & Biases experiment tracker automatically adds ZenML pipeline and step metadata to W&B runs and records W&B identifiers back to ZenML step metadata with configurable custom groups, job types, run configs, and run IDs - **added** — Add example demonstrating agentic human-in-the-loop pipelines with dynamic planning, task fan-out via step.map(), result summarization, human approval gates with zenml.wait(), and conditional branching - **added** — ZenML server supports structured logging with OpenTelemetry instrumentation and ZENML_CONSOLE_LOGGING_FORMAT environment variable with console, json, or text format options - **added** — Step and pipeline definitions now support string type annotations, either explicitly quoted or via from __future__ import annotations - **added** — Python package installations during Docker image building can now mount a build cache for faster builds - **added** — Interactive wait condition input with string schemas now accepts raw strings without quotes with automatic detection and handling - **changed** — Restored custom console log formatting for non-DEBUG output and added step-name prefixes in terminal output during execution - **changed** — ZENML_LOGGING_FORMAT variable is now deprecated in favor of ZENML_CONSOLE_LOGGING_FORMAT - **changed** — Improved scrolling behavior for the pipeline timeline view in dashboard - **fixed** — Pipeline runs now properly fail when the user pipeline function cannot be imported instead of staying in running state indefinitely - **fixed** — Keyboard interrupts during active wait conditions are now handled more gracefully without unclear error messages when server transitions to STOPPED status - **fixed** — Fixed user update endpoint bug where admins couldn't activate or deactivate user accounts due to incorrect field assignment ###### 🚀 Live Event Streaming for Pipeline Runs You can now stream custom events in real-time from your running pipelines! Call `zenml.streaming.publish()` from inside any step or dynamic pipeline to push events that can be consumed via Server-Sent Events (SSE). Enable this feature by setting `stream_broker_implementation_source` in your server configuration. The initial implementation includes a Redis-based broker with automatic catch-up, gap signaling, and idle cleanup. [PR #4804](https://github.com/zenml-io/zenml/pull/4804) ###### 📊 Pipeline Run Statistics Endpoint A new `POST /api/v1/runs/statistics` endpoint lets you query aggregated metrics across your pipeline runs. Group by status, pipeline, stack, user, time buckets (hour/day/week/month), metadata values, tags, and more. Calculate averages, sums, min/max over duration, step counts, cached steps, output artifacts, or custom numeric metadata. Perfect for building dashboards and analytics. [PR #4860](https://github.com/zenml-io/zenml/pull/4860) ###### 🎯 Richer Weights & Biases Integration The W&B experiment tracker now automatically adds ZenML pipeline and step metadata to your W&B runs, groups runs by pipeline execution, and records W&B identifiers back to ZenML step metadata. You can now configure custom groups, job types, run configs, explicit or deterministic run IDs, resume behavior, and pass through additional `wandb.init` kwargs for complete control over your experiment tracking. [PR #4838](https://github.com/zenml-io/zenml/pull/4838) ###### 🤖 Agentic Human-in-the-Loop Pipeline Example A new example demonstrates building dynamic agentic pipelines with human approval gates. The example shows how to plan agent tasks, fan them out with `step.map()`, summarize results, pause execution with `zenml.wait()` for human review, and branch the final action based on the decision. Includes clean lineage tracking with `Annotated` step outputs. [PR #4849](https://github.com/zenml-io/zenml/pull/4849) ###### 📝 Structured Logging and OpenTelemetry Instrumentation The ZenML server now supports structured logging with OpenTelemetry instrumentation. Configure console output with the new `ZENML_CONSOLE_LOGGING_FORMAT` environment variable, choosing between `console`, `json`, or `text` formats. Server logs use a clean structured layout with timestamps, levels, logger context, and optional JSON fields. [PR #4781](https://github.com/zenml-io/zenml/pull/4781) ###### 🛠️ Enhanced Developer Experience - **String Type Annotations**: Step and pipeline definitions now support string annotations, either explicitly quoted or via `from __future__ import annotations`. [PR #4843](https://github.com/zenml-io/zenml/pull/4843) - **Build Cache Mounting**: Python package installations during Docker image building can now mount a build cache for faster builds. [PR #4820](https://github.com/zenml-io/zenml/pull/4820) - **Improved Wait Condition Input**: When resolving wait conditions interactively with string schemas, you can now input raw strings without quotes—ZenML detects and handles this automatically. [PR #4845](https://github.com/zenml-io/zenml/pull/4845) - **Better Logging Control**: Restored custom console log formatting for non-DEBUG output, added step-name prefixes in terminal output during execution (disable with `ZENML_DISABLE_STEP_NAMES_IN_LOGS=true`), and kept stored logs clean and unformatted. The `ZENML_LOGGING_FORMAT` variable is now deprecated in favor of `ZENML_CONSOLE_LOGGING_FORMAT`. [PR #4851](https://github.com/zenml-io/zenml/pull/4851) ###### 🎨 Dashboard Improvements - Improved scrolling behavior for the pipeline timeline view. [PR #1053](https://github.com/zenml-io/zenml-dashboard/pull/1053)
Fixed - Pipeline runs now properly fail when the user pipeline function cannot be imported (e.g., due to missing packages), instead of staying in a `running` state indefinitely. [PR #4832](ht _[Truncated at 4000 characters — full notes: https://github.com/zenml-io/zenml/releases/tag/0.94.5]_ ### 0.94.4 - Date: 2026-05-12 - Version: 0.94.4 - Original notes: https://github.com/zenml-io/zenml/releases/tag/0.94.4 - Permalink: https://whatsnew.fyi/product/zenml/releases/0.94.4 - **added** — Add new Databricks step operator to run individual pipeline steps on Databricks - **added** — Support nested dynamic pipelines, allowing one dynamic pipeline to be called from within another - **added** — Add advanced training workload settings to Run:AI step operator including multiple mount types, workload templates, security context settings, port declarations, and parallelism configuration - **added** — Add parent run information display in the dashboard run details view - **changed** — Improve Kubernetes string handling with separate sanitization for DNS-style names and looser label-value rules for metadata - **changed** — Increase maximum allowed size for ZenML secrets stored in SQL secrets store to 64KB - **changed** — Add optional tag settings to Databricks orchestrator to label jobs and cluster resources for cost tracking, ownership, and governance - **fixed** — Fix signal handler registration during step execution to prevent errors when running many sync steps or interrupting pipelines - **fixed** — Fix redundant RBAC permission checks during response model dehydration - **fixed** — Allow step functions to use keyword-only arguments without causing failures - **fixed** — Fix Kubernetes orchestrator step pod configuration by replacing hardcoded lookups with proper settings calls - **fixed** — Improve Kubernetes job failure diagnostics to provide richer diagnostic information when dynamic pipeline jobs fail due to system issues ###### New Databricks Step Operator You can now run individual pipeline steps on Databricks using the new **Databricks step operator** [PR #4648](https://github.com/zenml-io/zenml/pull/4648). This is useful when you want specific steps to execute in the Databricks runtime while the rest of your pipeline uses a different orchestrator. The Databricks orchestrator also now supports optional tag settings to label jobs and cluster resources for cost tracking, ownership, and governance. ###### Nested Dynamic Pipelines Dynamic pipelines can now be nested, allowing you to call one dynamic pipeline from within another [PR #4775](https://github.com/zenml-io/zenml/pull/4775). This enables more modular and reusable pipeline designs. ###### Enhanced Run:AI Training Workload Configuration The Run:AI step operator now supports advanced training workload settings [PR #4780](https://github.com/zenml-io/zenml/pull/4780), including: - Multiple mount types (PVC, ConfigMap, Secret, NFS, S3, HostPath) - Workload templates via `workload_template_id` - Security context settings (UID/GID, non-root execution, seccomp, capabilities) - Port declarations and external URL exposure - Training workload `parallelism` and `completions` ###### Improved Kubernetes Job Failure Diagnostics When dynamic pipeline jobs fail due to system issues (such as OOM kills), ZenML now provides richer diagnostic information [PR #4800](https://github.com/zenml-io/zenml/pull/4800). This makes it easier to understand why Kubernetes terminated your pods. ###### Better Kubernetes Label Handling Kubernetes string handling has been improved with separate sanitization for DNS-style names/keys and looser label-value rules for metadata like run, pipeline, and step IDs [PR #4756](https://github.com/zenml-io/zenml/pull/4756). This makes it easier to navigate through runs in Kubernetes. ###### Increased Secret Size Limit The maximum allowed size for ZenML secrets stored in the SQL secrets store has been increased to 64KB [PR #4769](https://github.com/zenml-io/zenml/pull/4769). The limit applies to the combined size of all keys and values in a secret object. ###### Dashboard: Parent Run Display The dashboard now displays parent run information in the run details view when available [PR #1050](https://github.com/zenml-io/zenml-dashboard/pull/1050).
Fixed - **Signal handling during step execution**: Signal handlers are now properly unregistered after step execution, preventing strange errors when running many sync steps or interrupting pipelines [PR #4784](https://github.com/zenml-io/zenml/pull/4784). Dynamic pipeline steps running in isolated environments or step operators are no longer affected by signal handling from the orchestrator. - **RBAC performance**: Fixed redundant RBAC permission checks during response model dehydration [PR #4797](https://github.com/zenml-io/zenml/pull/4797). Previously, when permissions were already prefetched and denied, additional RBAC requests were sent unnecessarily for each sub-model. - **Keyword-only arguments in steps**: Step functions can now use keyword-only arguments without causing failures [PR #4798](https://github.com/zenml-io/zenml/pull/4798). - **Kubernetes orchestrator settings**: Fixed step pod configuration by replacing legacy hardcoded `orchestrator.kubernetes` lookups with proper `orchestrator.get_settings(...)` calls [PR #4803](https://github.com/zenml-io/zenml/pull/4803). Step pods now correctly apply orchestrator settings from canonical component keys while maintaining backward compatibility.
##### What's Changed * Add version 0.94.2 to legacy docs by @github-actions[bot] in https://github.com/zenml-io/zenml/pull/4766 * Add 0.94.3 to the migration tests by @github-actions[bot] in https://github.com/zenml-io/zenml/pull/4768 * Revise Run:AI service account setup and troubleshoo _[Truncated at 4000 characters — full notes: https://github.com/zenml-io/zenml/releases/tag/0.94.4]_ ### 0.94.3 - Date: 2026-04-24 - Version: 0.94.3 - Original notes: https://github.com/zenml-io/zenml/releases/tag/0.94.3 - Permalink: https://whatsnew.fyi/product/zenml/releases/0.94.3 - **added** — Add container engine abstraction and podman support - **added** — Include wait condition resolved at in DAG response - **added** — Add resource pools - **added** — Add event triggers - **added** — Add helper endpoint for trigger SourceType supported events - **added** — Add memory_call step type - **added** — Add acknowledgeable trigger dispatch status - **added** — Support max-runs stop criteria - **added** — Add Kubernetes gateway support - **added** — Add server-side replays - **added** — Allow multiple components of the same type for a stack - **changed** — Replace darglint with pydoclint for docstring linting - **changed** — Improve CLI commands for trigger dispatch status - **fixed** — Fix zizmor SHA/version mismatch - **fixed** — Fix broken docs header for triggers - **fixed** — Fix empty PR number in Claude Code review workflow - **fixed** — Fix making a secret public using CLI/client - **fixed** — Fix run status update condition - **fixed** — Fix StrEnum import breaking Python 3.10 - **fixed** — Fix python package fetching ##### What's Changed * Add version 0.94.1 to legacy docs by @github-actions[bot] in https://github.com/zenml-io/zenml/pull/4701 * Bump the minor-and-patch group with 2 updates by @dependabot[bot] in https://github.com/zenml-io/zenml/pull/4705 * Fix zizmor SHA/version mismatch and add zizmor to `lint.sh` by @strickvl in https://github.com/zenml-io/zenml/pull/4703 * Fix broken docs header - triggers by @Json-Andriopoulos in https://github.com/zenml-io/zenml/pull/4707 * Misc kitaru extensions by @schustmi in https://github.com/zenml-io/zenml/pull/4715 * Introduce a container engine abstraction and add podman support by @stefannica in https://github.com/zenml-io/zenml/pull/4651 * Include wait condition resolved at in DAG response by @schustmi in https://github.com/zenml-io/zenml/pull/4717 * Replace darglint with pydoclint for docstring linting by @strickvl in https://github.com/zenml-io/zenml/pull/4721 * Bump the minor-and-patch group with 4 updates by @dependabot[bot] in https://github.com/zenml-io/zenml/pull/4725 * Fix empty PR number in Claude Code review workflow by @strickvl in https://github.com/zenml-io/zenml/pull/4735 * Resource pools by @schustmi in https://github.com/zenml-io/zenml/pull/4465 * Feature/event triggers by @Json-Andriopoulos in https://github.com/zenml-io/zenml/pull/4692 * Helper endpoint: Trigger SourceType supported events by @Json-Andriopoulos in https://github.com/zenml-io/zenml/pull/4745 * Don't resolve zenml pypi version on flavor model generation by @stefannica in https://github.com/zenml-io/zenml/pull/4744 * Document Kubernetes permissions for various roles in running k8s pipelines by @stefannica in https://github.com/zenml-io/zenml/pull/4746 * Add docs for on-prem deployment of optional Pro features by @stefannica in https://github.com/zenml-io/zenml/pull/4751 * Remove the unique constraint from the `LogsSchema` by @bcdurak in https://github.com/zenml-io/zenml/pull/4729 * Add `memory_call` step type by @strickvl in https://github.com/zenml-io/zenml/pull/4724 * Add acknowledgeable trigger dispatch status and improve CLI commands by @stefannica in https://github.com/zenml-io/zenml/pull/4743 * Support for max-runs stop criteria by @Json-Andriopoulos in https://github.com/zenml-io/zenml/pull/4752 * Fix making a secret public using CLI/client by @schustmi in https://github.com/zenml-io/zenml/pull/4755 * Feature/add k8s gateway support by @safoinme in https://github.com/zenml-io/zenml/pull/4726 * Fix run status update condition by @schustmi in https://github.com/zenml-io/zenml/pull/4747 * Fix broken tests by @Json-Andriopoulos in https://github.com/zenml-io/zenml/pull/4758 * Fix StrEnum import breaking 3.10 by @Json-Andriopoulos in https://github.com/zenml-io/zenml/pull/4763 * Don't block while waiting for concurrent step inputs by @schustmi in https://github.com/zenml-io/zenml/pull/4699 * Fix python package fetching by @schustmi in https://github.com/zenml-io/zenml/pull/4762 * Allowing multiple components of the same type for a stack by @bcdurak in https://github.com/zenml-io/zenml/pull/4671 * Server-side replays by @schustmi in https://github.com/zenml-io/zenml/pull/4716 * Fix pydantic materializer double encoding by @schustmi in https://github.com/zenml-io/zenml/pull/4760 * Prepare release 0.94.3 by @github-actions[bot] in https://github.com/zenml-io/zenml/pull/4765 **Full Changelog**: https://github.com/zenml-io/zenml/compare/0.94.2...0.94.3