ZenML

AI

An MLOps framework for building portable, production-ready machine learning pipelines.

Latest 0.96.3 · by ZenMLWebsitezenml-io/zenml

Release activity

Release activity — 10 releases across 9 days since Apr 24, 2026. Each cell is one day; darker means more releases that day. Nothing is recorded before Apr 24, 2026. Older weeks are hidden at this screen width.
MayJunJulAug
SundayNo releases on Apr 26, 2026No releases on May 3, 2026No releases on May 10, 2026No releases on May 17, 2026No releases on May 24, 2026No releases on May 31, 2026No releases on Jun 7, 2026No releases on Jun 14, 2026No releases on Jun 21, 2026No releases on Jun 28, 2026No releases on Jul 5, 2026No releases on Jul 12, 2026No releases on Jul 19, 2026No releases on Jul 26, 2026No releases on Aug 2, 2026No releases on Aug 9, 2026
MondayNo releases on Apr 27, 2026No releases on May 4, 2026No releases on May 11, 2026No releases on May 18, 2026No releases on May 25, 2026No releases on Jun 1, 2026No releases on Jun 8, 2026No releases on Jun 15, 2026No releases on Jun 22, 2026No releases on Jun 29, 2026No releases on Jul 6, 2026No releases on Jul 13, 2026No releases on Jul 20, 2026No releases on Jul 27, 2026No releases on Aug 3, 2026No releases on Aug 10, 2026
TuesdayNo releases on Apr 28, 2026No releases on May 5, 20261 release on May 12, 2026No releases on May 19, 2026No releases on May 26, 20261 release on Jun 2, 2026No releases on Jun 9, 2026No releases on Jun 16, 2026No releases on Jun 23, 2026No releases on Jun 30, 2026No releases on Jul 7, 2026No releases on Jul 14, 2026No releases on Jul 21, 2026No releases on Jul 28, 2026No releases on Aug 4, 2026No releases on Aug 11, 2026
WednesdayNo releases on Apr 29, 2026No releases on May 6, 2026No releases on May 13, 2026No releases on May 20, 2026No releases on May 27, 2026No releases on Jun 3, 2026No releases on Jun 10, 20261 release on Jun 17, 2026No releases on Jun 24, 2026No releases on Jul 1, 2026No releases on Jul 8, 2026No releases on Jul 15, 2026No releases on Jul 22, 2026No releases on Jul 29, 2026No releases on Aug 5, 2026
ThursdayNo releases on Apr 30, 2026No releases on May 7, 2026No releases on May 14, 2026No releases on May 21, 2026No releases on May 28, 2026No releases on Jun 4, 2026No releases on Jun 11, 20261 release on Jun 18, 2026No releases on Jun 25, 20262 releases on Jul 2, 2026No releases on Jul 9, 2026No releases on Jul 16, 2026No releases on Jul 23, 2026No releases on Jul 30, 2026No releases on Aug 6, 2026
FridayNo releases on May 1, 2026No releases on May 8, 2026No releases on May 15, 2026No releases on May 22, 20261 release on May 29, 2026No releases on Jun 5, 2026No releases on Jun 12, 2026No releases on Jun 19, 2026No releases on Jun 26, 2026No releases on Jul 3, 2026No releases on Jul 10, 20261 release on Jul 17, 2026No releases on Jul 24, 2026No releases on Jul 31, 20261 release on Aug 7, 2026
SaturdayNo releases on May 2, 2026No releases on May 9, 2026No releases on May 16, 2026No releases on May 23, 2026No releases on May 30, 2026No releases on Jun 6, 2026No releases on Jun 13, 2026No releases on Jun 20, 2026No releases on Jun 27, 2026No releases on Jul 4, 2026No releases on Jul 11, 2026No releases on Jul 18, 2026No releases on Jul 25, 2026No releases on Aug 1, 2026No releases on Aug 8, 2026

10 releases since Apr 24, 2026, busiest day 2

Changelog

0.96.3

Added 5
  • Command steps can now run across multiple Kubernetes pods with the Kubernetes step operator by setting pod_count in KubernetesStepOperatorSettings to launch the step as an indexed job
  • ZenML now includes a local Docker sandbox with a unified settings model for containerized sandboxes, supporting file upload and download for testing containerized ZenML behavior locally
  • The Helm chart now supports server.pro.enrollmentKeySecretRef to inject ZenML Pro enrollment keys from an existing Kubernetes Secret
  • zenml login now accepts --stack flag to connect to a server and immediately set the active stack in one command
  • The dashboard timeline now shows steps that were not started because a run was cancelled and supports filtering for Not Started
Changed 3
  • Cloudpickle-materialized artifacts now store a SHA-256 hash when written and validate that hash before loading
  • Secret values in the dashboard no longer expand indefinitely in the UI and are truncated for readability with a direct copy action available
  • Local SQL stores backed by SQLite now wait up to 60 seconds for write locks instead of Python's 5-second default
Fixed 7
  • Artifact version deletion now correctly applies project scope when checking whether an artifact is unused on remote deployed servers
  • Exception reporting now includes the full traceback lineage instead of only the final exception traceback
  • Kubernetes orchestrator pods are no longer retried when the pipeline run is already in a finished state that cannot be retried
  • Prefix-based lookups now keep scoped and default filters properly constrained when matching by ID or name prefix
  • Generic and stack-deployment tokens now preserve the original service account or device provenance used to create them
  • ZenML's MLflow experiment tracker now behaves more reliably when managed runtimes inject MLflow environment variables such as MLFLOW_RUN_ID
  • Existing workspace-level API keys remain valid when a workspace service account is adopted by an organization-level service account with the same name
Runtime and orchestration
  • Multi-pod Kubernetes step operator jobs: Command steps can now run across multiple Kubernetes pods with the Kubernetes step operator. Set pod_count in KubernetesStepOperatorSettings to launch the step as an indexed job, making it easier to distribute command-style workloads across pods. PR #5104
  • Local Docker sandbox: ZenML now includes a local Docker sandbox, plus a unified settings model for containerized sandboxes. Local sandbox workflows also support file upload and download, making it easier to test containerized ZenML behavior locally before moving to remote infrastructure. PR #5102
Deployment, security, and artifact integrity
  • Enrollment keys from Kubernetes Secrets: The Helm chart now supports server.pro.enrollmentKeySecretRef, so ZenML Pro enrollment keys can be injected from an existing Kubernetes Secret instead of being stored inline in Helm release values. The secret reference is applied consistently to the server, migration, and worker containers. PR #5123
  • Cloudpickle artifact hash validation: Cloudpickle-materialized artifacts now store a SHA-256 hash when written and validate that hash before loading. This helps detect corrupted or unexpectedly modified artifact files earlier and fail with a clearer integrity signal. PR #5103
CLI and dashboard UX
  • Select a stack during login: zenml login now accepts --stack, allowing you to connect to a server and immediately set the active stack in one command. When used together with --project, ZenML applies the project first so the stack is resolved in the intended project context. PR #5125
  • Improved secret value display: Secret values in the dashboard no longer expand indefinitely in the UI and are truncated for readability. A direct copy action is now available, making it easier to work with long secret values without disrupting the page layout. PR #1104
  • Timeline updates for cancelled runs: The dashboard timeline now shows steps that were not started because a run was cancelled. The timeline filter also supports filtering for Not Started, making cancelled or partially executed runs easier to inspect. PR #1108
  • Fewer SQLite lock failures for local stores: Local SQL stores backed by SQLite now wait up to 60 seconds for write locks instead of using Python’s 5-second default. This reduces sqlite3.OperationalError: database is locked failures when concurrent steps, such as mapped dynamic pipeline steps, finish and publish artifacts at the same time. PR #5096
  • Artifact deletion on deployed servers: Artifact version deletion now correctly applies project scope when checking whether an artifact is unused. This fixes deletion through the API on remote deployed servers where the default project is disabled. PR #5100
  • Complete exception tracebacks: Exception reporting now includes the full traceback lineage instead of only the final exception traceback. This makes chained failures easier to debug because the original cause is preserved alongside the final error. PR #5098
  • Smarter Kubernetes dynamic pipeline retries: Kubernetes orchestrator pods are no longer retried when the pipeline run is already in a finished state that cannot be retried. This avoids unnecessary pod restarts that would immediately exit with Run is already finished. PR #5107
  • Correct scoped prefix lookups: Prefix-based lookups now keep scoped and default filters properly constrained when matching by ID or name prefix. This prevents unrelated entities from being returned, for example when looking up a schedule trigger by a prefix that does not match its name or ID. PR #5126
  • Generic token provenance and service account auth: Generic and stack-deployment tokens now preserve the original service account or device provenance used to create them. This fixes authentication failures for pipeline workloads using server-issued JWTs for workspace-local service accounts, especially with external authentication enabled. PR #5127
  • MLflow tracking with managed runtimes and Databricks: ZenML’s MLflow experiment tracker now behaves more reliably when managed runtimes inject MLflow environment variables such as MLFLOW_RUN_ID. This avoids accidentally resuming an inherited run when ZenML needs to create its own run, improving compatibility with Databricks-backed MLflow setups. PR #5122
  • API keys after service account adoption: Existing workspace-level API keys remain valid when a workspace service account is adopted by an organization-level service account with the same name. This allows teams to migrate service accounts gradually without interrupting workloads that still use older API keys. PR #5138
What's Changed
New Contributors

Full Changelog: https://github.com/zenml-io/zenml/compare/0.96.2...0.96.3

View originalPermalink
How 0.96.3 went

0.96.2

Added 7
  • Dynamic pipelines now support start_after parameter when calling steps to control which concurrently launched steps should wait for others before starting
  • Configure whether JSON-serializable raw values passed to steps in dynamic pipelines should be treated as parameters instead of artifacts through environment variables and explicit APIs
  • Dynamic pipelines now support CONTINUE_ON_FAILURE execution mode to allow queued or asynchronous work to continue when an async step fails
  • First-class DigitalOcean integration with support for DigitalOcean Spaces artifact stores and DigitalOcean Container Registry stack components
  • Helm chart now exposes server logging options and OpenTelemetry settings in values for console or JSON logging, service names, OTEL endpoints, and trace/metric/log configuration
  • Delete artifact version data through the server API from the UI or thin clients without direct stack access
  • Projects now support arbitrary project_metadata on create, update, and hydrated response models
Changed 7
  • start_after is now a reserved step keyword in dynamic pipelines
  • fileio.copy() now streams cross-filesystem copies in bounded chunks instead of reading entire files into memory
  • ZenML caches commonly reused project, store, stack, pipeline run, and completed step run responses in process to reduce server requests
  • DAG endpoint now does less unnecessary parsing and object construction when serving large pipeline graphs
  • FastAPI updated to 0.138.0, Starlette lower bound raised to 0.46.0, and OpenTelemetry packages updated for compatibility
  • Removed unused fastapi_utils dependency and cleaned up deprecated FastAPI response and lifespan usage
  • Dynamic pipelines now model implicit dependencies from newly launched steps to the last completed sync step
Fixed 4
  • User tokens issued before a password change are now rejected
  • Tokens derived from API keys are tied to the key generation so rotated keys no longer leave stale sessions usable
  • Dashboard now correctly rotates API keys when a non-zero retention period is configured
  • ZenML now properly handles in-memory Docker credentials when a local Docker credential store already has entries for the same registry
Known issues

Artifact version deletion via the API is broken in this release and will be fixed in 0.96.3. Artifact versions can still be deleted using the python SDK:

from zenml.client import Client

Client().delete_artifact_version(..., server_side=False)
Dynamic pipelines
  • Explicit start ordering for dynamic steps: Dynamic pipelines now support start_after=... when calling steps, letting you control which concurrently launched steps should wait for others before starting. This makes it easier to model ordering constraints without turning concurrent parts of a dynamic pipeline into fully synchronous execution. Note that start_after is now a reserved step keyword, so steps that previously used a parameter with this name will need to be updated. PR #4995
  • More flexible dynamic step inputs: You can now configure whether JSON-serializable raw values passed to steps in dynamic pipelines should be treated as parameters instead of artifacts. The new environment-variable threshold defaults to 0 to preserve existing behavior, while explicit APIs such as with_options(parameters=...) and ExternalArtifact(...) remain available when you want to force either behavior. PR #5079
  • Improved dynamic execution semantics: Dynamic pipelines now support CONTINUE_ON_FAILURE execution mode, allowing already queued or asynchronous work to continue when an async step fails. ZenML also models implicit dependencies from newly launched steps to the last completed sync step, making mixed sync/async dynamic pipelines execute in a more predictable order. PR #5052
Integrations and deployment
  • DigitalOcean integration: ZenML now includes a first-class digitalocean integration with support for DigitalOcean Spaces artifact stores and DigitalOcean Container Registry stack components. Spaces support builds on the existing S3-compatible implementation while handling DigitalOcean regions and endpoint generation for you. PR #5054
  • Helm chart logging and OpenTelemetry configuration: The ZenML Helm chart now exposes server logging options and OpenTelemetry settings directly in values. You can configure console or JSON logging, service names, OTEL endpoints, and enable or disable traces, metrics, and logs without custom chart modifications. PR #5048
Data and metadata management
  • Server-side artifact data deletion: ZenML can now delete artifact version data through the server API, not only from a full client with direct stack access. This enables artifact metadata and backing data to be deleted from the UI or from thin clients that do not have the artifact store stack component locally available. PR #5034
  • Project metadata: Projects now support arbitrary project_metadata on create, update, and hydrated response models. Metadata is stored as portable JSON, preserved when omitted, replaced when explicitly supplied, and can be cleared by sending an empty object. PR #5086
Performance and scalability
  • Lower-memory cross-filesystem copies: fileio.copy() now streams cross-filesystem copies in bounded chunks instead of reading the entire file into memory. This significantly reduces peak memory usage for local-to-remote and remote-to-local artifact operations, including PathMaterializer, directory copies, integration materializers, and code archive upload/download flows. PR #5031
  • Fewer server requests during runs: ZenML now caches commonly reused project, store, stack, pipeline run, and completed step run responses in process where safe. Local and in-process execution paths make substantially fewer server requests, which improves responsiveness for pipelines with many steps. PR #5036 PR #5038
  • Faster DAG endpoint on large runs: The DAG endpoint now does less unnecessary parsing and object construction when serving large pipeline graphs. In benchmarks on a DAG with thousands of nodes and edges, endpoint latency was reduced by roughly half. PR #5051
Security and dependencies
  • FastAPI, Starlette, and OpenTelemetry updates: ZenML now supports FastAPI 0.138.0, raises the lower Starlette bound to 0.46.0 to pick up security fixes, and updates OpenTelemetry packages for compatibility. The update also removes an unused fastapi_utils dependency and cleans up deprecated FastAPI response and lifespan usage. PR #5017 PR #5060
  • Token invalidation after credential changes: User tokens issued before a password change are now rejected, and tokens derived from API keys are tied to the key generation so rotated keys no longer leave stale sessions usable. The dashboard also correctly rotates API keys when a non-zero retention period is configured. PR #5025 PR #1088
  • Docker credentials supplied at runtime: ZenML now properly handles in-memory Docker credentials when a local Docker credential store already has entries for the same registry. This prevents the Docker Python client from silently preferring stale local credentials over credentials passed through ZenML. PR #5023
  • Remote image builds with restrictive .dockerignore files: ZenML no longer includes the root .dockerignore in generated build context archives for remote builders. This fixes builds on AWS CodeBuild, GCP Cloud Build, Kaniko, and similar builders when allowlist-style ignore patterns would otherwise exclude ZenML-generated files. PR #5033
  • Cleaner replay configurations: When replaying a pipeline run, ZenML now removes step parameters from the configuration if they are overridden by an input artifact. The displayed configuration no longer contains outdated values that were not actually used as step inputs. PR #5040
  • Deployment invocation with dict parameters: Dict-valued pipeline parameters sent to a deployment /invoke endpoint now replace compiled defaults instead of being recursively merged with them. This matches normal pipeline invocation behavior and prevents default keys from leaking into step inputs. PR #5042
  • Dynamic pipeline DAG race condition: ZenML now avoids a race where a step run could be visible in the database before its configuration was committed. This prevents intermittent 500 errors from the DAG endpoint while dynamic pipeline steps are being created. PR #5053
  • SSH orchestrator re-runs: SSH orchestrator container and Compose service names now avoid collisions across multiple runs of the same snapshot. Re-running from the dashboard, templates, or concurrent triggers no longer fails because an old container with the same name still exists on the host. PR #5082
What's Changed
New Contributors

Full Changelog: https://github.com/zenml-io/zenml/compare/0.96.1...0.96.2

View originalPermalink
How 0.96.2 went

0.96.1

Added 1
  • SSH orchestrator and SSH step operator for executing workloads on remote machines accessible via SSH
  • Run pipelines and steps over SSH: ZenML now includes an SSH orchestrator and SSH step operator for executing workloads on remote machines accessible via SSH (PR #4953). This makes it easier to use existing servers or on-prem infrastructure as execution targets without adopting a full cluster-based backend, while still managing runs through ZenML.
What's Changed

Full Changelog: https://github.com/zenml-io/zenml/compare/0.96.0...0.96.1

View originalPermalink
How 0.96.1 went

0.96.0

Added 8
  • Add Trackio experiment tracking integration for logging experiment data through Trackio's public API
  • Add Backblaze B2 as an S3-compatible artifact store option
  • Add baseten step operator flavor for running GPU workloads as Baseten Training jobs
  • Add generic OAuth2 service connector for authenticating external services using static tokens, client credentials, or client ID with refresh token
  • Add step_default_input_overrides parameter to override step inputs by name during pipeline replay
  • Add trigger cycle protection to detect and skip cyclic execution loops in Platform Event Trigger chains
  • Add optional sandbox cleanup configuration to destroy sandboxes automatically when sessions exit
  • Add configurable retry behavior for Kubernetes API calls in Kubernetes-based deployments
Changed 7
  • Raise minimum supported transformers version for Hugging Face compatibility
  • Update Azure integration to require newer Azure dependency versions and remove support for deprecated azureml-core library
  • Restrict service account and API key management to admins only in open-source ZenML server deployments without RBAC
  • Improve dashboard Timeline View with additional filtering options and more efficient connector selector
  • Optimize pipeline sorting by latest run query to avoid scanning all runs on large deployments
  • Improve efficiency of run, artifact, DAG, and model version queries to load only necessary data
  • Change expired API transaction cleanup to use bounded catch-up passes instead of fixed single deletes
Known Issues

If the Docker credential helper contains invalid credentials for a registry that ZenML is trying to use, those will have precendence over the credentials set by the service connector and will cause the following error: RuntimeError: Docker error: denied: Your authorization token has expired. Reauthenticate and try again... This will be fixed in the next release, in the meantime docker logout <REGISTRY> should fix it.

Breaking Changes
  • The minimum supported transformers version has been raised. If you use ZenML with Hugging Face/transformers, update your environment and dependency pins to a newer compatible transformers release before upgrading ZenML. PR #4976
  • The Azure integration now requires newer Azure dependency versions, and support for the deprecated azureml-core library has been fully removed. If you use ZenML on Azure, update your Azure-related dependency pins and migrate any remaining azureml-core usage to the currently supported Azure SDK packages before upgrading. PR #4987
  • In open-source ZenML server deployments without RBAC enabled, service account and API key management is now restricted to admins only. Non-admin users will no longer be able to manage service accounts or API keys they previously created, so move any required credentials and automation to admin-managed accounts as part of your upgrade. PR #5007
New integrations and execution backends
  • Trackio experiment tracking: ZenML now includes a Trackio experiment tracker integration, allowing pipelines to log experiment data through Trackio’s public API. This makes it easier to manage trial results and connect ZenML runs with Hugging Face-backed Trackio workflows such as datasets, spaces, and buckets. PR #4841
  • Backblaze B2 artifact store: You can now configure Backblaze B2 as a ZenML artifact store. This adds another S3-compatible storage option for teams that want to store pipeline artifacts in Backblaze infrastructure. PR #4791
  • Baseten step operator: ZenML now supports a baseten step operator flavor for running GPU workloads as Baseten Training jobs. It supports regular single-node steps with ZenML artifacts and logs, as well as multi-node distributed training through command steps that can consume Baseten’s distributed training environment variables. PR #4973
  • Generic OAuth2 service connector: A new OAuth2 service connector lets you authenticate external services using a static token, client credentials, or a client ID with refresh token. This provides a reusable connector option for services that expose OAuth2-based authentication. PR #4992
Workflow controls and platform operations
  • Replay input overrides by step name: When replaying a run, you can now use step_default_input_overrides to override a step input for every invocation of a step with the same name. Per-invocation step_input_overrides still take precedence, giving you both broad and targeted control during replay. PR #4978
  • Trigger cycle protection: ZenML now detects execution loops in Platform Event Trigger chains at the pipeline level. Cyclic trigger dispatches are skipped with the new SKIPPED_TRIGGER_CYCLE status, while unrelated downstream dispatches can continue normally and the affected cycle can be inspected through the SDK. PR #4971
  • Optional sandbox cleanup on exit: Sandbox sessions can now be configured to destroy the sandbox automatically when the session exits. The option defaults to False, preserving the behavior from previous releases unless you opt in. PR #4986
  • Configurable Kubernetes API retries: Kubernetes-based deployments can now configure retry behavior for Kubernetes API calls. This gives operators more control over resilience in clusters where transient API failures or throttling can occur. PR #5004
  • Dashboard filtering and connector selection improvements: The dashboard Timeline View now has additional filtering options, including more status filters. Component creation also gets a more efficient connector selector, making setup flows smoother in larger workspaces. PR #1084
Performance and scalability
  • Faster pipeline sorting by latest run: Listing pipelines sorted by latest run is now more efficient on large deployments. ZenML changed the query shape and supporting database indexing so the server no longer has to scan all runs for all pipelines just to compute the latest run timestamp. PR #4969
  • More efficient run and artifact queries: Several common server queries now load only the data they need and fetch related metadata more efficiently. This improves performance for DAG, pipeline run, step run, artifact version, and model version views, especially in workspaces with many entities. PR #4994
  • Catch-up cleanup for expired API transactions: Expired API transaction cleanup now works through bounded catch-up passes instead of a single fixed delete per interval. This helps servers recover from cleanup backlogs while keeping each database operation bounded, and also allows completed expired idempotency transactions to be safely reclaimed. PR #4943
  • Artifact store caching prevents server OOMs: The ZenML server now caches artifact store instances used for operations such as reading logs and visualizations. This avoids repeatedly rebuilding heavy storage clients and helps prevent memory growth that could previously lead to OOM kills on busy servers. PR #4974
  • Docker credentials for image builds and pushes: ZenML now configures Docker credentials correctly when building and pushing container images. This fixes cases where username/password credentials were not applied to the expected registry URI, which could cause authentication failures in build and push workflows. PR #5005
  • Secret backup and restore authorization: Secret backup and restore endpoints now enforce an explicit admin check when RBAC is disabled. This closes an authorization gap where an authenticated non-admin user could access admin-only secret operations in the default non-RBAC setup. PR #5009
  • Safer custom flavor loading: Custom flavor sources are now validated before server-side hydration. ZenML ensures the configured source resolves to a Flavor subclass before instantiation, preventing arbitrary zero-argument callables from being invoked during flavor loading. PR #5008
What's Changed
New Contributors

Full Changelog: https://github.com/zenml-io/zenml/compare/0.95.1...0.96.0

View originalPermalink
How 0.96.0 went

0.95.1

Changed 3
  • ZenML now falls back to the orchestrator image when steps use step operators in dynamic pipelines, matching the behavior used for isolated steps without step operators
  • Improved database query performance by adjusting how related data is loaded for common pipeline run and step run queries
  • ZenML now avoids writing back into the log store during shutdown to improve reliability for pipeline and step log collection
Fixed 2
  • Fixed an issue where running a step with a step operator in a dynamic pipeline could fail unless that step was explicitly listed in pipeline.depends_on
  • Fixed a deadlock that could happen during logging context shutdown when using fsspec-based artifact log stores with debug logs enabled
Dynamic pipelines with step operators

Dynamic pipeline execution is more reliable when steps use step operators.

  • Fixed an issue where running a step with a step operator in a dynamic pipeline could fail unless that step was explicitly listed in pipeline.depends_on.
  • ZenML now falls back to the orchestrator image in this case, matching the behavior already used for isolated steps without step operators. PR #4960
Faster pipeline and step run queries

Common pipeline run and step run views should now load more efficiently, especially on larger deployments.

  • Improved database query performance by adjusting how related data is loaded for common pipeline run and step run queries.
  • This helps avoid expensive query plans in MySQL for paginated run listings, making these queries more scalable. PR #4965
Logging stability

Logging shutdown is now safer when using artifact-backed log stores.

  • Fixed a deadlock that could happen during logging context shutdown when using fsspec-based artifact log stores with debug logs enabled.
  • ZenML now avoids writing back into the log store during shutdown, improving reliability for pipeline and step log collection. PR #4964
What's Changed

Full Changelog: https://github.com/zenml-io/zenml/compare/0.95.0...0.95.1

View originalPermalink
How 0.95.1 went

0.95.0

Added 7
  • Define steps and hooks with async def; ZenML now runs async functions on a fresh event loop for both normal and dynamic pipeline usage
  • Run arbitrary commands as pipeline steps with CommandStep(...), including non-Python commands and Python callables that do not require ZenML in the execution environment
  • Invoke deployments asynchronously: a new deployment endpoint can submit a pipeline run and return immediately instead of waiting for completion
  • Added the core Sandbox stack component abstraction for running untrusted or generated code in isolated sessions, including a built-in local flavor for subprocess-based execution
  • Added a kubernetes sandbox flavor where each sandbox session runs in a dedicated Kubernetes pod, with streamed command execution and support for re-attaching to running sessions
  • Added a Modal orchestrator flavor so complete ZenML pipelines can run on Modal, using Modal sandboxes for orchestration and step execution
  • Added targeted database indexes for common pagination and hydration query patterns across pipeline runs, snapshots, step configurations, step runs, and artifact versions
Changed 11
  • ZenML now supports Python 3.14, and environments using the local or server extras must accommodate the SQLModel upgrade from 0.18.0 to 0.38.0
  • Local MLflow tracking now uses a SQLite backend by default when no tracking_uri is configured
  • ZenML now requires opentelemetry-sdk==1.40.0 instead of 1.38.0
  • Step and pipeline hooks have been reworked into a new lifecycle-based hook system with persisted hook invocation records
  • ZenML server rate limiting no longer trusts raw X-Forwarded-For headers by default
  • CLI list commands now return the newest items first by default instead of the oldest first
Fixed 1
  • Pipeline execution may now raise different exception types depending on how step futures are awaited
Removed 1
  • The deprecated singular tag field has been removed from TaggableFilters
Known Issues
  • Enabling debug logs while having a GCP, Azure or S3 artifact log store will cause a deadlock at the end of a pipeline or step run. Fixed in 0.95.1.
Breaking Changes
  • PR #4844: ZenML now supports Python 3.14, and environments using the local or server extras must also accommodate the SQLModel upgrade from 0.18.0 to 0.38.0. If you depend on those extras, review and update any pinned SQLModel-related dependencies before upgrading.
  • PR #4900: Local MLflow tracking now uses a SQLite backend by default when no tracking_uri is configured. New tracking metadata is stored in <LOCAL_ARTIFACT_STORE>/mlflow.db and artifacts under the local artifact store, so users relying on the previous default local MLflow layout or behavior should update their local setup and migration expectations.
  • PR #4790: ZenML now requires opentelemetry-sdk==1.40.0 instead of 1.38.0. If your environment pins OpenTelemetry packages, update them to compatible versions before upgrading ZenML.
  • PR #4875: Step and pipeline hooks have been reworked into a new lifecycle-based hook system with persisted hook invocation records. If you use hooks or related internal APIs, review your existing integrations and update them to the new hook semantics and lifecycle events.
  • PR #4919: ZenML server rate limiting no longer trusts raw X-Forwarded-For headers by default. If you run ZenML behind an ingress or reverse proxy, make sure proxy header handling is explicitly configured so login rate limiting continues to use the correct client IPs.
  • PR #4459: CLI list commands now return the newest items first by default instead of the oldest first. If you have scripts or workflows that assumed the previous ordering, update them to explicitly sort or handle the new default order.
  • PR #4566: The deprecated singular tag field has been removed from TaggableFilters. Update any API or client code to use the supported tag filtering format instead of passing a single tag value.
  • PR #4950: Pipeline execution may now raise different exception types depending on how step futures are awaited. If you catch exceptions around pipeline execution, review and update your error-handling logic to account for StepExecutionException being raised in implicit await scenarios.
  • PR #4867: ZenML now requires modal>=1.4.0,<2.0.0 when using the Modal integration.
New ways to run code and pipelines

This release expands how you can execute work in ZenML, from async Python to arbitrary commands and new remote execution backends.

  • Define steps and hooks with async def; ZenML now runs async functions on a fresh event loop for both normal and dynamic pipeline usage. PR #4913
  • Run arbitrary commands as pipeline steps with CommandStep(...), including non-Python commands and Python callables that do not require ZenML in the execution environment. PR #4904
  • Invoke deployments asynchronously: a new deployment endpoint can submit a pipeline run and return immediately instead of waiting for completion. PR #4906
Sandboxes and Modal execution

ZenML now includes the core sandbox abstraction for isolated execution, plus new backend support for Kubernetes and Modal-based workloads.

  • Added the core Sandbox stack component abstraction for running untrusted or generated code in isolated sessions, including a built-in local flavor for subprocess-based execution. PR #4866
  • Added a kubernetes sandbox flavor where each sandbox session runs in a dedicated Kubernetes pod, with streamed command execution and support for re-attaching to running sessions. PR #4926
  • Added a Modal orchestrator flavor so complete ZenML pipelines can run on Modal, using Modal sandboxes for orchestration and step execution. PR #4915
Integrations and deployment improvements

Several integrations and deployment paths are more flexible and production-ready.

  • Kubernetes deployments now merge pod_settings.resources into the deployment template context, making it possible to set pod resource limits required by cluster policies such as OPA Gatekeeper constraints. PR #4523
  • Databricks-managed MLflow deployments now support machine-to-machine OAuth authentication via service principals. PR #4947
Performance and scalability

Common list and hydration operations should be faster and more reliable on larger ZenML deployments.

  • Improved list endpoint ordering so descending sorts can use matching index scans instead of forcing expensive mixed-direction database sorts. PR #4890
  • Added targeted database indexes for common pagination and hydration query patterns across pipeline runs, snapshots, step configurations, step runs, and artifact versions. PR #4942
  • Adjusted request timeout behavior so only deduplicated/cacheable requests may return a timeout or backpressure response while work continues in the background. PR #4942
Security and permissions

This release tightens authorization checks around API keys, stack deployments, secrets, and tag-resource relationships.

  • Service-account API key validation now handles omitted internal verification values and client-provided key values consistently, while preserving internal re-authentication behavior. PR #4920
  • GET /api/v1/stack-deployment/stack now verifies READ permissions for both the returned stack and its associated service connector before returning deployment metadata. PR #4917
  • Secret reference resolution now prevents users from attaching private secrets owned by others, or internal ZenML-managed secrets, to their own resources. PR #4923
  • Tag-resource endpoints now require UPDATE permissions on the referenced resource before tag relationships can be created or deleted, including batch operations. PR #4927
  • Tag-resource RBAC enforcement now lives in the RBAC store layer for more consistent behavior, and tag reads remain broadly available as server-wide resources. PR #4938
  • Fixed several dynamic pipeline edge cases around retries, stopping runs, and isolated step launch states:
    • Step failures that happen while launching a retry are now detected.
    • Steps no longer move to RETRYING if the run is already STOPPING or STOPPED.
    • Runs that fail while STOPPING now transition to STOPPED instead of FAILED.
    • Isolated steps now use PROVISIONING while they are being launched. PR #4916
  • Fixed an IndexError when step inputs annotated as bare list or tuple received multiple input artifacts. ZenML now loads each artifact using its stored data type, matching behavior for Any or unannotated inputs. PR #4929
  • Fixed GKE Kubernetes API endpoint selection in the GCP service connector by only using the DNS endpoint when it allows external traffic; otherwise, ZenML falls back to the IP-based endpoint. PR #4934
What's Changed
New Contributors

Full Changelog: https://github.com/zenml-io/zenml/compare/0.94.6...0.95.0

View originalPermalink
How 0.95.0 went

0.94.6

Fixed 3
  • Fixed GCP service connector failures when connecting to private GKE clusters that use Google's DNS-based control plane endpoint by connecting using the same method as gcloud container clusters get-credentials --dns-endpoint
  • Resolved an issue where ZenML would fail when no container engine was available even when users explicitly set skip_build=True in their DockerSettings
  • Include hook sources as string in snapshot config template
Infrastructure & Deployment Improvements
  • Enhanced GKE Private Cluster Support: Fixed GCP service connector failures when connecting to private GKE clusters that use Google's DNS-based control plane endpoint. ZenML now connects using the same method as gcloud container clusters get-credentials --dns-endpoint, ensuring reliable access to private clusters. PR #4856
  • Docker Build Requirements: Resolved an issue where ZenML would fail when no container engine was available, even when users explicitly set skip_build=True in their DockerSettings. The build checksum computation now correctly respects the skip_build flag. PR #4879
What's Changed

Full Changelog: https://github.com/zenml-io/zenml/compare/0.94.5...0.94.6

View originalPermalink
How 0.94.6 went

0.94.5

Added 8
  • Stream custom events in real-time from running pipelines via zenml.streaming.publish() with Server-Sent Events (SSE) support and Redis-based broker with automatic catch-up, gap signaling, and idle cleanup
  • Add POST /api/v1/runs/statistics endpoint to query aggregated metrics across pipeline runs with grouping by status, pipeline, stack, user, time buckets, metadata values, and tags
  • Weights & Biases experiment tracker automatically adds ZenML pipeline and step metadata to W&B runs and records W&B identifiers back to ZenML step metadata with configurable custom groups, job types, run configs, and run IDs
  • Add example demonstrating agentic human-in-the-loop pipelines with dynamic planning, task fan-out via step.map(), result summarization, human approval gates with zenml.wait(), and conditional branching
  • ZenML server supports structured logging with OpenTelemetry instrumentation and ZENML_CONSOLE_LOGGING_FORMAT environment variable with console, json, or text format options
  • Step and pipeline definitions now support string type annotations, either explicitly quoted or via from __future__ import annotations
  • Python package installations during Docker image building can now mount a build cache for faster builds
  • Interactive wait condition input with string schemas now accepts raw strings without quotes with automatic detection and handling
Changed 3
  • Restored custom console log formatting for non-DEBUG output and added step-name prefixes in terminal output during execution
  • ZENML_LOGGING_FORMAT variable is now deprecated in favor of ZENML_CONSOLE_LOGGING_FORMAT
  • Improved scrolling behavior for the pipeline timeline view in dashboard
Fixed 3
  • Pipeline runs now properly fail when the user pipeline function cannot be imported instead of staying in running state indefinitely
  • Keyboard interrupts during active wait conditions are now handled more gracefully without unclear error messages when server transitions to STOPPED status
  • Fixed user update endpoint bug where admins couldn't activate or deactivate user accounts due to incorrect field assignment
🚀 Live Event Streaming for Pipeline Runs

You can now stream custom events in real-time from your running pipelines! Call zenml.streaming.publish() from inside any step or dynamic pipeline to push events that can be consumed via Server-Sent Events (SSE). Enable this feature by setting stream_broker_implementation_source in your server configuration. The initial implementation includes a Redis-based broker with automatic catch-up, gap signaling, and idle cleanup. PR #4804

📊 Pipeline Run Statistics Endpoint

A new POST /api/v1/runs/statistics endpoint lets you query aggregated metrics across your pipeline runs. Group by status, pipeline, stack, user, time buckets (hour/day/week/month), metadata values, tags, and more. Calculate averages, sums, min/max over duration, step counts, cached steps, output artifacts, or custom numeric metadata. Perfect for building dashboards and analytics. PR #4860

🎯 Richer Weights & Biases Integration

The W&B experiment tracker now automatically adds ZenML pipeline and step metadata to your W&B runs, groups runs by pipeline execution, and records W&B identifiers back to ZenML step metadata. You can now configure custom groups, job types, run configs, explicit or deterministic run IDs, resume behavior, and pass through additional wandb.init kwargs for complete control over your experiment tracking. PR #4838

🤖 Agentic Human-in-the-Loop Pipeline Example

A new example demonstrates building dynamic agentic pipelines with human approval gates. The example shows how to plan agent tasks, fan them out with step.map(), summarize results, pause execution with zenml.wait() for human review, and branch the final action based on the decision. Includes clean lineage tracking with Annotated step outputs. PR #4849

📝 Structured Logging and OpenTelemetry Instrumentation

The ZenML server now supports structured logging with OpenTelemetry instrumentation. Configure console output with the new ZENML_CONSOLE_LOGGING_FORMAT environment variable, choosing between console, json, or text formats. Server logs use a clean structured layout with timestamps, levels, logger context, and optional JSON fields. PR #4781

🛠️ Enhanced Developer Experience
  • String Type Annotations: Step and pipeline definitions now support string annotations, either explicitly quoted or via from __future__ import annotations. PR #4843
  • Build Cache Mounting: Python package installations during Docker image building can now mount a build cache for faster builds. PR #4820
  • Improved Wait Condition Input: When resolving wait conditions interactively with string schemas, you can now input raw strings without quotes—ZenML detects and handles this automatically. PR #4845
  • Better Logging Control: Restored custom console log formatting for non-DEBUG output, added step-name prefixes in terminal output during execution (disable with ZENML_DISABLE_STEP_NAMES_IN_LOGS=true), and kept stored logs clean and unformatted. The ZENML_LOGGING_FORMAT variable is now deprecated in favor of ZENML_CONSOLE_LOGGING_FORMAT. PR #4851
🎨 Dashboard Improvements
  • Improved scrolling behavior for the pipeline timeline view. PR #1053
  • Pipeline runs now properly fail when the user pipeline function cannot be imported (e.g., due to missing packages), instead of staying in a running state indefinitely. PR #4832
  • Keyboard interrupts during active wait conditions are now handled more gracefully, preventing unclear error messages when the server transitions to STOPPED status. PR #4835
  • Fixed a copy-paste bug in the user update endpoint where admins couldn't activate or deactivate user accounts—the code was incorrectly writing is_admin instead of active to the safe update object. PR #4839
What's Changed
New Contributors

Full Changelog: https://github.com/zenml-io/zenml/compare/0.94.4...0.94.5

View originalPermalink
How 0.94.5 went

0.94.4

Added 4
  • Add new Databricks step operator to run individual pipeline steps on Databricks
  • Support nested dynamic pipelines, allowing one dynamic pipeline to be called from within another
  • Add advanced training workload settings to Run:AI step operator including multiple mount types, workload templates, security context settings, port declarations, and parallelism configuration
  • Add parent run information display in the dashboard run details view
Changed 3
  • Improve Kubernetes string handling with separate sanitization for DNS-style names and looser label-value rules for metadata
  • Increase maximum allowed size for ZenML secrets stored in SQL secrets store to 64KB
  • Add optional tag settings to Databricks orchestrator to label jobs and cluster resources for cost tracking, ownership, and governance
Fixed 5
  • Fix signal handler registration during step execution to prevent errors when running many sync steps or interrupting pipelines
  • Fix redundant RBAC permission checks during response model dehydration
  • Allow step functions to use keyword-only arguments without causing failures
  • Fix Kubernetes orchestrator step pod configuration by replacing hardcoded lookups with proper settings calls
  • Improve Kubernetes job failure diagnostics to provide richer diagnostic information when dynamic pipeline jobs fail due to system issues
New Databricks Step Operator

You can now run individual pipeline steps on Databricks using the new Databricks step operator PR #4648. This is useful when you want specific steps to execute in the Databricks runtime while the rest of your pipeline uses a different orchestrator. The Databricks orchestrator also now supports optional tag settings to label jobs and cluster resources for cost tracking, ownership, and governance.

Nested Dynamic Pipelines

Dynamic pipelines can now be nested, allowing you to call one dynamic pipeline from within another PR #4775. This enables more modular and reusable pipeline designs.

Enhanced Run:AI Training Workload Configuration

The Run:AI step operator now supports advanced training workload settings PR #4780, including:

  • Multiple mount types (PVC, ConfigMap, Secret, NFS, S3, HostPath)
  • Workload templates via workload_template_id
  • Security context settings (UID/GID, non-root execution, seccomp, capabilities)
  • Port declarations and external URL exposure
  • Training workload parallelism and completions
Improved Kubernetes Job Failure Diagnostics

When dynamic pipeline jobs fail due to system issues (such as OOM kills), ZenML now provides richer diagnostic information PR #4800. This makes it easier to understand why Kubernetes terminated your pods.

Better Kubernetes Label Handling

Kubernetes string handling has been improved with separate sanitization for DNS-style names/keys and looser label-value rules for metadata like run, pipeline, and step IDs PR #4756. This makes it easier to navigate through runs in Kubernetes.

Increased Secret Size Limit

The maximum allowed size for ZenML secrets stored in the SQL secrets store has been increased to 64KB PR #4769. The limit applies to the combined size of all keys and values in a secret object.

Dashboard: Parent Run Display

The dashboard now displays parent run information in the run details view when available PR #1050.

  • Signal handling during step execution: Signal handlers are now properly unregistered after step execution, preventing strange errors when running many sync steps or interrupting pipelines PR #4784. Dynamic pipeline steps running in isolated environments or step operators are no longer affected by signal handling from the orchestrator.

  • RBAC performance: Fixed redundant RBAC permission checks during response model dehydration PR #4797. Previously, when permissions were already prefetched and denied, additional RBAC requests were sent unnecessarily for each sub-model.

  • Keyword-only arguments in steps: Step functions can now use keyword-only arguments without causing failures PR #4798.

  • Kubernetes orchestrator settings: Fixed step pod configuration by replacing legacy hardcoded orchestrator.kubernetes lookups with proper orchestrator.get_settings(...) calls PR #4803. Step pods now correctly apply orchestrator settings from canonical component keys while maintaining backward compatibility.

What's Changed
New Contributors

Full Changelog: https://github.com/zenml-io/zenml/compare/0.94.3...0.94.4

View originalPermalink
How 0.94.4 went

0.94.3

Added 11
  • Add container engine abstraction and podman support
  • Include wait condition resolved at in DAG response
  • Add resource pools
  • Add event triggers
  • Add helper endpoint for trigger SourceType supported events
  • Add memory_call step type
Changed 2
  • Replace darglint with pydoclint for docstring linting
  • Improve CLI commands for trigger dispatch status
Fixed 7
  • Fix zizmor SHA/version mismatch
  • Fix broken docs header for triggers
  • Fix empty PR number in Claude Code review workflow
  • Fix making a secret public using CLI/client
  • Fix run status update condition
  • Fix StrEnum import breaking Python 3.10
  • Fix python package fetching
What's Changed

Full Changelog: https://github.com/zenml-io/zenml/compare/0.94.2...0.94.3

View originalPermalink
How 0.94.3 went

More in AI

View all
codex

codex release notes.

codex
Ollama

Get up and running with large language models locally.

Ollama
openclaw

Your own personal AI assistant. Any OS. Any Platform. The lobster way. 🦞

openclaw
superpowers

An agentic skills framework & software development methodology that works.

obra
ECC

The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.

affaan-m
hermes-agent

The agent that grows with you

NousResearch

Discussion