ZenML 0.94.5

0.94.5
Added 8
  • Stream custom events in real-time from running pipelines via zenml.streaming.publish() with Server-Sent Events (SSE) support and Redis-based broker with automatic catch-up, gap signaling, and idle cleanup
  • Add POST /api/v1/runs/statistics endpoint to query aggregated metrics across pipeline runs with grouping by status, pipeline, stack, user, time buckets, metadata values, and tags
  • Weights & Biases experiment tracker automatically adds ZenML pipeline and step metadata to W&B runs and records W&B identifiers back to ZenML step metadata with configurable custom groups, job types, run configs, and run IDs
  • Add example demonstrating agentic human-in-the-loop pipelines with dynamic planning, task fan-out via step.map(), result summarization, human approval gates with zenml.wait(), and conditional branching
  • ZenML server supports structured logging with OpenTelemetry instrumentation and ZENML_CONSOLE_LOGGING_FORMAT environment variable with console, json, or text format options
  • Step and pipeline definitions now support string type annotations, either explicitly quoted or via from __future__ import annotations
  • Python package installations during Docker image building can now mount a build cache for faster builds
  • Interactive wait condition input with string schemas now accepts raw strings without quotes with automatic detection and handling
Changed 3
  • Restored custom console log formatting for non-DEBUG output and added step-name prefixes in terminal output during execution
  • ZENML_LOGGING_FORMAT variable is now deprecated in favor of ZENML_CONSOLE_LOGGING_FORMAT
  • Improved scrolling behavior for the pipeline timeline view in dashboard
Fixed 3
  • Pipeline runs now properly fail when the user pipeline function cannot be imported instead of staying in running state indefinitely
  • Keyboard interrupts during active wait conditions are now handled more gracefully without unclear error messages when server transitions to STOPPED status
  • Fixed user update endpoint bug where admins couldn't activate or deactivate user accounts due to incorrect field assignment
🚀 Live Event Streaming for Pipeline Runs

You can now stream custom events in real-time from your running pipelines! Call zenml.streaming.publish() from inside any step or dynamic pipeline to push events that can be consumed via Server-Sent Events (SSE). Enable this feature by setting stream_broker_implementation_source in your server configuration. The initial implementation includes a Redis-based broker with automatic catch-up, gap signaling, and idle cleanup. PR #4804

📊 Pipeline Run Statistics Endpoint

A new POST /api/v1/runs/statistics endpoint lets you query aggregated metrics across your pipeline runs. Group by status, pipeline, stack, user, time buckets (hour/day/week/month), metadata values, tags, and more. Calculate averages, sums, min/max over duration, step counts, cached steps, output artifacts, or custom numeric metadata. Perfect for building dashboards and analytics. PR #4860

🎯 Richer Weights & Biases Integration

The W&B experiment tracker now automatically adds ZenML pipeline and step metadata to your W&B runs, groups runs by pipeline execution, and records W&B identifiers back to ZenML step metadata. You can now configure custom groups, job types, run configs, explicit or deterministic run IDs, resume behavior, and pass through additional wandb.init kwargs for complete control over your experiment tracking. PR #4838

🤖 Agentic Human-in-the-Loop Pipeline Example

A new example demonstrates building dynamic agentic pipelines with human approval gates. The example shows how to plan agent tasks, fan them out with step.map(), summarize results, pause execution with zenml.wait() for human review, and branch the final action based on the decision. Includes clean lineage tracking with Annotated step outputs. PR #4849

📝 Structured Logging and OpenTelemetry Instrumentation

The ZenML server now supports structured logging with OpenTelemetry instrumentation. Configure console output with the new ZENML_CONSOLE_LOGGING_FORMAT environment variable, choosing between console, json, or text formats. Server logs use a clean structured layout with timestamps, levels, logger context, and optional JSON fields. PR #4781

🛠️ Enhanced Developer Experience
  • String Type Annotations: Step and pipeline definitions now support string annotations, either explicitly quoted or via from __future__ import annotations. PR #4843
  • Build Cache Mounting: Python package installations during Docker image building can now mount a build cache for faster builds. PR #4820
  • Improved Wait Condition Input: When resolving wait conditions interactively with string schemas, you can now input raw strings without quotes—ZenML detects and handles this automatically. PR #4845
  • Better Logging Control: Restored custom console log formatting for non-DEBUG output, added step-name prefixes in terminal output during execution (disable with ZENML_DISABLE_STEP_NAMES_IN_LOGS=true), and kept stored logs clean and unformatted. The ZENML_LOGGING_FORMAT variable is now deprecated in favor of ZENML_CONSOLE_LOGGING_FORMAT. PR #4851
🎨 Dashboard Improvements
  • Improved scrolling behavior for the pipeline timeline view. PR #1053
  • Pipeline runs now properly fail when the user pipeline function cannot be imported (e.g., due to missing packages), instead of staying in a running state indefinitely. PR #4832
  • Keyboard interrupts during active wait conditions are now handled more gracefully, preventing unclear error messages when the server transitions to STOPPED status. PR #4835
  • Fixed a copy-paste bug in the user update endpoint where admins couldn't activate or deactivate user accounts—the code was incorrectly writing is_admin instead of active to the safe update object. PR #4839
What's Changed
New Contributors

Full Changelog: https://github.com/zenml-io/zenml/compare/0.94.4...0.94.5

View original

Upgraded? How did it go?

Discussion