Temporal v1.31.0

v1.31.0
Added 4
  • Add support for Serverless Workers on AWS Lambda with automatic invocation, scaling, and shutdown via the new Worker Controller Instance (WCI) component
  • Add immutable Principal field to workflow history events for server-computed, trustworthy attribution of actions
  • Add passwordCommand config option for SQL datastores to enable IAM-based authentication for cloud-managed databases
  • Add new current_chasm_executions table in core schema to support CHASM separate archetype ID spaces
Changed 3
  • Update core schema to MySQL v1.19, PostgreSQL v1.19, and SQLite v1.11
  • Update visibility schema to MySQL v1.14, PostgreSQL v1.14, and Elasticsearch v14 with new TemporalExternalPayloadSizeBytes and TemporalExternalPayloadCount fields
  • Overhaul Nexus error model to enable handlers and operations to convey their own messages
Removed 1
  • Remove Nexus feature flag and enable Nexus by default

Schema changes

[!WARNING] Before upgrading your Temporal Cluster to v1.31.0, you must upgrade core and visibility schema to the following:

  • Core:
    • MySQL schema v1.19
    • PostgreSQL schema v1.19
    • SQLite schema v1.11
  • Visibility:
    • Elasticsearch schema v14
    • MySQL schema v1.14
    • PostgreSQL schema v1.14

Please see our upgrade documentation for the necessary steps to upgrade your schemas.

Please see CHASM section below for more information about the core schema change if you have a custom persistence implementation.

Temporal database schema details

MySQL v1.19, PostgreSQL v1.19, SQLite v1.11 — adds a new current_chasm_executions table for CHASM separate archetype ID spaces:

CREATE TABLE current_chasm_executions(
  shard_id, namespace_id, business_id, archetype_id,
  run_id, create_request_id, state, status, start_version,
  start_time, last_write_version, data, data_encoding,
  PRIMARY KEY (shard_id, namespace_id, business_id, archetype_id)
);
Visibility schema details

MySQL v1.14, PostgreSQL v1.14 — adds TemporalExternalPayloadSizeBytes and TemporalExternalPayloadCount as generated columns with indexes.

Elasticsearch v14 — new index template version with TemporalExternalPayloadSizeBytes and TemporalExternalPayloadCount fields.

CHASM persistence schema change (custom persistence implementations only)

CHASM framework now supports separate businessID spaces for different archetypes and includes corresponding schema changes for SQL core databases. This is achieved by storing multiple current mutable state records in the database, one for each archetype.

If you maintain a custom persistence implementation (Cassandra-based or SQL-based), you must update it to support separate businessID spaces as well. All current mutable state related persistence requests now include a new ArchetypeID field, and your persistence implementation should read or update the current mutable state record for that archetype. Please refer to https://github.com/temporalio/temporal/pull/8907 (Cassandra) or https://github.com/temporalio/temporal/pull/8915 (SQL) for sample changes.

Serverless Workers (pre release)

This version of Temporal Server adds support for Serverless Workers. This feature allows Temporal Workers to run on serverless compute platforms - with AWS Lambda being the first supported platform. Temporal can automatically invoke, scale, and gracefully shut down (scaling to zero, if appropriate) Temporal Workers on your behalf based on workload volume and metrics. See the Serverless Workers documentation to get started.

Invocation is handled by a new server component, the Worker Controller Instance (WCI), which monitors Task Queues and dispatches invocations to a configured compute provider. WCI is disabled by default and must be enabled via the workercontroller.enabled dynamic config key. See Enable the Worker Controller Instance for complete configuration instructions.

Self-hosted setup requires network reachability from the Lambda execution environment to the Temporal frontend, AWS credentials for the server to assume customer IAM roles, and an IAM role in the target account granting lambda:GetFunction and lambda:InvokeFunction. See the self-hosted setup guide for full setup instructions.

Principal Attribution

Adds a server-computed, immutable Principal field to workflow history events, providing trustworthy attribution for "who did this?". Unlike the client-supplied identity field, Principal is derived from authenticated context and cannot be spoofed.

Principal has a Type / Name pair (e.g. jwt/alice@company.com, temporal/internal) and is surfaced in workflow history, CLI, Web UI, and history exports.

The default Authorizer populates Principal from the JWT sub claim. Users with custom Authorizer implementations can populate Principal by setting the new Principal field on authorization.Result.

To enable this feature, set the dynamic config system.enablePrincipalAttribution to true.

Cloud IAM auth for SQL datastores

https://github.com/temporalio/temporal/pull/9879 introduced new passwordCommand config option for SQL datastores as an alternative to the static password field. When set, Temporal executes the specified command and uses its stdout as the database password, re-running the command on each new physical connection so short-lived credentials stay fresh.

This enables IAM-based auth for cloud-managed databases (e.g. AWS RDS, GCP Cloud SQL) by invoking the provider's CLI. For example, configuring the passwordCommand to:

  • aws rds generate-db-auth-token for AWS RDS
  • gcloud sql generate-login-token for GCP Cloud SQL

Supported for MySQL and PostgreSQL. Token lifetime is managed through existing config maxConnLifetime.

Nexus

💥 Overhaul Nexus error model

Complete rework of how Nexus errors are handled and converted (https://github.com/temporalio/temporal/pull/9290).

This change enables Nexus handler errors and operation errors to convey their own messages as opposed to being wrappers for an underlying cause. The change was made to better support Nexus in all languages. There may be slight changes to errors / exception structures in the SDKs.

Remove Nexus feature flag and make Nexus work out of the box

Nexus is now always enabled (https://github.com/temporalio/temporal/pull/9512).

Use token based routing by default with token-based callback routing (https://github.com/temporalio/temporal/pull/9513).

Nexus caller timeouts

Support schedule-to-start and start-to-close timeouts for Nexus operations. SDK upgrade required.

Bug fixes
Metrics

Worker Versioning

Worker Versioning GA

Worker Deployment APIs are now fully GA. The following APIs have been in Public Preview since server v1.28.0. They are now considered GA and users can rely on the signature and behavior consistency going forward.

  • DescribeWorkerDeployment
  • DeleteWorkerDeployment
  • ListWorkerDeployments
  • SetWorkerDeploymentManager
  • DescribeWorkerDeploymentVersion
  • DeleteWorkerDeploymentVersion
  • SetWorkerDeploymentCurrentVersion
  • SetWorkerDeploymentRampingVersion
  • UpdateWorkerDeploymentVersionMetadata
⚠️💥 Sunsetting Worker Versioning V1 (Version Sets) and V2 (Assignment Rules)

The following APIs which have been deprecated since v1.28.0 are now officially sunsetted and their support will be removed in the next server version (v1.32.0). Users should migrate their applications to use the Worker Deployment and Worker Deployment Version APIs instead.

APIs to be removed in v1.32.0:

  • UpdateWorkerBuildIdCompatibility
  • GetWorkerBuildIdCompatibility
  • UpdateWorkerVersioningRules
  • GetWorkerVersioningRules
  • GetWorkerTaskReachability
New Experimental APIs

The following APIs are added in this release and are in Pre-Release stage as of now. At this stage, the APIs are considered experimental and may see breaking changes in the future:

  • CreateWorkerDeployment
  • CreateWorkerDeploymentVersion
  • UpdateWorkerDeploymentVersionComputeConfig
  • ValidateWorkerDeploymentVersionComputeConfig
Other Changes
  • Add new ContinueAsNew versioning behavior that continues workflows on the ramping version.
  • Fixed bug around stuck workflow after reset when WorkflowUpdate was used.
  • Add deployment name and build ID as separate labels on backlog metrics.
  • Rate-limit worker deployment read APIs and move ListWorkerDeployments to the visibility quota.
  • Cache task queue routing info in the History service to reduce RPC fan-out.
  • Version transition from Drained/Inactive to Draining state upon workflow start/moved via Versioning Override is disabled by default until the implementation is optimized in the next release.
  • Some other minor bug fixes or improvements.

CHASM

  • CHASM framework is enabled by default in this release, but applications built on top of it are NOT enabled by default to allow safe downgrades.
  • CHASM framework now supports separate businessID spaces for different archetypes and includes corresponding schema changes for SQL core databases. See the Schema Changes section above if you maintain a custom persistence implementation.

Standalone Activities

  • Standalone Activities (in public preview) allow activities to run independently of workflows. This feature is gated by the activity.enableStandalone dynamic config flag, which is disabled by default. Set it to true to enable.
  • New APIs and capabilities: DeleteStandaloneActivity API for deleting a standalone activity execution.
  • Durability improvements: server-generated request IDs are now preserved across restarts, a standby task discard handler has been added, and the 1-day retention limit has been removed.
  • PollActivityTaskQueueResponse now includes additional fields needed by workers running without a parent workflow, such as currentAttemptScheduledTime and namespace. Termination and cancellation identity is now propagated through failure info.
  • Multiple timing and retry bugs have been fixed, including the Start-To-Close timeout bug and its metric timestamps, incorrect RetryState for schedule timeouts, and TimerTaskStatus reset issues. Heart beating, RecordActivityTaskStarted, and various error codes also received fixes.
  • Internal task queue name validation has been added for standalone activities and all user-facing APIs.

Task Queue Priority and Fairness GA

  • Task queue priority and fairness features are now GA.
  • As part of that, the new matcher component is enabled by default, which means priority keys are respected by default. The new matcher is fully compatible and switching should be transparent, but if you have any issues you can switch back by setting matching.newUseMatcher to false (affected task queues will reload immediately).
  • As before, fairness can be enabled on a task queue, namespace, or cluster level with the matching.enableFairness dynamic config. Migration between fairness-enabled and fairness-disabled queues is enabled by default.

Cassandra 5 support

  • Temporal Server now supports Cassandra 5.0.4 and later.
  • Cassandra 5 versions earlier than 5.0.4 are not supported.

Archival

  • Custom history and visibility archiver implementations can now be plugged in via the new experimental WithCustomHistoryArchiverFactory and WithCustomVisibilityArchiverFactory server options.
  • Configuration for custom archivers can be supplied under archival.<history|visibility>.provider.customStores.<myScheme> in the server config file, where matches the URI scheme used in your archival URIs (e.g. myScheme://...).
  • See this README for more details.
Helpful links to get you started with Temporal

Temporal Docs Server Samples Server Helm Chart

Docker images

Server Admin-Tools

Full Changelog: https://github.com/temporalio/temporal/compare/v1.30.4...v1.31.0

View original

Upgraded? How did it go?

Discussion