Skip to main content

Temporal Proxy

View Markdown

The Temporal Proxy is a gRPC proxy that sits between your Temporal SDK Clients, Workers, and the Temporal Web UI on one side and one or more upstream Temporal Services on the other. It handles Namespace translation, TLS termination, and optional payload encryption so your applications can target a single local endpoint while the proxy routes each request to the right upstream, whether that is a local development Service, a self-hosted Service, or Temporal Cloud.

Why use it

Without the proxy, connection details leak into your application code. Every Worker and Client has to know the upstream's host, TLS material, credentials, and the exact Namespace name the upstream expects. That couples your code to an environment: moving between a local Service, a self-hosted deployment, and Temporal Cloud becomes a code change.

The proxy owns that concern instead. Workers talk plaintext to a single local endpoint using a short Namespace name, and the proxy adds TLS, credentials, and Namespace translation on the way out. Point a Worker at a different Namespace and it reaches a different upstream, with no change to the Worker.

How it works

The proxy is built from a gateway and one proxy per upstream, connected by unix sockets:

  • The gateway is the single inbound endpoint that every Worker, SDK Client, and the Web UI connects to.
  • Each upstream has its own proxy that handles communication with that destination.

For each request, the gateway:

  1. peeks the target Namespace without parsing the payload; it is codec-transparent and relays raw frames in both directions.
  2. picks an upstream: the first matching routing rule, otherwise the system upstream for Namespace-less calls, otherwise the default.
  3. hands the request to that upstream's proxy over a unix socket.

The per-upstream proxy then rewrites the local Namespace to the name the upstream expects, attaches that upstream's TLS and credentials, forwards to the Temporal Service, and translates the Namespace back on responses. When payload encryption is enabled, it also seals payloads on the way out and opens them on the way back, so the upstream only ever stores ciphertext.

Terms

TermMeaning
gatewayThe single inbound gRPC endpoint that every SDK Client, Worker, and the Web UI connects to. It routes each request to an upstream by Namespace and request metadata, and never parses payloads.
upstreamA configured destination the proxy forwards to: a Temporal Service (local dev, self-hosted, or Temporal Cloud), or another Temporal Proxy.
system upstreamThe upstream that handles Namespace-less requests, such as the SDK's GetSystemInfo call on connect.
extension serverA gRPC service you run that the proxy calls out to for a capability it has no built-in backend for. Today that means wrapping data encryption keys as a key management backend.
Temporal ServiceA Temporal frontend the proxy connects to.

Prerequisites

  • One or more upstream Temporal Services to route to, such as a local development Service, a self-hosted Service, or Temporal Cloud.
  • The hostPort address for each upstream.
  • Any credentials the upstreams require, such as a Temporal Cloud API key or mTLS certificates.
  • Go installed, if you build the proxy from source. The container image and Helm chart do not require a local Go toolchain.

Install the proxy

Install the proxy binary with Go:

go install github.com/temporalio/temporal-proxy/cmd/proxy@latest

Pin an explicit version instead of @latest, using a tag from the releases page:

go install github.com/temporalio/temporal-proxy/cmd/proxy@vX.Y.Z

Pull the container image:

docker pull temporalio/temporal-proxy:latest

Install with Helm from the Temporal Helm repo, optionally pinning a chart version with --version:

helm install temporal-proxy temporal-proxy \
--repo https://go.temporal.io/helm-charts

Each chart release deploys a proxy version by default. Override it with --set image.tag=vX.Y.Z. Supply the proxy configuration under the config key of a Helm values file, as described in Deploy to Kubernetes.

Run the proxy with a configuration file passed through the -c (or --config) flag:

proxy serve -c config.yaml

--config also reads the PROXY_CONFIG environment variable, which is how the Helm chart points the proxy at its mounted configuration. See Observability for the remaining flags.

Observability

The proxy serves Prometheus metrics at /metrics on :9090 and logs JSON to stderr. Both the metrics listener and the log level are set with flags on proxy serve, each with an environment variable equivalent:

FlagEnvironment variableDefaultSets
--config, -cPROXY_CONFIGnonePath to the configuration file. Required.
--levelLOG_LEVELinfoLog level: debug, info, warn, error.
--metrics-addrMETRICS_ADDR:9090The host:port serving /metrics.
--metrics-namespaceMETRICS_NAMESPACEtmprl_proxyPrometheus namespace prefixed onto metrics.

Metric names are <namespace>_<subsystem>_<name>, so with the default namespace the routing counter is tmprl_proxy_router_decisions_total. There are three subsystems:

SubsystemMetricLabelsReports
serverrequests_totalmethod, codeRPCs served, by gRPC status code
serverrequest_duration_secondsmethodEnd-to-end time serving an RPC
routerdecisions_totalupstream, outcomeRouting decisions, by chosen upstream
routerforwarding_errors_totalupstream, reasonForwarding failures the router originated
encryptionvault_ops_totaloperation, result, namespaceEnvelope operations, sealing and opening payloads
encryptionvault_ops_duration_secsoperation, namespaceTime per envelope operation, end to end
encryptiondek_ops_totaloperation, resultOutcome of the AES-256-GCM step alone
encryptiondek_ops_duration_secsoperationTime in the AES-256-GCM step alone
encryptionkek_ops_totalprovider, operation, resultDEK wrap and unwrap calls to your KMS
encryptionkek_ops_duration_secsprovider, operationTime spent wrapping and unwrapping DEKs
encryptiondek_rotations_totalreasonDEK rotations, by why the DEK was replaced
encryptiondek_cache_hits_totalnoneReads served from the decrypted-DEK cache
encryptiondek_cache_misses_totalnoneReads that required a KMS unwrap
encryptiondek_cache_sizenoneCurrent entries in the decrypted-DEK cache

The encryption metrics only move when payload encryption is configured. They are layered, so pick the one that matches the question you are asking:

  • vault_ops_* is the whole envelope operation end to end, including any KEK call and cache lookup, and is the pair to alert on. It carries the local Namespace.
  • dek_ops_* is the symmetric AES-256-GCM step by itself, with the KEK work excluded. Its result is that step's own outcome, so a payload that encrypts cleanly and then fails to wrap its DEK counts as a success here and an error under kek_ops_total, which keeps the blame with the KMS.
  • kek_ops_* is the calls to your KMS. Watch kek_ops_total{result="error"}, since a failure to wrap or unwrap a DEK fails the request that needed it.

dek_rotations_total splits by reason: initial for a Namespace's first DEK, scheduled for the renewBefore pre-rotation, and on_demand for a DEK replaced at request time because no fresh one was ready. A rising on_demand rate means rotation is falling behind, so raise renewBefore. Compare the cache counters against cacheSize to see whether the cache is absorbing read traffic.

In this guide