Temporal: Workflows and Cron Jobs

Overview

Temporal workflows are a good way of running timed actions and complex orchestration.

Temporal workflows can run on a schedule, which means we also use Temporal for distributed cron (jobs that only run once per hour/day/etc. even if there are multiple horizontally scaled instances of a service.)

Deployment

Temporal is complicated to run. It consists of multiple stateless services which use Postgres, and also depends on Elasticsearch for at-scale deployments. Therefore, we use Temporal Cloud, which is available in various AWS regions. Each Temporal Cloud namespace is isolated logically, so we do not need a separate Temporal Cloud account for single-tenant deployments.

Authentication and Authorization

We utilize two primary namespaces: one for staging/development and another one for production. Each application operates its own pool of workers to ensure sepration of resources. We use Temporal's Cloud with mTLS, where the client is authenticated using an end-entity certificate. For each application, we generate an end-entity certificate and sign it with the root certificate. Temporal Cloud requires us to upload the public key of the root certificate. Each time the client connects, it presents a certificate signed by the root certificate's key. This allows Temporal Cloud to verify the client's identity and use the same certificate to encrypt traffic.