Environment Variables and Secrets

Cloud Deployment

When our applications are deployed, environment variables/secrets are organized and injected entirely differently.

Cloudflare Pages

Frontend

Our frontend apps deployed to Cloudflare Pages don't use secrets exposed directly to the browser. Environment variables needed at build time are hardcoded via the Terraform config.

PivotAdmin

The Next.js app for PivotAdmin deployed to Cloudflare Pages uses Pages Functions and its environment variable/secrets system to store the private key used to sign Tunnel requests as well as other necessary backend values.

AWS ECS

Secrets are stored in SMM parameter store as 'secret strings'.

Non-secret environment variables are stored in the Terraform ECS task definition source code that is committed to the pivot-internal repository.

ECS uses the task definitions to launch each task so the environment variables in the Terraform config, including both static values hardcoded and the referenced SMM values are injected into each task.

Local Development

For local development, we hardcode local-only environment configuration into the Docker Compose and Nx configurations.

For apps that run in Docker Compose, the environment variables are defined in the docker-compose.yaml file. For apps that run locally, environment variables are defined in source controlled .env files in each app's subdirectory.

Overriding Values in Docker Compose

For various reasons, you may want to override an environment variable value without editing the docker-compose.yaml file or otherwise hardcoding the value. This is essential if you are using actual real secrets in local development, because you want those to be ignored by Git.

For example, third-party service APIs such as Mux and OpenAI are not able to be emulated locally, so null values are provided in the docker-compose.yaml file. If you are running a service that is dependent on such values, you'll need to override the variable value that is hardcoded into the docker-compose.yaml.

These overwridden values can be set using the --env-file CLI argument (opens in a new tab) when running docker compose watch (or up) using your own .env.local file, which will be automatically ignored by Git. This is as simple as creating a .env.local file in the root of the repository with just the values you want to override and then running docker compose --env-file .env.local watch .

Note that this only works to override the values of variables that already exist in at least one service in the docker-compose.yml file. A variable key has to be defined before its value can be overridden.

Overriding Values using Nx

Outside of Docker Compose, environment variables can also be set and overridden by Nx. By default, Nx will inject environment variables based on the .env file in each app directory, which is committed to source control. You can create a .env.local file to override any variable defined in the .env and this file will be ignored by Git.

Only those apps that are intended to be run via Nx in local development have .env files maintained in the repo. Backend services that run in Docker Compose do not, because Nx is not used at runtime, Docker is.

Summary of Local Environment Variables Rules

1. Don't add real secrets to docker-compose.yaml or .env (or otherwise edit those files if you don't intend to commit and push those changes).

2. If overriding or setting Docker Compose variables, create a .env.local file in the repository root and use the --env-file argument. This will not be committed to source control.

3. If overriding or setting variables in the context of directly running npx nx commands, create a .env.local file in the application's subdirectory which Nx will merge with the default environment variables at runtime. This will not be committed to source control.

4. .env is committed to source control. .env.local is not

If you remember one thing from this article let it be: don't add real secrets to docker-compose.yaml or .env, add them to .env.local for Docker and apps/*/.env.local for Nx.

AWS SDK Environment Variables

For services that use the AWS SDK such as Buzzbuzz, it is not necessary to provide an ACCESS_KEY_ID or SECRET_KEY as environment variables. Instead, in local development LocalStack dummy credentials can be hardcoded (if STAGE == 'local') and in production, the SDK can use the ECS task role assigned to the service.