Operations

This page covers running the platform after the initial deployment.

Upgrades

Upgrading is a helm upgrade with a new chart version and matching image tag. Because images track the chart's appVersion, moving to a new chart version moves all four images together.

helm upgrade workflows infrastructure/helm/workflows \
  -f my-values.yaml -n workflows --version <new-chart-version>

On every upgrade the migration job runs first as a pre-upgrade hook and must complete before the new pods roll out. If it fails, Helm halts the upgrade.

Rollback

helm history workflows -n workflows
helm rollback workflows <revision> -n workflows

Note that schema migrations are forward-only. A rollback of the application does not revert the database schema; ensure new schema changes remain backward-compatible with the previous app version, or restore the database from a backup taken before the upgrade.

Database migrations

Migrations run automatically as part of install and upgrade. To run them independently (for example, after restoring a database), you can re-run the chart with only migrations enabled, or set migrations.enabled: false to skip them during an upgrade when you have already applied them.

Always back up the database immediately before a migration in production.

Scaling

• Autoscaling is on by default. Tune minReplicas/maxReplicas and the CPU/memory targets per component in your values.
• For the worker, consider scaling on Temporal queue depth using worker.autoscaling.metrics (custom or external metrics) instead of CPU, so it scales with backlog rather than instantaneous CPU.
• Enable PodDisruptionBudgets (pdb.enabled: true) per component so node drains and cluster upgrades do not take all replicas down at once.

Secret rotation

Secrets live in your cloud secrets manager, so rotation happens there:

1. Write the new value to the provider secret (Secrets Manager or Secret Manager).
2. The External Secrets Operator resyncs on its refreshInterval (default 1h). To apply immediately, force a sync:

kubectl annotate externalsecret app-secrets -n workflows \
  force-sync=$(date +%s) --overwrite

3. Restart the affected workloads so they pick up the new environment values:

kubectl rollout restart deployment -n workflows

Backups and disaster recovery

• Database: use your provider's automated backups (RDS snapshots, Cloud SQL backups) or logical dumps on a schedule. The database is the only stateful component; the cluster workloads are stateless.
• Object storage: enable bucket versioning (the Terraform defaults do) so overwritten or deleted objects are recoverable.
• Configuration: keep your Terraform state and Helm values under version control (values without secrets). This makes the whole deployment reproducible.
• Recovery: to rebuild, re-apply Terraform, restore the database from a backup, and run helm upgrade --install. Because definitions are stateless in-cluster, recovery is Terraform plus a database restore plus a Helm install.

Monitoring and logs

# Pod status and recent events
kubectl get pods -n workflows
kubectl get events -n workflows --sort-by=.lastTimestamp

# Component logs
kubectl logs -f -l app.kubernetes.io/component=server -n workflows
kubectl logs -f -l app.kubernetes.io/component=worker -n workflows

The server exposes GET /health. Point your uptime and load-balancer health checks at it. Ship pod logs to your logging stack (CloudWatch, Cloud Logging, or your own).

Cost control and teardown

Cluster and networking dominate cost. To reduce spend in non-production environments without losing the hard-to-recreate resources:

• Scale to zero: scale the node group (or the deployments) to zero to stop compute charges while keeping the cluster, database, buckets, secrets, and auth.
• Delete the node group: remove worker nodes (the largest variable cost) while keeping the control plane, VPC, and identity intact. Rebuild by recreating the node group and running helm upgrade --install.
• Full teardown: delete the cluster entirely for maximum savings. Snapshot the database first if it lives in the cluster VPC. The Terraform-managed resources (buckets, secrets, KMS, auth, registry) survive and are cheap to keep.