Troubleshooting

Work from the symptom. Each entry lists the likely cause and the fix.

Pods start but crash: cannot connect to the database

Symptom: server or worker pods crash-loop; logs show a connection or auth error to PostgreSQL, or a missing PG_BASE_URL.

Cause: the secret is not attached to the pod, or PG_BASE_URL is missing or wrong.

Fix:

• Confirm the component has envFrom pointing at the synced secret. The chart does not attach secrets automatically.
• Confirm the secret exists and has the key:

  kubectl get secret app-secrets -n workflows -o jsonpath='{.data.PG_BASE_URL}' | base64 -d

• Check network reachability from the cluster to the database (security group, VPC, or Neon allowlist), and that the URL uses the correct database name (workflows) and SSL mode.

Database TLS error: unable to verify the first certificate

Symptom: connection fails with UNABLE_TO_VERIFY_LEAF_SIGNATURE or a leaf certificate error, typically against RDS or Cloud SQL.

Cause: the managed database uses an internal CA the client does not trust.

Fix: set the SSL mode to no-verify in the connection URL (for example ...?sslmode=no-verify) or provide the provider CA bundle. no-verify keeps the connection encrypted while skipping certificate chain verification.

Migration job fails and blocks the upgrade

Symptom: helm upgrade hangs or fails on the pre-upgrade migration job.

Causes and fixes:

• The job cannot reach the database: same checks as above (PG_BASE_URL, reachability). Inspect it with:

  kubectl get jobs -n workflows
  kubectl logs job/workflows-migration-<revision> -n workflows

• On GCP, the migration hook runs before the chart's ServiceAccount is created. Create the service account out-of-band and set serviceAccount.create: false.
• The image is the wrong architecture: build and push images for linux/amd64 if you mirror them from an Apple Silicon machine.

Ingress has no address

Symptom: kubectl get ingress -n workflows shows no ADDRESS, or no load balancer is created.

Causes and fixes:

• AWS: the AWS Load Balancer Controller is not installed or lacks IAM permissions. Confirm its deployment is running in kube-system and check its logs.
• GCP: GKE uses the kubernetes.io/ingress.class mechanism. Ensure className: gce and the required annotations (static IP name, managed certificate) are present, and that the static IP is global.

TLS certificate not provisioning (GCP)

Symptom: ManagedCertificate status is Provisioning or FailedNotVisible.

Cause: Google-managed certificates only issue after DNS resolves to the load balancer's IP.

Fix: create the A record from your domain to the reserved static IP, then wait (up to an hour). Verify with:

kubectl describe managedcertificate workflows-cert -n workflows

Image pull errors

Symptom: pods stuck in ImagePullBackOff or ErrImagePull.

Causes and fixes:

• Wrong global.imageRegistry, or the cluster cannot authenticate to the registry.
• AWS: the node role needs ECR pull permission (usually included by eksctl).
• GCP: grant the GKE node pool service account artifactregistry.reader; image pulls authenticate as the node service account, not the Workload Identity service account.
• Tag mismatch: confirm the image tag exists for the chart version you deploy.

Access denied to storage, KMS, or secrets

Symptom: the server logs AccessDenied when reading buckets, decrypting with KMS, or fetching secrets.

Cause: the workloads identity is not wired to the Kubernetes service account.

Fix:

• AWS (IRSA): the service account must carry the eks.amazonaws.com/role-arn annotation with the Terraform workloads_role_arn, and the cluster OIDC provider ARN must have been passed to Terraform.
• GCP (Workload Identity): the service account must carry the iam.gke.io/gcp-service-account annotation, and the Google service account must be bound to it (Terraform does this when the namespace and service account names match your values).

External Secret not syncing

Symptom: the target Kubernetes secret is empty or missing.

Fix:

kubectl get externalsecret -n workflows
kubectl describe externalsecret app-secrets -n workflows

Check that the SecretStore exists and authenticates, that remoteRef matches the provider secret path, and that the provider secret has values (not just the empty placeholders Terraform seeds). Force a resync:

kubectl annotate externalsecret app-secrets -n workflows force-sync=$(date +%s) --overwrite

Wrong cluster: gcloud auth error while running AWS commands

Symptom: a kubectl or helm command fails with failure while executing gcloud ... Reauthentication failed.

Cause: your kubeconfig current-context is on a GKE cluster while you intend to act on EKS (or vice versa). The context is persisted and shared across shells.

Fix:

kubectl config current-context           # see where you are
aws eks update-kubeconfig --name <cluster> --region <region>   # point at EKS
# or
gcloud container clusters get-credentials <cluster> --region <region> --project <project>

Getting help

When contacting support, include: the chart version and image tag, the cloud and region, kubectl get pods -n workflows, and the relevant component logs. Redact secrets before sharing.