Health endpoints
Three health endpoints, none of them authenticated.apps/api/app/main.py pings FerretDB on every request. degraded means the process is alive but the database is unreachable — the API answers, and every route that touches data fails. Treat degraded as down.
The runtime’s handler is a static response. It confirms the process is serving HTTP and nothing else: it does not check the API, MinIO, or any AI provider. A runtime returning ok can still fail every call. To test the path that matters, hit the answer webhook, which does reach the API to load the agent:
<Stream> URL. An error here is a real fault; /health would not have shown it.
The gateway reports degraded when a deployed slot’s upstream fails its check, and names which one in upstreams. It is a separate Compose project — see Overview.
Compose’s own healthchecks cover only postgres, minio, and redis. api, runtime, arq-worker, and campaign-orchestrator have no healthcheck: block, so docker compose ps shows them as running whether or not they are working. Probe them yourself:
Which logs matter
Every long-lived service uses thejson-file driver with max-size: 10m and max-file: 3 — 30 MB per container, then the oldest file is dropped. That rotation is set on postgres, ferretdb, api, minio, redis, arq-worker, campaign-orchestrator, and runtime.
Campaigns need both worker containers, and they answer different questions. The orchestrator decides when a batch runs; the ARQ worker runs it. A campaign stuck at zero progress is usually the worker; a campaign that stopped mid-list is usually the orchestrator. See Workers and orchestrator.
For a live call, follow both sides at once — the runtime logs the pipeline, the API logs what the runtime asked it for:
DEBUG=true in the root .env and recreating the stack. DEBUG reaches the API through env_file only and is deliberately not interpolated in docker-compose.yaml; the reason is in Environment variables.
Backups
VoicEra keeps state in four places. A backup that covers fewer than four is incomplete.
Back up the database logically, through
pg_dump against the voicera_oss_postgres container — not by copying the volume of a running Postgres:
-Fc (custom format); it restores selectively and compresses. Substitute your own MONGODB_USER for admin if you changed it — Compose reuses that value as POSTGRES_USER.
MinIO copies out with the mc client, using the same image the stack already pulls:
docker network ls — Compose prefixes app-network with the project name, which defaults to the directory name.
Chroma and Redis are file trees; copy them with a throwaway container while the stack is stopped:
voicera_oss_redis_data for Redis. Redis is worth capturing mainly so a restore does not resurrect stale queue entries; if you are willing to lose in-flight campaign batches, you can skip it and let the queue rebuild.
Restoring
Restore into a stack whose stores are running but whose application containers are stopped, so nothing writes underneath you.-Fc dump:
Rotating secrets
Three generated secrets, three very different rotation stories.
Rotating the Fernet key has no migration path in the code:
apps/api/app/services/secret_crypto.py decrypts with exactly one key and raises Failed to decrypt ProviderAuth credentials for anything the current key cannot open. To rotate deliberately, export the credentials you need first (GET /api/v1/auth/{provider} returns secrets unmasked to an admin), change the key, restart, and re-upsert.
Infrastructure passwords — MONGODB_PASSWORD, MINIO_ROOT_PASSWORD, REDIS_PASSWORD — need coordinated changes because Compose interpolates each into more than one place. MONGODB_PASSWORD is also POSTGRES_PASSWORD, and changing it against an initialised Postgres volume does not change the existing database user. REDIS_PASSWORD is the easy one: Compose rebuilds REDIS_URL from it, so changing it and recreating is enough. Details in Security hardening.
Capacity signals
VoicEra exposes no metrics endpoint. Capacity is read from the API, from Docker, and from the logs.
Scale in this order, because it matches where the load actually lands: runtime first (one WebSocket per live call), then the ARQ worker, then the API. The campaign orchestrator does not scale — see Production deployment.
Neither recordings nor call logs are ever deleted by VoicEra. Budget disk for the full retention you intend, and prune deliberately.
Restart and recovery
Routine restart, preserving all data:make restart runs application-down then application-up. application-down runs docker compose down with no -v, so the volumes survive. application-up ensures the three secrets exist in .env before starting, which is why it is preferred over a bare docker compose up.
Restarting one service in place, without touching the others:
restart after an environment change that the container reads at runtime; use up -d --build after a code or Dockerfile change.
What survives a restart, and what does not:
Losing the orchestrator’s in-memory state is recoverable by design: the completion monitor runs every 60 seconds, re-reads every
running campaign from the database, and falls back to last_activity_at, last_batch_scheduled_at, or started_at when it has no in-memory record. A campaign interrupted by a restart resumes within about a minute without intervention.
Recovery checks after any restart:
running still is. One that was paused by the circuit breaker before the restart stays paused — that state is in the database, and only POST /{campaign_id}/resume clears it.
If the API will not start, read its logs first. The failure is almost always one of: SECRET_KEY missing (Compose refuses to interpolate and the stack never starts), FerretDB not yet accepting connections, or a startup exception from initialize_database, which is raised and logged before the process exits.