Runbook

Rotate the production database credentials

Quarterly rotation of api_rw on prod-pg-1. Zero downtime — but only if step 4 finishes before you touch step 5.

When
Quarterly
Takes
~15 min
Downtime
None
Needs
db-admin

Before you start

All four. They take a minute together.

  • You are on the VPN and reach the primary.
    psql -h prod-pg-1 -c '\conninfo'
  • The deploy freeze is off.
    ./bin/freeze status # => open
  • No migration is running.
    ./bin/migrations running # => 0
  • You can write to the secret store.
    vault kv get -field=version \ secret/prod/db

Stop if any is false Rotating during a migration leaves the migrator holding a dead connection. Wait it out.

The procedure

Two roles exist at once for a few minutes. That overlap is the design — nothing restarts, nothing has to be timed.

  1. Announce it

    A rotation nobody expected costs more than the one you announce.

    ./bin/announce "rotating api_rw,
  ~15 min, no downtime expected"posted to #eng-oncall

  2. Generate the new secret

    Written to a staging slot the application does not read yet.

    export NEW=$(openssl rand -hex 24)vault kv put secret/prod/db/next \
  password="$NEW"Key       Value
version   8

  3. Create the successor role

    It inherits from the api_rw group role, so you never re-grant table by table. 

    psql -h prod-pg-1 \
  -f sql/rotate.sql \
  -v pw="$NEW" -v role=api_rw_q3CREATE ROLE
GRANT

  4. Point the application at it

    This is the step that matters. The sidecar reloads on its own, no restart. Until it returns, the old role is still serving traffic. 

    vault kv put secret/prod/db \
  username=api_rw_q3 \
  password="$NEW"./bin/wait-for-rollout \
  --key secret/prod/db \
  --timeout 120sall 14 pods on version 9 (47s)

Verify — between steps 4 and 5

Zero on the old role for five straight minutes. That is the bar. 

psql -h prod-pg-1 -c "select usename,
  count(*) from pg_stat_activity
  where datname='app' group by 1" api_rw_q3 | 42
 api_rw_q2 |  0

If the old role still has connections: a pod missed the reload. Restart it.

  1. Retire the old role

    Only once the verification check has been clean for five minutes.

    psql -h prod-pg-1 \
  -c "drop role api_rw_q2"DROP ROLE

    If it says objects depend on it: something still owns tables as that role. Stop and escalate — never REASSIGN OWNED on production alone.

If it goes wrong

Rollback — safe at any point before step 5

The old secret version and the old role both still exist, so this is a sixty-second undo — not a deploy. 

vault kv rollback -version=8 \
  secret/prod/db./bin/wait-for-rollout \
  --key secret/prod/dball 14 pods back on version 8 (39s)

When to escalate

  • Connections have not drained after 10 minutes. Page /page data-platform and leave both roles in place.
  • DROP ROLE fails for any reason. Stop and page. Two live roles is safe; half-rotated is not.