Postgres High Availability & Internal DNS

Temps runs high-availability PostgreSQL as a pg_auto_failover cluster: one monitor plus two or more data members, one of which is elected primary. Linked apps get a multi-host POSTGRES_URL with target_session_attrs=read-write, so after a failover the database driver reconnects to whichever member is the new primary. Cluster DNS adds stable *.temps.local names that follow the primary within about 6 seconds; it is experimental and off by default (cluster_dns.enabled), and you need it once cluster members run on worker nodes.

Create an HA cluster

HA clusters are a PostgreSQL-only option for managed services. A cluster needs:

• Worker nodes. The topology choice only appears in the dashboard once at least one worker node has joined with temps join (see Multi-node). For real failover, put each member on a different node; members on the control plane share one machine.
• One monitor and at least two data members. The monitor coordinates failover; pg_auto_failover elects one data member as primary at runtime and the rest replicate from it.

From the dashboard:

1. Open Services and click Create Service
2. Choose PostgreSQL, then the Cluster (HA) topology
3. Add the monitor and two or more data members, and assign each to a node
4. Click Create. Every member runs the gotempsh/postgres-ha:18-bookworm-walg image

From the API, send topology: "cluster" with a members list:

curl -X POST "https://your-temps-instance/api/external-services" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "mydb",
    "service_type": "postgres",
    "parameters": {},
    "topology": "cluster",
    "members": [
      { "role": "monitor", "node_id": 1 },
      { "role": "replica", "node_id": 2 },
      { "role": "replica", "node_id": 3 }
    ]
  }'

Link the cluster to a project like any other service. Temps creates a per-project database on the current primary and injects POSTGRES_URL, POSTGRES_HOST, POSTGRES_PORT, POSTGRES_DB, POSTGRES_USER and POSTGRES_PASSWORD. POSTGRES_URL lists every data member:

postgresql://user:password@member-1:5432,member-2:5432/app_db?target_session_attrs=read-write

libpq, psycopg, node-postgres, sqlx and pgx parse this multi-host form. Go's lib/pq does not; use pgx instead.

When every member runs on the control plane, the hosts in POSTGRES_URL are container names that Docker resolves on its own. Once any member runs on a worker node, they are per-member *.temps.local names (mydb-1.mydb.temps.local), which resolve only after you turn on cluster DNS under Worker Nodes → Cluster DNS.

Backups and restore

Cluster backups use WAL-G: a base backup pushed from the current primary plus continuous WAL archiving to your S3 source. Temps writes the WAL-G credentials to every data member, so archiving keeps working after a failover promotes another member. WAL archiving only starts after the primary's next restart (archive_mode is a restart-only setting); the base backup is on S3 straight away.

An in-place restore tears down the members, seeds the primary from the latest WAL-G backup and rebuilds the cluster with the same names, so the connection string does not change. Two limits apply:

• Single host only. Restore is refused while any member runs on a worker node. Move all members to the control plane first. Backups have no such limit.
• Latest backup only. Point-in-time recovery is not available for clusters yet.

The phase-by-phase detail is in Set Up Managed Services; schedules and S3 sources are covered in Backups.

How cluster DNS works

The internal DNS plane has two halves: a control-plane registry that decides what each name should resolve to, and a per-node resolver that serves those answers to containers.

The per-node resolver

The temps-dns-resolver component runs a Hickory-based DNS server embedded in the Temps agent (temps-agent) on every node. It:

• Serves authoritative A/AAAA/CNAME answers for the *.temps.local zone over both UDP and TCP.
• Forwards out-of-zone queries (package mirrors, third-party APIs) to upstream public resolvers, so containers that use it as their first nameserver can still reach the public internet. The host's own DNS servers are added after it as fallbacks (up to three nameservers in total).
• Listens on the bridge gateway IP (port 53), so every container on the node sees it as a nameserver. On worker nodes the agent's resolver also binds 127.0.0.53:53 for host-local lookups. The control plane runs its own resolver, fed directly from its database, that binds only the bridge gateway: on Ubuntu 127.0.0.53 belongs to systemd-resolved.
• Persists every applied zone generation to a local zone.json snapshot, so it can serve stale-but-correct records across resolver restarts and control-plane outages.

The resolver never reads the database directly. It stays in sync by long-polling the control plane:

GET  {control_plane_url}/api/internal/nodes/{node_id}/dns/changes?since={generation}
POST {control_plane_url}/api/internal/nodes/{node_id}/dns/ack

It sends since=N (its last-applied generation), receives a diff or a full snapshot, applies it, then ACKs the generation it applied.

The control-plane registry

On the control plane, the DnsRegistry service (in the temps-dns crate) is the sole writer to the service_endpoints table. Every mutation bumps a cluster-wide monotonic generation counter stored in a dedicated dns_generation singleton table, incremented at write time via UPDATE dns_generation SET current = current + 1 ... RETURNING current inside the same transaction — not derived from MAX(service_endpoints.generation), so the counter survives full-table deletes and rebuilds. This is what makes the since=N long-poll diff well-defined.

Because Docker assigns a fresh IP to a container on every create, the registry rewrites a cluster member's records on every container start, not just first creation — using an atomic delete-and-insert so consumers never see a window with both the old and new IP present.

Record tiers

Records come in tiers:

The cluster VIP (<svc>.temps.local) returns one A record per healthy data member — the resolver returns all matching records, so a single lookup gets the whole set. Combined with libpq's target_session_attrs=read-write, writes land on the primary while plain read connections fan out across the set.

When a member has no overlay IP — for example the pg_auto_failover monitor running on the control plane, or a single-host setup — the registry falls back to the node's underlay address plus the published host port, so the FQDN still resolves through Docker's port forward.

Failover behavior

A per-cluster role reconciler runs on the control plane on a fixed tick (TICK_INTERVAL = 5s). On each tick it queries the pg_auto_failover monitor and atomically rewrites the Tier 3 role/VIP records. Reconcilers are re-spawned at plugin startup for every running cluster, so they survive control-plane restarts.

After a failover/promotion, the reconciler refreshes the role-aliased records on its next tick. Application connections to <svc>.temps.local (or primary.<svc>.temps.local) then follow the new primary without a restart. End-to-end, the promotion-to-DNS-flip is bounded at roughly 6 seconds: the 5s reconciler tick plus the ~1s agent sync long-poll.

Container wiring

Cluster member containers are wired to the resolver automatically. When the agent creates a cluster member container, it sets Docker's HostConfig.dns to the gateway IP of the temps0 overlay network, so *.temps.local resolves natively from inside the container.

This fails open: on single-host setups where the temps0 overlay isn't bootstrapped, DNS injection is skipped and Docker's default resolver is used.

Configuration

Once cluster_dns.enabled is on, the resolver itself is configured by the agent at startup — there are no further user-facing flags for it. These are the defaults baked into ResolverConfig:

Worked example

With cluster DNS enabled and a Postgres HA cluster running, an app deployed on the same Temps instance can connect through the cluster names instead of the injected multi-host POSTGRES_URL. The members and zone are created for you; you only need the names.

Connect to the cluster VIP

Point your connection string at <svc>.temps.local and let libpq route writes to the primary:

postgresql://appuser:password@mydb.temps.local:5432/appdb?target_session_attrs=read-write

• mydb.temps.local resolves to all healthy data members (multi-A round-robin).
• target_session_attrs=read-write makes the client connect to whichever member accepts writes (the primary).

Pin reads to replicas, writes to the primary

If you want to send read traffic explicitly to replicas:

# Writes — always the current primary
postgresql://appuser:password@primary.mydb.temps.local:5432/appdb

# Reads — fan out across healthy secondaries
postgresql://appuser:password@replica.mydb.temps.local:5432/appdb

Verify resolution from inside a container

From any container on the node, or from the host by querying the bridge gateway IP (the first nameserver in a container's /etc/resolv.conf):

# From inside a cluster member container
nslookup mydb.temps.local
nslookup primary.mydb.temps.local

# From the host, against the resolver on the bridge gateway IP
dig @<bridge-gateway-ip> mydb.temps.local

After a failover, re-run the primary.mydb.temps.local lookup — within roughly 6 seconds it returns the new primary's address, and existing connection strings keep working unchanged.

What happens after a failover

The full sequence, from the moment the primary goes down to clients reconnecting, is:

1. Primary is detected down. pg_auto_failover's monitor notices the current primary has stopped reporting healthy.
2. A replica is promoted. pg_auto_failover promotes a healthy secondary to primary; the monitor's pgautofailover.node view now reports the new primary.
3. The A record is rewritten. On its next 5s reconciler tick, the role reconciler queries the monitor, sees the new primary, and atomically rewrites the primary.<svc>.temps.local (and VIP) records to the new address.
4. Existing connections drop. primary.<svc>.temps.local is a plain A record, not a connection proxy, so live TCP sessions to the old primary are not migrated — they fail when the old primary goes away.
5. Clients re-resolve and reconnect. A client whose connection dropped re-resolves the name (the short 5s TTL means it gets the new address quickly) and reconnects to the new primary automatically — no connection-string change.

Concretely, watching the record flip:

# Before failover
nslookup primary.mydb.temps.local
# → 172.20.5.10  (old primary)

# Primary fails; pg_auto_failover promotes a replica.
# On the reconciler's next 5s tick, it rewrites the primary A record
# (TTL 5s), and the agent applies it on its next ~1s sync poll.

# After failover — same command, new address
nslookup primary.mydb.temps.local
# → 172.20.5.11  (new primary)

End-to-end this is bounded at roughly 6 seconds (the 5s reconciler tick plus the ~1s agent sync). Because primary.<svc>.temps.local is a plain A record — not a connection proxy — live TCP sessions to the old primary do not migrate: a client whose connection drops simply re-resolves the name and reconnects to the new primary, with no connection-string change. The short 5s TTL keeps stale answers from outliving the flip.

Notes & gotchas

• The /api prefix is load-bearing. The DNS sync handlers are declared without it, but the plugin runtime mounts plugin routes under /api. The served paths are /api/internal/nodes/{node_id}/dns/changes and /api/internal/nodes/{node_id}/dns/ack.
• Stale-but-correct over down. Every resolver failure mode is "keep serving the last snapshot." If the control plane is unreachable, the resolver serves the records from its zone.json snapshot rather than failing lookups.
• DNS injection fails open. On single-host setups without the temps0 overlay, the agent does not set HostConfig.dns, and containers fall back to Docker's default resolver. Cluster FQDNs only resolve natively when the overlay is up.
• Underlay fallback. Members without an overlay IP (such as the monitor on the control plane) resolve to the node's underlay address plus the published host port, so the FQDN still works through Docker's port forward.
• <svc>.temps.local is not a single VIP. It is a multi-A answer across all healthy data members. Routing writes to the primary relies on target_session_attrs=read-write, not on the DNS layer choosing one address.

Next steps

For standalone Postgres, see Managed PostgreSQL; for restore modes and point-in-time recovery, see Restore & Recovery. Guide: Self-hosted PostgreSQL high availability with pg_auto_failover walks through the whole setup, from creating the cluster to failover and backups.