June 25, 2026 (3w ago)
Written by Temps Team
Last updated June 25, 2026 (3w ago)
To issue TLS certificates on demand for wildcard-DNS subdomains such as sslip.io, Temps issues each certificate lazily at the first TLS handshake instead of provisioning every hostname up front. Because the Temps operator does not control the sslip.io DNS zone, it cannot complete DNS-01 for a wildcard certificate. Pingora's certificate_callback finds no certificate, fails the first handshake quickly, and starts a Let's Encrypt HTTP-01 order in the background. A later connection succeeds after the certificate is available.
TL;DR: On the first TLS handshake for an eligible hostname, Pingora's
certificate_callbackfinds no cert, fails the handshake fast, and starts a background Let's Encrypt HTTP-01 order; the retry succeeds. The guards are what make it safe: a zone gate, in-flight dedup so a flood of connections to one new host fires exactly one issuance, a concurrency semaphore, an hourly cap, exponential backoff, and a 120-second issuance timeout. The hard scoping rule: issue only for stable hostnames, never ephemeral per-deployment ones, or every deploy churns certs and trips rate limits.
The clean way to do multi-tenant HTTPS is a wildcard certificate. You prove control of example.com once via a DNS-01 challenge (drop a TXT record at _acme-challenge.example.com) and Let's Encrypt hands you a *.example.com cert that covers every subdomain you'll ever create. One cert, infinite hostnames, zero per-tenant work.
This is exactly the basic flow covered in How to Set Up Custom Domains With Automatic SSL: you own the apex, you control DNS, you get a wildcard.
The wildcard approach falls apart the moment you don't control DNS for the apex. The motivating case in Temps is sslip.io, the wildcard-DNS service that resolves 1-2-3-4.sslip.io to 1.2.3.4 with no configuration.
It's perfect for giving a fresh server instant working hostnames before the user has wired up a real domain. But you can't DNS-01 a wildcard for sslip.io. You don't own the zone, and nobody will let you put a TXT record at its apex.
So every hostname needs its own certificate, proven via HTTP-01 (serve a token at http://<host>/.well-known/acme-challenge/<token>). That leaves two bad options and one good one:
| Approach | When certs are issued | Problem |
|---|---|---|
| Wildcard (DNS-01) | Once, up front | Impossible: you don't control the sslip.io zone |
| Pre-provision every hostname | At deploy time, eagerly | You don't know the hostnames ahead of time, and most are never visited. Wasteful, and it burns rate limits. |
| On-demand (HTTP-01 at handshake) | First time a browser connects | Needs careful guards, but issues exactly the certs that are actually used |
On-demand issuance is the only approach that fits the constraint. The cost is complexity: you're now doing certificate provisioning inside the latency-sensitive path of a TLS handshake, and you have to make absolutely sure a burst of traffic to a brand-new hostname doesn't turn into a stampede of duplicate ACME orders.
Blocking the handshake while you fetch a certificate is a non-starter. An ACME order takes 5–30 seconds; holding a TLS connection open that long ties up a proxy worker, times out browsers and load balancers, and a burst of new hostnames exhausts your connection budget instantly.
So: fail fast and issue in the background. When Pingora's certificate callback can't find a cert for an eligible hostname, it returns an error immediately, aborting the handshake. In the same breath it enqueues a background issuance.
The browser sees a connection failure and does what browsers do: it retries. By the time the retry lands (or the one after it), the certificate exists and the handshake completes normally.
Pingora's TLS callback is where all 3 decisions happen: in-memory cert store lookup (fast path), eligibility check, then a non-blocking enqueue.
TLS callback:
DynamicCertLoader::certificate_callbackincrates/temps-proxy/src/server.rsreads the SNI, loads the cert from an in-memory fast path, and — if no cert exists — callsmanager.try_enqueue(&sni, None)to hand work off to the background consumer. No blocking I/O. — verified atcrates/temps-proxy/src/server.rsline 129
Everything expensive (the ACME order, the HTTP-01 challenge, the database writes) happens off the handshake path.
This is the same shape as the wake-on-request pattern behind scale-to-zero dev environments: the first request pays a small, bounded penalty (a failed handshake, or a cold-start wake), the work happens asynchronously, and the retry finds everything ready. On-demand TLS is scale-to-zero for certificates.
A naive version of this feature is a denial-of-service vector against your own ACME account. The danger is obvious once you picture it: a browser opens 6 parallel connections to a new hostname (HTTP/2 preconnect, prefetch, favicon, the works), each one fails the handshake, each one triggers issuance, and now you've fired 6 identical ACME orders for one cert. Multiply by a page full of links to other new hosts and you've blown through Let's Encrypt's rate limits in seconds.
The OnDemandCertManager in crates/temps-proxy/src/on_demand_cert.rs exists to make that impossible. Its try_enqueue method layers 5 guards before a background task ever touches ACME:
OnDemandCertManager::try_enqueue(crates/temps-proxy/src/on_demand_cert.rs): zone subdomain gate, in-flight DashMap dedup (only the first concurrent caller proceeds), per-domain backoff check (readson_demand_backoff_until), hourly issuance cap — all synchronous, all before a tokio task is spawned. The background consumer then acquires a concurrency semaphore and wrapsDomainService::provision_on_demandin a 120-secondtimeout. — verified incrates/temps-proxy/src/on_demand_cert.rs
Each guard handles a distinct failure mode:
| Guard | Failure mode it prevents |
|---|---|
| Zone gate | Issuing certs for hostnames that have no business getting one |
| In-flight dedup | A connection flood to one new host firing N duplicate orders |
| Concurrency semaphore | A burst of distinct new hosts saturating your ACME concurrency |
| Hourly cap | Slow, sustained abuse grinding through your weekly rate-limit budget |
| Exponential backoff | Hammering Let's Encrypt for a host that keeps failing (bad DNS, etc.) |
| 120s issuance timeout | A wedged order pinning a semaphore permit and starving everything else |
provision_on_demand lives on DomainService and is a new trigger in front of the existing LetsEncryptProvider that powers ordinary custom-domain issuance. The only new persistence is an on_demand_cert_attempts audit table (one row per attempt, success or failure) and an on_demand_backoff_until column on domains.
Every guard above is defensive plumbing. The decision that makes or breaks this feature is which hostnames are eligible at all, and the answer is: only stable hostnames.
The console host and the per-environment aliases get on-demand certs. Ephemeral per-deployment hostnames (the ones marked is_calculated in deployment_domains, which change on every single deploy) never do.
The reasoning is brutal arithmetic. If a per-deployment hostname were eligible, every deploy would mint a brand-new hostname, the first visit would trigger a fresh ACME order, and a team deploying 20 times a day would be issuing 20 throwaway certificates a day, each one used for a few hours and then abandoned. That's not just wasteful; it's a fast path into Let's Encrypt's rate limits.
There is also a shared-rate-limit concern. Let's Encrypt applies its certificates-per-registered-domain limit to the registered domain. Hostnames such as app.192-0-2-1.sslip.io share the registered domain sslip.io, so unrelated users can draw from the same issuance bucket. Churning ephemeral certificates can therefore affect more than one Temps installation.
So eligibility is computed when routes are built and stamped onto each route as a cert_eligible boolean, never resolved in the hot path.
cert_eligibleonRouteInfo(crates/temps-routes/src/route_table.rs):truefor stable hostnames (console host, per-environment aliases),falsefor ephemeral per-deployment hostnames and custom user domains. By the time the TLS callback runs, it's already a boolean on the route — no DB query. — verified incrates/temps-routes/src/route_table.rs
That detail came directly out of the security review.
The security review before shipping changed the design in 3 concrete ways.
First, we removed a per-request database query from the proxy's request_filter hot path. The first draft looked up domain eligibility from the database on every request; that's a Postgres query on the most latency-sensitive code in the platform. The fix: precompute the cert_eligible flag onto the route when routes are built, so the hot path reads a boolean.
Pingora's latency budget is tight (it's part of why we chose Rust for the proxy in the first place).
Second, we added the 120-second issuance timeout. Without it, a single ACME order that hangs (a slow validation, a network partition to Let's Encrypt) would hold a semaphore permit indefinitely.
A handful of those and the semaphore is fully occupied by zombies, and no new host can ever get a cert. The timeout guarantees every permit is released within a bounded window.
Third, we aligned the per-IP limiter and the trust-boundary docs with reality. The original docs described limits that the code didn't quite enforce; the review forced the documentation and the actual limiter behavior into agreement. Trust-boundary docs that lie are worse than no docs, because someone will build on the lie.
There was also a quieter but load-bearing dependency fix: we upgraded instant-acme from 0.7.2 to 0.8.5. The old version couldn't deserialize challenge JSON from modern ACME servers: it failed with missing field 'token' because the challenge object shape had drifted.
On-demand issuance is all challenge handling, so a broken challenge deserializer is a hard blocker. The upgrade fixed it.
If issuance happens invisibly in the background, you need a way to see what it did; otherwise debugging "why isn't my hostname getting a cert" is pure guesswork.
Every issuance attempt writes a row to on_demand_cert_attempts, success or failure, with the host, the outcome, and the timestamp. That table is queryable from the CLI:
# What's the cert status for on-demand hosts?
temps domain cert-status app-prod.1-2-3-4.sslip.io
# List every domain currently using on-demand issuance.
temps domain list --on-demand
And the user-facing experience during the gap matters too. While a certificate is being provisioned, a plain HTTPS request can't succeed yet (there's no cert).
So on port 80 the proxy serves an honest 503 with a provisioning or failed status, so a human hitting the URL sees "we're getting your certificate" rather than a raw connection reset. It's a small thing, but it's the difference between a confusing failure and an explained one.
We also added real, Docker-gated end-to-end tests that issue genuine certificates against a Pebble ACME server, covering both the HTTP-01 on-demand path and, for contrast, a DNS-01 wildcard. Mocking ACME is how you ship a cert system that passes every test and fails against a real server, because the failure modes live in the protocol details. Pebble lets us exercise the actual handshake-to-cert loop in CI.
On-demand HTTPS is a niche tool, and it's worth being clear about that. If you control DNS for your apex domain, get a wildcard and move on: it's simpler, it's one cert, and it sidesteps every guard in this article. The basics in How to Set Up Custom Domains With Automatic SSL cover that case completely.
You reach for handshake-time issuance only when all three of these are true: you can't get a wildcard (you don't control the apex), you don't know the hostnames ahead of time, and the set of live hostnames is sparse enough that pre-provisioning would be mostly wasted work. sslip.io deployments hit all three. When they do, lazy issuance at the handshake is the pattern that fits, provided you build the dedup, the semaphore, the cap, the backoff, and the timeout, and you scope it to stable hostnames only.
Temps is open source under Apache 2.0 and free to self-host (the proxy, the ACME machinery, the on-demand manager, and everything else ship in a single ~80MB Rust binary you run on your own server). If you want to read the actual OnDemandCertManager or the certificate callback, it's all in the open at github.com/gotempsh/temps. Temps Cloud runs the same code for about $6/mo flat, no per-seat fees.
Issue a certificate for each eligible hostname with HTTP-01 when the hostname is first requested. In Temps, the TLS callback fails the initial handshake without blocking on network I/O and queues the ACME order in the background. The proxy serves the HTTP-01 challenge, and a later connection succeeds after the certificate is stored. In-flight deduplication, a concurrency limit, an hourly cap, backoff, and a timeout prevent repeated handshakes from creating duplicate orders.
ACME wildcard certificates require DNS-01. That challenge requires creating a TXT record in the relevant DNS zone. A Temps operator can use an sslip.io hostname but does not control the sslip.io zone, so the operator cannot publish the required challenge record. HTTP-01 works per hostname because Temps can serve the challenge response over HTTP.
The first TLS connection for a new hostname cannot complete because no certificate exists yet. Temps fails that handshake quickly and issues the certificate in the background rather than keeping the connection open during the ACME order. The client must connect again after issuance completes. Later connections use the stored certificate without repeating the issuance path.