Set Up Backups & Monitoring

This tutorial configures automated backups and uptime monitoring for your Temps instance. By the end, your data will be backed up to S3 on a schedule, you will receive alerts when something goes down, and you will know how to restore from a backup.

What you will build

By the end of this tutorial, you will have:

• An S3-compatible storage bucket connected to Temps for off-site backups
• A manual backup of your Temps database and all managed services
• An automated backup schedule that runs nightly
• A notification channel (email, Slack, or webhook) for alerts
• An uptime monitor that checks your application every 60 seconds
• Experience restoring from a backup

Time required: approximately 20 minutes.

What this tutorial covers and does not cover:

Prerequisites

You need:

• A running Temps instance with at least one deployed project (complete Your First Deployment first)
• S3-compatible storage — any of the following:
  • AWS S3 bucket
  • MinIO instance (self-hosted)
  • DigitalOcean Spaces
  • Cloudflare R2
  • Any S3-compatible object storage
• Credentials for your S3 storage: access key ID, secret access key, bucket name, region, and endpoint URL (for non-AWS providers)

Connect S3 storage

Temps stores backups in S3-compatible object storage. You need to connect a storage source before you can create backups.

1. In the Temps dashboard sidebar, click Settings (at the bottom)
2. Find the Backups section
3. Click Add S3 Source
4. Fill in your storage details:

Click Save. When you create a source, Temps tests the connection and auto-creates the bucket before saving the source. If validation fails, verify your credentials and ensure the endpoint is reachable.

Default source and connection testing

Temps tracks a single default S3 source so schedules and on-demand backups have a destination without you re-selecting it each time:

• The first source you create is automatically the default, regardless of what you pass.
• Marking another source as default (the Set Default action) atomically swaps the default inside a database transaction. A partial-unique index guarantees at most one source is ever the default.

You can test a source two ways:

• Test a saved source — uses the stored credentials and reports whether the bucket is reachable. Does not create anything.
• Preview-test before saving — checks a prospective configuration from the form by attempting a list. It does not create the bucket or store any credentials, so you can validate keys before committing them.

The S3 sources are managed through a CRUD API under /api. All routes require a bearer token and the listed permission:

Run your first backup

With storage connected, run a manual backup to verify everything works.

1. On the Backups settings page, find your S3 source
2. Click the Run Backup button (or use the actions menu)
3. Select Full as the backup type

Temps starts the backup immediately. The process:

1. Temps database — The internal PostgreSQL database (with TimescaleDB extension) is backed up first.
2. Managed services — Every managed service you have provisioned (PostgreSQL databases, Redis instances, S3 storage) is backed up individually.
3. Metadata — A metadata.json file is generated with checksums, sizes, timestamps, and service inventory.
4. Upload — Everything is uploaded to your S3 bucket and the backup index is updated.

You can watch the progress on the page. When the status changes to Completed, your first backup is done.

Verify the backup

Check that files were written to your S3 bucket. The backup is stored at:

{bucket_path}/backups/{YYYY}/{MM}/{DD}/{backup-uuid}/
  ├── backup.sql.gz       # Temps database dump (gzip compressed)
  ├── metadata.json        # Checksums, sizes, timestamps
  └── ...                  # External service backups (if any)

There is also an index file at {bucket_path}/backups/index.json that lists all backups.

Verify from the CLI

If you have the Temps CLI installed, you can list backups using your configured backup source. First create a backup source (if you have not done so already), then list backups for it.

Create a backup source (AWS S3):

bunx @temps-sdk/cli backup source create \
  --name my-s3-source \
  --bucket your-backup-bucket \
  --region us-east-1 \
  --access-key YOUR_ACCESS_KEY \
  --secret-key YOUR_SECRET_KEY

Create a backup source (non-AWS, e.g. MinIO):

bunx @temps-sdk/cli backup source create \
  --name my-minio-source \
  --bucket your-backup-bucket \
  --endpoint https://your-minio-server:9000 \
  --access-key YOUR_ACCESS_KEY \
  --secret-key YOUR_SECRET_KEY

List backups for a source or schedule:

# List by schedule
bunx @temps-sdk/cli backup list --schedule-id YOUR_SCHEDULE_ID

# List all backups for a source
bunx @temps-sdk/cli backup source backups --id YOUR_SOURCE_ID

Create a backup schedule

Manual backups are useful for one-off situations (before a migration, before an upgrade). For day-to-day protection, set up an automated schedule.

1. On the Backups settings page, click Add Schedule
2. Configure the schedule:

Click Create. The schedule starts immediately — the next backup will run at the time specified by your cron expression.

Choose what a schedule backs up

By default a schedule backs up every database plus the Temps control plane, but you can scope it. The create, edit, and schedule-detail pages expose two controls:

• All databases (recommended) / Specific databases — a radio. All databases fans every run out to every external service on the host, including databases you add in the future. Specific databases restricts the run to the databases you explicitly attach via the checkbox list.
• Also back up the Temps control plane — a switch. When on, every run also produces a control_plane backup of Temps' own Postgres; when off, that step is skipped.

Both options map to boolean columns on the schedule (target_all_services and include_control_plane), both default to true. Temps rejects a schedule that would back up nothing: if the control plane is off and "All databases" is off and no databases are attached, create/update fail with a validation error. Flipping back to "All databases" clears any explicit per-database membership ("all means all"). On upgrade, existing schedules backfill to the prior behavior — back up every database plus the control plane.

The explicit per-database membership is also exposed through the API (all routes under /api, bearer token required):

Attach and detach write audit logs.

Recommended schedules

Disable or enable a schedule

You can pause a schedule without deleting it. On the Backups page, use the toggle or actions menu to Disable or Enable a schedule. Disabled schedules retain their configuration and history but do not run.

How retention is enforced

Retention is enforced two ways, and the app-side path is the source of truth:

• App-side (primary). BackupService::enforce_retention sweeps enabled schedules and deletes backups older than each schedule's retention period. This is what keeps your storage clean and is the source of truth on providers that don't support tagging.
• Bucket lifecycle rules (best-effort). So expired backups are still removed even while Temps is offline, Temps also reconciles per-bucket lifecycle rules from your current schedules. Each backup object is tagged on upload (via a separate PutObjectTagging call, not an upload header) with temps-managed=true, temps-retention-days (the schedule's retention in days, or the literal never for ad-hoc backups or non-positive retentions), and traceability tags. Temps then emits one tag-filtered lifecycle rule per distinct retention value (e.g. temps-retention-7d, temps-retention-30d) that expires objects after that many days. Tag filters were chosen over key prefixes so existing backup keys are untouched and restore keeps working; objects written before this feature lack the tags and are simply ignored by the rules.

Reconcile runs fire-and-forget on schedule create and delete (and on update only when retention or enabled changed), plus an hourly drift sweep that re-pushes the desired state.

Set up a notification channel

Before setting up monitoring, configure where Temps should send alerts. Temps supports three notification channels: email, Slack, and webhooks.

Option A: Email notifications

Temps supports three email provider types: AWS SES, Scaleway, and a generic SMTP provider. Use generic SMTP when you only have plain SMTP credentials — AWS SES SMTP endpoints, Sendgrid, Mailgun, Postmark, a self-hosted Postfix, or a local Mailpit/Mailhog instance for testing.

1. In the sidebar, click Settings
2. Find the Notifications (or Email Providers) section
3. Click Add Provider and choose the provider type. For generic SMTP (the Server icon), configure your relay:

Generic SMTP providers send via the lettre library. Creating one sends a SmtpCredentialsRequest to POST /api/email-providers.

Edit an email provider

Email providers are editable in place — the Edit dropdown item is now active for SES, Scaleway, and SMTP providers. Editing sends a partial update to PATCH /api/email-providers/{id} (requires the email_providers:write permission).

• name, region, is_active, and credentials are individually updatable.
• provider_type is immutable — the Edit dialog shows it read-only, and sending a credential block for a different type is rejected (Cannot change provider type from … to …).
• Omitting a credential block preserves the stored secret, so you can rename a provider or change its region without re-typing passwords. Secret fields render blank with "Leave blank to keep current" helper text.
• Updates emit an EMAIL_PROVIDER_UPDATED audit event that records only the names of the changed fields (e.g. ["name","region","credentials"]), never their values.

Option B: Slack notifications

1. Create a Slack Incoming Webhook for the channel you want alerts in
2. In Temps, click Add Provider and choose Slack
3. Paste the webhook URL (must start with https://hooks.slack.com/)
4. Optionally set a channel name for display purposes

Slack notifications include color-coded attachments: red for errors, orange for warnings, blue for informational.

Option C: Webhook notifications

For integrations with Discord, PagerDuty, or any HTTP endpoint:

1. Click Add Provider and choose Webhook
2. Enter the URL of your webhook endpoint
3. Optionally configure:
   • HTTP Method — POST (default), PUT, or PATCH
   • Headers — Custom headers for authentication (e.g. Authorization: Bearer your-token)
   • Timeout — Maximum wait time in seconds (default: 30)

Webhook payloads are JSON:

{
  "id": "notif_abc123",
  "title": "Backup Failed",
  "message": "Nightly backup schedule 'production-daily' failed",
  "type": "error",
  "priority": "high",
  "severity": "error",
  "timestamp": "2026-02-28T02:00:15Z",
  "metadata": {
    "schedule_id": 1,
    "schedule_name": "production-daily",
    "backup_type": "full"
  }
}

Test your notification channel

After creating a provider, click the Test button. Temps sends a test notification to verify the connection works. You should receive a test message within a few seconds.

Both test emails are sent with the standard Temps template (dark header, status badge, details table, footer):

• Email-provider test (the "Send Test Email" action on an email provider) arrives with subject [Temps] Email provider test — <name> and renders the provider type as a human-readable label (AWS SES / Scaleway / SMTP). The provider name, region, and type are HTML-escaped before rendering, so a provider name like <script>…</script> can't inject markup into the inbox.
• Notification-provider health-check arrives as a multipart/alternative message (plaintext + HTML) with subject [Temps] Notification provider health check. The body shows an "Notification provider is reachable" title and a details table with the instance hostname, the SMTP relay as host:port, the From address, and a timestamp. The instance label is resolved from TEMPS_HOSTNAME, then TEMPS_PUBLIC_HOSTNAME, then the literal temps instance. The health check no longer fails on an empty recipient list or a single malformed recipient address (the SMTP connection succeeding is still a valid signal), and it logs which stage failed (connection vs send) so you can act on a misconfiguration.

Configure notification preferences

Temps lets you fine-tune what triggers notifications. In Settings > Notifications > Preferences, you can control:

• Backup notifications — failures (enabled by default), successes, S3 connection issues, retention violations
• Domain notifications — SSL certificate expiration (30 days before), domain expiration, DNS changes
• Runtime notifications — deployment failures, build errors, runtime error spikes
• Weekly digest — a summary email sent every Monday at 9 AM with deployment stats, error trends, and performance metrics (see below)

Weekly digest

The weekly digest is a scheduled summary email that gives admin users a high-level view of how their Temps instance performed over the past week. It replaces the need to manually check each project for activity.

What it contains:

• Deployment stats — total successful and failed deployments across all projects for the week, giving a quick sense of release cadence and reliability
• Error trends — error event counts broken down by project, so you can spot which projects had a spike before digging into logs
• Performance metrics — aggregated request-level performance data for the week

When it is sent: every Monday at 9 AM (server local time).

Who receives it: all admin users on the instance.

Subject line format: [Temps] Weekly digest — week of <date>

Requirements:

• An email provider must be configured (AWS SES, Scaleway, or SMTP — the same provider used for transactional auth email)
• The Weekly digest toggle must be enabled in Settings > Notifications > Preferences

If no email provider is configured, the digest is silently skipped — no error is logged and no fallback notification is sent. To start receiving it, add an email provider and make sure the preference is on.

Create an uptime monitor

Uptime monitors check your application's health at regular intervals and alert you when something goes down.

1. Open your project in the dashboard
2. Click Monitors in the project sidebar
3. Click Add Monitor
4. Configure the monitor:

Click Create. Temps runs the first check immediately and then continues at the interval you specified.

How health checks work

Each check:

1. Sends an HTTP GET request to the monitor URL
2. Waits up to 10 seconds for a response (within a 30-second client timeout)
3. Retries up to 3 times with exponential backoff (100ms, 200ms, 400ms) before marking as failed
4. Maps the response to a status:

Failure threshold and incidents

A single failed check does not trigger an alert. By default, Temps requires 2 consecutive failures before creating an incident. This avoids false alarms from network blips.

When an outage is confirmed:

1. The monitor status changes to Down
2. An incident is created with severity based on the failure type (Minor for degraded, Major for down)
3. A notification is sent to all configured channels
4. The incident appears on the project's status overview

When the monitor recovers:

1. The monitor status returns to Operational
2. The incident is automatically resolved
3. A recovery notification is sent

View uptime history

On the Monitors page, each monitor shows:

• Current status — Operational, Degraded, or Down
• Response time — Average, P50, P75, P95, P99 percentiles
• Uptime history — Time-bucketed data showing operational/degraded/down periods
• Incident history — All past incidents with severity, duration, and resolution time

Simulate a failure

To verify your monitoring and notification setup is working end-to-end, you can temporarily break your application and watch the system respond.

Option 1: Stop the container (recommended)

If you have SSH access to your server:

# Find your application's container
docker ps | grep your-project-name

# Pause it temporarily (does not destroy it)
docker pause <container-id>

Wait for 2-3 minutes (two check intervals). You should:

1. See the monitor status change to Down in the dashboard
2. Receive an outage notification on your configured channel
3. See an incident created in the project's Monitors section

Now resume the container:

docker unpause <container-id>

Within one check interval (60 seconds), you should:

1. See the monitor status return to Operational
2. Receive a recovery notification
3. See the incident automatically resolved

Option 2: Deploy a broken version

Push a commit that returns a 500 error from your health endpoint, deploy it, wait for the alert, then fix and redeploy. This tests the full loop including deployments, but takes longer.

Restore from a backup

The point of backups is restoring from them. Test this now while everything is working, so you are prepared if you ever need it for real.

Restore via the CLI

The CLI is the primary way to restore backups. It handles downloading from S3, decompressing, and restoring both the Temps database and all managed services.

Step 1: List available backups

# List backups by schedule ID
bunx @temps-sdk/cli backup list --schedule-id YOUR_SCHEDULE_ID

# Or list all backups for a backup source
bunx @temps-sdk/cli backup source backups --id YOUR_SOURCE_ID

Note the backup-id (UUID) of the backup you want to restore.

Step 2: Restore the full backup

bunx @temps-sdk/cli services restore \
  --id YOUR_SERVICE_ID \
  --backup-id YOUR_BACKUP_UUID

To restore into a new service instead of overwriting the existing one, add --new-service:

bunx @temps-sdk/cli services restore \
  --id YOUR_SERVICE_ID \
  --backup-id YOUR_BACKUP_UUID \
  --new-service

Step 3: Restart Temps

After a restore, restart the Temps service to pick up the restored data:

sudo systemctl restart temps

Restore a single managed service

If you only need to restore one specific service (e.g. a PostgreSQL database that was corrupted), use services restore targeting that service's ID:

bunx @temps-sdk/cli services restore \
  --id YOUR_SERVICE_ID \
  --backup-id YOUR_BACKUP_UUID

For point-in-time recovery (Postgres only), add --pitr with an ISO 8601 timestamp:

bunx @temps-sdk/cli services restore \
  --id YOUR_SERVICE_ID \
  --backup-id YOUR_BACKUP_UUID \
  --pitr "2026-05-01T03:00:00Z"

What you have accomplished

Your Temps instance now has a complete protection layer:

Additional protections that Temps runs automatically in the background:

• Disk space monitoring — Watches free space on the data volume and alerts before it fills up (see below).
• Backup failure alerts — High-priority notifications if a scheduled backup fails.
• SSL expiration warnings — Notifications 30 days before certificates expire.
• Weekly digest — A summary of deployment stats, error trends, and performance metrics sent every Monday at 9 AM to admin users (requires an email provider and the preference enabled).

Disk space monitoring

Temps runs a background monitor that watches free space on the data volume and notifies you before it fills up. It is enabled by default and uses these settings (configurable under Settings → System Monitoring):

When an alert fires, the severity is derived from usage: ≥ 95% is Critical, ≥ 90% is High, otherwise Normal. The notification title is Disk Space Alert: {mount_point} at {usage}% and the body tells you to free up disk space or raise the threshold in Settings → System Monitoring.

The monitor only runs when a notification provider is configured — alerts are skipped entirely if none is set up. While monitoring is disabled, the loop re-checks settings every 60 seconds.

You can also read current disk usage on demand without sending any notifications:

GET /api/settings/disk-status

This read-only endpoint (requires the SettingsRead permission) returns a DiskSpaceCheckResult with checked_at, enabled, threshold_percent, the per-mount disks usage, and any alerts (mounts meeting or exceeding the threshold). The dashboard uses it to surface a low-disk-space warning.

What to explore next