Docs · Get started

Updates and maintenance

Upgrades verify signatures, back up first, check health and roll back on their own. This page covers updates, backups, secrets, status and removal, and which command works on which provider.

Which command works where

Taskgcpgcp-firebaseSelf-hosted
upgradeyesyesyes
Install update from the adminyes, via the upgrader jobyes, via the upgrader jobyes, via the update agent (Linux with systemd)
backupon-demand Cloud SQL backupon-demand Cloud SQL backupone .tar.gz archive
Restoreprinted gcloud sql backups restore commandprinted gcloud sql backups restore commandrestore --from … --yes
rotate-secretyesyesnot available
statusyesinstall job progress onlystatus-self-hosted (basic info)
destroydeletes the projectnot availablenot available

All commands run through the same package: bunx @caelo-cms/provisioning@<version> <command>. Name the version: without it, bunx may keep running an old copy it downloaded earlier. For Google Cloud installs, run the commands on a machine that knows the install. That is the machine you installed from, or any machine after bunx @caelo-cms/provisioning attach --project <project-id>. Self-hosted commands run from the directory that holds .caelo/.

Settings, Caelo updates

The admin page Settings, Caelo updates (/settings/updates) shows your version, whether a newer release is out, and its release notes. For an Owner it also shows:

  • Install update, which you confirm with your Caelo password. The admin never installs anything itself. On Google Cloud it starts the upgrader, a Cloud Run job in your project. On a self-hosted install it writes a request that the update agent on your server picks up. A timeline shows each step, and Update history keeps every attempt.
  • The exact terminal command for your install, as an alternative.
  • Automatic updates. By default Caelo tells you about a release and you install it ("notify"). You can let it install security fixes, patch releases and minor releases on its own, each with its own switch. An automatic install waits 24 hours after a release comes out (2 hours for security fixes), and you can limit it to a maintenance window. If an automatic install fails, automatic updates pause until an Owner resumes them.
  • Some releases are never installed automatically: releases with destructive database changes, releases that change your cloud setup, and major releases. The signed release information says which kind a release is, and the updates page tells you what to do.

The AI can tell you which version runs and whether an update is out. It cannot install one or change the policy.

Upgrade from the terminal

Google Cloud:

bunx @caelo-cms/provisioning@<version> upgrade --version <version> --install <domain>

Self-hosted, from the directory that holds .caelo/ (without --install, which names Google Cloud installs):

bunx @caelo-cms/provisioning@<version> upgrade --version <version>

The version appears twice on purpose. @<version> runs the installer of the release you are moving to, and --version rolls exactly that release even if a newer one ships in the meantime. Without --version, upgrade takes the newest stable release. --channel rc or --channel beta opts into pre-releases.

What every upgrade does, in this order:

  1. Verify. Resolves the release images to digests and checks their cosign signatures. A failed check changes nothing. You need cosign installed.
  2. Back up. Google Cloud: an on-demand Cloud SQL backup, whose id is printed. Self-hosted: pg_dump of both databases into .caelo/backups/. If the backup fails, the upgrade stops.
  3. Migrate. Runs the database migrations with the new release's image while the current release keeps serving.
  4. Switch and check. Rolls the admin, then the gateway, and checks the health of each.
  5. Roll back if needed. If a health check fails, the previous images are started again. The database is not restored automatically: the backup id (Google Cloud) or path (self-hosted) is printed for the case that the old release no longer works with the migrated schema. Restoring overwrites everything written since the backup.

On Google Cloud an upgrade also brings the infrastructure up to what the new release expects, such as permissions and secrets. Run it signed in with gcloud auth login as an account that may change the project and can open the admin, because the health check goes through Google's Identity-Aware Proxy. --skip-health-check skips the deep check, for example when moving to a release that predates it. Re-running upgrade is safe on every provider.

Releases that change your Google Cloud setup cannot be installed by the in-admin upgrader. The updates page then gives you an Open Cloud Shell link and the commands to run there.

The self-hosted update agent

On Linux with systemd, init and upgrade set up a systemd timer that runs as your user once a minute. It picks up update requests from the admin, verifies the release's signed manifest and the new installer's npm provenance, and then runs the same upgrade as above. It reaches Postgres through its own restricted login on the loopback interface only. Manage it with:

bunx @caelo-cms/provisioning@<version> agent install
bunx @caelo-cms/provisioning@<version> agent status
bunx @caelo-cms/provisioning@<version> agent logs
bunx @caelo-cms/provisioning@<version> agent uninstall

The agent needs cosign and npm on its PATH, and your user in the docker group. The agent and a hand-run upgrade never roll at the same time.

Backups and restore

Google Cloud (gcp and gcp-firebase)

WhatHow it is protected
Database (both databases on one Cloud SQL instance)Automated backup every night at 03:00. The last 7 are kept.
Point-in-time recoverygcp-firebase: always on, for the last 7 days. gcp: only with high availability, which is off by default.
Before every upgradeAn extra Cloud SQL backup. If it fails, the upgrade changes nothing.
Uploaded mediaThe media bucket keeps older versions of every file, with no expiry. A deleted or overwritten file stays recoverable in the bucket. Older versions cost storage. Caelo has no restore command for media.
Published sitegcp-firebase: every published build is kept as a Firebase Hosting version, and the live site can be rolled back to an earlier build. gcp: the site bucket has no versions. The site can be rebuilt from the database, but rolling the live site back to an earlier build does not work yet.
Install state (Pulumi state and install files)A versioned bucket in your project. Older versions are kept for 30 days.
SecretsNot backed up by Caelo. They stay in Secret Manager in your project.

backup takes an extra backup now and prints the matching restore command:

bunx @caelo-cms/provisioning@<version> backup --install <domain>
# prints e.g.: gcloud sql backups restore <id> --restore-instance=<instance> --project=<project>

There is no restore command on Google Cloud. You run the printed gcloud command yourself. It overwrites the whole database instance, both databases, and everything written since the backup is lost. Caelo does not wrap point-in-time recovery; use Cloud SQL's own tools for that.

Self-hosted

bunx @caelo-cms/provisioning@<version> backup --to caelo-backup.tar.gz
bunx @caelo-cms/provisioning@<version> restore --from caelo-backup.tar.gz --yes

The archive holds a pg_dump of both databases, the published site and the uploaded media. The admin and the gateway stop while it runs, so the parts match: a short outage. The archive does not contain .caelo/config.json, on purpose, so you can copy the archive off the server. Keep a copy of that file yourself, because without it the stored AI provider keys in a backup cannot be decrypted. restore checks the archive first, replaces both databases and the files, migrates, and starts everything with health checks. If it stops half-way, run it again.

There is no schedule and no point-in-time recovery. Schedule backup yourself, for example as a daily cron job, and copy the archives off the server.

Rotate secrets (Google Cloud)

bunx @caelo-cms/provisioning@<version> rotate-secret <name> --install <domain>

Names: postgres-password, public-role-password, internal-secret, tool-approval-secret, staging-gate-secret. The command stores a new value in Secret Manager, updates the database role for the password secrets, and rolls the services that read it. Rotating staging-gate-secret signs every staging session out. Your AI provider key is changed in the admin under Settings, AI.

The old name gateway-role-password is still accepted but has no effect since 0.10.51, because the gateway no longer logs into the content database. It will be removed.

Status

bunx @caelo-cms/provisioning@<version> status

On gcp this shows the admin's Cloud Run URL, the Cloud SQL state, your URLs, the installer version and whether a newer release exists. On gcp-firebase it currently shows only the install job's progress. On a self-hosted install use status-self-hosted for the basic facts and docker compose -f .caelo/docker-compose.yml ps for container health.

Remove an install

  • gcp: destroy asks twice (the second time you type the domain) and then deletes the whole Google Cloud project. Google keeps a deleted project restorable for 30 days.
  • gcp-firebase: destroy does not delete anything yet. Delete the project in the Google Cloud console or with gcloud projects delete <project-id>.
  • Self-hosted: there is no command. Stop the stack with docker compose -f .caelo/docker-compose.yml down; adding -v also deletes the database volume.

In every case the local install directory (~/.caelo-<install-id>/ or .caelo/) stays until you delete it.

Something wrong or missing on this page? Open an issue on GitHub