## Summary This PR creates a comprehensive self-hosted JWT key rotation how-to for SpacetimeDB with reproducible defaults, identity-preserving rotation guidance, and host-scoped data migration runbooks. It is written for both operators and automation workflows, with explicit guardrails for production-like VM environments. ## Key Changes - Adds a top-level assumptions/risk section for a multi-host topology (`prod`, `test`, `dev`, optional `local`) including backup prerequisites before rotation or sync. - Defines an opinionated, end-to-end path contract for keys: - host source: `./.generated/spacetimedb-keys` - runtime mount: `/etc/spacetimedb` - startup args pinned to `/etc/spacetimedb/id_ecdsa` and `/etc/spacetimedb/id_ecdsa.pub` - Includes reproducible quickstart scaffolding: - OpenSSL commands for compatible ES256/P-256 key generation - non-container and Docker startup examples - rotation/verify command sequence and marker checks - Documents one unified tooling surface around `spacetimedb-tooling.ts` for: - rotate (`--dry-run`, `--yes`) - verify (`--verify-only`) - token continuity (`--resign-token-only`) - explicit token/key path overrides (`--publisher-cli-toml-path`, `--private-key-path`) - Clarifies 401 vs 403 behavior and why identity drift causes publish failures after rotation. - Covers all rotation strategies in one place: - clean-slate rotation (stateless/dev) - identity-preserving rotation (stateful) - OIDC-backed identity model for production - Adds host-scoped migration runbooks for staged sync (`prod -> test -> dev`) with destination-side token re-signing semantics (run on destination host context, including SSH examples). - Explicitly separates conceptual topology guidance from currently implemented sync primitives in reference tooling to avoid over-claiming. - Adds operational guardrails for sync promotions: - `rsync --delete` is destructive - stop/start ordering around sync - required acceptance gates: key parity, destination re-sign, restart/redeploy, publish marker validation - Expands verification and automation content: - PEM/parity/fingerprint/newline checks - AI/automation contract with inputs, outputs, required command order, and success/failure markers - troubleshooting flow for `InvalidSignature` and ownership mismatch errors ## Why This Matters The most common failure mode in self-hosted fleets is identity drift: signatures validate, but `spacetime publish` still fails with `403 Forbidden` because destination ownership no longer matches the publishing identity after key rotation or data movement. The updated how-to makes these operator requirements explicit: 1. preserve identity continuity across rotations and host promotions 2. re-sign destination tokens after `rsync` in destination host context 3. treat restart/redeploy and publish markers as promotion gates Following this runbook prevents regressions where key material is correct but publish still fails due to stale signatures or owner mismatch state. ## Why JWT Key Rotation Is Essential for Self-Hosters In self-hosted deployments, operators are the signing authority. Rotation is both a security control and an operational correctness requirement. ### Security value - reduces exposure window for compromised private keys - invalidates old/stolen tokens after rotation - mitigates risk from snapshots, backups, clones, and long-lived VM access ### Operational value - keeps key material consistent across multi-VM environments - reduces publish outages after promotion/sync operations - prevents avoidable `401`/`403` failures in routine release workflows ## Scope - docs-only change - public-safe examples only (no internal org names, private vault IDs, or private secret prefixes) - includes non-container and Docker workflows, self-publish/versioning guidance, and marker-based validation (`PUBLISH_SUCCESS`, `PUBLISH_FAILED`) - includes data migration workflow guidance (`rsync` + destination re-sign + restart/redeploy + publish checks) --------- Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
SpacetimeDB Documentation
This repository contains the markdown files which are used to display documentation on our website. This documentation is built using Docusaurus.
Making Edits
To make changes to our docs, you can open a pull request in this repository. You can typically edit the files directly using the GitHub web interface, but you can also clone our repository and make your edits locally.
Instructions
- Fork our repository
- Clone your fork:
git clone ssh://git@github.com/<username>/SpacetimeDB
cd SpacetimeDB/docs
- Make your edits to the docs that you want to make + test them locally (See Testing Locally)
- Commit your changes:
git add .
git commit -m "A specific description of the changes I made and why"
- Push your changes to your fork as a branch
git checkout -b a-branch-name-that-describes-my-change
git push -u origin a-branch-name-that-describes-my-change
- Go to our GitHub and open a PR that references your branch in your fork on your GitHub
CLI Reference Section
To regenerate the CLI reference section, run pnpm generate-cli-docs.
Docusaurus Documentation
For more information on how to use Docusaurus, see the Docusaurus documentation.
Testing Locally
Installation
- Make sure you have Node.js installed (version 22 or higher is recommended).
- Clone the repository and navigate to the
docsdirectory. - Install the dependencies:
pnpm install - Run the development server:
pnpm dev, which will start a local server and open a browser window. All changes you make to the markdown files will be reflected live in the browser.
Cutting Docs Versions
Use Docusaurus versioning to snapshot the current docs into versioned_docs.
- From
docs/, cut a version:
pnpm docusaurus docs:version <version-name>
Example:
pnpm docusaurus docs:version 1.12.0
This updates:
docs/versions.jsondocs/versioned_docs/version-<version-name>/docs/versioned_sidebars/version-<version-name>-sidebars.json
After cutting, update docs/docusaurus.config.ts as needed:
lastVersionfor the default version at/docsversions.currentlabel/path for prerelease docsversions['<version-name>']label/banner for the stable snapshot
Re-cutting a Version From an Older Commit
If you need a version snapshot from an old commit (instead of current docs/docs), use:
./docs/scripts/get-old-docs.sh <commit> <version-name>
Example:
./docs/scripts/get-old-docs.sh e45cf891c20d87b11976e1d54c04c0e4639dbe81 1.12.0
The script creates a temporary worktree, snapshots docs from that commit, and copies the generated versioned_docs artifacts back into your current branch.
Rewriting Absolute Links to Version-Safe Relative Links
Absolute links like /quickstarts/react can resolve to the default docs version. To keep links inside the current version, rewrite internal links to relative paths.
Dry run:
pnpm --dir docs rewrite-links
Apply changes:
pnpm --dir docs rewrite-links:write
This script rewrites internal absolute links in:
docs/docs(current/prerelease docs)docs/versioned_docs/version-*(all version snapshots)
Adding new pages
All of our directory and file names are prefixed with a five-digit number which determines how they're sorted. We started with the hundreds place as the smallest significant digit, to allow using the tens and ones places to add new pages between. When adding a new page in between two existing pages, choose a number which:
- Doesn't use any more significant figures than it needs to.
- Is approximately halfway between the previous and next page.
For example, if you want to add a new page between 00300-foo and 00400-bar, name it 00350-baz. To add a new page between 00350-baz and 00400-bar, prefer 00370-quux or 00380-quux, rather than 00375-quux, to avoid populating the ones place.
To add a new page after all previous pages, use the smallest multiple of 100 larger than all other pages. For example, if the highest-numbered existing page is 01350-abc, create 01400-def.
Best practices
- Use relative links for linking between documentation pages, as this will ensure that links work correctly with versioning and localization.
License
This documentation repository is licensed under Apache 2.0. See LICENSE.txt for more details