mirror of
https://github.com/clockworklabs/SpacetimeDB.git
synced 2026-07-25 11:32:30 -04:00
bradleyshep
fb0a458d4f
# Description of Changes AI app generation benchmark comparing SpacetimeDB vs PostgreSQL (Express + Socket.io + Drizzle ORM). Same AI model (Claude Sonnet 4.6), same prompts, same chat app, two backends. Upgraded through 12 feature levels, manually graded at each level, bugs fixed, all costs measured via OpenTelemetry. Results viewable at: https://spacetimedb.com/llms-benchmark-sequential-upgrade ## Benchmark harness (`tools/llm-sequential-upgrade/`) - `run.sh`: orchestrates headless Claude Code sessions for code generation, sequential upgrades, and bug fixes. Tracks all API costs via OTel. Supports `--upgrade`, `--fix`, `--composed-prompt`, `--resume-session` modes. - `grade.sh` / `grade-agents.sh`: grading harnesses for manual testing of generated apps. - `docker-compose.otel.yaml`: OTel collector + PostgreSQL services. - `generate-report.mjs` / `parse-telemetry.mjs`: aggregate per-session telemetry into cost reports. - Backend guidelines in `backends/`: SpacetimeDB SDK reference, config templates, server setup docs, PostgreSQL setup with Drizzle/Socket.io guidance. **After https://github.com/clockworklabs/SpacetimeDB/pull/4740 merges, we will likely want to update this so that it reads backend and SDK guidance from SKILLS** ## Two complete benchmark runs **Run 1 (20260403):** Original methodology. **Run 2 (20260406):** Refined methodology with domain bias removed from SpacetimeDB SDK docs and PostgreSQL instructions made feature-spec-neutral. **Note: no meaningful changes in results were observed with these changes. Domain familiarity biases were very small and almost certainly not the cause of STDB's major gains over PG stack.** Each run contains full L1-L12 app source for both backends, level snapshots preserving state before each upgrade, and per-session OTel cost summaries. ## 12 feature levels | Level | Feature | |---|---| | L1 | Basic Chat + Typing + Read Receipts + Unread Counts | | L2 | Scheduled Messages | | L3 | Ephemeral Messages | | L4 | Message Reactions | | L5 | Message Editing with History | | L6 | Real-Time Permissions (kick, ban, promote) | | L7 | Rich User Presence | | L8 | Message Threading | | L9 | Private Rooms + Direct Messages | | L10 | Room Activity Indicators | | L11 | Draft Sync | | L12 | Anonymous to Registered Migration | ## Results | | Run 1 (20260403) | Run 2 (20260406) | |---|---|---| | **SpacetimeDB total cost** | $13.33 | $12.62 | | **PostgreSQL total cost** | $17.80 | $19.68 | | **SpacetimeDB bugs** | 5 | 2 | | **PostgreSQL bugs** | 19 | 8 | | **SpacetimeDB fix sessions** | 4 | 1 | | **PostgreSQL fix sessions** | 17 | 10 | Both runs agree: SpacetimeDB apps are cheaper to build, have fewer bugs, and require fewer fix iterations. The refined methodology (Run 2) widened the cost gap and **confirmed the advantage is structural, not an artifact of domain-biased SDK docs.** ## Performance benchmark (`perf-benchmark/`) Stress throughput tool that fires concurrent writers at peak saturation against the AI-generated `send_message` handlers. | Tier | SpacetimeDB (avg) | PostgreSQL (avg) | Ratio | |---|---|---|---| | AI-generated (as-shipped) | 5,267 msgs/sec | 694 msgs/sec | 7.6x | | PG rate limit removed | 5,267 msgs/sec | 1,070 msgs/sec | 4.9x | | Optimized (same features kept) | 25,278 msgs/sec | 1,139 msgs/sec | 22x | The gap widens with optimization because SpacetimeDB's bottleneck is fixable code patterns in the reducer while PostgreSQL's bottleneck is architectural (sequential network round-trips to an external database). Optimized reference code with all features preserved is in `perf-benchmark/results/optimized-reference/`. ## Data handling Per-session cost summaries (`cost-summary.json`, `COST_REPORT.md`, `metadata.json`) are committed. Raw OTel telemetry (`raw-telemetry.jsonl`) containing PII is excluded via `.gitignore` and stored privately. # API and ABI breaking changes None. All changes are in `tools/llm-sequential-upgrade/`. No production code, library, or SDK changes. # Expected complexity level and risk **1 - Trivial.** Self-contained benchmarking tooling and data. No interaction with production code. # Testing - [x] L1-L12 upgrades completed on all 4 apps (2 backends x 2 runs) with OTel cost capture - [x] All levels manually graded after each upgrade; bugs filed and fixed via the harness - [x] Methodology refinement between runs validated (domain bias removal, feature-neutral instructions) - [x] Stress benchmarks run across both runs x 3 tiers (as-shipped, rate-limit-removed, optimized) - [x] Optimized benchmarks verified to preserve all original features - [x] Sensitive data (PII in raw telemetry) removed from repo and gitignored - [ ] Reviewer: spot-check that METRICS_DATA.json / METRICS_REPORT.json numbers match the telemetry cost-summary.json files --------- Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com> Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Perf Benchmark - PG vs STDB Chat Apps
Runtime performance harness for the Level 12 chat apps the LLM built in the sequential upgrade benchmark. Measures messages-per-second throughput and latency so we have showcase numbers for the marketing one-pager.
This is not a synthetic benchmark of PostgreSQL vs SpacetimeDB. It's a benchmark of the apps the LLM built on each stack, run as-is.
What it tests
| Scenario | What it measures |
|---|---|
stress |
N writers flooding send_message for D seconds. Sustained msgs/sec + p99 latency. |
realistic |
M users at human cadence (5-15s jitter) for D seconds. Sustained msgs/sec + latency under realistic load. |
Setup
npm install
# Generate SpacetimeDB bindings against the target Level 12 app's backend.
# Re-run this if you change which app you're benchmarking.
spacetime generate --lang typescript --out-dir src/module_bindings \
--module-path ../sequential-upgrade/sequential-upgrade-20260406/spacetime/results/chat-app-20260406-153727/backend/spacetimedb
Prerequisites for running
The target apps must already be running:
- Postgres:
cd <pg-app>/server && npm run dev(Express on:6001), plus theexhaust-test-postgres-1Docker container (port 6432). - SpacetimeDB: local
spacetime startrunning, and the target module must be published (the apps publish themselves automatically when generated).
Run
# PG stress, 30s, 20 writers
npm run run -- --backend pg --scenario stress --writers 20 --duration 30
# STDB stress, 30s, 50 writers
npm run run -- --backend stdb --scenario stress --writers 50 --duration 30 \
--module chat-app-20260406-153727
# Both throughput scenarios for one backend
npm run run -- --backend pg --scenario all
npm run run -- --backend stdb --scenario all --module chat-app-20260406-153727
Results land in results/<timestamp>/<backend>-<scenario>.json.
Saved optimized-reference snapshots also live under
results/optimized-reference/.
Tracked reference implementations and methodology live in
optimized-reference/.
Caveats
- The PG app's
send_messagehandler enforces a 500ms-per-user rate limit in application code. Each PG writer can therefore issue at most ~2 msgs/sec. Throughput scales with writers, not with cadence. The harness paces writers at ~510ms to avoid drops. SpacetimeDB has no equivalent limit, so its per-writer ceiling is much higher. - Numbers reflect what shipped from the LLM, on a single dev machine, against a local DB. They are not the theoretical ceiling of either backend.
- Each connection in the harness uses the same Node process clock, so fan-out latency is meaningful (no clock skew across machines).