bradleyshep
7143ed9722
# Description of Changes Refresh this stale PR against current `master`. Several original items were already applied upstream or conflict with recent keynote-2 work (#4616, #4647, #4678, #4682, #4698, #4703, #4743, #4745, #4753, #4757), so those are dropped. What remains is the still-relevant subset, rebased onto the current file structures. **README.md:** - Use `pnpm run demo` in Quick Demo (consistency with pnpm workspace) - Add `--concurrency` and `--alpha` to demo options - Add `--` separator to `docker compose run` bench example - Fix hardware config punctuation (add comma before "OS:") - Remove redundant Quick Start section; replace with link to DEVELOP.md for prerequisites and CLI reference - Add symlink for license **DEVELOP.md:** - Use `pnpm run` throughout (demo, prep, bench) instead of `npm run` - Drop the `-- ` pass-through after `pnpm run bench` (not needed with pnpm; matches the `#4703` testing examples) - Add Rust to Prerequisites - Add explicit list of valid connector names (`convex`, `spacetimedb`, `bun`, `postgres_rpc`, `cockroach_rpc`, `sqlite_rpc`, `supabase_rpc`, `planetscale_pg_rpc`) - Update CLI reference defaults to match methodology (seconds: 1→10, concurrency: 10→50) - Condense `docker compose run` bench example to a single line with `--` separator; fix `npm prep` → `pnpm run prep` **src/opts.ts:** (CLI parsing moved here in `#4703`; original PR targeted the now-gone inline parsers in `cli.ts`/`demo.ts`) - `parseBenchOptions`: bench `--seconds` default `1` → `10` - `parseDemoOptions`: demo `--concurrency` default `10` → `50` **.env.example:** - Comment out `USE_DOCKER=1` and `SKIP_CONVEX=1` so demo defaults (convex, spacetimedb) work out of the box - Comment out `CONVEX_USE_SHARDED_COUNTER=1` (still a supported knob, just off by default) # Dropped as superseded by master - Rust Client README section tweaks (heading capitalization, `bottlnecked`/`then` typo fixes) — section was removed by `#4753` - Rename `SPACETIME_METRICS_ENDPOINT` → `USE_SPACETIME_METRICS_ENDPOINT` — master's `src/config.ts` still reads the original name - Connector-name fixes in examples (`sqlite` → `sqlite_rpc`, `postgres` → `postgres_rpc`) — already corrected on master # API and ABI breaking changes None. # Expected complexity level and risk **1** – Documentation and default-value changes. No functional changes to core logic. # Testing - [x] `pnpm install` in `templates/keynote-2/` succeeds - [x] `pnpm run bench --help` / `pnpm run demo --help` render with valid-connec
SpacetimeDB Benchmark Suite
A benchmark suite comparing SpacetimeDB against traditional web application stacks for transactional workloads.
Quick Demo
See SpacetimeDB's performance advantage with one command:
pnpm install
pnpm run demo
The demo compares SpacetimeDB and Convex by default, since both are easy for anyone to set up and run locally without additional infrastructure. Other systems (Postgres, CockroachDB, SQLite, etc.) are also supported but require more setup. The demo checks that required services are running (prompts you to start them if not), seeds databases, and displays animated results.
Options: --systems a,b,c | --seconds N | --concurrency N | --alpha N | --skip-prep | --no-animation
Note: demo always runs the built-in test-1 scenario. Use bench if you need to specify a test name directly.
Note: demo selects targets with --systems; bench filters test connectors with --connectors.
Results Summary
All tests use 50 concurrent connections with a transfer workload (read-modify-write transaction between two accounts).
| System | TPS (~0% Contention) | TPS (~80% Contention) |
|---|---|---|
| SpacetimeDB | 107,850 | 103,590 |
| SQLite + Node HTTP + Drizzle | 7,845 | 7,652 |
| Bun + Drizzle + Postgres | 7,115 | 2,074 |
| Postgres + Node HTTP + Drizzle | 6,429 | 2,798 |
| Supabase + Node HTTP + Drizzle | 6,310 | 1,268 |
| CockroachDB + Node HTTP + Drizzle | 5,129 | 197 |
| PlanetScale + Node HTTP + Drizzle | 477 | 30 |
| Convex | 438 | 58 |
Key Finding: SpacetimeDB achieves ~14x higher throughput than the next best option (SQLite RPC) and maintains nearly identical performance under high contention (only ~4% drop), while traditional databases suffer significant degradation (CockroachDB drops 96%).
Note: SpacetimeDB runs on ARM architectures (including Apple M-series Macs), but has not yet been optimized for them.
Contention Impact
The chart above shows TPS vs Zipf Alpha (contention level). Higher alpha values concentrate more transactions on fewer "hot" accounts, increasing contention. SpacetimeDB maintains consistent performance regardless of contention level, while traditional database architectures show significant degradation.
Methodology
All systems were tested with out-of-the-box default settings - no custom tuning, no configuration optimization. This reflects what developers experience when they first adopt these technologies.
For cloud services, we tested paid tiers to give them their best chance:
- PlanetScale: PS-2560 (32 vCPUs, 256 GB RAM), single node, us-central1.
- Supabase: Pro tier
- Convex: Pro tier
Test Architecture
All benchmarks follow an apples-to-apples comparison using the same architecture pattern:
Client → Web Server (HTTP) → ORM (Drizzle) → Database
Or for integrated platforms (SpacetimeDB, Convex):
Client → Integrated Platform (compute + storage colocated)
This ensures we're measuring real-world application performance, not raw database throughput.
The Transaction
Each transaction performs a fund transfer between two accounts:
- Read both source and destination account balances
- Verify sufficient funds in source account
- Debit source account
- Credit destination account
- Commit transaction with row-level locking
This is a classic read-modify-write workload that tests transactional integrity under concurrent access.
Test Command
docker compose run --rm bench -- --seconds 10 --concurrency 50 --alpha XX --connectors YY
--seconds 10: Duration of benchmark run--concurrency 50: Number of concurrent client connections--alpha 0: ~0% contention (uniform account distribution)--alpha 1.5: ~80% contention (Zipf distribution concentrating on hot accounts)--stdb-compression none|gzip: SpacetimeDB client compression mode (default:none)
Hardware Configuration
Server Machine (Variant A - PhoenixNAP):
- s3.c3.medium bare metal instance - Intel i9-14900k 24 cores (32 threads), 128GB DDR5 Memory, OS: Ubuntu 24.04
Server Machine (Variant B - Google Cloud):
- c4-standard-32-lssd (32 vCPUs, 120 GB Memory) OS: Ubuntu 24.04
- RAID 0 on 5 Local SSDs
- Region: us-central1
Client Machine:
- c4-standard-32 (32 vCPUs, 120 GB Memory) OS: Ubuntu 24.04
- Region: us-central1
- Runs on a separate machine from the server
Note: All services (databases, web servers, benchmark runner) except Convex local dev backend run in the same Docker environment on the server machine.
Why Separate Client Machines?
Running clients on separate machines ensures:
- Network round-trip latency is measured (realistic production scenario)
- Client CPU/memory doesn't compete with server resources
- Results reflect actual deployment conditions
Account Seeding
- 100,000 accounts seeded before each benchmark
- Initial balance: 10,000,000 per account
- Zipf distribution controls which accounts are selected for transfers
Technical Notes
Why SpacetimeDB Outperforms Traditional Stacks
The primary bottleneck in traditional web application architectures is the round-trip latency between the application server and database:
Traditional: Client → Server → Database → Server → Client
↑___________↑
Network round-trip per query
SpacetimeDB eliminates this by colocating compute and storage:
SpacetimeDB: Client → SpacetimeDB (compute + storage) → Client
This architectural difference means SpacetimeDB can execute transactions in microseconds rather than milliseconds, resulting in order-of-magnitude performance improvements.
Client Pipelining
The benchmark supports pipelining for all clients - sending multiple requests without waiting for responses. This maximizes throughput by keeping connections saturated.
Confirmed Reads (withConfirmedReads)
SpacetimeDB supports withConfirmedReads mode which ensures transactions are durably committed before acknowledging to the client. The benchmark results shown use withConfirmedReads = ON for fair comparison with databases that provide similar durability guarantees.
Cloud vs Local Results
PlanetScale results (~477 TPS) demonstrate the significant impact of cloud database latency. When the database is accessed over the network (even within the same cloud region), round-trip latency dominates performance. This is why SpacetimeDB's colocated architecture provides such dramatic improvements.
Systems Tested
| System | Architecture |
|---|---|
| SpacetimeDB | Integrated platform. |
| SQLite + Node HTTP + Drizzle | Node.js HTTP server → Drizzle ORM → SQLite |
| Bun + Drizzle + Postgres | Bun HTTP server → Drizzle ORM → PostgreSQL |
| Postgres + Node HTTP + Drizzle | Node.js HTTP server → Drizzle ORM → PostgreSQL |
| Supabase + Node HTTP + Drizzle | Node.js HTTP server → Drizzle ORM → Supabase (Postgres) |
| CockroachDB + Node HTTP + Drizzle | Node.js HTTP server → Drizzle ORM → CockroachDB |
| PlanetScale + Node HTTP + Drizzle | Node.js HTTP server → Drizzle ORM → PlanetScale (Cloud) |
| Convex | Integrated platform |
Running the Benchmarks
See DEVELOP.md for prerequisites, configuration, and full CLI reference.
The distributed TypeScript SpacetimeDB workflow is documented there as Run the distributed TypeScript SpacetimeDB benchmark.
Output
Benchmark results are written to ./runs/ as JSON files with TPS and latency statistics.
License
See repository root for license information.