# Description of Changes
Makes the keynote benchmark job reusable so that it can be invoked and
run in other CI environments.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
2
# Testing
Refactor. Relies on existing coverage.
### Note 1: this requires a website PR to merge
### Note 2:
I was able to run all workflow smoke tests successfully, including
golden validation and dry-run benchmarks, except for the C# dry-run
benchmark path. C# golden validation passes, but the C# benchmark dry
run still fails intermittently/consistently on the runner despite
several attempts to align its build/publish setup with the known-good
smoketest path.
```
gh workflow run llm-benchmark-periodic.yml `
--repo ClockworkLabs/SpacetimeDB `
--ref bradley/fix-validate-goldens-ci `
-f model_set=explicit `
-f models="openrouter:openai/gpt-5.4-mini" `
-f languages=rust,csharp,typescript `
-f modes=guidelines `
-f tasks=t_000_empty_reducers `
-f dry_run=true
```
# Description of Changes
This updates the LLM benchmark automation and runner plumbing.
- Move periodic LLM benchmark and golden validation workflows from
daily/nightly to weekly Monday UTC runs.
- Add manual workflow inputs for benchmark smoke runs:
- model set: website-managed, local defaults, or explicit models
- languages, modes, categories, tasks
- dry-run mode
- Build the local TypeScript SDK before TypeScript benchmark/golden
validation runs.
- Add support for fetching active/available benchmark models from the
website API via `--model-source remote`.
- Keep explicit `--models ...` working for manual/local overrides.
- Add OpenRouter preflight checks before benchmark execution:
- checks key/account credits when available
- probes the selected model when credit balance cannot be checked
- supports `OPENROUTER_ALLOW_UNCHECKED_CREDITS=1` escape hatch
- supports `OPENROUTER_MIN_CREDITS` / `LLM_MIN_CREDITS`
- Force scheduled benchmark workflow runs through OpenRouter with
`LLM_VENDOR=openrouter`, while preserving direct OpenAI support for
local/manual use.
- Improve benchmark publishing isolation:
- isolated SpacetimeDB CLI root per publish
- serialized C# benchmark publish concurrency
- local NuGet package references for generated C# benchmark projects
- Windows/PATH handling for TypeScript `pnpm`
- Update default benchmark model routes to current model names/ids.
- Update TypeScript golden answers for current SDK shape.
# API and ABI breaking changes
None.
This adds benchmark-runner/workflow behavior and CLI options, but does
not change SpacetimeDB runtime API or ABI.
# Expected complexity level and risk
3/5
The changes are mostly isolated to the LLM benchmark runner and GitHub
workflows, but the risk is moderate because they touch CI execution
paths, local SDK build assumptions, website-managed model resolution,
OpenRouter routing, and generated module publish behavior across Rust,
C#, and TypeScript.
The most sensitive pieces are:
- GitHub Actions workflow dispatch/manual input behavior.
- Remote model registry parsing from the website.
- C# benchmark publish behavior on the self-hosted runner.
# Testing
- [x] `cargo check -p xtask-llm-benchmark --bin llm_benchmark`
- [x] `cargo test -p xtask-llm-benchmark --bin llm_benchmark`
- [x] `cargo test -p xtask-llm-benchmark
parses_active_available_model_routes`
- [x] Manual GitHub Actions golden validation smoke runs for Rust, C#,
and TypeScript.
- [ ] Run a dry-run periodic benchmark workflow from this branch with
one explicit OpenRouter model, one task, and all languages.
- [ ] Run a website-dispatched dry-run benchmark and verify it sends
`model_set=explicit` plus selected model/task inputs.
---------
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@clockworklabs.io>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
Reviewing recent benchmark runs, it appears that #5071 probably
regressed TPS by around 3-5%. I don't want to revert that change because
it has implications for replication, and so for now we'll just live with
the slight regression.
# API and ABI breaking changes
None
# Expected complexity level and risk
0
# Testing
N/A
# Description of Changes
Use an isolated server process per SDK test instead of a single process
for all of the tests. In addition to reducing the memory footprint of
each test run, this should also allow for more parallelism among the
individual tests.
# API and ABI breaking changes
None
# Expected complexity level and risk
1.5
# Testing
The SDK test suite should continue to work
## Summary
- Add a CLA Gate workflow that publishes a repository-owned commit
status.
- Mirror CLA Assistant license/cla status on pull requests and status
events.
- Publish CLA Gate=success on merge_group runs, because entries have
already passed PR checks before entering the queue.
## Required settings change after merge
- Remove license/cla from required checks.
- Add CLA Gate as the required CLA check.
This keeps CLA enforcement before merge queue while removing CLA
Assistant from the merge-group critical path.
## Test plan
- Parsed .github/workflows/cla-gate.yml with Ruby YAML loader.
- Ran git diff --check.
---------
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
# Description of Changes
(Moving this to a tools repo)
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
None
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
AI app generation benchmark comparing SpacetimeDB vs PostgreSQL (Express
+ Socket.io + Drizzle ORM). Same AI model (Claude Sonnet 4.6), same
prompts, same chat app, two backends. Upgraded through 12 feature
levels, manually graded at each level, bugs fixed, all costs measured
via OpenTelemetry.
Results viewable at:
https://spacetimedb.com/llms-benchmark-sequential-upgrade
## Benchmark harness (`tools/llm-sequential-upgrade/`)
- `run.sh`: orchestrates headless Claude Code sessions for code
generation, sequential upgrades, and bug fixes. Tracks all API costs via
OTel. Supports `--upgrade`, `--fix`, `--composed-prompt`,
`--resume-session` modes.
- `grade.sh` / `grade-agents.sh`: grading harnesses for manual testing
of generated apps.
- `docker-compose.otel.yaml`: OTel collector + PostgreSQL services.
- `generate-report.mjs` / `parse-telemetry.mjs`: aggregate per-session
telemetry into cost reports.
- Backend guidelines in `backends/`: SpacetimeDB SDK reference, config
templates, server setup docs, PostgreSQL setup with Drizzle/Socket.io
guidance.
**After https://github.com/clockworklabs/SpacetimeDB/pull/4740 merges,
we will likely want to update this so that it reads backend and SDK
guidance from SKILLS**
## Two complete benchmark runs
**Run 1 (20260403):** Original methodology.
**Run 2 (20260406):** Refined methodology with domain bias removed from
SpacetimeDB SDK docs and PostgreSQL instructions made
feature-spec-neutral.
**Note: no meaningful changes in results were observed with these
changes. Domain familiarity biases were very small and almost certainly
not the cause of STDB's major gains over PG stack.**
Each run contains full L1-L12 app source for both backends, level
snapshots preserving state before each upgrade, and per-session OTel
cost summaries.
## 12 feature levels
| Level | Feature |
|---|---|
| L1 | Basic Chat + Typing + Read Receipts + Unread Counts |
| L2 | Scheduled Messages |
| L3 | Ephemeral Messages |
| L4 | Message Reactions |
| L5 | Message Editing with History |
| L6 | Real-Time Permissions (kick, ban, promote) |
| L7 | Rich User Presence |
| L8 | Message Threading |
| L9 | Private Rooms + Direct Messages |
| L10 | Room Activity Indicators |
| L11 | Draft Sync |
| L12 | Anonymous to Registered Migration |
## Results
| | Run 1 (20260403) | Run 2 (20260406) |
|---|---|---|
| **SpacetimeDB total cost** | $13.33 | $12.62 |
| **PostgreSQL total cost** | $17.80 | $19.68 |
| **SpacetimeDB bugs** | 5 | 2 |
| **PostgreSQL bugs** | 19 | 8 |
| **SpacetimeDB fix sessions** | 4 | 1 |
| **PostgreSQL fix sessions** | 17 | 10 |
Both runs agree: SpacetimeDB apps are cheaper to build, have fewer bugs,
and require fewer fix iterations. The refined methodology (Run 2)
widened the cost gap and **confirmed the advantage is structural, not an
artifact of domain-biased SDK docs.**
## Performance benchmark (`perf-benchmark/`)
Stress throughput tool that fires concurrent writers at peak saturation
against the AI-generated `send_message` handlers.
| Tier | SpacetimeDB (avg) | PostgreSQL (avg) | Ratio |
|---|---|---|---|
| AI-generated (as-shipped) | 5,267 msgs/sec | 694 msgs/sec | 7.6x |
| PG rate limit removed | 5,267 msgs/sec | 1,070 msgs/sec | 4.9x |
| Optimized (same features kept) | 25,278 msgs/sec | 1,139 msgs/sec |
22x |
The gap widens with optimization because SpacetimeDB's bottleneck is
fixable code patterns in the reducer while PostgreSQL's bottleneck is
architectural (sequential network round-trips to an external database).
Optimized reference code with all features preserved is in
`perf-benchmark/results/optimized-reference/`.
## Data handling
Per-session cost summaries (`cost-summary.json`, `COST_REPORT.md`,
`metadata.json`) are committed. Raw OTel telemetry
(`raw-telemetry.jsonl`) containing PII is excluded via `.gitignore` and
stored privately.
# API and ABI breaking changes
None. All changes are in `tools/llm-sequential-upgrade/`. No production
code, library, or SDK changes.
# Expected complexity level and risk
**1 - Trivial.** Self-contained benchmarking tooling and data. No
interaction with production code.
# Testing
- [x] L1-L12 upgrades completed on all 4 apps (2 backends x 2 runs) with
OTel cost capture
- [x] All levels manually graded after each upgrade; bugs filed and
fixed via the harness
- [x] Methodology refinement between runs validated (domain bias
removal, feature-neutral instructions)
- [x] Stress benchmarks run across both runs x 3 tiers (as-shipped,
rate-limit-removed, optimized)
- [x] Optimized benchmarks verified to preserve all original features
- [x] Sensitive data (PII in raw telemetry) removed from repo and
gitignored
- [ ] Reviewer: spot-check that METRICS_DATA.json / METRICS_REPORT.json
numbers match the telemetry cost-summary.json files
---------
Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
## Summary
- Add a small `Retry CLA Assistant` workflow for cases where
`license/cla` is missing or pending after the normal PR checks have gone
green.
- Keep the workflow thin: it checks out trusted default-branch code,
determines the PR number from the GitHub event, and runs `cargo ci
retry-cla-assistant --pr-number <number>`.
- Put the retry gate and CLA Assistant recheck call in the existing Rust
`tools/ci` crate.
- Trigger it from PR updates, any workflow completion, manual dispatch
with a PR number, and a 15-minute scheduled fallback sweep.
## Behavior
The workflow passes explicit PR numbers into the Rust command. For
scheduled runs, it enumerates open `master` PRs in the workflow and
invokes the Rust command for each one.
The `cargo ci retry-cla-assistant --pr-number <number>` command only
calls CLA Assistant's recheck endpoint when the PR is open, non-draft,
targets `master`, the head SHA is at least 10 minutes old, at least one
check run exists, all reported check runs are green, no non-CLA commit
status is non-green, and `license/cla` is missing/pending/failing.
The `workflow_run` trigger is intentionally unfiltered, so adding or
renaming CI workflows does not require changing this workflow.
It calls
`https://cla-assistant.io/check/{owner}/{repo}?pullRequest={number}` and
polls `license/cla` for up to 3 minutes.
## Safety
- The workflow uses `pull_request_target`, but it does not check out or
execute PR code. It checks out the trusted default branch before running
the CI tool.
- It ignores completion of the `Retry CLA Assistant` workflow itself.
- It intentionally does not forge a `license/cla` status.
## Validation
- Parsed the workflow YAML locally with Ruby/Psych.
- Ran `cargo fmt --package ci`.
- Ran `cargo check -p ci`.
- Ran `cargo ci retry-cla-assistant --help` and verified `--pr-number`
is required.
- Tested the first Rust version against live PR #5164 metadata and
caught that a hard-coded required-check list was too strict; replaced
that list with a reported-checks-all-green gate.
Addresses #5215.
---------
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
# Description of Changes
Part of https://github.com/clockworklabs/SpacetimeDB/issues/4970.
Doing this part in a separate PR because it turns out that this test was
failing, so this PR also includes the fixes.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
CI passes (but only with the "code" changes).
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
A sort of followup to/unrevert of #4884, an alternative to #4927.
In #3832, we made FunctionBudget a different unit than EnergyQuanta, but
there were still many places where we treated them the same and directly
converted between them. I got confused by that in #4884, and now this PR
properly separates them and corrects the v8 energy calculation.
Also, since we now benchmark the same module in rust and typescript in
the form of the keynote benchmark, this patch adds a new assertion to
that test that the fuel usage of both modules is within 2X of each
other.
# Expected complexity level and risk
3 - touches energy calculation without reverting the problematic #4884,
but I'm confident it's correct this time.
# Testing
- [x] Verified that the conversion factors make sense for wasmtime and
for v8.
- [x] Assert that typescript and rust keynote bench runs have similar
cpu usage
---------
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
# Description of Changes
This addresses issue #5121 by updating the `bump-versions` cargo command
to include:
* Regenerating case conversion test bindings to update embedded CLI
version
* Updating the version in the Godot SDK's `.csproj`
* Updating the version in the 2 republishing tests in C# regression
tests' `.csproj`
# API and ABI breaking changes
No API or ABI changes
# Expected complexity level and risk
1 - Small and strait forward
# Testing
- [X] Ran `cargo bump-versions --all 2.3.0-test --accept-snapshots`
locally from the root of the repo, and confirmed that ~25 files being
modified.
* 22 where related to the version bump.
* 3 files in the case-conversion-test-client typescript module bindings
where updated as part of the regenerating steps.
# Description of Changes
the `README.md` (generated by `cargo ci self-docs`) now properly
includes the value names passed to args.
This change was made by AI.
# API and ABI breaking changes
None. CI only.
# Expected complexity level and risk
1
# Testing
CI, including the docs check, passes
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
The keynote benchmark check has had a couple spurious failures in CI
recently, namely for the rust module. I set a higher TPS threshold for
rust than I did for typescript since rust is a bit faster, but it looks
like the threshold I set is within the margin of error, so this change
drops it to 275K - exactly what it is for typescript.
A failure now almost certainly represents a real performance regression
and requires deeper investigation.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
0 (one liner)
# Testing
Should have no more spurious failures for this CI job.
# Description of Changes
Adds support to Rust modules and the SpacetimeDB host for defining HTTP
handlers and registering them to routes.
## User-facing API
In a Rust module, users can annotate functions with the new macro
`#[spacetimedb::http::handler]`. A function annotated this way must
accept exactly two arguments, of types `&mut
spacetimedb::http::HandlerContext` and `spacetimedb::http::Request`
(which is a type alias for `http::Request<spacetimedb::http::Body>`. It
must also return `spacetimedb::http::Response` (which is a type alias
for `http::Response<spacetimedb::http::Body>`).
Once the user has defined an HTTP handler, they can register it to a
route by annotating a function with `#[spacetimedb::http::router]`. Such
a function must take no arguments and return a
`spacetimedb::http::Router`. (The original design put this annotation on
a `static` variable rather than a function, but that turned out to be
undesirable because it required that constructing a `Router` be
`const`.) `Router` exposes various methods for registering handlers to
routes.
All of a database's user-defined routes are exposed under
`/v1/database/:name_or_identity/route/{*path}`.
## Example
See [the new
smoketest](https://github.com/clockworklabs/SpacetimeDB/blob/phoebe/http-handlers-webhooks/crates/smoketests/tests/smoketests/http_routes.rs)
for a more exhaustive example.
A simpler example, which stores arbitrary byte data in a table via a
`POST` request, returns an ID, and then retrieves that same data via a
`GET` request with a query parameter:
```rust
#[spacetimedb::table(accessor = data)]
struct Data {
#[primary_key]
#[auto_inc]
id: u64,
body: Vec<u8>,
}
#[spacetimedb::http::handler]
fn insert(ctx: &mut HandlerContext, request: Request) -> Response {
let body: Vec<u8> = request.into_body().into_bytes().into();
let id = ctx.with_tx(|tx| tx.db.data().insert(Data { id: 0, body: body.clone() }).id);
Response::new(Body::from_bytes(format!("{id}")))
}
#[spacetimedb::http::handler]
fn retrieve(ctx: &mut HandlerContext, request: Request) -> Response {
let id = request
.uri()
.query()
.and_then(|query| query.strip_prefix("id="))
.and_then(|id| u64::from_str(id).ok())
.unwrap();
let body = ctx.with_tx(|tx| tx.db.data().id().find(id).map(|data| data.body));
if let Some(body) = body {
Response::new(Body::from_bytes(body))
} else {
Response::builder().status(404).body(Body::empty()).unwrap()
}
}
#[spacetimedb::http::router]
fn router() -> Router {
Router::new().post("/insert", insert).get("/retrieve", retrieve)
}
```
## Design and implementation notes
- As mentioned above, the router is registered via a function, not a
`static` or `const` item. This is because `static` or `const`
initializers must be `const`, and it turns out to be a pain to make all
of the `Router` constructors be `const fn`s.
- The `#[handler]` macro clobbers the original function name with a
`const` variable of type `HttpHandler`. This is unfortunate, but AFAICT
necessary, 'cause we need to pass the string identifier for the handler
to the `Router`, not the function pointer, and Rust allows no (stable
and reliable) way to get a unique string identifier out of a function
item/value, nor to attach data or implement traits for function
items/values. The alternative(s) would involve changing the signature of
the `Router` methods to have uglier and more complex callsites, e.g.
like `.get("/retrieve", retrieve::handler())`, `.get("/retrieve",
handler!(retrieve))` or `.get::<retrieve>("/retrieve")`. I believe that
registering handlers will be much more common than calling their
functions, so I've chosen to make it so that registering them gets the
convenient syntax, even though the inability to call them directly will
be somewhat surprising.
- I haven't wired up energy handling or timing metrics for handler
execution to anywhere. Procedures are still in the same boat.
- HTTP requests to user-defined routes bypass the usual SpacetimeDB auth
middleware, meaning that the host does not validate (or inspect in any
way) `Authorization` headers in requests before invoking the
user-defined handler. This is required to allow arbitrary
user-programmable handling of `Authorization` headers, including those
in formations which SpacetimeDB would reject. As a result of this,
`HandlerContext` doesn't expose a `sender` or `sender_connection_id`.
- HTTP route paths may consist only of a very restrictive set of
characters. I've chosen this set to keep our options open in the future
to add additional syntax to routes, like for registering wildcard
segments and path parameters:
- ASCII digits.
- ASCII letters.
- `-_~/`.
- The internal data structure that represents a `Router` is currently a
`Vec<Route>`, meaning that resolving a request to a route is
`O(num_routes)`. Registering a route checks against each previous route
for uniqueness, meaning that constructing a router is `O(num_routes ^
2)`. There are TODO comments to use a trie, but I think this can wait,
as I expect most databases to register few routes.
- Commit 999a7c317 contains a fix to a mostly-unrelated bug where a few
bindings introduced by the SATS derive macros were unhygienic and not in
a reserved namespace, leading to name conflicts. I discovered this
'cause I tried writing an HTTP handler named `index` to serve the
index/root of a website and it broke.
## Still TODO
- [x] Resolve various TODO comments in the diff.
- [x] Documentation.
- [x] C# bindings support.
- [x] C++ bindings support.
- [x] V8 host support.
- [x] TypeScript bindings support.
# API and ABI breaking changes
New APIs, currently flagged as `unstable`, which will eventually need
stability guarantees. No (intentional) breaking changes, or changes to
existing APIs at all.
# Expected complexity level and risk
3? Changes to our HTTP routing to support the user-defined routes.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] New smoketest of the behavior!
- [x] I dunno, maybe try hosting a simple webpage and see how it works?
- [x] Build a test app with Stripe integration.
- @aasoni did this.
---------
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
# Description of Changes
Adds a new required ci check for keynote-2 benchmark regressions. The
test runs for 60s and fails if throughput < 300K TPS.
Note, this check will be flaky as long as it's running concurrently with
other CI jobs. It may need a dedicated runner/host machine. Although it
may be sufficient to only schedule one runner/VM to a single host
machine at a time. I'll need to sync with @jdetter to determine the best
way forward here.
UPDATE: We're using a dedicated runner. See the **Testing** section.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
2
Mainly copy-paste from the other CI workflows.
# Testing
This job now uses `spacetimedb-benchmark-runner` which is entirely
dedicated to this one CI job. I've tested this at different times of the
day when the CI runners are under load and not. The performance is
consistent and the test isn't flaky. It has passed every time.
# Description of Changes
A follow-on to #5133 that fixes another Godot flake that was observed
after #5133 merged. Specifically a C# SDK regen flake where the Godot
package restore path reused `sdks/csharp/packages`, which is also the
Unity-visible package cache.
During `cargo regen csharp dlls`, the Godot flow previously did this:
- cleared `sdks/csharp/packages/godotsharp`
- cleared `sdks/csharp/obj~/godot`
- restored the Godot project back into `sdks/csharp/packages`
- packed the Godot project using that same Unity-visible package area
That meant a Godot-only cleanup could delete package content under the
Unity SDK package cache. If a previous GodotSharp restore was partial or
stale, regen tried to fix it by deleting `packages/godotsharp`, but that
path is visible to Unity and can cause dirty/missing package content
during regen.
With this change, Godot restore state is now isolated under
`sdks/csharp/obj~/godot/packages`, and the regen flow becomes:
- clear only `sdks/csharp/obj~/godot`
- restore the Godot project into `sdks/csharp/obj~/godot/packages`
- pack the Godot project under the same path
Clearing `obj~/godot` now removes both Godot intermediates and
Godot-only NuGet packages, while leaving the normal Unity SDK package
cache under `sdks/csharp/packages` untouched.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
1
# Testing
- [x] `unity-testsuite` passes
# Description of Changes
Harden `cargo regen csharp dlls` against flaky Godot restore/build state
in the Unity test suite.
I suspect this flake is runner specific. Some Unity runners appear to
have incomplete Godot package state, where
`sdks/csharp/packages/godotsharp` exists but is missing
`GodotSharp.dll`. In that state, NuGet can still report restore success
and generate a `project.assets.json` that contains `GodotSharp` metadata
but no usable compile asset.
The regen flow now removes the Godot MSBuild intermediate directory
(`sdks/csharp/obj~/godot`) before restoring
`SpacetimeDB.ClientSDK.Godot.csproj`. It also clears the targeted
`GodotSharp` package cache (`sdks/csharp/packages/godotsharp`) so NuGet
cannot reuse a partially restored package missing `GodotSharp.dll`.
After restore, `regen` parses `project.assets.json` and verifies that
`GodotSharp` is present and that the referenced `GodotSharp.dll` exists
before running `dotnet pack --no-restore`.
This should either prevent the flaky missing-`Godot` namespace failure,
or fail earlier with a clearer restore/package-state error.
NOTE/WARNING: This code was entirely AI generated.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
~
# Testing
- [x] `unity-testsuite` passes (I ran it at least 5 times to try to get
it scheduled to different runners)
# Description of Changes
Very small additions to the C# SDK specific for Godot.
I wrote the Godot Blackholio tutorial and updated the Unity one.
I added the image assets necessary for the tutorial.
I added the files for the Godot demo.
# API and ABI breaking changes
No breaking changes.
# Expected complexity level and risk
1. There's really no risk for current systems or projects, it's all new
additions to support nicely(-ish) Godot, new Blackholio demo for Godot
and updates to the tutorials.
# Testing
- [X] Follow the tutorial and verify everything works.
- [X] Build for Windows and verify it works.
- [x] Build for Linux and verify it works.
- [x] Build SDK
- [x] Publish to Nuget
---------
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
See https://github.com/clockworklabs/SpacetimeDB/issues/5083 for
motivation.
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] `cargo run -pspcaetimedb-cli -- init --template nodejs-ts` no
longer creates a directory with `.npmrc`
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Due to the relatively frequent supply chain attacks on especially npm
packages, we're instituting a minimum package age in the whole repo.
- Globally set a minimum npm package age in CI
- Best-effort set npm package age using `.npmrc` beside any
`package.json`
- Add CI checks that pnpm version and minimum package age values are the
same everywhere
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] CI passes
- [x] if I remove a `.npmrc` then `cargo ci lint` fails
- [x] if I change a value in `.npmrc` then `cargo ci lint` fails
---------
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
LLM benchmark infrastructure improvements and new benchmark tasks.
**Runner & scoring:**
- Add retry logic with backoff for LLM API calls (rate limits,
502/503/504, timeouts)
- Fix `generation_duration_ms` to only time the successful attempt, not
retries+sleep delays
- Add `--dry-run` flag to run benchmarks without saving results
- Add OpenRouter client as unified fallback when direct vendor keys
aren't set
- Add web search mode via OpenRouter `:online` suffix
- Extract shared OpenAI-compatible response types into `oa_compat.rs`
- Add `ReducerCallBothScorer` for calling reducers on both golden and
LLM databases
- Set `max_tokens` on OpenRouter and Meta clients to prevent silent
truncation
**Model routing:**
- Add `ModelRoute` with display name, vendor, API model, and OpenRouter
model ID
- Support ad-hoc model IDs via `--models vendor:model` without static
registration
- Add model name normalization (OpenRouter IDs, case variants →
canonical display names)
**Context modes:**
- Add `guidelines`, `cursor_rules`, `search`, `no_context` modes with
`is_empty_context_mode()` helper
- Add mode-specific prompt preambles
- Consolidate mode alias normalization (`none`/`no_guidelines` →
`no_context`)
**CI workflows:**
- Add `llm-benchmark-periodic.yml` for scheduled nightly runs with
per-language failure tracking
- **Note**: The periodic workflow requires `OPENROUTER_API_KEY`,
`LLM_BENCHMARK_UPLOAD_URL`, and `LLM_BENCHMARK_API_KEY` as GitHub
secrets.
- Add `llm-benchmark-validate-goldens.yml` for validating golden answers
still compile
**Results & summary:**
- Add `cmd_status` to show incomplete benchmark combinations with rerun
commands
- Add `cmd_analyze` for LLM-powered failure analysis
- Split `normalize_details_file` from `write_summary_from_details_file`
- Derive task categories from filesystem for summary generation
- Add timestamp tracking (`started_at`/`finished_at`) and token usage
**New benchmark tasks:**
- 30 new tasks across auth, data_modeling, queries, basics, and schema
categories
- Updated/fixed existing task prompts and golden answers
# API and ABI breaking changes
None. Internal tooling only.
# Expected complexity level and risk
2 — Changes are scoped to the LLM benchmark CLI tool
(`xtask-llm-benchmark`) and CI workflows. No impact on SpacetimeDB core.
# Testing
- [x] `cargo check -p xtask-llm-benchmark` — zero errors, zero warnings
- [x] Dry run: `llm_benchmark run --lang typescript --modes no_context
--tasks t_001 --models openai:gpt-5-mini --dry-run` — ran end-to-end,
confirmed no results saved to disk
- [ ] Verify periodic workflow runs successfully on next scheduled
trigger
---------
Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
# Description of Changes
Merged the `upgrade-version-check.yml` into `ci.yml`, and moved the
business logic under `cargo ci`.
I would also be very open to just removing this test until we choose to
define a better suite of tests for `cargo bump-version`.
# API and ABI breaking changes
None. CI only.
# Expected complexity level and risk
1
# Testing
- [x] Ran it locally. It made a diff
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Just removing an old script that we would like to maintain via a cargo
subcommand now
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
None
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Migrate these checks into `cargo ci`:
- Check that packages are publishable
- Docs test
- TypeScript - Tests
# API and ABI breaking changes
None. CI only.
# Expected complexity level and risk
2
# Testing
- [ ] CI passes
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Removed some "if we're on windows" checks in the CI so that we're always
running through `cargo ci update-flow`.
# API and ABI breaking changes
None. CI only.
# Expected complexity level and risk
1.
# Testing
- [x] Upgrade flow tests pass
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Just moving code into functions so it can be reused, no functional
changes.
# API and ABI breaking changes
None. CI only.
# Expected complexity level and risk
1
# Testing
None
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Several args for `cargo ci` had empty helptext, because we were only
printing the explicitly-annotated helptext. This PR updates it so that
inline helptext also shows in the README.
# API and ABI breaking changes
None. CI only.
# Expected complexity level and risk
1
# Testing
- [x] Updated README has more helptext
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Merge the `TypeScript - Lint` CI job into `cargo ci lint`.
Note that this removes the custom caching for the pnpm store, but we're
planning to overhaul our CI cache approach anyway.
# API and ABI breaking changes
None. CI only.
# Expected complexity level and risk
1
# Testing
- [x] Lint step passes on this PR
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
To make `cargo ci test` more properly include the full test logic.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
1
# Testing
honestly none
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
To make `cargo ci` more properly the full test logic.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
1
# Testing
honestly none
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
This PR adapts the Rust SDK test suite to work with the wasm version
added in https://github.com/clockworklabs/SpacetimeDB/pull/4089 (which
I've closed in favor of this PR).
Most of the changes revolve around wasm's different async semantics -
everything runs in one thread, so things that relied on background
threads didn't work directly. Several tests would lock up because
something in them blocked synchronously, which blocked any background
work from progressing.
We moved the test-clients contents into a `test_handlers.rs` so that it
could be called from both `main` (for native tests) and `lib` (for wasm
tests). To show what actually changed, use:
```bash
git diff --no-index -- <(git show origin/master:sdks/rust/tests/procedure-client/src/main.rs) sdks/rust/tests/procedure-client/src/test_handlers.rs
```
(or similar for other test-clients)
# API and ABI breaking changes
None, I think/hope.
# Expected complexity level and risk
2
# Testing
- [x] I've augmented the CI to also run the test suite with the `web`
feature
---------
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Thales R <thlsrmsdev@gmail.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Adds `tools/templates/` scripts to derive template metadata from
manifests and generate READMEs from quickstart docs. Replaces slug-based
`builtWith` with manifest-derived data and hardcoded quickstart mappings
with discovery from docs.
**Manifest-based `builtWith`** (`update-template-jsons.ts`)
- Reads `package.json`, `Cargo.toml`, and `.csproj` to populate
`builtWith` in `.template.json`.
- Scoped npm packages normalize to scope (`@angular/core` → `angular`).
Excludes `@types/*`. Adds `nodejs` only for nodejs-ts when `@types/node`
is present.
- Root manifests processed before subdirs; primary framework first (e.g.
`react` before `spacetimedb` in react-ts). Dependencies reordered in
package.json where needed.
**Dynamic quickstart discovery** (`generate-template-readmes.ts`)
- Discovers template → quickstart by parsing `--template X` from files
in `docs/docs/00100-intro/00200-quickstarts/`.
- Optional `quickstart` override in `.template.json`; must resolve under
quickstarts dir.
- chat-react-ts has no quickstart; uses manual README.
**New:** `tools/templates/` (update-template-jsons.ts,
generate-template-readmes.ts, README, package.json, pnpm-lock).
**Modified:** all `templates/*/.template.json` (added `builtWith`),
new/generated `templates/*/README.md`.
# API and ABI breaking changes
None.
# Expected complexity level and risk
**1** – Dev tooling only. No runtime or API changes. Scripts are
isolated; failures only affect generated metadata and READMEs.
# Testing
- [ ] `cd tools/templates && pnpm run generate` completes without errors
- [ ] Spot-check `builtWith` and generated READMEs for a few templates
# Description of Changes
- Updated the Unreal SDK and generated Unreal bindings for the websocket
2.0 protocol/model
- Reworked DbConnectionBase to handle the updated message shapes
- Switched subscription handling over to new message types and
QuerySetId
- Updated reducer to ReducerResult, removal of callbacks, and set
reducer flags
- Added event table support
- Baked in multi-module support replacing [the old
PR](<https://github.com/clockworklabs/SpacetimeDB/pull/3417>)
- Added functionality to generate module support for multiple folders in
the Unreal project (add <module>.Build.cs, <module>.h, <module>.cpp)
using the --module-name
- Add new configuration option for spacetime generate to handle module
prefix
- Regenerated Unreal Blackholio/TestClient/QuickstartChat bindings
- Rebuilt Unreal Blackholio's consume entity to use event tables
- Updated migration documentation
- Updated the version bump tool to impact C++
# API and ABI breaking changes
- Unreal websocket/message handling updated to the new protocol
- Unreal generation now expects a real .uproject target and will stop
immediately if project
metadata is invalid instead of continuing past setup issues.
# Expected complexity level and risk
3 - A large set of changes to update the websocket/message handling
along with heavy codegen changes to handle multi-module support
# Testing
Test coverage of the Unreal SDK will need expansion in a future ticket
once our issues with flakiness on CI is resolved.
- [x] Updated Unreal Blackholio
- [x] Ran full Unreal SDK test suite
- [x] Built new test project using the new `--module-prefix`
- [x] Run through Unreal Blackholio (C++ and Blueprint)
- [x] Rebuilt Unreal Blackholio with multi-module, and duplicate
generated module testing side-by-side modules that would overlap
# Review Question(s)
- [x] Updates to `spacetime init` have made the tutorial a little
confusing with pathing for the Unreal Blackholio tutorial. To fix though
we'd have to update all the commands to be more explicit, or update the
tutorial `spacetime init` to use `--project-path .` to keep pathing
simpler, thoughts?
---------
Signed-off-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: Ryan <r.ekhoff@clockworklabs.io>
# Description of Changes
`cargo ci lint` was running `cargo fmt`, but that didn't pick up all
files. As a result, our pre-commit hook (which just runs `rustfmt` on
any changed `.rs` files) would change otherwise-untouched files in merge
commits.
This PR addresses that discrepancy by having `cargo ci lint` run
`rustfmt` on all tracked `.rs` files.
The entire diff is just `rustfmt` changes except for the changes in
`tools/ci/src/main.rs`.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
2
# Testing
- [x] `cargo ci lint` fails
- [x] `cargo ci lint` passes after running `rustfmt` on everything
- [x] `cargo fmt --all` doesn't cause any diff after doing the above
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Our flake.nix relies on Crane, which begins by scanning the repo for
`Cargo.toml` files in order to pre-compute the full set of dependencies
in order to record them in the Nix store. A previous PR, #4413,
introduced a `Cargo.toml` which was intentionally invalid to our
repository, with a script that modified it as part of a test. This
`Cargo.toml` was excluded from our workspace, but unfortunately, Crane's
`buildDepsOnly` doesn't respect the workspace, and just searches the
whole repository. I consider this a bug in Crane, but in the interest of
doing useful work on SpacetimeDB in the near future rather than spending
hours hacking on my build script, this commit changes the `Cargo.toml`
in question to be valid at rest, so that Crane doesn't get angry due to
failing to parse it.
# API and ABI breaking changes
N/a
# Expected complexity level and risk
1
# Testing
- [x] `nix build` and `nix develop` work locally.
- [ ] @bradleyshep should please run `cargo llm run` to his satisfaction
to verify that I haven't broken it.
# Description of Changes
LLM benchmark updates for local development:
- **Local SDK paths**: Templates use relative paths to workspace crates
(`crates/bindings`, `crates/bindings-csharp`,
`crates/bindings-typescript`) instead of published packages, so the
bench runs against local SDK changes.
- **NODEJS_DIR support**: On Windows (e.g. nvm4w), if `pnpm` is not on
PATH, the bench uses `NODEJS_DIR` to locate `pnpm` and prepends it to
PATH for subprocesses.
- **Refactor**: Extracted `relative_to_workspace()` in `templates.rs`
and removed noisy `NODEJS_DIR` logging in `publishers.rs`.
- **Benchmark results**: Updated `docs/llms/llm-comparison-details.json`
and `docs/llms/llm-comparison-summary.json`.
# API and ABI breaking changes
None.
# Expected complexity level and risk
**2** — Local-only changes to the benchmark tool. Templates now require
local SDKs to be built (especially TypeScript: `pnpm build` in
`crates/bindings-typescript`). No impact on published SDKs or runtime.
# Testing
- [ ] Run `cargo llm run --lang rust --modes docs --providers openai`
from repo root
- [ ] Run TypeScript benchmarks with `pnpm build` in
`crates/bindings-typescript` first
- [ ] On Windows with nvm4w, set `NODEJS_DIR` if `pnpm` is not on PATH
and run TypeScript benchmarks
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: clockwork-labs-bot <bot@clockworklabs.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
Previously `bump-versions` would push a failing commit because the
smoketests had a diff. This fixes that by having `bump-versions` update
that lockfile.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
- [x] Ran `cargo bump-versions --all 3.0.0` and confirmed that the
lockfile was updated
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Smoketests were running twice because `tests/foo.rs` automatically gets
turned into a test binary, but we also had a `mod.rs` that listed each
file, so they were getting tested via that as well.
This was first broken in
https://github.com/clockworklabs/SpacetimeDB/pull/4184.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
- [x] CI passes
- [x] If I run `cargo ci smoketests` locally, I still see the CLI tests
running
- [x] If I run `cargo ci smoketests` locally, I do not see a particular
CLI test name appearing more than once (e.g.
`cli_cannot_publish_breaking_change_without_flag`)
- [x] `cargo ci smoketests check-mod-list` passes
- [x] `cargo ci smoketests check-mod-list` fails if I `touch
crates/smoketests/tests/smoketests/foo.rs`
---------
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
# Description of Changes
`cargo bump-versions` works properly with prerelease versions. Before it
would quietly drop the `-foo` or `+foo` suffix.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
- [x] Ran the script and bumped the versions for the 2.0.0 prerelease
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Make the `global.json` files under `templates` into literal copies of
the root one, instead of symlinks. The symlinks were causing template
breakage when the CLI was built under windows.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Changing a template's global.json causes `cargo ci
global-json-policy` to fail
- [x] Making a template's global.json into a symlink also causes `cargo
ci global-json-policy` to fail
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
All types are now defined and exported in `module_bindings/types.ts`,
meanings there's only one module you need to import from to access
types.
# Expected complexity level and risk
2
# Testing
n/a, no change in behavior.
# Description of Changes
- Updated all markdown files that call out `#[spacetimedb::table(name`
to `#[spacetimedb::table(accessor`
- Updated the LLM details on the same change
# API and ABI breaking changes
N/A
# Expected complexity level and risk
1 - Small rework in documentation to use the new
# Testing
- [x] Did a small double check on name vs accessor locally
- [x] Codex tested all the LLM answers
# Description of Changes
A few small changes to pin the version:
- Updated the CMakeLists back from 2.0.0 to 1.12.0
- Updated the Unreal Blackholio tutorial documentation to target the
1.12.0 branch and not master
- Updated the Unreal Blackholio tutorial to call out 2.0 is coming soon
- Disabled the upgrade-version tool code path for C++ upgrades, but left
logic in place for the near future
# API and ABI breaking changes
N/A
# Expected complexity level and risk
1 - Docs change, config change, and blocking update path for C++
# Testing
- [x] Reviewed the docs
- [x] Tested update command locally
- [x] Tested `spacetime init` for `basic-cpp` and successfully built
# Description of Changes
Implementation of #4295
Convert existing `Name` attribute to `Accessor` to support new Canonical
Case conversation of 2.0
# API and ABI breaking changes
Yes, in C# modules, we no longer use the attribute name `Name`, it
should now be `Accessor`
# Expected complexity level and risk
1
# Testing
- [X] Build and tested locally
- [X] Ran regression tests locally
## Description of Changes
This PR primarily affects the `bindings-macro` and `schema` crates to
review:
### Core changes
1. Replaces the `name` macro with `accessor` for **Tables, Views,
Procedures, and Reducers** in Rust modules.
2. Extends `RawModuleDefV10` with a new section for:
* case conversion policies
* explicit names
New sections are not validated in this PR so not functional.
3. Updates index behavior:
* Index names are now always **system-generated** for clients. Which
will be fixed in follow-up PR when we start validating RawModuleDef with
explicit names.
* The `accessor` name for an index is used only inside the module.
## Breaking changes (API/ABI)
1. **Rust modules**
* The `name` macro must be replaced with `accessor`.
2. **Client bindings (all languages)**
* Index names are now system-generated instead of using explicitly
provided names.
**Complexity:** 3
A follow-up PR will reintroduce explicit names with support for case
conversion.
---------
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <bot@clockworklabs.com>
# Description of Changes
This PR implements support for the `spacetime.json` configuration file
that can be used to set up common `generate` and `publish` targets. An
example of `spacetime.json` could look like this:
```
{
"dev_run": "pnpm dev",
"generate": [
{ "out-dir": "./foobar", "module-path": "region-module", "language": "c-sharp" },
{ "out-dir": "./global", "module-path": "global-module", "language": "c-sharp" },
],
"publish": {
"database": "bitcraft",
"module-path": "spacetimedb",
"server": "local",
"children": [
{ "database": "region-1", "module-path": "region-module", server: "local" },
{ "database": "region-2", "module-path": "region-module", server: "local" }
]
}
}
```
With this config, running `spacetime generate` without any arguments
would generate bindings for two targets: `region-module` and
`global-module`. `spacetime publish` without any arguments would publish
three modules, starting from the parent: `bitcraft`, `region-1`, and
`region-2`. On top of that, the command `pnpm dev` would be executed
when using `spacetime dev`.
It is also possible to pass additional command line arguments when
calling the `publish` and `generate` commands, but there are certain
limitations. There is a special case when passing either a module path
to generate or a module name to publish. Doing that will filter out
entries in the config file that do not match. For example, running:
```
spacetime generate --project-path global-module
```
would only generate bindings for the second entry in the `generate`
list.
In a similar fashion, running:
```
spacetime publish region-1
```
would only publish the child database with the name `region-1`
Passing other existing arguments is also possible, but not all of the
arguments are available for multiple configs. For example, when running
`spacetime publish --server maincloud`, the publish command would be
applied to all of the modules listed in the config file, but the
`server` value from the command line arguments would take precedence.
Running with arguments like `--bin-path` would, however, would throw an
error as `--bin-path` makes sense only in a context of a specific
module, thus this wouldn't work: `spacetime publish --bin-path
spacetimedb/target/debug/bitcraft.wasm`. I will throw an error unless
there is only one entry to process, thus `spacetime publish --bin-path
spacetimedb/target/debug/bitcraft.wasm bitcraft` would work, as it
filters the publish targets to one entry.
# API and ABI breaking changes
None
# Expected complexity level and risk
3
The config file in itself is not overly complex, but when coupled with
the CLI it is somewhat tricky to get right. There are also some changes
that I had to make to how clap arguments are validated - because the
values can now come from both the config file and the clap config, we
can't use some of the built-in validations like `required`, or at least
I haven't found a clean way to do so.
# Testing
I've added some automated tests, but more tests and manual testing is
coming.
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: bradleyshep <148254416+bradleyshep@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: = <cloutiertyler@gmail.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
joshua-spacetime