# Description of Changes
`Cargo.lock` was getting stale in our smoketests modules, because we
don't check it anywhere. I added a check.
# API and ABI breaking changes
CI only.
# Expected complexity level and risk
1
# Testing
- [x] new CI step fails without the `Cargo.lock` changes
(<https://github.com/clockworklabs/SpacetimeDB/actions/runs/21960035403/job/63435284426?pr=4282>)
- [x] new CI step passes with the `Cargo.lock` changes
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
This adds the v2 websocket protocol and adds support on the server side.
For context on many of the changes/decisions, you can look at the
discussion on https://github.com/clockworklabs/SpacetimeDB/pull/4023.
To restate some of the key changes:
- The reducer event information is no longer sent with transaction
updates (because we don't want to broadcast reducer call information
anymore).
- If a client calls a reducer, they are sent a `ReducerResult` which
includes the outcome of the reducer call and and related row updates for
queries that the client is subscribed to.
- We no longer dedupe queries that appear in multiple query sets for the
same client. This is because we are moving toward per-query storage.
- Related to that, Unsubscribe requests have an option to send the
related rows. We need this for now, since clients don't have per-query
storage implemented yet.
- We don't have the json format in v2.
Notes for reviewers:
- This moves around the messages in
`crates/client-api-messages/src/websocket` (into `common`, `v1`, and
`v2`), and this renaming of existing messages adds a lot of noise to the
PR.
- In many places, I chose to duplicate a lot of code to have a v1
version and a v2 version. I went with this to make it easier to remove
the v1 version in the future (hopefully we can just fully delete most of
the v1 functions).
- `module_subscription_manager.rs` has probably has the biggest changes,
since we now track queries by query_set_id, and we get to remove some
complexity of v1's FormatSwitch.
<!-- Please describe your change, mention any related tickets, and so on
here. -->
# API and ABI breaking changes
The v1 protocol still works, though we won't send the reducer event info
for v10 modules.
# Expected complexity level and risk
4. This touches a lot of places.
# Testing
Unit testing is pretty minimal for the new code paths. I've done some
manual e2e testing with the typescript quickstart, and this has been
tested with a different branch implementing the v2 rust client.
---------
Co-authored-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: Jeffrey Dallatezza <jeffreydallatezza@gmail.com>
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
- Version upgrade to 2.0.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
1 - this is just a version bump
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] License file has been updated
- [x] CI passing
---------
Signed-off-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
Apparently we ran too many unity testsuite jobs today and we got this:
```
Unable to find image 'unityci/editor:ubuntu-2022.3.32f1-linux-il2cpp-3' locally
ubuntu-2022.3.32f1-linux-il2cpp-3: Pulling from unityci/editor
docker: error from registry: You have reached your unauthenticated pull rate limit. https://www.docker.com/increase-rate-limit
```
So now we login before pulling to increase our limit.
# API and ABI breaking changes
None
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
1 - this just logs us into docker before running the test
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] CI still passes
# Description of Changes
- Updated the chat tutorial to include blocks for C++ server code
- Updated the Python smoketests to allow C++ with the quickstart check
# API and ABI breaking changes
N/A
# Expected complexity level and risk
1 - Small changes to get the C++ modules to be tested through quickstart
# Testing
- [x] Ran the tests locally
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
Our releases are currently broken because the machine that built the
release is based off of ubuntu 24.04. I've now changed it back to
`ubuntu-22.04` so that we don't have this issue.
```
boppy@geralt:~/clockwork/SpacetimeDB/crates/update$ curl -sSf https://install.spacetimedb.com | sh
Downloading installer...
/tmp/tmp.6i9sLQU0ff/spacetime-install: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by /tmp/tmp.6i9sLQU0ff/spacetime-install)
boppy@geralt:~/clockwork/SpacetimeDB/crates/update$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.5 LTS
Release: 22.04
Codename: jammy
```
# API and ABI breaking changes
None
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
1 - we had fixed this before but we (meaning me) broket this again
recently.
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
I have not tested this myself but I know this fixes the issue because I
was the one who broke it.
# Description of Changes
Blocks procedures from requesting private ip ranges after dns
resolution.
Adds a new cargo feature to `spacetimedb-standalone` permitting loopback
http requests in test environments only.
# API and ABI breaking changes
None
# Expected complexity level and risk
2. I may have missed a range.
# Testing
- [x] Unit tests for IP address matching
- [x] Smoketests for blocking a private IP address
# Description of Changes
CI is currently failing in a bunch of places since we tweaked the Rust
versions available in our CI runner, and it revealed that many of our
workflows don't set the default rust toolchain.
# API and ABI breaking changes
None. CI only.
# Expected complexity level and risk
1
# Testing
- [x] CI passes in this PR
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
This adds C++ server bindings (/crate/bindings-cpp) to allow writing C++
20 modules.
- Emscripten WASM build system integration with CMake
- Macro-based code generation (SPACETIMEDB_TABLE, SPACETIMEDB_REDUCER,
etc)
- All SpacetimeDB types supported (primitives, Timestamp, Identity,
Uuid, etc)
- Product types via SPACETIMEDB_STRUCT
- Sum types via SPACETIMEDB_ENUM
- Constraints marked with FIELD* macros
# API and ABI breaking changes
None
# Expected complexity level and risk
2 - Doesn't heavily impact any other areas but is complex macro C++
structure to support a similar developer experience, did have a small
impact on init command
# Testing
- [x] modules/module-test-cpp - heavily tested every reducer
- [x] modules/benchmarks-cpp - tested through the standalone (~6x faster
than C#, ~6x slower than Rust)
- [x] modules/sdk-test-cpp
- [x] modules/sdk-test-procedure-cpp
- [x] modules/sdk-test-view-cpp
- [x] Wrote several test modules myself
- [x] Quickstart smoketest [Currently in progress]
- [ ] Write Blackholio C++ server module
---------
Signed-off-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: Ryan <r.ekhoff@clockworklabs.io>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
https://github.com/clockworklabs/SpacetimeDB/pull/4209 changed our v8
patch to fix release builds.. but it turns out that the csharp-testsuite
uses both debug _and_ release builds.
# API and ABI breaking changes
None. CI only.
# Expected complexity level and risk
1
# Testing
I dunno mannnn
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Required in order to bump rolldown to its latest version, which I'm
planning to do afterwards. I also figure we may as well do this before
releasing a major version.
# Expected complexity level and risk
2
# Testing
n/a
# Description of Changes
We were using `cargo install` but doing our hacky v8 dance for debug
build outputs. `cargo install` runs in release mode though.
This came up in https://github.com/clockworklabs/SpacetimeDB/pull/4205.
# API and ABI breaking changes
None. CI only
# Expected complexity level and risk
1
# Testing
🤷
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
This PR translates all of our Python smoketests into Rust tests which
can be run from `cargo run`
## Motivation
The purpose of this fivefold:
1. All developers on the team are familiar with Rust
2. It simplifies our devops because we can drop Python as a dependency
to run the tests
3. You can now run all tests in the repo through the single `cargo test`
interface
4. Because we use the `SpacetimeDbGuard` and `cargo test`/`cargo
nextest` we can easily parallelize the smoke tests
5. The smoketests can now use machinery imported from SpacetimeDB crates
(e.g. `bsatn` etc.)
IMPORTANT NOTE!
There are several ways to implement the smoke tests in Rust (none are
great):
1. A separate xtask specifically for the smoke tests
- This doesn't solve the problem of the CLI tests which also use the
`guard` crate
- Idiosyncratic way to run the smoke tests as opposed to cargo test
- Does NOT resolve the cargo within cargo problem because we still have
to build the test modules with cargo
2. A `build.rs` script in `guard` which first builds the executables as
a compile step for compiling guard
- Deadlocks on a cargo lock file conflict (Outer cargo compiles guard →
runs build.rs, inner cargo tries to acquire the build directory lock,
outer cargo holds the directory lock, deadlock)
- If you fix the deadlock by using different target dirs, it still looks
stuck on building guard because it's actually compiling all of
spacetimedb-standalone and spacetimedb-cli.
- Still technically runs cargo inside of cargo.
3. Add `spacetimedb-cli` and `spacetimedb-standalone` as an artifact
dependency of the guard crate
- Has good and clear output but requires +nightly when running the
smoketests and CLI tests, otherwise won't do the right thing. See
https://github.com/rust-lang/cargo/issues/9096
4. Compile the executables at runtime during the tests themselves where
the first test takes a lock while the executables are building using
cargo within cargo
- Makes the tests look like they're taking a long time when they're just
waiting for the build to complete
- Requires relatively complex locking machinery across
binaries/tests/processes
5. A two step solution where the developer has to build the binaries
before calling the smoke tests
- Very error prone
None of these are good. `xtask` is not bad, but doesn't enable us to run
other integration tests in other crates (e.g. the CLI)
(3) is the correct solution and has the best user experience, but it
requires nightly and I don't want to introduce that for all of our
tests.
I have chosen to do a combination of (1) and (4). You will now run the
smoketests with `cargo smoketest`. If you run `cargo test --all` (or use
`guard`) without doing `cargo smoketest` it will fall back to (4) which
compiles the executables at runtime. Running `cargo build` is the **only
way** to ensure that the executables are not stale because of the
internal fingerprint checking. Everything else is fragile not robust.
NOTE! There is no way to avoid cargo within cargo and have the smoke
tests be run as cargo tests because the modules under test must be
compiled with cargo.
# API and ABI breaking changes
Note that this is a BREAKING CHANGE to `cargo test --all`. The
smoketests are now part of `cargo test --all` unless you specifically
exclude them.
# Expected complexity level and risk
3, this is partially AI translated. We need to carefully review to
ensure the semantics have not regressed.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [ ] <!-- maybe a test you want to do -->
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
# Description of Changes
This moves a bunch of stuff from `lib` back into `server` and `sdk`, and
removes all but one global variable from the server sdk in preparation
for export-based reducer definition.
# Expected complexity level and risk
2 - a pretty big refactor, but it's mostly just code movement.
# Testing
- [x] Refactor, so automated tests are sufficient.
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
This workflow change migrates us from docker container runners to the
new system which uses virtual machines via qemu. This solves a lot of
complexities around docker-in-docker and also allows us to easily run
windows VMs in the future.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
0 - this only updates our workflow and has no impact on the code.
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] CI is passing
---------
Signed-off-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
This has been popping up in more places, so I've added this in more
places.
# API and ABI breaking changes
None. CI only.
# Expected complexity level and risk
1
# Testing
🤷
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
We have had several `global.json` files introduced that aren't symlinks
to the root `global.json`.
This PR fixes that drift, and adds a CI check.
# API and ABI breaking changes
None.
# Expected complexity level and risk
2
# Testing
- [x] `cargo ci global-json-policy` succeeds
- [x] `cargo ci global-json-policy` fails if you add a new `global.json`
file somewhere
- [x] Rest of CI passes, confirming that templates are still working
properly
---------
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
# Description of Changes
Skip the LLM check because it's causing headaches for getting PRs
merged.
# API and ABI breaking changes
CI only
# Expected complexity level and risk
1
# Testing
- [x] It didn't run on this PR.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
(jdetter): Github marked Ryan's PR as merged because I rebased it
without updating the base branch first - my bad.
Original PR: https://github.com/clockworklabs/SpacetimeDB/pull/4120
This PR fixes C#-related CI failures when testing against `v1.12.0`
before the corresponding NuGet packages are available on nuget.org.
* CI: Ensure local C# NuGet packages are used consistently
* Pack `crates/bindings-csharp/{BSATN.Runtime,Runtime}` with `-c
Release` so the configured local package sources (`bin/Release`)
actually contain the `.nupkg`s.
* Run `./sdks/csharp/tools~/write-nuget-config.sh` in CI to generate a
`NuGet.Config` that maps `SpacetimeDB.BSATN.Runtime` /
`SpacetimeDB.Runtime` to those local sources (with `nuget.org` as
fallback for non-SpacetimeDB dependencies).
* Add an explicit `dotnet restore --configfile NuGet.Config` step before
tests, and `run dotnet test --no-restore`, so restore always uses the
intended config and does not “accidentally” restore from `nuget.org`.
* `write-nuget-config.sh`: Make NuGet config discovery + mapping
deterministic
* Write `NuGet.Config` (capitalized) and include `<clear />` + an
explicit `nuget.org` source to avoid inherited sources and to keep
`PackageSourceMapping` functional.
* Also write a repo-root `NuGet.Config` so `dotnet publish/pack` invoked
from templates (outside `sdks/csharp/`) still discovers the same
override behavior.
* Smoketests quickstart: avoid `PackageSourceMapping` restore failures
* Ensure smoketest helper packing uses `Release`, and when repo-root
`NuGet.Config` exists, perform `dotnet restore --configfile ...` +
`dotnet pack --no-restore` so pack/restore are evaluated with the same
sources/mapping.
* When editing configs, prefer `NuGet.Config` casing for Linux discovery
and ensure `nuget.org` exists as a source for the fallback mapping.
# API and ABI breaking changes
No changes
# Expected complexity level and risk
1
# Testing
- [X] Locally tested concept after simulating local repro.
- [x] Confirmed CI passes in this branch
---------
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
This fixes 2 issues with the upgrade version tool:
1. The typescript bindings need to be updated otherwise the typescript
test in CI will fail
2. The snapshots need to be updated
When the version upgrade tool check in CI runs, snapshot changes are
accepted automatically via the `--accept-snapshots` cli argument. When
you are running this tool locally without `--accept-snapshots` you will
be asked to manually review the snapshot changes before doing a final
test to make sure the snapshots are correct.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
1 - this just updates the version upgrade tool
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Version bump 1.12.0 worked:
https://github.com/clockworklabs/SpacetimeDB/pull/4084
- [x] CI passes
---------
Signed-off-by: John Detter <4099508+jdetter@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
# Description of Changes
This PR renames the templates to always use shorthand for the language,
specify a framework (or console) if necessary, and shorten the naming in
general
# Expected complexity level and risk
1
# Testing
I've tested generating templates manually
---------
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
@cloutiertyler when you fix the llm benchmarks feel free to revert this
PR.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
0
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] CI runs even if llm benchmark fails
# Description of Changes
Adds TypeScript as a third language for LLM benchmark tests alongside
Rust and C#, and fixes table naming convention mismatches.
**TypeScript Support:**
- Added `Lang::TypeScript` variant with camelCase naming conventions
- Created TypeScript project template (`templates/typescript/server/`)
with package.json, tsconfig.json, and index.ts
- Added TypeScript publisher that uses `spacetime build` and `spacetime
publish`
- Created 22 TypeScript task prompts and golden answer files for all
benchmark tests
- Updated prompt discovery to find `tasks/typescript.txt` files
**Table Naming Fix:**
- Standardized on singular table names across all languages:
- Rust: `user` (snake_case singular)
- C#: `User` (PascalCase singular)
- TypeScript: `user` (camelCase singular)
- Updated `table_name()` helper to convert singular names to appropriate
case per language
- Updated all spec.rs files to use `table_name("user", lang)` instead of
hardcoded `"users"`
**CI/Hashing Improvements:**
- Added `compute_processed_context_hash()` for language-specific hash
computation after tab filtering
- Updated CI check to verify both `rustdoc_json` and `docs` modes for
Rust
- Fixed `--hash-only` mode to skip golden builds
# API and ABI breaking changes
None - these are internal benchmark tooling changes only.
# Expected complexity level and risk
**Complexity: 2**
The changes add a new language following existing patterns for Rust and
C#. The table naming fixes are straightforward find-and-replace style
updates. Low risk since this only affects the benchmark tooling, not the
core SpacetimeDB codebase.
# Testing
- [x] `cargo build -p xtask-llm-benchmark` compiles successfully
- [x] All 22 TypeScript golden modules build and publish successfully
- [x] Rust and C# benchmarks unaffected by changes
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
- This will eliminate a lot of the test flakes with the unity testsuite.
A huge amount of the test flakes here were that we were trying to
acquire more than 2 unity license seats at a time which is not allowed.
This will have the largest impact when the runners are busy.
The downside here: This will cause jobs to wait which is what we want
right now. Jobs may wait to queue for several minutes while other jobs
finish but I don't expect this to have a big impact on the total time a
PR will spend in the merge queue.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
1
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Unity CI passes even if multiple jobs are running at the same time
# Description of Changes
Major documentation overhaul focusing on tables, column types, and
indexes.
**Quickstart Guides:**
- Updated React, TypeScript, Rust, and C# quickstarts with table/reducer
examples
- Fixed CLI syntax (positional `--database` argument)
- Improved template consistency across languages
**Tables Documentation:**
- Added "Why Tables" section explaining table-oriented design philosophy
(tables as fundamental unit, system tables, data-oriented design
principles)
- Added "Physical and Logical Independence" section explaining how
subscription queries use the relational model independently of physical
storage
- Added brief sections linking to related pages (Visibility,
Constraints, Schedule Tables)
- Renamed "Scheduled Tables" to "Schedule Tables" throughout (tables
store schedules; reducers are scheduled)
**Column Types:**
- Split into dedicated page with unified type reference table
- Added "Representing Collections" section (Vec/Array vs table
tradeoffs)
- Added "Binary Data and Files" section for Vec<u8> storage patterns
- Added "Type Performance" section (smaller types, fixed-size types,
column ordering for alignment)
- Added complete example struct demonstrating all type categories
- Renamed "Structured" category to "Composite"
**Indexes:**
- Complete rewrite with textbook-style documentation
- Added "When to Use Indexes" guidance
- Documented single-column and multi-column index syntax (field-level
and table-level)
- Comprehensive range query examples with correct TypeScript `Range`
class syntax
- Explained multi-column index prefix matching semantics
- Added index-accelerated deletion examples
- Included index design guidelines
**Styling:**
- Added CSS for table border radius and row separators
- Created Check component for green checkmarks in tables
# API and ABI breaking changes
None. Documentation only.
# Expected complexity level and risk
1 - Documentation changes only, no code changes.
# Testing
- [ ] Verify docs build without errors
- [ ] Review rendered pages for formatting issues
- [ ] Confirm code examples are syntactically correct
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Signed-off-by: John Detter <4099508+jdetter@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
This also has a few minor changes to fix build errors for the
`test-app`.
# Description of Changes
Updates the typescript-test CI job to build some packages that weren't
being built before.
This also updates the root-level `pnpm generate/format/lint/build`
commands to also apply to templates.
# Expected complexity level and risk
1
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Augmented the concurrency group to not have unrelated comments cancel
the `/update-llm-benchmark` command.
# API and ABI breaking changes
CI only
# Expected complexity level and risk
1
# Testing
I think this can only be tested once merged? Comments on this PR still
seem to use the old workflow.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Make the GH username for this job correspond to an actual GH user, so we
can have it sign our CLA.
# API and ABI breaking changes
CI only
# Expected complexity level and risk
1
# Testing
I think we could test this by having clockworklabs-bot create an
external PR, if we think that's worth it. Alternatively we could merge
and test in prod.
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
We would like to move all of the templates to a central directory
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
---------
Co-authored-by: spacetimedb-bot <spacetimedb-bot@users.noreply.github.com>
# Description of Changes
Introduce a new **LLM benchmarking app** and supporting code.
* **CLI:** `llm` with subcommands `run`, `routes list`, `diff`,
`ci-check`.
* **Runner:** executes globally numbered tasks; filters by `--lang`,
`--categories`, `--tasks`, `--providers`, `--models`.
* **Providers/clients:** route layer (`provider:model`) with HTTP LLM
Vendor clients; env-driven keys/base URLs.
* **Evaluation:** deterministic scorers (hash/equality, JSON
shape/count, light schema/reducer parity) with clear failure messages.
* **Results:** stable JSON schema; single-file HTML viewer to
inspect/filter/export CSV.
* **Build & guards:** build script for compile-time setup;
* **Docs:** `DEVELOP.md` includes `cargo llm …` usage.
This PR is the initial addition of the app and its modules (runner,
config, routes, prompt/segmentation, scorers, schema/types,
defaults/constants/paths/hashing/combine, publishers, spacetime guard,
HTML stats viewer).
### How it works
1. **Pick what to run**
* Choose tasks (`--tasks 0,7,12`), or a language (`--lang rust|csharp`),
or categories (`--categories basics,schema`).
* Optionally limit vendors/models (`--providers …`, `--models …`).
2. **Resolve routes**
* Read env (API keys + base URLs) and build the active set (e.g.,
`openai:gpt-5`).
3. **Build context**
* Start Spacetime
* Publish golden answer modules
* Prepare prompts and send to LLM model
* Attempt to publish LLM module
4. **Execute calls**
* Run the selected tasks within each test against selected models and
languages.
5. **Score outputs**
* Apply deterministic scorers (hash/equality, JSON shape/count, simple
schema/reducer checks).
* Record the score and any short failure reason.
6. **Update results file**
* Write/update the single results JSON with task/route outcomes,
timings, and summaries.
# API and ABI breaking changes
None. New application and modules; no existing public APIs/ABIs altered.
# Expected complexity level and risk
**4/5.** New CLI, routing, evaluation, and artifact format.
* External model APIs may rate-limit/timeout; concurrency tunable via
`LLM_BENCH_CONCURRENCY` / `LLM_BENCH_ROUTE_CONCURRENCY`.
# Testing
I ran the full test matrix and generated results for every task against
every vendor, model, and language (rust + C#). I also tested the CI
check locally using [act](https://github.com/nektos/act).
**Please verify**
* [ ] `llm run --tasks 0,1,2` (explicit `run`)
* [ ] `llm run --lang rust --categories basics` (filters)
* [ ] `llm run --categories basics,schema` (multiple categories)
* [ ] `llm run --lang csharp` (language switch)
* [ ] `llm run --providers openai,anthropic --models "openai:gpt-5
anthropic:claude-sonnet-4-5"` (provider/model limits)
* [ ] `llm run --hash-only` (dry integrity)
* [ ] `llm run --goldens-only` (test goldens only)
* [ ] `llm run --force` (skip hash check)
* [ ] `llm ci-check`
* [ ] Stats viewer loads the JSON; filtering and CSV export work
* [ ] CI works as intended
---------
Signed-off-by: bradleyshep <148254416+bradleyshep@users.noreply.github.com>
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@aol.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: spacetimedb-bot <spacetimedb-bot@users.noreply.github.com>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
Introduce a hacky workaround to our `csharp-testsuite` to address
missing `librusty_v8.a`: manually check for that file and manually build
the package if it's missing.
# API and ABI breaking changes
CI-only change
# Expected complexity level and risk
1
# Testing
- [x] Locally tested removing the `librusty_v8.a` and then running
`cargo clean -p v8 && cargo build -p v8`, and this does seem to repair
it.
- [x] The CI has run with a cache that is "broken", but successfully
passes `csharp-testsuite`
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Disable `cache-on-failure` as we think that it's contributing to
mysterious `rusty_v8` linker issues.
I bumped the prefix key so that PRs with this change don't share caches
with PRs missing this change.
# API and ABI breaking changes
CI only.
# Expected complexity level and risk
1
# Testing
None. We'll have to see if we stop having issues once this is merged.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Make `cargo ci` work properly on Windows, in preparation for
https://github.com/clockworklabs/SpacetimeDB/pull/3702.
# API and ABI breaking changes
No. CI-only.
# Expected complexity level and risk
2. Not trivial, but not complicated.
# Testing
- [x] CI output seems to be genuinely running the tests, and it's
passing on Windows
- [x] Make a change to `crates/bindings-csharp` and see that `cargo ci
test` fails
- [x] I can manually run a minimal `cargo ci smoketests` invocation on a
Windows machine
---------
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Kasama <robertoaall@gmail.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
We were using `rust-toolchain` in some places, and `rust-toolchain-file`
in others. I think Rust released a new version, which made the
`rust-toolchain` parts break with:
```
info: downloading component 'rustfmt'
info: removing previous version of component 'rust-src'
info: rolling back changes
error: could not rename component file from '/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/test/src/term/terminfo/searcher' to '/root/.rustup/tmp/a4eo07uz83vsyfhk_dir/bk': Invalid cross-device link (os error 18)
Error: Process completed with exit code 1.
```
(Separately, this breakage is confusing.. we'll probably run into this
again when we roll forward our rust version..)
# API and ABI breaking changes
None. CI-only change.
# Expected complexity level and risk
1
# Testing
- [x] CI passes
- [x] There are no more instances of `rust-toolchain`:
```
$ grep -rIP 'rust-toolchain(?!-file)' .github/workflows
.github/CODEOWNERS:/rust-toolchain.toml @cloutiertyler
```
(on `master`, this finds the instances we changed)
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
This changes the ci runs to execute `cargo ci` instead of running
commands directly from the github workflow.
The goal here is to unify the commands under `cargo ci` so that it's
easier and more intuitive to run locally
# API and ABI breaking changes
There are no API/ABI changes.
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
Complexity: 1
It is not a complex change as it is mostly localized to the ci runs and
is easily reversible if something goes wrong. The biggest risk here is
to have future CI runs break, which can be remediated by reverting these
changes.
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] run `cargo ci` and its subcommands locally
- [x] run the github workflow against this branch to check if the CI
jobs are working properly.
---------
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Signed-off-by: Roberto Pommella Alegro <robertoaall@gmail.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
This has 2 benefits:
1. If the Unity test fails because of a license issue then we don't have
to re-run the C# tests again as part of this flow. Re-running the Unity
tests will be much faster if that's the only thing the job is doing.
2. These tests will run faster because they will now run in parallel as
separate CI jobs.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
1
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Both tests pass
---------
Signed-off-by: John Detter <4099508+jdetter@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
Sets a 2 hour timeout for the smoketests. If the smoketests are taking
more than 2 hours they likely will never finish and we should just time
them out.
# API and ABI breaking changes
None
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
1
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x]
https://github.com/clockworklabs/SpacetimeDB/actions/runs/19697510980/job/56425630374?pr=3772
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
Just moving more jobs to the custom runner.
|Test|Before|After|% Change|
|---|---|---|---|
|Docs - Build + Test|2m|1m|No change|
|Docs - Publish|2m|<2m|No change|
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
1 - This is very low risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
I ran a publish test and it worked just fine:
<https://github.com/clockworklabs/SpacetimeDB/actions/runs/19515454265/job/55865908554>
- [x] CI still passes
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Move anything running on `spacetimedb-runner` to
`spacetimedb-new-runner`
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
- [x] CI passes
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Our automated discord posts have a more explicit call to action for the
PR author.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Did a test post and it looks fine
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Stop the CLI doc check from failing on `Cargo.lock` diffs, which it was
doing before.
# API and ABI breaking changes
CI only.
# Expected complexity level and risk
1
# Testing
- [x] CI still fails elsewhere if `Cargo.lock` needs updating
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
This adds a github actions workflow for attaching artifacts to a
release. After this PR merges it can manually be invoked in the github
actions page.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
1 - This can't really break anything other than the release artifacts
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
It worked when I tested it just now. Look at the release here:
https://github.com/clockworklabs/SpacetimeDB/releases
Plus you can check the job output here:
https://github.com/clockworklabs/SpacetimeDB/actions/runs/19578784486/job/56071106846?pr=3724
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
- This has 2 benefits:
1. `ubuntu-latest` recently changed to 24.04 instead of 22.04 so that
broke the last CLI on some systems. This will fix that issue.
2. Performance - this should build much much faster.
# API and ABI breaking changes
None
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
2 - if the CLI fails to build this might be why.
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
- [x] Tested via today's release:
https://github.com/clockworklabs/SpacetimeDB/actions/runs/19550648024/job/55980960439
# Description of Changes
I thought this was covered by the `NEUTRAL` case, but apparently
`SKIPPED` is separate.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
We'll see when it merges 🤷
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
These tests have been flaky, so we're disabling them until they're
working more consistently.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
- [x] Tests are indeed skipped? 🤷
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Zeke Foppa