# Description of Changes
This adds C++ server bindings (/crate/bindings-cpp) to allow writing C++
20 modules.
- Emscripten WASM build system integration with CMake
- Macro-based code generation (SPACETIMEDB_TABLE, SPACETIMEDB_REDUCER,
etc)
- All SpacetimeDB types supported (primitives, Timestamp, Identity,
Uuid, etc)
- Product types via SPACETIMEDB_STRUCT
- Sum types via SPACETIMEDB_ENUM
- Constraints marked with FIELD* macros
# API and ABI breaking changes
None
# Expected complexity level and risk
2 - Doesn't heavily impact any other areas but is complex macro C++
structure to support a similar developer experience, did have a small
impact on init command
# Testing
- [x] modules/module-test-cpp - heavily tested every reducer
- [x] modules/benchmarks-cpp - tested through the standalone (~6x faster
than C#, ~6x slower than Rust)
- [x] modules/sdk-test-cpp
- [x] modules/sdk-test-procedure-cpp
- [x] modules/sdk-test-view-cpp
- [x] Wrote several test modules myself
- [x] Quickstart smoketest [Currently in progress]
- [ ] Write Blackholio C++ server module
---------
Signed-off-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: Ryan <r.ekhoff@clockworklabs.io>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
Same changes as https://github.com/clockworklabs/SpacetimeDB/pull/4101
but for `connection_id`.
# API and ABI breaking changes
API Breaking
# Expected complexity level and risk
1
# Testing
Pure refactor. Tests and docs have been updated, but no new tests added.
# Description of Changes
Makes the `sender` field on `ViewContext` private and exposes a
`sender()` method. Does the same for `ReducerContext` and
`ProcedureContext`.
The purpose of this change: So that the host can determine if/when a
view invokes or reads the `sender`.
Currently, because `sender` is a field, the host assumes that it is
always read. This means views must be materialized per client, even if
the view doesn't actually depend on `sender`, resulting in data
duplication.
The initial solution for this problem was `AnonymousViewContext` which
doesn't have a `sender` field. The better solution is to make `sender` a
method so that it can call into the host and record when it's actually
invoked.
Note, this patch only updates the module API, so the current
implementation does not change. `ViewContext` views are still duplicated
across clients. Changing this requires a new host syscall and for
`sender()` to invoke that syscall. This however is backwards compatible
and can be done anytime after the module APIs for the other languages
(C#, TypeScript, C++) are updated.
Also note that `ReducerContext` and `ProcedureContext` were updated
purely for consistency. There are currently no plans to track reads of
`sender` in these contexts.
# API and ABI breaking changes
Breaks the rust module api.
# Expected complexity level and risk
1
# Testing
N/A
# Description of Changes
When reading the docs, I noticed that even though TypeScript is included
in most of the examples, the text at the top of the 'Key Architecture'
section doesn't mention it as a language that can be used to write
Modules.
This change simply adds TypeScript to the list of languages that can be
used to write Modules.
This is my first ever pull request, so my apologies if this change is
considered too small.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
I've not done any testing considering the size of the changes.
Signed-off-by: Luuk Domhof <60610562+ExpLuuk@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
# Description of Changes
Same changes as https://github.com/clockworklabs/SpacetimeDB/pull/4101
but for `connection_id`.
# API and ABI breaking changes
API Breaking
# Expected complexity level and risk
1
# Testing
Pure refactor. Tests and docs have been updated, but no new tests added.
# Description of Changes
Tiny addition to the llm context to explain that maincloud publishing is
free of charge
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Ran claude code with updated context and it performed better in the
publishing step
# Description of Changes
Update ai rules general mdc file to explain deployment and maincloud
# API and ABI breaking changes
No
# Expected complexity level and risk
1
# Testing
Ran with claudecode
Implements the client-api changes for the organizations feature.
The interesting part of this PR are the `teams` smoketests.
There is one subtlety with deletion.
Note that I haven't (yet) covered the interaction between collaborators
and organization existing for the same database.
# Description of Changes
The PR implements the following updates:
- Create Svelte framework sdk/integration
- Add Svelte Quickstart Docs
- Create `svelte-ts` template
<!-- Please describe your change, mention any related tickets, and so on
here. -->
# Screenshots
- `svelte-ts` template
<img width="1511" height="863" alt="Screenshot1"
src="https://github.com/user-attachments/assets/f266dc63-228c-4bff-96f6-b780ce53167f"
/>
- Svelte Quickstart Docs
<img width="1451" height="893" alt="Screenshot2"
src="https://github.com/user-attachments/assets/3456a60b-60d7-4232-94fc-152a020f675b"
/>
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [ ] <!-- maybe a test you want to do -->
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
---------
Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
# Description of Changes
Make `Sender` a method on [Reducer|View|Procedure|Tx]Context in C#.
Equivalent changes to
https://github.com/clockworklabs/SpacetimeDB/pull/4101.
# API and ABI breaking changes
API breaking
# Expected complexity level and risk
1
# Testing
Pure refactor, no additional testing.
# Description of Changes
Makes the `sender` field on `ViewContext` private and exposes a
`sender()` method. Does the same for `ReducerContext` and
`ProcedureContext`.
The purpose of this change: So that the host can determine if/when a
view invokes or reads the `sender`.
Currently, because `sender` is a field, the host assumes that it is
always read. This means views must be materialized per client, even if
the view doesn't actually depend on `sender`, resulting in data
duplication.
The initial solution for this problem was `AnonymousViewContext` which
doesn't have a `sender` field. The better solution is to make `sender` a
method so that it can call into the host and record when it's actually
invoked.
Note, this patch only updates the module API, so the current
implementation does not change. `ViewContext` views are still duplicated
across clients. Changing this requires a new host syscall and for
`sender()` to invoke that syscall. This however is backwards compatible
and can be done anytime after the module APIs for the other languages
(C#, TypeScript, C++) are updated.
Also note that `ReducerContext` and `ProcedureContext` were updated
purely for consistency. There are currently no plans to track reads of
`sender` in these contexts.
# API and ABI breaking changes
Breaks the rust module api.
# Expected complexity level and risk
1
# Testing
N/A
# Description of Changes
Addresses https://github.com/clockworklabs/SpacetimeDB/issues/4131.
I did not thread the new option into other commands such as `spacetime
init` (they just retain the previous behavior). I think this is a
relatively niche use case, and it's easy to work around either way (by
just using `spacetime login` first) so I'd wait until someone asks us to
thread it through to other places.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
- [x] Ran `spacetime login --no-browser` and saw that it didn't open a
browser
- [x] Ran `spacetime login` and saw that it still opened a browser
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
Two documentation improvements:
1. **Reducers documentation**: Clarified that using global/static
variables in reducers is **undefined behavior**, not just "values won't
persist". Added six specific reasons why this is undefined:
- Fresh execution environments
- Module updates
- Concurrent execution
- Crash recovery
- Non-transactional updates
- Replay safety
2. **Access permissions documentation**: Replaced the "Combining Both
Techniques" example that used indexes on Option fields (which
SpacetimeDB doesn't support) with a working example that filters by a
required `department` field instead.
# API and ABI breaking changes
None. Documentation only.
# Expected complexity level and risk
1 - Documentation changes only.
# Testing
- [ ] Verify the reducers warning is clear and accurate
- [ ] Verify the access permissions example compiles and makes sense
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
Version upgrade to `v1.12.0`.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
1 - this is just a version upgrade
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
The testsuite failures are fixed by
https://github.com/clockworklabs/SpacetimeDB/pull/4120
- [x] License has been properly updated including version number and
date
- [x] CI passes
---------
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
The PR implements the following updates:
- Create Vue framework sdk
- Add Vue Quickstart Docs
- Create `vue-ts` template
# Screenshots
- `vue-ts` template
<img width="1512" height="862" alt="Screenshot"
src="https://github.com/user-attachments/assets/15c8209b-ec7f-4f4a-a5b4-5174ddd068be"
/>
- Vue Quickstart Docs
<img width="1392" height="854" alt="image"
src="https://github.com/user-attachments/assets/57650232-81fa-43d3-be5a-135aa1799f05"
/>
<!-- Please describe your change, mention any related tickets, and so on
here. -->
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [ ] <!-- maybe a test you want to do -->
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
---------
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
Try fixing this again? It seems to pass on PRs if re-run.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
- [x] It passes on this PR now 🤷
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
the TS reference was slightly missing. you need assign a variable to the
reducer to call it cross context
# Description of Changes
slight reference fix
Signed-off-by: kormann <49917710+DKormann@users.noreply.github.com>
# Description of Changes
This PR renames the templates to always use shorthand for the language,
specify a framework (or console) if necessary, and shorten the naming in
general
# Expected complexity level and risk
1
# Testing
I've tested generating templates manually
---------
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
Adds TypeScript as a third language for LLM benchmark tests alongside
Rust and C#, and fixes table naming convention mismatches.
**TypeScript Support:**
- Added `Lang::TypeScript` variant with camelCase naming conventions
- Created TypeScript project template (`templates/typescript/server/`)
with package.json, tsconfig.json, and index.ts
- Added TypeScript publisher that uses `spacetime build` and `spacetime
publish`
- Created 22 TypeScript task prompts and golden answer files for all
benchmark tests
- Updated prompt discovery to find `tasks/typescript.txt` files
**Table Naming Fix:**
- Standardized on singular table names across all languages:
- Rust: `user` (snake_case singular)
- C#: `User` (PascalCase singular)
- TypeScript: `user` (camelCase singular)
- Updated `table_name()` helper to convert singular names to appropriate
case per language
- Updated all spec.rs files to use `table_name("user", lang)` instead of
hardcoded `"users"`
**CI/Hashing Improvements:**
- Added `compute_processed_context_hash()` for language-specific hash
computation after tab filtering
- Updated CI check to verify both `rustdoc_json` and `docs` modes for
Rust
- Fixed `--hash-only` mode to skip golden builds
# API and ABI breaking changes
None - these are internal benchmark tooling changes only.
# Expected complexity level and risk
**Complexity: 2**
The changes add a new language following existing patterns for Rust and
C#. The table naming fixes are straightforward find-and-replace style
updates. Low risk since this only affects the benchmark tooling, not the
core SpacetimeDB codebase.
# Testing
- [x] `cargo build -p xtask-llm-benchmark` compiles successfully
- [x] All 22 TypeScript golden modules build and publish successfully
- [x] Rust and C# benchmarks unaffected by changes
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
Major documentation overhaul focusing on tables, column types, and
indexes.
**Quickstart Guides:**
- Updated React, TypeScript, Rust, and C# quickstarts with table/reducer
examples
- Fixed CLI syntax (positional `--database` argument)
- Improved template consistency across languages
**Tables Documentation:**
- Added "Why Tables" section explaining table-oriented design philosophy
(tables as fundamental unit, system tables, data-oriented design
principles)
- Added "Physical and Logical Independence" section explaining how
subscription queries use the relational model independently of physical
storage
- Added brief sections linking to related pages (Visibility,
Constraints, Schedule Tables)
- Renamed "Scheduled Tables" to "Schedule Tables" throughout (tables
store schedules; reducers are scheduled)
**Column Types:**
- Split into dedicated page with unified type reference table
- Added "Representing Collections" section (Vec/Array vs table
tradeoffs)
- Added "Binary Data and Files" section for Vec<u8> storage patterns
- Added "Type Performance" section (smaller types, fixed-size types,
column ordering for alignment)
- Added complete example struct demonstrating all type categories
- Renamed "Structured" category to "Composite"
**Indexes:**
- Complete rewrite with textbook-style documentation
- Added "When to Use Indexes" guidance
- Documented single-column and multi-column index syntax (field-level
and table-level)
- Comprehensive range query examples with correct TypeScript `Range`
class syntax
- Explained multi-column index prefix matching semantics
- Added index-accelerated deletion examples
- Included index design guidelines
**Styling:**
- Added CSS for table border radius and row separators
- Created Check component for green checkmarks in tables
# API and ABI breaking changes
None. Documentation only.
# Expected complexity level and risk
1 - Documentation changes only, no code changes.
# Testing
- [ ] Verify docs build without errors
- [ ] Review rendered pages for formatting issues
- [ ] Confirm code examples are syntactically correct
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Signed-off-by: John Detter <4099508+jdetter@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
I believe that local users do not have API tokens for OpenAI, so the
existing hint was not helpful. Apparently the correct path is to post
`/update-llm-benchmark` on the PR and let the CI take care of it.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
None
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
Fix LLM benachmarks in rust since they used Result instead of ResultRow
in the request to the LLM making it always fail.
1. see the answer's file there it is ResultRow.
2. Result is a keyword so it will always fail
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
1
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
Run /update-llm-benchmark to see if more passes.
---------
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Update the path of dirName to match the correct path.
# API and ABI breaking changes
None.
# Expected complexity level and risk
0.0001
# Testing
Tested locally by running Docusaurus
# Description of Changes
- Add an "Authentication" section under "Core concepts"
- Add an introduction about how and explanations about common topics
- Move claims usage out of SpacetimeAuth docs
- Update SpacetimeAuth docs to the new version of the dashboard
# API and ABI breaking changes
None.
# Expected complexity level and risk
2
# Testing
Tested locally using Docusaurus
---------
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
- Made spacetime dev <database> a positional argument and deprecated
--database
- Fixed double connection in React SDK
- Added a more descriptive error message to unresolved table name.
# API and ABI breaking changes
Deprecates `--database`. Still works, but it prints with a warning.
# Expected complexity level and risk
2
# Testing
- [x] I have tested that the double render fix works in React
---------
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
This PR does several small things:
1. It removes the explicit `h1` tags on every page, and either uses the
side bar title directly, or puts it in the frontmatter
2. It merges what are currently called quickstarts into a single Chat
App Tutorial
3. It creates new quickstarts which just use `spacetime dev --template`
to get you up and running quickly
4. It adds a "The Zen of SpacetimeDB" page much like the Zen of Python
which goes over the 5 key principles of SpacetimeDB
5. It reorders all Tabs groups so that the ordering is `TypeScript`,
`C#`, `Rust`, `Unreal`, `C++`, `Blueprints` (order of decreasing
popularity).
6. It improves the sidebar navigation by having categories act as
overview pages, and also fixes the breadcrumbs
7. It fixes various small typos and issues
8. Closes#3610 and adds cursor rules files generally
9. It fixes general styling on the docs page by bring it inline with the
UI design:
Old:
<img width="1678" height="958" alt="image"
src="https://github.com/user-attachments/assets/f36efee6-b81a-4463-a179-da68b3a7152e"
/>
New:
<img width="1678" height="957" alt="image"
src="https://github.com/user-attachments/assets/f430f77d-0663-47f2-9727-45cbfe10e4c7"
/>
https://github.com/user-attachments/assets/adc5a78a-ada8-45b5-8078-a45cb81477a3
# API and ABI breaking changes
This PR does NOT change any old links. It does add new pages though.
# Expected complexity level and risk
3 - it's a large change. I manually tested the TypeScript Chat App
Tutorial but I have not gone through the Rust and C# quickstarts.
However, we have testing on the quickstarts and this is text only so can
be carefully reviewed.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Ran through each step of the Chat App TypeScript tutorial to
ensure it is working
- [x] Ran and tested the styles and the functionality of the side bar
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: spacetimedb-bot <spacetimedb-bot@users.noreply.github.com>
Co-authored-by: clockworklabs-bot <clockworklabs-bot@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
Updated the link to target SpacetimeDB repository instead of archived
blackholio repository.
Update the repository structure in the README.md to match the current
one (also sort it alphabetically).
# API and ABI breaking changes
NA
# Expected complexity level and risk
1
# Testing
Visit https://spacetimedb.com/docs/unreal/. this link is already used a
bit below for the text "Blackhol.io"
---------
Signed-off-by: homersimpsons <guillaume.alabre@gmail.com>
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
The syntax is not supported like shows and must be included in the
`table` macro.
<!-- Please describe your change, mention any related tickets, and so on
here. -->
# API and ABI breaking changes
None.
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
0 No runtime change
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Code compiles now.
---------
Signed-off-by: Kilian Strunz <93079615+kistz@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
We would like to move all of the templates to a central directory
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
---------
Co-authored-by: spacetimedb-bot <spacetimedb-bot@users.noreply.github.com>
# Description of Changes
Introduce a new **LLM benchmarking app** and supporting code.
* **CLI:** `llm` with subcommands `run`, `routes list`, `diff`,
`ci-check`.
* **Runner:** executes globally numbered tasks; filters by `--lang`,
`--categories`, `--tasks`, `--providers`, `--models`.
* **Providers/clients:** route layer (`provider:model`) with HTTP LLM
Vendor clients; env-driven keys/base URLs.
* **Evaluation:** deterministic scorers (hash/equality, JSON
shape/count, light schema/reducer parity) with clear failure messages.
* **Results:** stable JSON schema; single-file HTML viewer to
inspect/filter/export CSV.
* **Build & guards:** build script for compile-time setup;
* **Docs:** `DEVELOP.md` includes `cargo llm …` usage.
This PR is the initial addition of the app and its modules (runner,
config, routes, prompt/segmentation, scorers, schema/types,
defaults/constants/paths/hashing/combine, publishers, spacetime guard,
HTML stats viewer).
### How it works
1. **Pick what to run**
* Choose tasks (`--tasks 0,7,12`), or a language (`--lang rust|csharp`),
or categories (`--categories basics,schema`).
* Optionally limit vendors/models (`--providers …`, `--models …`).
2. **Resolve routes**
* Read env (API keys + base URLs) and build the active set (e.g.,
`openai:gpt-5`).
3. **Build context**
* Start Spacetime
* Publish golden answer modules
* Prepare prompts and send to LLM model
* Attempt to publish LLM module
4. **Execute calls**
* Run the selected tasks within each test against selected models and
languages.
5. **Score outputs**
* Apply deterministic scorers (hash/equality, JSON shape/count, simple
schema/reducer checks).
* Record the score and any short failure reason.
6. **Update results file**
* Write/update the single results JSON with task/route outcomes,
timings, and summaries.
# API and ABI breaking changes
None. New application and modules; no existing public APIs/ABIs altered.
# Expected complexity level and risk
**4/5.** New CLI, routing, evaluation, and artifact format.
* External model APIs may rate-limit/timeout; concurrency tunable via
`LLM_BENCH_CONCURRENCY` / `LLM_BENCH_ROUTE_CONCURRENCY`.
# Testing
I ran the full test matrix and generated results for every task against
every vendor, model, and language (rust + C#). I also tested the CI
check locally using [act](https://github.com/nektos/act).
**Please verify**
* [ ] `llm run --tasks 0,1,2` (explicit `run`)
* [ ] `llm run --lang rust --categories basics` (filters)
* [ ] `llm run --categories basics,schema` (multiple categories)
* [ ] `llm run --lang csharp` (language switch)
* [ ] `llm run --providers openai,anthropic --models "openai:gpt-5
anthropic:claude-sonnet-4-5"` (provider/model limits)
* [ ] `llm run --hash-only` (dry integrity)
* [ ] `llm run --goldens-only` (test goldens only)
* [ ] `llm run --force` (skip hash check)
* [ ] `llm ci-check`
* [ ] Stats viewer loads the JSON; filtering and CSV export work
* [ ] CI works as intended
---------
Signed-off-by: bradleyshep <148254416+bradleyshep@users.noreply.github.com>
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@aol.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: spacetimedb-bot <spacetimedb-bot@users.noreply.github.com>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
Closes#3659
# API and ABI breaking changes
Remove route and alter the semantics of the `call` route on both server
and `cli`
# Expected complexity level and risk
1
# Testing
- [x] Publish module with `procedures` and observe calling the `cli` the
result is print.
---------
Co-authored-by: Phoebe Goldman <phoebe@goldman-tribe.org>
# Description of Changes
Closes: #3895
- Added third tier expanding folder
- Merged most into intro + core concepts
- Collapsed modules + database into one
- Moved several documents to how-to to streamline the core concepts
- Added warning to RLS to use views
- refactored all links
- Moved How To and References into Developer Resources
- Fixed TypeScript View code
Sample: (Getting Started will start expanded due to being the initial
page)
<img width="293" height="746" alt="image"
src="https://github.com/user-attachments/assets/6abc3a0d-1ae2-4886-bda7-11f5565afecf"
/>
# API and ABI breaking changes
N/A
# Expected complexity level and risk
1
# Testing
- [X] Updated the quickstarts.py and generate-cli tooling
---------
Signed-off-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
# Description of Changes
With the refactor docs I had left the original module folder and
mistakenly didn't remove it as requested as per Tyler's comment:
https://github.com/clockworklabs/SpacetimeDB/pull/3877#pullrequestreview-3585291570:
> Also please remove the "old files" before merging.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
1
# Testing
Removal of documents there is no test to run
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
# Description of Changes
Add back the instructions for regenerating CLI docs, which were removed
in https://github.com/clockworklabs/SpacetimeDB/pull/3343. I also made a
script for it.
This also fixes the CI checking this file, which was silently broken in
the same PR.
I have **not** verified that this works in Git Bash in Windows.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
- [x] CI passes
- [x] CI fails if I change the CLI reference
- [x] CLI reference looks visually reasonable on a local `pnpm dev`
---------
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
The `.png` logo image has been replaced by a `.svg` and needed to be
updated.
# API and ABI breaking changes
Not applicable
# Expected complexity level and risk
0.000001
# Testing
Run `pnpm dev` and check that the logo is correctly displayed in the top
left corner
# Description of Changes
This makes it possible to insert new pages between existing pages
without renaming the existing pages.
I also added a section to docs/README.md with procedure for how to name
new pages, which I'll paste here:
### Adding new pages
All of our directory and file names are prefixed with a five-digit
number which determines how they're sorted.
We started with the hundreds place as the smallest significant digit, to
allow using the tens and ones places to add new pages between.
When adding a new page in between two existing pages, choose a number
which:
- Doesn't use any more significant figures than it needs to.
- Is approximately halfway between the previous and next page.
For example, if you want to add a new page between `00300-foo` and
`00400-bar`, name it `00350-baz`. To add a new page between `00350-baz`
and `00400-bar`, prefer `00370-quux` or `00380-quux`, rather than
`00375-quux`, to avoid populating the ones place.
To add a new page after all previous pages, use the smallest multiple of
100 larger than all other pages. For example, if the highest-numbered
existing page is `01350-abc`, create `01400-def`.
# API and ABI breaking changes
N/a
# Expected complexity level and risk
1 - @JulienLavocat said this wouldn't break anything.
# Testing
None.
---------
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
Shubham Mishra