mirror of
https://github.com/clockworklabs/SpacetimeDB.git
synced 2026-06-27 16:30:35 -04:00
bot/docs-audit
74 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 clockwork-labs-bot
|
84cfe5a920 |
Fix TypeScript table handle casing (#5286)
## Compatibility note This PR changes the generated TypeScript table/view handles for snake_case module accessors to the target-language TypeScript accessor casing, but keeps the old snake_case handles as deprecated aliases. Generated TS clients now expose the intended TypeScript accessor casing: - `ctx.db.databaseTree` - `tables.loggedOutPlayer` - `tables.myProgress` The old snake_case handles continue to work and are marked deprecated in the generated type surface: - `ctx.db.database_tree` still works as an alias for `ctx.db.databaseTree` - `tables.logged_out_player` still works as an alias for `tables.loggedOutPlayer` - `tables.my_progress` still works as an alias for `tables.myProgress` The database canonical names stay unchanged. For example, the generated TypeScript handle is camelCase, but the table metadata still has `name: 'logged_out_player'`. ## Remaining API breakage risk This should be non-breaking for normal generated TypeScript client usage, including: - direct table access through `conn.db.snake_case` - callback contexts like `ctx.db.snake_case` - exported query builders like `tables.snake_case` - type inference for deprecated aliases There are still a few edge cases where users may notice an API shape change: - Code that enumerates generated table handles with `Object.keys`, `Object.entries`, `for...in`, or similar will now see both the target-language handle and the deprecated snake_case alias. - Code with pathological table/view accessor collisions may not get every possible alias. The normal case is `logged_out_player` -> generated handle `loggedOutPlayer` plus deprecated alias `logged_out_player`. Pathological cases are shapes like: - `foo_bar` and `fooBar`: both want the generated TypeScript handle `fooBar`, so generated clients cannot provide two distinct `tables.fooBar` entries. - `foo_bar` and some other table/view whose target-language handle is already `foo_bar`: the deprecated `foo_bar` alias for `foo_bar` -> `fooBar` would shadow the other generated handle, so the generator skips that alias. - Code that compares the exact generated `index.ts` text, emitted declaration text, or public type names will see new helper/types such as `DbView`, `Tables`, and the alias metadata. TypeScript modules themselves are not expected to break from this unless they consume generated TypeScript client bindings or depend on exact generated client object keys. ## Terminology The casing proposal uses **canonical name** for the database/internal name. That is language-independent. It uses **accessor name** for the source/module/client-facing identifier that codegen derives language-specific handles from. This PR keeps database canonical names unchanged. It changes the generated TypeScript accessor handles to match TypeScript casing while retaining the old generated handles as deprecated aliases. ## Why The TypeScript code generator was using `table.accessor_name` and `view.accessor_name` directly as object keys in `tablesSchema`. That preserved the raw module accessor spelling instead of applying the target-language `Case::Camel` conversion for TypeScript handles. Per the casing policy, client codegen should use the server accessor name as its source and apply the target-language conversion. ## What changed - Convert generated TypeScript table and view handle keys with `Case::Camel`. - Generate deprecated snake_case aliases for table/view handles when the old generated handle differs from the target-language TypeScript handle. - Keep runtime table metadata and database canonical names unchanged. - Avoid duplicate runtime table definitions. - Add a type-only aliased schema so callback contexts infer deprecated aliases too. - Add regression coverage for TypeScript-cased handles and deprecated aliases. - Update checked-in generated TS bindings and references that change under this fix. Generated code in this repo that changes as a result: - `crates/bindings-typescript/test-app/src/module_bindings/index.ts` - `crates/bindings-typescript/test-react-router-app/src/module_bindings/index.ts` - `crates/bindings-typescript/test-solid-router/src/module_bindings/index.ts` - `crates/bindings-typescript/case-conversion-test-client/src/module_bindings/index.ts` - `demo/Blackholio/client-ts/src/module_bindings/index.ts` - `templates/hangman-react-ts/src/module_bindings/index.ts` - `templates/money-exchange-react-ts/src/module_bindings/index.ts` - `crates/codegen/tests/snapshots/codegen__codegen_typescript.snap` I also updated the corresponding client/test references to the generated TypeScript-cased handles. ## Verification - `cargo fmt --all --check` - `cargo test -p spacetimedb-codegen typescript` - `pnpm --dir crates/bindings-typescript test -- tests/client_query.test.ts tests/db_connection.test.ts` - `pnpm --dir crates/bindings-typescript exec vitest run --typecheck.enabled tests/client_query.test.ts tests/db_connection.test.ts` - `git diff --check` The explicit Vitest typecheck command still reports existing global typecheck errors from unrelated files loaded by the test project, including missing `spacetimesys@2.x` ambient modules and existing test type issues, but the touched alias tests report `Type Errors no errors` and the earlier generated-code `declare override` issue is gone. --------- Signed-off-by: Ryan <r.ekhoff@clockworklabs.io> Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com> Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com> Co-authored-by: Ryan <r.ekhoff@clockworklabs.io> |
||
|
Jason Larabie
|
95e61f415c |
Unreal SDK query builder (#4810)
# Description of Changes - Added a client-side query builder for the Unreal SDK - Added typed query-builder subscriptions for Unreal C++ via SubscriptionBuilder.AddQuery(...).Subscribe() - Added generated Blueprint query-builder support with source query nodes, column nodes, predicates, Where, AddQuery, and Subscribe - Added Blueprint autocasts at the AddQuery boundary so source-specific Blueprint queries can connect cleanly into the generic subscription builder for better devex in Blueprint - Synced the Unreal query-builder core with the C++ module implementation - Updated the copied Unreal core query-builder headers to match the shared C++ - Moved Unreal-specific literal/type adapters out of the shared core copy and into Unreal-specific expansion code - Added Unreal SDK test coverage and documentation for the new query-builder surface - Added new test harnesses to start to match the View/ViewPk tests - Added documentation for the Unreal query builder - Refactor generation for modules with more than 255 reducers, required to get the TestClient back in working order # API and ABI breaking changes - No intended API or ABI breaking changes to released Unreal SDK behavior - Adds a new public client-side query-builder API to the Unreal SDK for C++ and Blueprint - Raw SQL subscriptions remain available # Expected complexity level and risk 3 - Adds new query-builder surface with large changes to the code-gen, the risk here is keeping the C++ module core mirrored to the SDK in the future # Testing What I've done so far: - [x] Updated and ran the following tests: - [x] TestClientEditor - [x] TestViewClientEditor - [x] TestViewPkClientEditor - [x] Ran the Unreal harness suites covering the query-builder client paths - [x] Verified the generated Unreal query-builder surface in both C++ and Blueprint testing supported fields with a small sample project Should be done at least once by a reviewer: - [ ] Re-run the full `sdk-unreal-test-harness` as it's disabled in CI - [ ] Play with the C++ and Blueprint versions in a small Unreal project --------- Signed-off-by: Jason Larabie <jason@clockworklabs.io> Co-authored-by: Ryan <r.ekhoff@clockworklabs.io> Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com> |
||
|
Phoebe Goldman
|
5c04860649 |
Implement HTTP handlers / webhooks in Rust modules (#4636)
# Description of Changes
Adds support to Rust modules and the SpacetimeDB host for defining HTTP
handlers and registering them to routes.
## User-facing API
In a Rust module, users can annotate functions with the new macro
`#[spacetimedb::http::handler]`. A function annotated this way must
accept exactly two arguments, of types `&mut
spacetimedb::http::HandlerContext` and `spacetimedb::http::Request`
(which is a type alias for `http::Request<spacetimedb::http::Body>`. It
must also return `spacetimedb::http::Response` (which is a type alias
for `http::Response<spacetimedb::http::Body>`).
Once the user has defined an HTTP handler, they can register it to a
route by annotating a function with `#[spacetimedb::http::router]`. Such
a function must take no arguments and return a
`spacetimedb::http::Router`. (The original design put this annotation on
a `static` variable rather than a function, but that turned out to be
undesirable because it required that constructing a `Router` be
`const`.) `Router` exposes various methods for registering handlers to
routes.
All of a database's user-defined routes are exposed under
`/v1/database/:name_or_identity/route/{*path}`.
## Example
See [the new
smoketest](https://github.com/clockworklabs/SpacetimeDB/blob/phoebe/http-handlers-webhooks/crates/smoketests/tests/smoketests/http_routes.rs)
for a more exhaustive example.
A simpler example, which stores arbitrary byte data in a table via a
`POST` request, returns an ID, and then retrieves that same data via a
`GET` request with a query parameter:
```rust
#[spacetimedb::table(accessor = data)]
struct Data {
#[primary_key]
#[auto_inc]
id: u64,
body: Vec<u8>,
}
#[spacetimedb::http::handler]
fn insert(ctx: &mut HandlerContext, request: Request) -> Response {
let body: Vec<u8> = request.into_body().into_bytes().into();
let id = ctx.with_tx(|tx| tx.db.data().insert(Data { id: 0, body: body.clone() }).id);
Response::new(Body::from_bytes(format!("{id}")))
}
#[spacetimedb::http::handler]
fn retrieve(ctx: &mut HandlerContext, request: Request) -> Response {
let id = request
.uri()
.query()
.and_then(|query| query.strip_prefix("id="))
.and_then(|id| u64::from_str(id).ok())
.unwrap();
let body = ctx.with_tx(|tx| tx.db.data().id().find(id).map(|data| data.body));
if let Some(body) = body {
Response::new(Body::from_bytes(body))
} else {
Response::builder().status(404).body(Body::empty()).unwrap()
}
}
#[spacetimedb::http::router]
fn router() -> Router {
Router::new().post("/insert", insert).get("/retrieve", retrieve)
}
```
## Design and implementation notes
- As mentioned above, the router is registered via a function, not a
`static` or `const` item. This is because `static` or `const`
initializers must be `const`, and it turns out to be a pain to make all
of the `Router` constructors be `const fn`s.
- The `#[handler]` macro clobbers the original function name with a
`const` variable of type `HttpHandler`. This is unfortunate, but AFAICT
necessary, 'cause we need to pass the string identifier for the handler
to the `Router`, not the function pointer, and Rust allows no (stable
and reliable) way to get a unique string identifier out of a function
item/value, nor to attach data or implement traits for function
items/values. The alternative(s) would involve changing the signature of
the `Router` methods to have uglier and more complex callsites, e.g.
like `.get("/retrieve", retrieve::handler())`, `.get("/retrieve",
handler!(retrieve))` or `.get::<retrieve>("/retrieve")`. I believe that
registering handlers will be much more common than calling their
functions, so I've chosen to make it so that registering them gets the
convenient syntax, even though the inability to call them directly will
be somewhat surprising.
- I haven't wired up energy handling or timing metrics for handler
execution to anywhere. Procedures are still in the same boat.
- HTTP requests to user-defined routes bypass the usual SpacetimeDB auth
middleware, meaning that the host does not validate (or inspect in any
way) `Authorization` headers in requests before invoking the
user-defined handler. This is required to allow arbitrary
user-programmable handling of `Authorization` headers, including those
in formations which SpacetimeDB would reject. As a result of this,
`HandlerContext` doesn't expose a `sender` or `sender_connection_id`.
- HTTP route paths may consist only of a very restrictive set of
characters. I've chosen this set to keep our options open in the future
to add additional syntax to routes, like for registering wildcard
segments and path parameters:
- ASCII digits.
- ASCII letters.
- `-_~/`.
- The internal data structure that represents a `Router` is currently a
`Vec<Route>`, meaning that resolving a request to a route is
`O(num_routes)`. Registering a route checks against each previous route
for uniqueness, meaning that constructing a router is `O(num_routes ^
2)`. There are TODO comments to use a trie, but I think this can wait,
as I expect most databases to register few routes.
- Commit
|
||
|
DexterKoelson
|
59ac77970f |
Add useProcedure React hook and export procedures from codegen (#4752)
Closes #4751 # Description of Changes - Add `useProcedure` React hook that mirrors `useReducer` which returns a stable, typed callback that queues calls until the connection is ready - Add `ProcedureParamsType` and `ProcedureReturnType` utility types to `type_utils.ts` - Update TypeScript codegen to emit `export const procedures = __convertToAccessorMap(proceduresSchema.procedures)` in generated module bindings, matching the existing pattern for reducers ```ts import { procedures } from './module_bindings'; import { useProcedure } from 'spacetimedb/react'; const doSomeThing = useProcedure(procedures.doSomeThing); const result = await doSomeThing({ foo: "..." }); ``` # API and ABI breaking changes None. Additive only — new hook export and new codegen line. # Expected complexity level and risk Low. The hook is a near-copy of `useReducer` adapted for procedure signatures. The codegen change adds one line following the identical pattern used for reducers. # Testing - [x] TypeScript SDK compiles with no new errors (`tsc --noEmit`) - [x] All 170 existing tests pass (17 test files) - [x] Tested end-to-end in a React app calling a procedure --------- Co-authored-by: Jason Larabie <jason@clockworklabs.io> |
||
|
Noa
|
fbec2761ab |
[TS] Fix access before initialization in codegen (#4709)
# Description of Changes Fixes #4584. Now we check, fully recursively, whether compound `AlgebraicTypeUse`s contain a ref. # Expected complexity level and risk 1 # Testing - [x] Added a test to ensure correct codegen --------- Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com> |
||
|
Zeke Foppa
|
10a4779b13 |
wasm support for Rust SDK (#4183)
# Description of Changes This PR adapts the Rust SDK test suite to work with the wasm version added in https://github.com/clockworklabs/SpacetimeDB/pull/4089 (which I've closed in favor of this PR). Most of the changes revolve around wasm's different async semantics - everything runs in one thread, so things that relied on background threads didn't work directly. Several tests would lock up because something in them blocked synchronously, which blocked any background work from progressing. We moved the test-clients contents into a `test_handlers.rs` so that it could be called from both `main` (for native tests) and `lib` (for wasm tests). To show what actually changed, use: ```bash git diff --no-index -- <(git show origin/master:sdks/rust/tests/procedure-client/src/main.rs) sdks/rust/tests/procedure-client/src/test_handlers.rs ``` (or similar for other test-clients) # API and ABI breaking changes None, I think/hope. # Expected complexity level and risk 2 # Testing - [x] I've augmented the CI to also run the test suite with the `web` feature --------- Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com> Co-authored-by: Thales R <thlsrmsdev@gmail.com> Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com> |
||
|
Jason Larabie
|
f8d6d76ee4 |
Update Unreal SDK to websocket 2.0 (#4497)
# Description of Changes - Updated the Unreal SDK and generated Unreal bindings for the websocket 2.0 protocol/model - Reworked DbConnectionBase to handle the updated message shapes - Switched subscription handling over to new message types and QuerySetId - Updated reducer to ReducerResult, removal of callbacks, and set reducer flags - Added event table support - Baked in multi-module support replacing [the old PR](<https://github.com/clockworklabs/SpacetimeDB/pull/3417>) - Added functionality to generate module support for multiple folders in the Unreal project (add <module>.Build.cs, <module>.h, <module>.cpp) using the --module-name - Add new configuration option for spacetime generate to handle module prefix - Regenerated Unreal Blackholio/TestClient/QuickstartChat bindings - Rebuilt Unreal Blackholio's consume entity to use event tables - Updated migration documentation - Updated the version bump tool to impact C++ # API and ABI breaking changes - Unreal websocket/message handling updated to the new protocol - Unreal generation now expects a real .uproject target and will stop immediately if project metadata is invalid instead of continuing past setup issues. # Expected complexity level and risk 3 - A large set of changes to update the websocket/message handling along with heavy codegen changes to handle multi-module support # Testing Test coverage of the Unreal SDK will need expansion in a future ticket once our issues with flakiness on CI is resolved. - [x] Updated Unreal Blackholio - [x] Ran full Unreal SDK test suite - [x] Built new test project using the new `--module-prefix` - [x] Run through Unreal Blackholio (C++ and Blueprint) - [x] Rebuilt Unreal Blackholio with multi-module, and duplicate generated module testing side-by-side modules that would overlap # Review Question(s) - [x] Updates to `spacetime init` have made the tutorial a little confusing with pathing for the Unreal Blackholio tutorial. To fix though we'd have to update all the commands to be more explicit, or update the tutorial `spacetime init` to use `--project-path .` to keep pathing simpler, thoughts? --------- Signed-off-by: Jason Larabie <jason@clockworklabs.io> Co-authored-by: Ryan <r.ekhoff@clockworklabs.io> |
||
|
joshua-spacetime
|
a39b81b087 |
[C#] Primary keys for query builder views (#4626)
# Description of Changes Same change set as https://github.com/clockworklabs/SpacetimeDB/pull/4614, just targeting master. The return type of a query builder view is now a special SATS product type `{ __query__: T }`. A view with this return type now has a primary key if `T` has a primary key. This means that client codegen will generate an `OnUpdate` callback for such views. It will also generate query builder index bindings for the primary key column. # API and ABI breaking changes None. Old modules with query builder views still work, they just don't have primary keys. # Expected complexity level and risk 2 # Testing Added equivalent tests to the ones that were added in #4573 and #4572. |
||
|
Phoebe Goldman
|
6882232108 |
Add a mode to the Rust SDK with additional logging to a file (#4566)
# Description of Changes
In the Rust client SDK, with this PR, doing
`.with_debug_to_file("path.txt")` on the connection builder will result
in the SDK logging additional verbose info to `path.txt`.
# API and ABI breaking changes
Adds a new user-facing-ish API to the Rust client SDK.
# Expected complexity level and risk
2? Some possibility of deadlock due to adding a new mutex, but this is
explicitly not for production use.
# Testing
- [x] Ran `chat-console-rs` locally with this enabled and got some debug
logs out of it.
|
||
|
joshua-spacetime
|
6eaf06b6b5 |
[Rust] Primary keys for query builder views (#4572)
# Description of Changes Query builder views return a subset of rows from a physical table. If that table has a primary key, then so should the view. What this means concretely is that the view should expose the same api as the table, specifically as it relates to the primary key column. With that in mind, this patch commits the following changes: 1. Annotates `ViewDef` with a `primary_key` 2. Updates the return type of query builder views in the raw module def to a special product type 3. Adds an index for the primary key on the view's backing table 4. Updates the query planner to use this index 5. Updates rust client codegen to generate `on_update` for such views # API and ABI breaking changes None Old `impl Query` views compiled with an older version of SpacetimeDB will continue to work as they did before - without a primary key. # Expected complexity level and risk 3 # Testing - [x] New rust sdk integration suite exercising `on_update` for PK views and semijoin scenarios - [x] Smoketests for PK views and semijoin scenarios |
||
|
joshua-spacetime
|
c5743cfc8d |
Add implicit query builder conversions from bool to BoolExpr (#4547)
# Description of Changes Adds implicit query builder conversions from `bool` to `BoolExpr` so that you can write: ```rust ctx.from.user().r#where(|u| u.online) ``` instead of ```rust ctx.from.user().r#where(|u| u.online.eq(true)) ``` Also removes `NullableCol` and `NullableIxCol` types from C# query builder. # API and ABI breaking changes None # Expected complexity level and risk 1 # Testing Unit and smoketests |
||
|
joshua-spacetime
|
33f3ea18e2 |
Make accessor required for table-level index defs in typescript (#4525)
# Description of Changes
This patch contains two main changes:
1. It makes `accessor` a required argument for table-level index defs in
typescript to align with rust.
2. It removes unsound index typing in the TypeScript bindings by
splitting index data into two explicit representations:
- `indexes`: declarative user config (`IndexOpts`) used for type
inference
- `resolvedIndexes`: normalized runtime metadata (`UntypedIndex`)
derived from `RawTableDefV10`
`TableCacheImpl` now builds index accessors from `resolvedIndexes`
instead of re-casting `indexes` at runtime.
This addressed the following comment:
```
// TODO: horrible horrible horrible. we smuggle this
`Array<UntypedIndex>`
// by casting it to an `Array<IndexOpts>` as `TableToSchema` expects.
// This is then used in `TableCacheImpl.constructor` and who knows where
else.
// We should stop lying about our types.
```
> Why?
We were conflating two different concepts under `tableDef.indexes`:
1. declared table-level index options authored by users
2. resolved runtime index definitions (including field-level inferred
indexes)
This required unsafe casts (`T['idxs']` / `UntypedIndex`) and made the
type model unsound.
Note, (2) was largely ai assisted.
# API and ABI breaking changes
Technically breaks the module api, although I believe this is the
behavior that is outlined in the spec, and so the current behavior
should really be considered a bug.
# Expected complexity level and risk
2
# Testing
Added unit tests covering the following:
1. Table-level explicit index without accessor throws.
2. Table-level duplicate accessor throws.
3. Table-level explicit accessor is accepted and used.
4. Field-level `.index(...)` derives accessor from the field name
(`displayName`).
|
||
|
Noa
|
e3582131fe |
Migrate to Rust 2024 (#3802)
# Description of Changes It'd be best to review this commit-by-commit, and using [difftastic](https://difftastic.wilfred.me.uk) to easily tell when changes are minor in terms of syntax but a line based diff doesn't show that. # Expected complexity level and risk 3 - edition2024 does bring changes to drop order, which could cause issues with locks, but I looked through [all of the warnings that weren't fixed automatically](https://gistcdn.githack.com/coolreader18/80485ae5c5f82de1784229cce2febb26/raw/ba80f3fecda66ceb34f4f7ad73b98ea02d4893a2/warnings.html) and couldn't find any issues. # Testing n/a; internal code change |
||
|
Jason Larabie
|
14f79910ee |
Update C++ module bindings to RawModuleDefV10 (#4461)
# Description of Changes
- Migrated the C++ module-definition assembly path to V10-first
internals:
- Added v10_builder and module_type_registration systems.
- Switched Module::__describe_module__ to serialize RawModuleDef with
V10 payload.
- Updated macro registration pipeline to register through V10
- Added explicit naming support across macro surface (*_NAMED variants
for reducer/procedure/
view and field/index macros).
- Reworked multi-column index macros (FIELD_MultiColumnIndex,
FIELD_MultiColumnIndex_NAMED) with
migration alias.
- Added SPACETIMEDB_SETTING_CASE_CONVERSION(...) to support case
conversion policy
- Error-path hardening by adding explicit constraint-registration error
tracking and preinit validation
- Codegen updates:
- Updated C++ moduledef regen to V10 builder types.
- Adjusted C++ codegen duplicate-variant wrapper generation to emit
proper product-type
wrappers.
- Test/harness updates:
- type-isolation-test runner now defaults to focused V10 regression
checks; --v9 runs broader
legacy/full suite.
- Added focused modules for positive/negative V10 checks:
- test_multicolumn_index_valid
- error_multicolumn_missing_field
- error_default_missing_field
- Re-enabled C++ paths in sdks/rust/tests/test.rs procedure/view/test
suites.
# API and ABI breaking changes
- Refactor of the underlying module definition
- New *_NAMED variant macros for explicit canonical naming
- FIELD_NamedMultiColumnIndex renamed to FIELD_MultiColumnIndex
# Expected complexity level and risk
3 - Large set of changes moving over to V10 with underlying changes to
make future updates a little easier
# Testing
- [x] Ran the type isolation test and expanded it
- [x] Ran the spacetimedb-sdk test framework to confirm no more drift
between C++ and other module languages
- [x] Ran Unreal test suite though not really applicable
- [x] New app creation with `spacetime init --template basic-cpp`
- [x] Ran describe module tests against Rust + C# matching with C++ on
the /modules/sdk-test* modules to find any possible mis-alignment
# Review
- [x] Another look at the new features with C++
- [x] Thoughts on *_NAMED macros, I couldn't come up with a better
solution with C++20
|
||
|
Shubham Mishra
|
a327f91d30 |
TS: fix sourceName in table schema (#4449)
# Description of Changes fixes: https://github.com/clockworklabs/SpacetimeDB/issues/4433. Test will be added in: https://github.com/clockworklabs/SpacetimeDB/pull/4450/changes # API and ABI breaking changes NA # Expected complexity level and risk 2 |
||
|
Shubham Mishra
|
ba15373948 |
typescript: canonical naming for reducer and procedure. (#4371)
# Description of Changes Canonical naming for reducer and procedure. It shouldn't have the `name` field there but if it's there it should be used as canonical name. # API and ABI breaking changes Yes # Expected complexity level and risk 1 |
||
|
Shubham Mishra
|
e2f8a60759 |
Case conversion (#4263)
# Description of Changes Update the Default casing policy to `snake_case` for `RawModuleDefV10`. Messy PR contains changes at different places, so that CI can pass: Here are the main changes as follows: - `bindings-macro` & `bindings` crate: `name` macro in Indexes for canonical name and supply it to `RawModuleDefV10` via `ExplicitNames`. - `bindings-typescript`: - Changes has been reviewed through this PR - https://github.com/clockworklabs/SpacetimeDB/pull/4308. - `binding-csharp`: a single line change to pass `sourceName` of index instead of null. - `codegen`: - Changes has been merged from branch - https://github.com/clockworklabs/SpacetimeDB/pull/4337. - Except a fix in rust codegen to use canonical name in Query buillder instead of accessor. - `lib/db/raw_def`: Extends `RawDefModuleV10` structure to support case conversion. - `schema` crate: - `validate/v9` - Nothing itself should change or changes in v9 validation logic but the file contains a `CoreValidator` which is shared with `validate/v10`. No test have t be updated to `validate/v9` which ensures we aren't regressing it. - `validate/v10`: This is the main meat, look at the new tests added in bottom to understand what it does. - Rest of the files are either test updates or module bindings. ## Testing: 1. Extensive unit tests have been added to verify generated `ModuleDef` is correct. 2. I have done some e2e testing to verify rust codegen with rust and typescript modules. 3. I would have like to do more testing for other codegens , I am continue doing . I have removed `sql.py` smoketest, as that seems to be already migated in new framework and was headache to update. ## Expected complexity level and risk 4, It could have side-effect which aren't easily visible. - - - --------- Signed-off-by: Shubham Mishra <shivam828787@gmail.com> Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io> Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com> Co-authored-by: clockwork-labs-bot <bot@clockworklabs.com> Co-authored-by: joshua-spacetime <josh@clockworklabs.io> Co-authored-by: Noa <coolreader18@gmail.com> Co-authored-by: = <cloutiertyler@gmail.com> Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@clockworklabs.io> Co-authored-by: Jason Larabie <jason@clockworklabs.io> Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io> |
||
|
Noa
|
098afaf1a5 |
[TS] Improve autogen autocompletion and typing (#4309)
# Description of Changes All types are now defined and exported in `module_bindings/types.ts`, meanings there's only one module you need to import from to access types. # Expected complexity level and risk 2 # Testing n/a, no change in behavior. |
||
|
Shubham Mishra
|
a0ccf2d1cd |
Add accessor_name field in ModuleDef (#4323)
# Description of Changes Update Schema Defs to add `accessor_name` Look at `crates/schema/src/def.rs`. All other file updates are due to changing existing field names. # API and ABI breaking changes NA # Expected complexity level and risk 1 |
||
|
Shubham Mishra
|
e4098f98d9 |
Rust: macro change name -> accessor (#4264)
## Description of Changes
This PR primarily affects the `bindings-macro` and `schema` crates to
review:
### Core changes
1. Replaces the `name` macro with `accessor` for **Tables, Views,
Procedures, and Reducers** in Rust modules.
2. Extends `RawModuleDefV10` with a new section for:
* case conversion policies
* explicit names
New sections are not validated in this PR so not functional.
3. Updates index behavior:
* Index names are now always **system-generated** for clients. Which
will be fixed in follow-up PR when we start validating RawModuleDef with
explicit names.
* The `accessor` name for an index is used only inside the module.
## Breaking changes (API/ABI)
1. **Rust modules**
* The `name` macro must be replaced with `accessor`.
2. **Client bindings (all languages)**
* Index names are now system-generated instead of using explicitly
provided names.
**Complexity:** 3
A follow-up PR will reintroduce explicit names with support for case
conversion.
---------
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <bot@clockworklabs.com>
|
||
|
Piotr Sarnacki
|
4c962b9170 |
spacetime.json config implementation (#4199)
# Description of Changes
This PR implements support for the `spacetime.json` configuration file
that can be used to set up common `generate` and `publish` targets. An
example of `spacetime.json` could look like this:
```
{
"dev_run": "pnpm dev",
"generate": [
{ "out-dir": "./foobar", "module-path": "region-module", "language": "c-sharp" },
{ "out-dir": "./global", "module-path": "global-module", "language": "c-sharp" },
],
"publish": {
"database": "bitcraft",
"module-path": "spacetimedb",
"server": "local",
"children": [
{ "database": "region-1", "module-path": "region-module", server: "local" },
{ "database": "region-2", "module-path": "region-module", server: "local" }
]
}
}
```
With this config, running `spacetime generate` without any arguments
would generate bindings for two targets: `region-module` and
`global-module`. `spacetime publish` without any arguments would publish
three modules, starting from the parent: `bitcraft`, `region-1`, and
`region-2`. On top of that, the command `pnpm dev` would be executed
when using `spacetime dev`.
It is also possible to pass additional command line arguments when
calling the `publish` and `generate` commands, but there are certain
limitations. There is a special case when passing either a module path
to generate or a module name to publish. Doing that will filter out
entries in the config file that do not match. For example, running:
```
spacetime generate --project-path global-module
```
would only generate bindings for the second entry in the `generate`
list.
In a similar fashion, running:
```
spacetime publish region-1
```
would only publish the child database with the name `region-1`
Passing other existing arguments is also possible, but not all of the
arguments are available for multiple configs. For example, when running
`spacetime publish --server maincloud`, the publish command would be
applied to all of the modules listed in the config file, but the
`server` value from the command line arguments would take precedence.
Running with arguments like `--bin-path` would, however, would throw an
error as `--bin-path` makes sense only in a context of a specific
module, thus this wouldn't work: `spacetime publish --bin-path
spacetimedb/target/debug/bitcraft.wasm`. I will throw an error unless
there is only one entry to process, thus `spacetime publish --bin-path
spacetimedb/target/debug/bitcraft.wasm bitcraft` would work, as it
filters the publish targets to one entry.
# API and ABI breaking changes
None
# Expected complexity level and risk
3
The config file in itself is not overly complex, but when coupled with
the CLI it is somewhat tricky to get right. There are also some changes
that I had to make to how clap arguments are validated - because the
values can now come from both the config file and the clap config, we
can't use some of the built-in validations like `required`, or at least
I haven't found a clean way to do so.
# Testing
I've added some automated tests, but more tests and manual testing is
coming.
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: bradleyshep <148254416+bradleyshep@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: = <cloutiertyler@gmail.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
|
||
|
Tyler Cloutier
|
3f58b5951b |
Implement event tables (server, Rust/TS/C# codegen + client SDKs) (#4217)
## Summary
Implements event tables end-to-end: server datastore, module bindings
(Rust/TypeScript/C#), client codegen (Rust/TypeScript/C#), client SDKs
(Rust/TypeScript/C#), and integration tests.
Event tables are tables whose rows are ephemeral — they persist to the
commitlog and are delivered to V2 subscribers, but are NOT merged into
committed state. Rows are only visible within the transaction that
inserted them. This is the mechanism that replaces reducer event
callbacks in 2.0.
## What's included
### Server
- `is_event` flag on `RawTableDefV10`, `TableDef`, `TableSchema`
- Event table rows recorded in TxData but skipped during committed state
merge
- Commitlog replay treats event table inserts as no-ops
- Migration validation rejects changing `is_event` between module
versions
- `SELECT * FROM *` excludes event tables
- V1 WebSocket subscriptions to event tables rejected with upgrade
message
- V2 subscription path delivers event table rows correctly
- `CanBeLookupTable` trait — event tables cannot be lookup tables in
semijoins
- Runtime view validation rejects event tables
### Module bindings
- **Rust**: `#[spacetimedb::table(name = my_events, public, event)]`
- **TypeScript**: `table({ event: true }, ...)`
- **C#**: `[Table(Event = true)]`
### Client codegen (`crates/codegen/`)
- **Rust**: Generates `EventTable` impl (insert-only) for event tables,
`Table` impl for normal tables. `CanBeLookupTable` emitted for non-event
tables.
- **TypeScript**: Emits `event: true` in generated table schemas.
`ClientTableCore` type excludes `onDelete`/`onUpdate` for event tables
via conditional types.
- **C#**: Generates classes inheriting from `RemoteEventTableHandle`
(which hides `OnDelete`/`OnBeforeDelete`/`OnUpdate`) for event tables.
### Client SDKs
- **Rust**: `EventTable` trait with insert-only callbacks, client cache
bypass, `count()` returns 0, `iter()` returns empty
- **TypeScript**: Event table cache bypass in `table_cache.ts` — fires
`onInsert` callbacks but doesn't store rows. Type-level narrowing
excludes delete/update methods.
- **C#**: `RemoteEventTableHandle` base class hides delete/update
events. Parse/Apply/PostApply handle `EventTableRows` wire format, skip
cache storage, fire only `OnInsert`.
### Tests
- 9 datastore unit tests (insert/delete/update semantics, replay,
constraints, indexes, auto-inc, cross-tx reset)
- 3 Rust SDK integration tests (basic events, multiple events per
reducer, no persistence across transactions)
- Codegen snapshot tests (Rust, TypeScript, C#)
- Trybuild compile tests (event tables rejected as semijoin lookup
tables)
## Deferred
- `on_delete` codegen for event tables (server only sends inserts;
client synthesis deferred)
- Event tables in subscription joins / views (well-defined but
restricted for now)
- C++ SDK support
- RLS integration test
## API and ABI breaking changes
- `is_event: bool` added to `RawTableDefV10` (appended, defaults to
`false` — existing modules unaffected)
- `CanBeLookupTable` trait bound on semijoin methods in query builder
(all non-event tables implement it, so existing code compiles unchanged)
- `RemoteEventTableHandle` added to C# SDK (new base class for generated
event table handles)
## Expected complexity level and risk
3 — Changes touch the schema pipeline end-to-end and all three client
SDKs, but each individual change is straightforward. The core risk area
is the committed state merge skip in `committed_state.rs`. Client SDK
changes are additive (new code paths for event tables, existing paths
unchanged).
## Testing
- [x] `cargo clippy --workspace --tests --benches` passes
- [x] `cargo test -p spacetimedb-codegen` (snapshot tests)
- [x] `cargo test -p spacetimedb-datastore --features
spacetimedb-schema/test -- event_table` (9 unit tests)
- [x] `pnpm format` passes
- [x] Rust SDK integration tests pass (`event_table_tests` module)
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
|
||
|
clockwork-labs-bot
|
927d4c4020 |
[2.0 Breaking] Update C# client SDK for V2 WebSocket format (#4293)
Re-opened from #4289 which was accidentally merged into its base branch (`rekhoff/csharp-module-defs-v10-no-et`) instead of `master`. This is the same PR, now correctly targeting `master`. --- Original description from #4289: This PR migrates the C# SDK client path to WebSocket v2 semantics and aligns behavior with the Rust SDK direction for 2.0, including follow-on fixes in generation/tests. See #4289 for full description. --------- Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io> Co-authored-by: Jason Larabie <jason@clockworklabs.io> Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com> |
||
|
Noa
|
c044d96a7a |
[TS] schema() takes an object (#4273)
# Description of Changes Implements the rest of the casing proposal. # Expected complexity level and risk <!-- How complicated do you think these changes are? Grade on a scale from 1 to 5, where 1 is a trivial change, and 5 is a deep-reaching and complex change. This complexity rating applies not only to the complexity apparent in the diff, but also to its interactions with existing and future code. If you answered more than a 2, explain what is complex about the PR, and what other components it interacts with in potentially concerning ways. --> # Testing <!-- Describe any testing you've done, and any testing you'd like your reviewers to do, so that you're confident that all the changes work as expected! --> - [ ] <!-- maybe a test you want to do --> - [ ] <!-- maybe a test you want a reviewer to do, so they can check it off when they're satisfied. --> |
||
|
Tyler Cloutier
|
2ec07a3f70 |
Standardize query builder syntax across Rust, TypeScript, and C# (Server/Client) (#4261)
# Description of Changes Standardizes the query builder API across all three language SDKs (Rust, TypeScript, C#) for consistency. **Rust:** - Rename `Query` struct to `RawQuery`, make `Query` a trait with `fn into_sql(self) -> String` - All builder types (`Table`, `FromWhere`, `LeftSemiJoin`, `RightSemiJoin`) implement `Query<T>` trait - Views can return `-> impl Query<T>` instead of specifying exact builder types - The `#[view]` macro auto-detects `impl Query<T>` and rewrites to `RawQuery<T>` - Add `Not` variant to `BoolExpr` with `.not()` method **TypeScript:** - Add `ne()` to `ColumnExpression` - Refactor `BooleanExpr` to `BoolExpr` class with chainable `.and()`, `.or()`, `.not()` methods - Make builders valid queries directly (`.build()` deprecated but still works) - Deprecate `from()` wrapper — use `tables.person.where(...)` directly - Merge `query` export into `tables` so table refs are also query builders - Add subscription callback form: `subscribe(ctx => ctx.from.person.where(...))` - Unify `useTable` with query builder syntax; deprecate `filter.ts` **C#:** - Add `Not()` method to `BoolExpr<TRow>` - Add `IQuery<TRow>` interface implemented by all builder types (`Table`, `FromWhere`, `LeftSemiJoin`, `RightSemiJoin`, `Query`) - Add `ToSql()` to all builder types so `.Build()` is no longer required - Update `AddQuery` to accept `IQuery<TRow>` instead of `Query<TRow>` # API and ABI breaking changes - Rust: `Query<T>` is now a trait (was a struct). The struct is renamed to `RawQuery<T>`. This is a breaking change for any code that used `Query<T>` as a type directly. - TypeScript: `BooleanExpr` is now a `BoolExpr` class (was a discriminated union type). The `query` export is deprecated in favor of `tables`. - C#: `AddQuery` now accepts `Func<QueryBuilder, IQuery<TRow>>` instead of `Func<QueryBuilder, Query<TRow>>`. Existing `.Build()` calls still work since `Query<TRow>` implements `IQuery<TRow>`. # Expected complexity level and risk 3 — Changes touch multiple language SDKs and codegen, but each individual change is straightforward. The Rust macro rewrite for `impl Query<T>` detection is the most complex piece. All existing `.build()`/`.Build()` calls continue to work. # Testing - [x] `cargo test -p spacetimedb-query-builder` — 16/16 tests pass - [x] `cargo check -p spacetimedb` — clean, no warnings - [x] `cargo check` on views-query, views-sql, views-basic, views-trapped smoketest modules — all clean - [x] `cargo test -p spacetimedb-codegen codegen_csharp` — snapshot updated, passes - [x] `npm test` (TypeScript) — 101/101 tests pass - [x] C# QueryBuilder tests — new tests for `Not()`, `IQuery<T>` interface - [ ] CI passes |
||
|
Phoebe Goldman
|
1592dec8af |
Update Rust client SDK for V2 WebSocket format (#4257)
# Description of Changes
Update the Rust client SDK to use the new V2 WebSocket format, and
present the V2 user-facing API.
## Reducer events
### Remove on-reducer callbacks
It's no longer possible to observe reducers called by other clients by
registering callbacks with `ctx.reducers.on_{my_reducer}`. We no longer
code-generate those methods, or the associated
`ctx.reducers.remove_on_{my_reducer}`. Internal plumbing for storing and
invoking those callbacks is also removed.
### Add specific reducer invocation callbacks
In addition to the previous way to invoke reducers,
`ctx.reducers.{my_reducer}(args...)`, we add a method that registers a
callback to run after the reducer is finished. This method has the
suffix `_then`, as in `ctx.reducers.{my_reducer}_then(args...,
callback)`.
The callback will accept two arguments:
- `ctx: &ReducerEventContext`, the same context as was previously passed
to on-reducer callbacks.
- `status: Result<Result<(), String>, InternalError>`, denoting the
outcome of the reducer.
- `Ok(Ok(())` means the reducer committed. This corresponds to
`ReducerOutcome::Ok` or `ReducerOutcome::Okmpty` in the new WS format.
- `Ok(Err(message))` means the reducer returned an "expected" or "user"
error. This corresponds to `ReducerOutcome::Err` in the new WS format.
- `Err(internal_error)` means something went wrong with host execution.
This corresponds to `ReducerOutcome::InternalError` in the new WS
format.
Internally, the SDK stores the callbacks in its `ReducerCallbacks` map.
This is keyed on `request_id: u32`, a number that is generated for each
reducer call (from an `AtomicU32` that we increment each time), and
included in the `ClientMessage::CallReducer` request. The
`ServerMessage::ReducerResult` includes the same `request_id`, so the
SDK pops out of the `ReducerCallbacks` and invokes the appropriate
callback when processing that message.
These new callbacks are very similar to the existing procedure
callbacks.
### The `Event` exposed to row callbacks
Row callbacks caused by a reducer invoked by this client will see
`Event::Reducer`, the same as they would prior to this PR. These
callbacks will be the result of a `ServerMessage::ReducerResult` with
`ReducerOutcome::Ok`. In order to expose the reducer name and arguments
to this event, the client stores them in its `ReducerCallbacks` map,
alongside the callback for when the reducer is complete.
Row callbacks caused by any other reducer, or any non-reducer
transaction, are now indistinguishable to the client. These will see
`Event::Transaction`, which is renamed from the old
`Event::UnknownTransaction`.
### Less metadata in `ReducerEvent`
Some metadata is removed from `ReducerEvent`, as the V2 WebSocket format
no longer publishes it, even to the caller.
## `CallReducerFlags` are removed
All machinery for setting, storing and applying call reducer flags is
removed from the SDK, as the new WS format does not have any non-default
flags.
## Requesting rows in unsubscribe
When sending a `ClientMessage::Unsubscribe`, we always request that the
server include the matching rows in its response
`ServerMessage::UnsubscribeApplied`. This saves us having to update the
SDK to store query sets separately, at least for now. (We'll do that
later.)
## Handling rows
The new SDK does some additional parsing to wrangle rows in the new
WebSocket format into the same internal data structures as before,
rather than re-writing the client cache. (We'll do that later.)
Specifically, parsing of `DbUpdate` is changed so that:
- We parse raw `TransactionUpdate` into the generated `DbUpdate` type,
which requires an additional loop compared to the previous version, to
cope with the new WS format's dividing updates by query set. We define a
function `transaction_update_iter_table_updates` which encapsulates this
nested loop in an iterator.
- We have two new functions for parsing raw `QueryRows` into the
generated `DbUpdate` type, one for when they come from a
`SubscribeApplied`, and the other when they come from an
`UnsubscribeApplied`. `QueryRows` from `SubscribeApplied` translate to a
`DbUpdate` of all inserts, while one from `UnsubscribeApplied` will be
all deletes.
## Legacy subscriptions
"Legacy subscriptions" are removed. These were only used for
`subscribe_to_all_tables`, which as of now is stubbed. I will follow up
with a change to re-implement `subscribe_to_all_tables` by
code-generating a list of all known tables, and having it subscribe to
`select * from {table}` for every table in that list.
## `subscribe_to_all_tables` via a list
Previously, `subscribe_to_all_tables` worked by sending a legacy
subscription with the query `SELECT * FROM *`, which the host had
special handling to expand to subscribing to all tables. As legacy
subscriptions are no longer usable in V2 clients, this can't work.
Instead, we code-generate `SpacetimeModule::ALL_TABLE_NAMES`, a list of
all the known table names. `subscribe_to_all_tables` then maps across
this list to construct a list of queries in the form `SELECT * FROM
{table_name}`, and subscribes to all of those queries. This has the
upside that defining a new table in the module without regenerating
client bindings will no longer result in the client seeing rows of
tables it does not know about and cannot parse.
## Light mode removed
Light mode is no longer meaningful in the V2 WS format, so all code
related to it is removed.
## Internal changes
### Renamed WS messages
The SDK's internal code is updated to account for various renames:
- `QueryId` -> `QuerySetId`, `query_id` -> `query_set_id`.
- `SubscribeMulti` -> `Subscribe`, `UnsubscribeMulti` -> `Unsubscribe`.
## Incidental changes in this PR, not necessary for other client SDKs
### Don't filter out empty ranges in `RowSizeHint`
The Rust implementation of `RowSizeHint` in `BsatnRowList` got regressed
in the base branch to not work with zero-sized rows. This change fixes
that.
# API and ABI breaking changes
Boy howdy is it!
# Expected complexity level and risk
3? Changes ended up being less complicated than I feared, but we do have
some fiddly code here, and we have internal dependencies on the SDK.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Updated automated test suite.
- Known failures:
- [ ] `subscribe_all_select_star`, which is currently broken because
it's trying to subscribe to rows from private tables. #4241 will fix
this.
---------
Co-authored-by: Jeffrey Dallatezza <jeffreydallatezza@gmail.com>
Co-authored-by: = <cloutiertyler@gmail.com>
|
||
|
Phoebe Goldman
|
98585e858d |
Rename with_module_name -> with_database_name (#4267)
# Description of Changes Modules are like programs, databases are like processes. Your client connects to a remote database, not a remote module. <!-- Please describe your change, mention any related tickets, and so on here. --> # API and ABI breaking changes Yep! # Expected complexity level and risk 2? I made this change with find + replace, and it's possible that I borked the edit, or missed some reference somehow. It seems unlikely, however, that this change will have deeper implications we haven't considered. # Testing N/a |
||
|
Jason Larabie
|
c59ee1ddc3 |
[2.0 Breaking] Add --include-private and default private tables to not generate (#4241)
# Description of Changes Updated the codegen table/function iteration functions to take in a parameter to check visibility in all locations for the supported languages. - Updated the util.rs functions for iterating tables/functions to check for a CodegenVisibility enum (IncludePrivate, or OnlyPublic) - Added a new CodegenOptions struct to pass around the CodegenVisibility and future flags, defaulted visibility to OnlyPublic - Updated the CLI to return a list of all private tables not included (added a TODO to check the --include-private opt): ```bash Optimising module with wasm-opt... Build finished successfully. Skipping private tables during codegen: secret_note, secret_order, secret_person. Generate finished successfully. ``` # API and ABI breaking changes Technically API breaking as the private tables will no longer be available. (GitHub labels are not working at the moment) # Expected complexity level and risk 1 - Simple change the testing took longer # Testing Turns out when you remove private tables you invalidate most of the module_bindings across the system! - [x] Rust test SDK for all languages - [x] C# SDK tests - [x] C# dotnet tests - [x] Updated and checked snap files - [x] Updated Blackholio (Unreal + Unity) module_bindings and tested - [x] Ran Unreal SDK tests --------- Signed-off-by: Jason Larabie <jason@clockworklabs.io> |
||
|
joshua-spacetime
|
a78b056fcc |
Reorganize types generated for typescript clients (#4258)
NOTE: Cherry-picking https://github.com/clockworklabs/SpacetimeDB/pull/4127 from the `2.0-breaking-changes` branch. ## Original PR Description This changes generated types in ts client bindings. We currently generate a few different types of types: reducer args, procedure args, rows, and user defined types. To avoid potential conflicts between these types (for example, if a user defined a type called `FooRow`, and also had a tabled named `foo`, we would end up with two types named `FooRow`), this puts each set of types in a different file and namespace. We also stopped exporting the `xxxRow` types, because there is always another type generated for those. We now have a `types` directory, which has an `index.ts` with user defined types, along with `reducers.ts` and `procedures.ts` for the types generated for reducer/procedure parameters. ``` import type * as Types from './module_bindings/types'; var currentMessages: Types.Message[] = []; ``` or ``` import { type Message } from './module_bindings/types'; var currentMessages: Message[] = []; ``` This has a couple other changes: - For procedure and reducer types, this adds a suffix of `Args`, since we may want types for the return values in the future. - For all of the types, instead of exposing the schema object, we are now giving the typescript type (e.g. `export type Message = __Infer<typeof MessageRow>;`). I couldn't think of a reason for users to want the schema object, so this should save users from needing to do all of the `Infer` boilerplate. This is a breaking change for v2. 2. This only changes typescript, and it should generally make thing easier to use. --------- Co-authored-by: Jeffrey Dallatezza <jeffreydallatezza@gmail.com> |
||
|
Jason Larabie
|
52b6c66fa1 |
Add C++ Bindings (#3544)
# Description of Changes This adds C++ server bindings (/crate/bindings-cpp) to allow writing C++ 20 modules. - Emscripten WASM build system integration with CMake - Macro-based code generation (SPACETIMEDB_TABLE, SPACETIMEDB_REDUCER, etc) - All SpacetimeDB types supported (primitives, Timestamp, Identity, Uuid, etc) - Product types via SPACETIMEDB_STRUCT - Sum types via SPACETIMEDB_ENUM - Constraints marked with FIELD* macros # API and ABI breaking changes None # Expected complexity level and risk 2 - Doesn't heavily impact any other areas but is complex macro C++ structure to support a similar developer experience, did have a small impact on init command # Testing - [x] modules/module-test-cpp - heavily tested every reducer - [x] modules/benchmarks-cpp - tested through the standalone (~6x faster than C#, ~6x slower than Rust) - [x] modules/sdk-test-cpp - [x] modules/sdk-test-procedure-cpp - [x] modules/sdk-test-view-cpp - [x] Wrote several test modules myself - [x] Quickstart smoketest [Currently in progress] - [ ] Write Blackholio C++ server module --------- Signed-off-by: Jason Larabie <jason@clockworklabs.io> Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com> Co-authored-by: Ryan <r.ekhoff@clockworklabs.io> Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com> |
||
|
Shubham Mishra
|
bbb2335eac |
RawModuleDefV10 Scheduled functions should not be callable (#4179)
# Description of Changes - Codegen to skip creating reducer and procedure file if `visibility == FunctionVisiblity::Internal` - `UpdateHost` to check for visibility before excuting reducers and procedures. # API and ABI breaking changes NA, master is not exposing new raw module version yet. # Expected complexity level and risk 1. While the current patch is simple, could there be other places where visibility should also be applied? # Testing Smoketest can be in `2.0-breaking-changes` branch once we merge this. Exising test should cover for regression |
||
|
Mazdak Farrokhzad
|
3d3c99f8db |
Shrink JsWorkerRequest & use the right HashMap/Set (#4150)
# Description of Changes This PR shrinks `JsWorkerRequest` so that it is (almost) as small as the call reducer request. To do that, a bunch of trivial changes had to be done to auth code, that mostly revolves around `String` -> `Box<str>`. This should help the auth code, but that is incidental. The main goal was to improve throughput through the request tx/rx channel for V8, which is taking quite a bit of time in flamegraphs. I also noticed while making this change that the wrong hash map was being used in a bunch of places, so I fixed all of those. A follow up PR will shrink the reply side to fit within a cache line. Yet another follow up PR will change the channel to replace flume with `fibre::spsc`. # API and ABI breaking changes None # Expected complexity level and risk 2, fairly trivial changes. # Testing Covered by existing tests. |
||
|
Ryan
|
4f0a21f948 |
C# client typed query builder (#3982)
# Description of Changes This PR implements the C# client-side typed query builder, as assigned in https://github.com/clockworklabs/SpacetimeDB/issues/3759. Key pieces: * Added a small C# runtime query-builder surface in the client SDK (`sdks/csharp/src/QueryBuilder.cs`): * `Query` (wraps the generated SQL string) * `Table<TRow, TCols, TIxCols>` (entry point for `All()` / `Where(...)`) * `Col<TRow, TValue>` and `IxCol<TRow, TValue>` (typed column references) * `BoolExpr` (typed boolean expression composition) * SQL identifier quoting + literal formatting helpers (`SqlFormat`) * `Join(...)` with `WhereLeft(...)` / `WhereRight(...)` * `LeftSemijoin(...)` / `RightSemijoin(...)` with `Where(...)` chaining * Extended C# client bindings codegen (`crates/codegen/src/csharp.rs`) to generate: * Per-table/view `*Cols` and `*IxCols` helper classes used by the typed query builder. * A generated per-module `QueryBuilder` with a `From` accessor for each table/view, producing `Table<...>` values. * A generated `TypedSubscriptionBuilder` which collects `Query<TRow>.Sql` values and calls the existing subscription API. * An `AddQuery(Func<QueryBuilder, Query> build)` entry point off `SubscriptionBuilder`, mirroring the proposal’s Rust API. * Fixed a codegen naming collision found during regression testing: * `*Cols`/`*IxCols` helpers are now named after the table/view accessor name (PascalCase) instead of the row type, since multiple tables/views can share the same row type (e.g. alias tables / views returning an existing product type). * Kept `Cols`/`IxCols` off the public surface: * `Table.Cols` and `Table.IxCols` are internal, so consumers only access columns via the `Where(...)`/join predicate lambdas. C# usage examples (mirroring the proposal’s Rust examples) 1) Typed subscription flow (no raw SQL) ```csharp void Subscribe(SpacetimeDB.Types.DbConnection conn) { conn.SubscriptionBuilder() .OnApplied(ctx => { /* ... */ }) .OnError((ctx, err) => { /* ... */ }) .AddQuery(qb => qb.From.Users().Build()) .AddQuery(qb => qb.From.Players().Build()) .Subscribe(); } ``` 2) Typed `WHERE` filters and boolean composition ```csharp conn.SubscriptionBuilder() .OnApplied(ctx => { /* ... */ }) .OnError((ctx, err) => { /* ... */ }) .AddQuery(qb => qb.From.Players().Where(p => p.Name.Eq("alice").And(p.IsOnline.Eq(true))).Build()) .Subscribe(); ``` 3) “Admin can see all, otherwise only self” (proposal’s “player” view logic, but client-side) ```csharp Identity self = /* ... */; conn.SubscriptionBuilder() .AddQuery(qb => qb.From.Players().Where(p => p.Identity.Eq(self) ) ) .Subscribe(); ``` 4) Index-column access for query construction (IxCols) ```csharp conn.SubscriptionBuilder() .AddQuery(qb => qb.From.Players().Where( qb.From.Players().IxCols.Identity.Eq(self) ) ) .Subscribe(); ``` # API and ABI breaking changes None. * Additive client SDK runtime types. * Additive client bindings codegen output. * No wire-format changes. # Expected complexity level and risk 2 - Low to moderate * Mostly additive code + codegen. * The main risk is correctness/compat of generated SQL strings and name/casing conventions across languages; this is mitigated by targeted unit tests + full C# regression test runs. # Testing - [X] Ran run-regression-tests.sh successfully after regenerating C# bindings. - [X] Ran C# unit tests using `dotnet test sdks/csharp/tests~/tests.csproj -c Release` - [X] Added a new unit test suite (`sdks/csharp/tests~/QueryBuilderTests.cs`) validating: * Identifier quoting / escaping * Literal formatting (including `Identity`/`ConnectionId`/`Uuid` hex literals; `U128` integer literal) * null + enum unsupported behavior throws * Boolean expression parenthesization (`And`/`Or`/`Not`) * `Where(...)` overloads including `IxCols`-based predicates * left/right semijoin SQL formatting and predicate chaining |
||
|
Mazdak Farrokhzad
|
d6bc325244 |
Define TableName and ReducerName backed by EcoString (#4137)
# Description of Changes The first commit defines a type `TableName` that is used in e.g., `TxData` and where determined profitable and necessary to do this change. `TableName` is backed by [`ecow::EcoString`](https://docs.rs/ecow/0.2.6/ecow/string/struct.EcoString.html) which affords O(1) clones and 15 bytes of inline storage and `mem::size_of::<EcoString>() == 16`. The second commit does the same for `ReducerName`. This is also used in reducer execution. Together, these commits increase TPS by around 5-7k TPS. # API and ABI breaking changes None # Expected complexity level and risk 1 # Testing Covered by existing tests. |
||
|
Shubham Mishra
|
2560846f22 |
Rust: client query builder (#4003)
# Description of Changes Client Query builder for rust, as per proposal - https://github.com/clockworklabs/SpacetimeDBPrivate/pull/2356. 1. Pach moves query builder to its separate crate, so that it can be shared between module and sdk. 2. Implements `TypedSubscriptionBuilder` in `sdks/rust` as mentioned in proposal 3. Modify codegen to extend types to support query builder as mentioned in proposal 4. a test # API and ABI breaking changes NA, additive changes. # Expected complexity level and risk 2 # Testing Added a test. --------- Signed-off-by: Shubham Mishra <shivam828787@gmail.com> Co-authored-by: joshua-spacetime <josh@clockworklabs.io> |
||
|
Jeffrey Dallatezza
|
c5e5b3b5c8 |
Add a typed query builder for the typescript client (#4021)
# Description of Changes This moves the query builder code out of the `server` package and into `lib` so it can be shared by the client and server. I put the query builder in the `index.ts` of module bindings as a `query` object that can be imported. The typescript `test-app` has an example of using it with the subscription builder. This is branched off of `https://github.com/clockworklabs/SpacetimeDB/pull/3980`. # API and ABI breaking changes This extends the client subscription builder API to allow `string | RowTypedQuery<any, any>`, so existing client code should be fine. # Expected complexity level and risk 1.5. This is low risk. # Testing I manually tested that the test app still works locally. |
||
|
Jason Larabie
|
effb471b68 |
Add Unreal Result type to support C++ and Blueprint (#3834)
# Description of Changes - Updated Unreal codegen to work with ::Result to match how ::Option works to create a new struct for each type pair identified - Added new integration tests to confirm Ok state for all new types added to sdk-tests - Added new UE_SPACETIMEDB_RESULT macro to capture de/serialization for the new structs # API and ABI breaking changes N/A # Expected complexity level and risk 2 - Added new type generated from Result which creates new structs for each one identified # Testing - [x] Updated the integration tests to include the new Result types --------- Co-authored-by: Mario Alejandro Montoya Cortes <mamcx@elmalabarista.com> |
||
|
Mazdak Farrokhzad
|
8544e6cf02 |
Add Hash indices (#3976)
# Description of Changes Fixes https://github.com/clockworklabs/SpacetimeDB/issues/1122. Adds hash indices and exposes them through `#[index(hash)]` for Rust modules, with support for typescript and C# to come in follow ups. On the client/sdk side, for now, any index is backed via a BTree/native index as it is the most general. A hash index may only be queried through point scan and never ranged scans. Attempting a ranged scan results in an error, with the mechanism implemented in the previous PR (https://github.com/clockworklabs/SpacetimeDB/pull/3974). # API and ABI breaking changes None # Expected complexity level and risk 2? # Testing A test for ensuring that hash indices cannot be used for range scans is added. Tests exercising hash indices will come in the next PR. |
||
|
Brougkr
|
09ff117faa |
fix(codegen/unreal): eliminate UObject memory leak in reducer event handlers (#3987)
## Summary - Fixes critical memory leak in Unreal C++ generated code where `NewObject<UReducer>()` is called for every reducer event but never cleaned up - At 30Hz server tick rate (common for game servers), this creates ~30 orphaned UObjects per second, causing continuous memory growth - Solution: Pass stack-allocated `FArgs` struct directly via new `InvokeWithArgs()` method instead of allocating UObjects ## Problem The generated `ReducerEvent` handler in `SpacetimeDBClient.g.cpp` creates a new UObject for every reducer event: ```cpp FArgs Args = ReducerEvent.Reducer.GetAs...(); UReducer* Reducer = NewObject<UReducer>(); // LEAK - never cleaned up! Reducer->Param = Args.Param; Reducers->Invoke...(Context, Reducer); ``` Unreal's garbage collector cannot keep up at high frequencies. Attempts to fix with `MarkAsGarbage()`, `ConditionalBeginDestroy()`, and transient package flags all failed because GC timing is unpredictable. ## Solution Eliminate UObject allocation entirely by passing `FArgs` directly: ```cpp FArgs Args = ReducerEvent.Reducer.GetAs...(); Reducers->Invoke...WithArgs(Context, Args); // Zero allocation! ``` The original `Invoke()` methods that take `UReducer*` are preserved for backwards compatibility. ## Changes 1. **Header generation** (~line 2515): Add `InvokeWithArgs()` method declaration 2. **ReducerEvent handler** (~line 3194): Call `InvokeWithArgs()` instead of creating UObject 3. **Implementation** (~line 3703): Add `InvokeWithArgs()` that extracts params from `FArgs` struct ## Test Plan - [x] Verified with 30Hz `TickGameScheduled` reducer - UObject count remains stable - [x] Memory growth eliminated (was ~1000 UObjects per 14 seconds) - [x] Backwards compatibility preserved - existing `Invoke()` methods still work ## Before/After **Before**: `OBJS` count in Unreal stats continuously increasing, memory growing unbounded **After**: `UObjDelta=0` - no UObject growth from reducer events --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: Jason Larabie <jason@clockworklabs.io> |
||
|
Brougkr
|
30ab035fbd |
fix(codegen): Initialize UPROPERTY fields in generated Unreal C++ code (#3990)
## Summary UE5 strict mode requires all UPROPERTY fields to be explicitly initialized, otherwise the engine logs "property not initialized properly" errors at startup. This PR ensures generated Unreal C++ code properly initializes all UPROPERTY fields. ### Changes - Modified `cpp_ty_init_fmt_impl` to accept `module` parameter for enum type resolution - Added initialization for enum types (PlainEnum) using the first variant as default value - Updated all call sites in reducer/procedure/struct generation to pass the module ### Before ```cpp UPROPERTY(BlueprintReadWrite, Category="SpacetimeDB") EMyEnumType SomeEnumField; // ERROR: property not initialized properly ``` ### After ```cpp UPROPERTY(BlueprintReadWrite, Category="SpacetimeDB") EMyEnumType SomeEnumField = EMyEnumType::FirstVariant; // OK ``` ### Affected generated code - Reducer args structs (`FXxxReducerArgs`) - Reducer classes (`UXxxReducer`) - Procedure args structs (`FXxxProcedureArgs`) - Product type structs (`FXxxType`) ## Test plan - [x] Build codegen crate successfully - [x] Generate bindings for a project with enum types - [x] Verify generated code has proper initializers - [x] Build Unreal project without "property not initialized properly" errors --- Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: Jason Larabie <jason@clockworklabs.io> |
||
|
Mario Montoya
|
82d5a4f6c0 |
Implement SpacetimeType for Result<T, E> (#3790)
# Description of Changes Closes #3673 *NOTE*: C++ part will be in another PR # Expected complexity level and risk 2 Adding a new type touch everywhere # Testing - [x] Adding smoke and unit test --------- Signed-off-by: Ryan <r.ekhoff@clockworklabs.io> Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io> Co-authored-by: Phoebe Goldman <phoebe@goldman-tribe.org> Co-authored-by: Jason Larabie <jason@clockworklabs.io> |
||
|
Mario Montoya
|
8fb0bcf922 |
Add UUID built-in convenience type to SpacetimeDB (#3538)
# Description of Changes Closes [#3290](https://github.com/clockworklabs/SpacetimeDB/issues/3290). Adds a new "special" type to SATS, `UUID`, which is represented as the product `{ __uuid__: u128 }`. Adds versions of this type to all of our various languages' module bindings libraries and client SDKs, and updates codegen to recognize it and output references to those named library types. Adds methods for creating new UUIDs according to the V4 (all random) and V7 (timestamp, monotonic counter and random) specifications. # API and ABI breaking changes We add a new type # Expected complexity level and risk 2 it impacts all over the code # Testing - [x] Extends the Rust and Unreal SDK tests, and the associated `module-test` modules in Rust, C# and TypeScript, with uses of UUIDs. - [x] Extends the C# SDK regression tests with uses of UUIDs. - [x] Extends the TypeScript test suite with tests with uses of UUIDs. --------- Signed-off-by: Mario Montoya <mamcx@elmalabarista.com> Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io> Co-authored-by: Jason Larabie <jason@clockworklabs.io> Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com> |
||
|
Jeffrey Dallatezza
|
a1607ce523 |
Add Reducer to the end of reducer arg types to avoid collisions (#3889)
# Description of Changes This helps with the issue reported in https://github.com/clockworklabs/SpacetimeDB/issues/3811. Right now we have a type representing the reducer, and a type for the reducer args, and both have the same name. This adds `Reducer` to the end of the args type, which is similar to what we are doing for procedure arguments or the `Row` suffix for tables. This will still cause some potential problems, since someone could have a type that ends in `Reducer` (or `Row`), but this will fix the majority of issues that are currently breaking people. This also has some changes to get the basic react example to build. # API and ABI breaking changes This is technically a breaking change if people are using this for type annotations (which doesn't seem too likely), but it should be an easy one for people to fix. # Expected complexity level and risk 1. # Testing I tested the quickstart. |
||
|
Noa
|
4d23fe3e8d |
[TS] Fix useTable casing issues (#3853)
# Description of Changes Fixes #3807. Not sure what the best way to model this in the API is (another argument to `t.row()`?), but it does work. # Expected complexity level and risk 1 # Testing - [ ] <!-- maybe a test you want to do --> - [ ] <!-- maybe a test you want a reviewer to do, so they can check it off when they're satisfied. --> |
||
|
Noa
|
bea7a7dc68 |
[TS] Use 'override' when overriding class items (#3852)
# Description of Changes This should fix part of #3503. Adds an override modifier to generated code and fixes a warning from the angular compiler. # Expected complexity level and risk 1 # Testing - [x] Code works fine with the added `override` modifier. - [ ] Perhaps we should have an angular test project? |
||
|
Noa
|
afe169ac4a |
Fix the issues with scheduling procedures (#3816)
# Description of Changes This reapplies the patch from #3704, and fixes the issues that were causing it to deadlock. The reason it was deadlocking was that it allowed for the following sequence of events: * `SchedulerActor::handle_queued()` begins mutable tx * `ModuleHost::disconnect_client()` submits call to `call_reducer(tx: None)` * scheduler submits call to `call_reducer(tx: Some)` * `WasmModuleInstance::disconnect_client` now has to try to take tx lock, but the scheduler's call_reducer already holds it and is behind it in the queue So, I moved most of the logic from `handle_queued` back to being executed in the module worker thread, but kept the code in `scheduler.rs` so that it can all be reasoned about locally. Fixes #3645. Should I uncomment the implementation of `ExportFunctionForScheduledTable for F: Procedure` now? # Expected complexity level and risk 2 - there's a chance that this patch hasn't fully fixed the deadlock issue from #3704, but I'm quite confident. # Testing - [x] Manually verified that deadlock no longer occurs - previously, `while true; do python -m smoketests schedule_reducer -k test_scheduled_table_subscription; done` would freeze up in only 2 or 3 iterations, but now it can run for 10 minutes without issues. |
||
|
Tyler Cloutier
|
1630f0c553 |
Fixes reported issues with the TypeScript SDK (#3737)
# Description of Changes Fixes: - [This issue](https://discord.com/channels/1037340874172014652/1398209084699709492/1441556875647647816) by exporting the `SubscriptionHandle` type with the `REMOTE_MODULE` type applied. - [This issue](https://discord.com/channels/1037340874172014652/1398209084699709492/1441559246213746749) by converting to `camelCase` for column names in code generation. - Fixes an issue where `onMyReducer` callbacks were passing arguments as variadic params, while the types indicated they would be passed as an object. `onMyReducer((ctx, argA, argB, argC) => {})` vs onMyReducer((ctx, { argA, argB, argC}) => {})` - [Fixes an issue](https://github.com/clockworklabs/SpacetimeDB/issues/3503#issuecomment-3566715928) where the table type name was used instead of the table name in code generation for constructing tables. - [Fixes issue](https://discord.com/channels/1037340874172014652/1398209084699709492/1441886030436499466) with `ScheduleAt` being used in non-table types. - [Fixes issue](https://github.com/clockworklabs/SpacetimeDBPrivate/issues/2168) where template projects do not use the correct lifecycle reducer setup # API and ABI breaking changes Adds a new export, and fixes casing in code genreation. # Expected complexity level and risk 2 # Testing - I have tested that the `SubscriptionHandle` is correctly exported - I have tested that the constraint name is now output in `camelCase` - I have tested that `onMyReducer` callbacks now return a single argument - I have tested that the table name (and view name) is now used instead of the type name for code generation - I have tested that the new lifecycle reducers correctly compile |
||
|
John Detter
|
d26f3a10a9 |
Revert "Procedures: fix scheduling (#3704)" (#3774)
This reverts commit
|
||
|
Mazdak Farrokhzad
|
b2e37e8008 |
Procedures: fix scheduling (#3704)
# Description of Changes Reworks how `SchedulerActor::handle_queued` works so that it first determines the parameters of the call to a reducer or the parameters of the call to the procedure. This also enables the removal of the special case `call_scheduled_reducer`. Fixes #3645. # API and ABI breaking changes None # Expected complexity level and risk 2 # Testing A test `schedule_procedure` is added. --------- Co-authored-by: Noa <coolreader18@gmail.com> Co-authored-by: Phoebe Goldman <phoebe@goldman-tribe.org> Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io> |
||
|
Noa
|
174f5286d3 |
[TS] Client-side procedure bindings (#3765)
# Description of Changes <!-- Please describe your change, mention any related tickets, and so on here. --> # Expected complexity level and risk 2 # Testing - [x] The types for generated module bindings look right - [ ] Actually test the client behavior (how?) |