# Description of Changes
This PR updates Rust SDK subscription unsubscribe handling to return an
error instead of panicking when the internal pending mutation channel is
closed.
## Motivation
`SubscriptionHandle::unsubscribe_then` currently calls `unwrap()` after
sending an unsubscribe mutation through the internal pending mutation
channel. If the connection or subscription manager has already shut
down, that send can fail with a disconnected channel error, causing the
application to panic.
I encountered this in [`bevy_stdb`](https://github.com/onx2/bevy_stdb),
which tracks subscriptions across reconnects and unsubscribes old
handles after applying a new subscription for the same key.
## Changes
- Replaces `unwrap()` on `pending_mutation_sender.unbounded_send(...)`
with error propagation.
- Returns `crate::Error::Internal(...)` when the unsubscribe mutation
cannot be sent.
- Updates local unsubscribe state only after the unsubscribe mutation is
successfully queued.
# API and ABI breaking changes
None expected.
This changes an internal error path from panicking to returning an
existing `crate::Error::Internal(...)` through the existing
`crate::Result<()>` return type.
# Expected complexity level and risk
Complexity: 1/5
This is a small, localized change. The method already returns
`crate::Result<()>`, so replacing the `unwrap()` with error propagation
fits the existing API shape.
Risk is low. Callers now receive an error if the internal pending
mutation channel is closed instead of panicking. Existing `AlreadyEnded`
and `AlreadyUnsubscribed` behavior is unchanged.
# Testing
- [x] Verified that `unsubscribe_then` returns
`Err(crate::Error::Internal(...))` instead of panicking when the pending
mutation channel is disconnected.
- [ ] Verified that successful unsubscribe behavior is unchanged.
- [ ] Verified that `AlreadyEnded` is still returned when the
subscription has already ended.
- [ ] Verified that `AlreadyUnsubscribed` is still returned when
unsubscribe was already requested.
---------
Signed-off-by: Jeff Rooks <onx2rj@gmail.com>
# Description of Changes
This pull request improves the handling of connection lifecycle events
in the Rust client SDK for SpacetimeDB, particularly distinguishing
between connection failures and disconnections. It introduces a new
`ConnectionLifecycle` state machine to track connection progress,
ensures that the correct callback (`on_connect_error` or
`on_disconnect`) is invoked based on the connection state.
**Changes**
* `ConnectionLifecycle` enum to track the connection state
(`Connecting`, `Connected`, `Ended`)
* Refactored error handling so that if a connection fails before
establishment, the `on_connect_error` callback is invoked; if the
connection fails after establishment, the `on_disconnect` callback is
invoked. See `end_connection`.
* Updated where disconnections are handled
(`advance_one_message_blocking`, `advance_one_message_async`, and
message processing) to use `finish_connection`
* Improved handling of user-initiated disconnects during the connection
process to avoid reporting them as connection errors and to ensure
proper cleanup.
# API and ABI breaking changes
I guess maybe if people relied on the `on_connect_error` to actually
fire the `on_disconnect` then this changes that behavior.
# Expected complexity level and risk
Maybe a 2? Seems pretty low risk but I'm still new to the codebase,
please double check.
This doesn't fix the websocket issues, that'll be for another day. I
noticed websocket.rs has some places it just drops and the error isn't
handled properly. We could technically surface that information and run
our callbacks with more specific error messages.
# Testing
I had an agent build and run loads of tests for this but didn't commit
those since it would have made the PR massive. I was planning on testing
locally though to see if I could trigger a connection failure at some
point, maybe via an invalid access token.
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
- Bumps version to 2.2.0
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
- 1 - this is just a version bump
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Version number is correct (`2.2.0`)
- [x] BSL license file has been updated with the new date and version
number
---------
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
`UClientCache::ApplyDiff` (`sdks/unreal/.../DBCache/ClientCache.h`) has
an asymmetry between its insert and delete paths.
**Phase 1 (deletes)** correctly only emits a `Diff.Deletes` entry when
the row's refcount transitions to 0 — overlapping subscriptions just
decrement.
**Phase 2 (inserts)** always appends to `Diff.Inserts`, regardless of
whether the row was already cached:
```cpp
// before
FRowEntry<RowType>* Entry = Table->Entries.Find(Key);
if (!Entry) { /* refcount = 1 */ }
else { /* refcount + 1 */ }
Diff.Inserts.Add(Key, *NewRow); // ← fires for both branches
```
`BroadcastDiff` then fires `OnInsert` for every `Diff.Inserts` entry, so
any table subscribed by two overlapping queries (e.g. a global `SELECT *
FROM t` plus a per-row `WHERE id = ...`) re-fires its insert handler on
every later subscription apply — once per cached row, every time. Game
code that does work in `OnInsert` (positioning, spawning, snapping to
terrain) re-runs and clobbers state that was meant to be set once.
The intent is documented in `RowEntry.h`: *"Wrapper storing a row value
with a reference count used by overlapping subscriptions."* Phase 1
follows that design; phase 2 doesn't.
### Fix
Move the `Diff.Inserts.Add` into the `!Entry` branch only, so it fires
only on the absent → refcount=1 transition:
```cpp
if (!Entry) {
Table->Entries.Add(Key, FRowEntry<RowType>{NewRow, 1});
Diff.Inserts.Add(Key, *NewRow);
}
else {
Table->Entries.Add(Key, FRowEntry<RowType>{NewRow, Entry->RefCount + 1});
}
```
### Why real updates still work
Cache keys are BSATN row-bytes, not primary keys. A real update arrives
as a `(delete old_bytes, insert new_bytes)` pair where `old_bytes ≠
new_bytes` — so the insert side still takes the `!Entry` branch and gets
a `Diff.Inserts` entry. `FTableAppliedDiff::DeriveUpdatesByPrimaryKey`
then pairs the delete and insert by PK into
`UpdateInserts`/`UpdateDeletes`, and `OnUpdate` (not `OnInsert`) fires,
exactly as today.
Edge cases:
| Scenario | Phase 2 branch | Result | Correct? |
|---|---|---|---|
| New row | `!Entry` | `OnInsert` | ✓ |
| Real update (different bytes) | `!Entry` | `OnInsert`+`OnDelete`
reconciled to `OnUpdate` by PK | ✓ |
| Overlapping sub re-delivers cached row | `else` | refcount bump, no
event | ✓ (was broken — fired duplicate `OnInsert`) |
| Trivial update (identical bytes) | `else` | refcount bump | irrelevant
— server diffs identical rows away before emitting |
# API and ABI breaking changes
None. Purely internal cache bookkeeping. Existing
`OnInsert`/`OnDelete`/`OnUpdate` semantics are preserved for all
non-overlapping cases; the only behavior change is that overlapping
subscriptions stop emitting duplicate `OnInsert` events for
already-cached rows — which matches the documented `RowEntry` refcount
contract.
# Expected complexity level and risk
**1.** Two lines moved into a branch; comments updated. Mirrors logic
already present and known-correct in phase 1.
# Testing
Reproduced and validated downstream in an Unreal project. The repro
setup is straightforward to replicate against any module:
1. A table `t` with ~150 rows.
2. A global subscription `SELECT * FROM t`, applied first.
3. A diagnostic actor that binds `t.OnInsert`, then every 10s submits a
new overlapping subscription (e.g. `SELECT * FROM t` again, or any
`SELECT * FROM t WHERE 'id' = X` covering already-cached rows) and
counts the `OnInsert` events that arrive in that round.
Expected: round 1 fires once per row that is *new to the cache*;
subsequent rounds against already-cached rows fire 0.
Observed (~161 rows cached after initial load):
```
Pre-fix Post-fix
Global sub (empty → 161) 161 161 ← genuine inserts; unchanged
Round 1 (overlapping) 134 0 ← 134 cached dupes
Round 2 (overlapping) 134 0
Round 3 (overlapping) 134 0
Round 4–6 (overlapping) 134 each 0 each
```
The genuine "empty cache → 161 entries" wave on the initial global
subscription is unaffected — same `OnInsert` count both pre- and
post-fix. Only the duplicate fires from later overlapping subscriptions
on already-cached rows are eliminated. `OnUpdate` still fires correctly
when underlying rows actually change.
- [x] Reviewer: confirm an existing real-update (different bytes, same
PK) test still produces `OnUpdate` and not `OnInsert`+`OnDelete`.
- [x] Reviewer: confirm any test that relies on `OnInsert` firing on
every subscription apply (rather than only on cache 0→1 transition) is
not present — if it exists, it was relying on the bug.
# Description of Changes
Tests for case conversion.
# API and ABI breaking changes
NA
# Expected complexity level and risk
1
---------
Co-authored-by: clockwork-labs-bot <bot@clockworklabs.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
Make Timestamp a FilterableValue in Rust, C#, and Typescript. I'm not
sure this is changing all the places because we have the server and the
client in those 3 languages.
# API and ABI breaking changes
It's an additive change.
# Expected complexity level and risk
3. There are some designs decisions, like comparing timestamp to
strings/numbers.
# Testing
Added unit tests for the 3 languages.
# Description of Changes
- Updated the Unreal SDK test harness to allow Unreal Editor to work
with MacOS
- Updated the Unreal SDK test handler to work with Nil as it's a special
case
# API and ABI breaking changes
N/A
# Expected complexity level and risk
1 - Small changes to the Unreal SDK tests
# Testing
- [x] Ran full suite of tests on Mac for Unreal SDK
- [x] Ran full suite of tests on Windows + Linux for Unreal SDK to
confirm no regression
---------
Co-authored-by: Jason Larabie <jasonlarabie@Mac.lan>
# Description of Changes
Reapply changes from #4515 after reversion
# API and ABI breaking changes
No API or ABI changes
# Expected complexity level and risk
2 - This PR change itself is trivial, as it just reimplements #4515,
however as #4515 had broken the `quickstart` smoketest, this should be
considered when reviewing this PR.
# Testing
- [X] Tested against `python3 -m smoketests quickstart` locally
---------
Signed-off-by: Ryan <r.ekhoff@clockworklabs.io>
Co-authored-by: Tyler Cloutier <cloutiertyler@aol.com>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
This is a copy of #4006 with only the updates to the way Unreal handles
ticks. As per @brougkr findings:
- **FTickableGameObject initialization order bug** - Replaced with
FTSTicker for reliable tick registration:
- Removed FTickableGameObject inheritance
- Added FTSTicker::FDelegateHandle for manual tick registration
- Added destructor to clean up ticker registration
- Added OnTickerTick() method
### Issue
FTickableGameObject registers itself in its constructor BEFORE
UDbConnectionBase's constructor body runs. Even with
ETickableTickType::Never, UE's GENERATED_BODY() macro can interfere with
base class initialization order, causing the default constructor to be
called instead.
# API and ABI breaking changes
- Refactor of an underlying component of the SDK and non-breaking.
# Expected complexity level and risk
2 - This changes the structure of how the database tooling can
auto-tick, but is invisible to the developer
# Testing
As this changes a core feature I tested all aspects:
- [x] Reproduced the bug and confirmed against `master` and this branch
to see the fix working
- [x] Ran full suite of Unreal tests
- [x] Manually tested Unreal Blackholio
# Description of Changes
Revert the following PRs that have caused some breakage:
```
a32cffa76 Finish refactoring out replay (#4850)
d639be0af Replay: some code motion & reuse `ReplayCommittedState` (#4849)
78d6b6f7d Update NativeAOT-LLVM infrastructure to current ABI (#4515)
d5c1738c1 Better module backtraces for panics and whatnot (#577)
6f23b19f3 Wait for database update to become durable (#4846)
81c9eab86 Add `spacetime lock/unlock` to prevent accidental database deletion (#4502)
809aebd7c Move field `replay_table_updated` to `ReplayCommittedState` (#4807)
21b58ef99 Update axum (#2713)
b5cadff7a Extract replay stuff out of `CommittedState`, part 1 (#4804)
```
I also updated the Python smoketests for breakage introduced in
https://github.com/clockworklabs/SpacetimeDB/pull/4502. Reverting that
PR caused conflicts, so this fix is more straightforward.
# API and ABI breaking changes
Maybe kind of, but we haven't released any of these.
# Expected complexity level and risk
1
# Testing
Ask @bfops about testing
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
## Summary
- Update the experimental NativeAOT-LLVM build path
(`EXPERIMENTAL_WASM_AOT=1`) to include all host function imports from
ABI versions 10.0 through 10.4
- Fix the compiler package reference to work on both Windows x64 and
Linux x64 (was hardcoded to Windows only)
- Add a CI smoketest to verify AOT builds work on Linux x64
## Context
See #4514 for the full writeup on the C# AOT situation. The
`wasi-experimental` workload that all C# module builds depend on is
deprecated and removed from .NET 9+. NativeAOT-LLVM is the recommended
path forward for ahead-of-time compilation of C# to WebAssembly.
The existing NativeAOT-LLVM support (added by RReverser in #713) was
stale: missing imports added since then and a Windows-only package
reference.
## Changes
**`SpacetimeDB.Runtime.targets`:**
- Add 10 missing `WasmImport` declarations across spacetime_10.0 through
10.4
- Replace `runtime.win-x64.Microsoft.DotNet.ILCompiler.LLVM` with
`runtime.$(NETCoreSdkPortableRuntimeIdentifier).Microsoft.DotNet.ILCompiler.LLVM`
so it resolves correctly on Linux x64 as well
- Use explicit version strings instead of the `$(SpacetimeNamespace)`
variable
**`ci.yml`:**
- Add AOT build smoketest step in the `csharp-testsuite` job
## Test plan
- [x] CI smoketest passes: `EXPERIMENTAL_WASM_AOT=1 dotnet publish -c
Release` builds successfully on Linux x64
- [ ] Existing C# tests continue to pass (no changes to the default
interpreter path)
---------
Signed-off-by: Ryan <r.ekhoff@clockworklabs.io>
Co-authored-by: Ryan <r.ekhoff@clockworklabs.io>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
Adds a `sdk-test-procedure-cs` test to the other test modules in
`sdks/rust/tests/test.rs`.
Functionally `sdk-test-procedure-cs` performs all the operations as the
Rust test `sdk-test-procedure` but through C#, fulfilling the
requirements of Issue #3951
# API and ABI breaking changes
No API or ABI changes, this only adds a test.
# Expected complexity level and risk
1
# Testing
- [X] Ran `cargo test -p spacetimedb-sdk csharp_procedures` locally
without errors.
## Summary
On macOS/Apple platforms, Objective-C++ defines `Nil` as a macro
(`#define Nil nullptr` in `objc/objc.h`). Unreal Engine compiles all C++
as Objective-C++ on Apple platforms, so `FSpacetimeDBUuid::Nil()` in
`Builtins.h` collides with this macro and produces:
```
error: expected member name or ';' after declaration specifiers; 'nullptr' is a keyword in Objective-C++
```
Every UE project using the SpacetimeDB Unreal SDK on macOS hits this
build failure.
## Affected platforms
- macOS (any version) — Unreal Engine 5.x compiles as Objective-C++ on
Apple platforms
- Affects all SpacetimeDB Unreal SDK versions that expose
`FSpacetimeDBUuid::Nil()`
## Reproduction steps
1. Create or open any Unreal Engine project on macOS
2. Add the SpacetimeDB Unreal SDK (v2.1.0)
3. Build the project
4. Observe the build failure in `Builtins.h` at
`FSpacetimeDBUuid::Nil()`
## Fix
Added `#pragma push_macro` / `#undef Nil` / `#pragma pop_macro` guards
in `Builtins.h`:
- **Before struct definitions**: Save and undefine the `Nil` macro so it
doesn't interfere with `FSpacetimeDBUuid::Nil()` and any other
identifiers using `Nil`
- **After the last closing brace**: Restore the original `Nil` macro so
downstream Objective-C++ translation units are unaffected
This is the standard UE pattern for handling platform macro collisions
(similar to how UE itself handles `check`, `verify`, `TEXT`, etc. macro
conflicts).
## Test plan
- [ ] Verify macOS UE build succeeds with the SDK included
- [ ] Verify Windows/Linux builds are unaffected (the `#ifdef Nil` guard
is a no-op when the macro is not defined)
- [ ] Verify `FSpacetimeDBUuid::Nil()` and `NilUuid()` Blueprint
functions work correctly at runtime
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
This is the implementation to resolve#4424
# Description of Changes
* Add support for `IEnumerable<T>` as a valid C# View return type in
codegen, so view implementations can return query results without
forcing `ToList()`. The generator now detects `IEnumerable<T>`,
serializes it via the list serializer.
* Snapshots have been updated with a test to confirm that
`IEnumerable<T>` rather than checking it is unsupported (which we had
added when removing support during the transition away from `Query<T>`
usage.
* Refreshes docs/examples to reflect the new supported signature.
# API and ABI breaking changes
No breaking changes. This is additive support for an existing .NET
interface.
# Expected complexity level and risk
2 - Low
# Testing
- [X] Rebuilt CLI locally, ran regression tests, and `dotnet tests` in
C# module bindings.
- [X] Tested updated docs code snippets to ensure they resolve in an IDE
and publish.
---------
Signed-off-by: Ryan <r.ekhoff@clockworklabs.io>
# Description of Changes
This PR adapts the Rust SDK test suite to work with the wasm version
added in https://github.com/clockworklabs/SpacetimeDB/pull/4089 (which
I've closed in favor of this PR).
Most of the changes revolve around wasm's different async semantics -
everything runs in one thread, so things that relied on background
threads didn't work directly. Several tests would lock up because
something in them blocked synchronously, which blocked any background
work from progressing.
We moved the test-clients contents into a `test_handlers.rs` so that it
could be called from both `main` (for native tests) and `lib` (for wasm
tests). To show what actually changed, use:
```bash
git diff --no-index -- <(git show origin/master:sdks/rust/tests/procedure-client/src/main.rs) sdks/rust/tests/procedure-client/src/test_handlers.rs
```
(or similar for other test-clients)
# API and ABI breaking changes
None, I think/hope.
# Expected complexity level and risk
2
# Testing
- [x] I've augmented the CI to also run the test suite with the `web`
feature
---------
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Thales R <thlsrmsdev@gmail.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Bump versions to 2.1.0
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
CI
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
- Updated the Unreal SDK and generated Unreal bindings for the websocket
2.0 protocol/model
- Reworked DbConnectionBase to handle the updated message shapes
- Switched subscription handling over to new message types and
QuerySetId
- Updated reducer to ReducerResult, removal of callbacks, and set
reducer flags
- Added event table support
- Baked in multi-module support replacing [the old
PR](<https://github.com/clockworklabs/SpacetimeDB/pull/3417>)
- Added functionality to generate module support for multiple folders in
the Unreal project (add <module>.Build.cs, <module>.h, <module>.cpp)
using the --module-name
- Add new configuration option for spacetime generate to handle module
prefix
- Regenerated Unreal Blackholio/TestClient/QuickstartChat bindings
- Rebuilt Unreal Blackholio's consume entity to use event tables
- Updated migration documentation
- Updated the version bump tool to impact C++
# API and ABI breaking changes
- Unreal websocket/message handling updated to the new protocol
- Unreal generation now expects a real .uproject target and will stop
immediately if project
metadata is invalid instead of continuing past setup issues.
# Expected complexity level and risk
3 - A large set of changes to update the websocket/message handling
along with heavy codegen changes to handle multi-module support
# Testing
Test coverage of the Unreal SDK will need expansion in a future ticket
once our issues with flakiness on CI is resolved.
- [x] Updated Unreal Blackholio
- [x] Ran full Unreal SDK test suite
- [x] Built new test project using the new `--module-prefix`
- [x] Run through Unreal Blackholio (C++ and Blueprint)
- [x] Rebuilt Unreal Blackholio with multi-module, and duplicate
generated module testing side-by-side modules that would overlap
# Review Question(s)
- [x] Updates to `spacetime init` have made the tutorial a little
confusing with pathing for the Unreal Blackholio tutorial. To fix though
we'd have to update all the commands to be more explicit, or update the
tutorial `spacetime init` to use `--project-path .` to keep pathing
simpler, thoughts?
---------
Signed-off-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: Ryan <r.ekhoff@clockworklabs.io>
# Description of Changes
This bug was introduced when we added the v2 subscription protocol.
Before this change, when a v2 client disconnected or unsubscribed, other
clients could stop receiving updates for sender-scoped views they were
subscribed to.
This happened because `SubscriptionManager::remove_all_subscriptions()`
was cleaning up v2 subscription state correctly, but we would also run
the legacy v1 cleanup loop which mixed v1 and v2 queries. This meant the
same query hash could be processed again in the v1 cleanup loop after
the v2 removal had already happened. In the sender-view case, that
second pass could remove metadata (`indexes` / `search_args`) for active
subscribers.
Specifically, we could have the following sequence:
1. Client B disconnects.
2. The v2 loop removes B’s v2 subscription for `query_hash`.
3. `query_hash` now has no subscribers, so
`remove_query_from_tables(query_hash)` runs once.
4. That compatibility loop then visits `query_hash` again via
`subscription_ref_count.keys()`.
5. `query_hash` still has no subscribers, so
`remove_query_from_tables(query_hash)` runs a second time.
and when `remove_query_from_tables(query_hash)` runs a 2nd time, it
calls `index_ids.delete_index_ids_for_query()` which is refcounted, not
idempotent. So calling it twice for the same query decrements the shared
index count twice, and the 2nd decrement drops an index that client A’s
query still needs.
The fix was to:
1. Collect the disconnected client’s v1 query hashes from
`client_info.v1_subscriptions`
2. Run the v2 cleanup loop over `client_info.v2_subscriptions` (as we
did before this change)
3. Run the old compatibility cleanup loop only over the deduplicated v1
hashes
# API and ABI breaking changes
None
# Expected complexity level and risk
2
Small fix to a rather intricate bug. Most of the code in this patch is
testing code.
# Testing
This bug was not reproducible with the current smoketest harness,
because `spacetime subscribe` still uses the `v1` subscription protocol.
I added a Rust SDK test, which did reproduce the bug before the fix,
because the Rust SDK uses the `v2` subscription protocol.
# Description of Changes
Same change set as
https://github.com/clockworklabs/SpacetimeDB/pull/4614, just targeting
master.
The return type of a query builder view is now a special SATS product
type `{ __query__: T }`. A view with this return type now has a primary
key if `T` has a primary key. This means that client codegen will
generate an `OnUpdate` callback for such views. It will also generate
query builder index bindings for the primary key column.
# API and ABI breaking changes
None. Old modules with query builder views still work, they just don't
have primary keys.
# Expected complexity level and risk
2
# Testing
Added equivalent tests to the ones that were added in #4573 and #4572.
# Description of Changes
In the Rust client SDK, with this PR, doing
`.with_debug_to_file("path.txt")` on the connection builder will result
in the SDK logging additional verbose info to `path.txt`.
# API and ABI breaking changes
Adds a new user-facing-ish API to the Rust client SDK.
# Expected complexity level and risk
2? Some possibility of deadlock due to adding a new mutex, but this is
explicitly not for production use.
# Testing
- [x] Ran `chat-console-rs` locally with this enabled and got some debug
logs out of it.
# Description of Changes
Query builder views return a subset of rows from a physical table. If
that table has a primary key, then so should the view. What this means
concretely is that the view should expose the same api as the table,
specifically as it relates to the primary key column.
With that in mind, this patch commits the following changes:
1. Annotates `ViewDef` with a `primary_key`
2. Updates the return type of query builder views in the raw module def
to a special product type
3. Adds an index for the primary key on the view's backing table
4. Updates the query planner to use this index
5. Updates rust client codegen to generate `on_update` for such views
# API and ABI breaking changes
None
Old `impl Query` views compiled with an older version of SpacetimeDB
will continue to work as they did before - without a primary key.
# Expected complexity level and risk
3
# Testing
- [x] New rust sdk integration suite exercising `on_update` for PK views
and semijoin scenarios
- [x] Smoketests for PK views and semijoin scenarios
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
- Bumps version to 2.0.4
# API and ABI breaking changes
None - this is just a version bump
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
1 - this is just a version bump
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] License file has been updated with correct version + time
---------
Signed-off-by: John Detter <4099508+jdetter@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
# Description of Changes
Adds implicit query builder conversions from `bool` to `BoolExpr` so
that you can write:
```rust
ctx.from.user().r#where(|u| u.online)
```
instead of
```rust
ctx.from.user().r#where(|u| u.online.eq(true))
```
Also removes `NullableCol` and `NullableIxCol` types from C# query
builder.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Unit and smoketests
# Description of Changes
- Migrated the C++ module-definition assembly path to V10-first
internals:
- Added v10_builder and module_type_registration systems.
- Switched Module::__describe_module__ to serialize RawModuleDef with
V10 payload.
- Updated macro registration pipeline to register through V10
- Added explicit naming support across macro surface (*_NAMED variants
for reducer/procedure/
view and field/index macros).
- Reworked multi-column index macros (FIELD_MultiColumnIndex,
FIELD_MultiColumnIndex_NAMED) with
migration alias.
- Added SPACETIMEDB_SETTING_CASE_CONVERSION(...) to support case
conversion policy
- Error-path hardening by adding explicit constraint-registration error
tracking and preinit validation
- Codegen updates:
- Updated C++ moduledef regen to V10 builder types.
- Adjusted C++ codegen duplicate-variant wrapper generation to emit
proper product-type
wrappers.
- Test/harness updates:
- type-isolation-test runner now defaults to focused V10 regression
checks; --v9 runs broader
legacy/full suite.
- Added focused modules for positive/negative V10 checks:
- test_multicolumn_index_valid
- error_multicolumn_missing_field
- error_default_missing_field
- Re-enabled C++ paths in sdks/rust/tests/test.rs procedure/view/test
suites.
# API and ABI breaking changes
- Refactor of the underlying module definition
- New *_NAMED variant macros for explicit canonical naming
- FIELD_NamedMultiColumnIndex renamed to FIELD_MultiColumnIndex
# Expected complexity level and risk
3 - Large set of changes moving over to V10 with underlying changes to
make future updates a little easier
# Testing
- [x] Ran the type isolation test and expanded it
- [x] Ran the spacetimedb-sdk test framework to confirm no more drift
between C++ and other module languages
- [x] Ran Unreal test suite though not really applicable
- [x] New app creation with `spacetime init --template basic-cpp`
- [x] Ran describe module tests against Rust + C# matching with C++ on
the /modules/sdk-test* modules to find any possible mis-alignment
# Review
- [x] Another look at the new features with C++
- [x] Thoughts on *_NAMED macros, I couldn't come up with a better
solution with C++20
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
- Version upgrade 2.0.3
# API and ABI breaking changes
- None, this is a version bump
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
- 1 - this is a version bump
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] License has been updated
- [x] Version number is correct (2.0.3)
# Description of Changes
Bumping all versions to 2.0.2 for the release.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
CI
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Just bumping versions to 2.0.1.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
- [ ] CI passes
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
This fixes an issue with the C# implementation of Query Builder
requiring `Build()` to return a query, and the return type being
`Query<TRow>` rather than `IQuery<TRow>`.
# Description of Changes
1. Runtime/query-builder
* Removed the concrete `Query<TRow>` carrier type and every `.Build()`.
Query builder shapes now expose only `IQuery<TRow>` plus `ToSql()`.
* Ensured all builder entry points (tables, joins, filters) continue to
return `IQuery<TRow>`.
2. Source generator + bindings
* Updated `ViewDeclaration` analysis to treat any return type
implementing `SpacetimeDB.IQuery<TRow>` as a SQL view.
* Dispatcher generation now emits
`ViewResultHeader.RawSql(returnValue.ToSql())` . This eliminates a
`Query<TRow>` special-case.
3. Tests, fixtures, regression module
* Converted the C# query-builder unit tests, codegen fixtures, and
regression-test server views to call `ToSql()`/return `IQuery<TRow>`.
* Added coverage proving `RightSemiJoin` (and friends) still satisfy
`IQuery<TRow>`.
4. CLI templates & generated bindings
* Regenerated/edited C# template bindings so
`SubscriptionBuilder.AddQuery` accepts `Func<QueryBuilder,
IQuery<TRow>>` and captures SQL via `ToSql()`.
# API and ABI breaking changes
While technically API breaking, this actually brings the API closer to
the intended design.
* `Query<TRow>` has been removed from the public surface area; any
previous references (including `.Build()` and `.Sql`) must be replaced
with the builder instance itself plus `.ToSql()`.
* View methods must now return an `IQuery<TRow>` (or any custom type
implementing it) when producing SQL for the host.
* Generated C# client bindings now expect typed subscription callbacks
to produce `IQuery<TRow>`, aligning the client SDK with the new runtime
contract.
# Expected complexity level and risk
3 - Medium: Touches runtime, codegen, fixtures, and templates. Risk is
mitigated by parity with Rust semantics and comprehensive test updates,
but downstream modules must recompile to adopt the new interface.
# Testing
- [X] Built CLI and ran regression tests locally with removed `.Build()`
- [X] Ran `dotnet test .\sdks\csharp\tests~\tests.csproj -c Release`
with all tests passing
- [X] Ran `dotnet test
.\crates\bindings-csharp\Codegen.Tests\Codegen.Tests.csproj -c Release`
with all tests passing
# Description of Changes
Update the Default casing policy to `snake_case` for `RawModuleDefV10`.
Messy PR contains changes at different places, so that CI can pass:
Here are the main changes as follows:
- `bindings-macro` & `bindings` crate: `name` macro in Indexes for
canonical name and supply it to `RawModuleDefV10` via `ExplicitNames`.
- `bindings-typescript`:
- Changes has been reviewed through this PR -
https://github.com/clockworklabs/SpacetimeDB/pull/4308.
- `binding-csharp`: a single line change to pass `sourceName` of index
instead of null.
- `codegen`:
- Changes has been merged from branch -
https://github.com/clockworklabs/SpacetimeDB/pull/4337.
- Except a fix in rust codegen to use canonical name in Query buillder
instead of accessor.
- `lib/db/raw_def`: Extends `RawDefModuleV10` structure to support case
conversion.
- `schema` crate:
- `validate/v9` - Nothing itself should change or changes in v9
validation logic but the file contains a `CoreValidator` which is shared
with `validate/v10`. No test have t be updated to `validate/v9` which
ensures we aren't regressing it.
- `validate/v10`: This is the main meat, look at the new tests added in
bottom to understand what it does.
- Rest of the files are either test updates or module bindings.
## Testing:
1. Extensive unit tests have been added to verify generated `ModuleDef`
is correct.
2. I have done some e2e testing to verify rust codegen with rust and
typescript modules.
3. I would have like to do more testing for other codegens , I am
continue doing .
I have removed `sql.py` smoketest, as that seems to be already migated
in new framework and was headache to update.
## Expected complexity level and risk
4, It could have side-effect which aren't easily visible.
- - -
---------
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <bot@clockworklabs.com>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
Co-authored-by: Noa <coolreader18@gmail.com>
Co-authored-by: = <cloutiertyler@gmail.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@clockworklabs.io>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
# Description of Changes
Prior to this commit, if a reducer call failed during a test in the Rust
SDK test suite, the client would hang, as it generally didn't register
callbacks when invoking reducers. This could mask some bugs, causing the
tests to fail with TIMEOUT rather than a useful message. For example, we
relied on consistency between all of the test modules, which was broken
by our not having updated the C++ bindings library to use the new
case-conversion scheme. This meant that the client was sending reducer
calls with incorrect reducer names to at least one database per test
run, which would result in a `Panic` status or an `Err(Err(_))` outcome.
With this commit, the SDK test clients are updated to inspect the exit
status of reducers they invoke, and (except for intentional reducer
failures) to panic the test client when encountering a reducer error or
panic.
# API and ABI breaking changes
N/a
# Expected complexity level and risk
1
# Testing
- [x] Automated tests passed locally.
- [x] Intentionally broke reducer calls via the 2.0 WS API, causing them
to skip the call and unconditionally respond with `Status::Panic`.
Observed "loud" failures in the test suite with non-`TIMEOUT` output
that looked more debuggable to me.
# Description of Changes
Basic tests for templates, the test tries to generate each of the
template, build and publish the SpacetimeDB part, and build or type
check the client side.
This PR also includes bindings for all of the clients as it's needed for
the tests to work and given the choices to either generate during the
test run or include the bindings in git we chose the latter. This makes
it consistent across the templates as some of them already included
bindings.
# Expected complexity level and risk
1
---------
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
# Description of Changes
* Extend the C# regression client to cover compound Query Builder
predicates (range filters + boolean combinators) so we ensure the
generated SQL matches the documentation promises. See: `Program.cs`
* Updated the C# documentation snippets to use `.AddQuery(...)` over raw
SQL. See: `00400-subscriptions.md` and `00400-sdk-api.md`
# API and ABI breaking changes
No changes to API or ABI
# Expected complexity level and risk
1
# Testing
- [X] Ran regression tests locally
# Description of Changes
Implementation of #4295
Convert existing `Name` attribute to `Accessor` to support new Canonical
Case conversation of 2.0
# API and ABI breaking changes
Yes, in C# modules, we no longer use the attribute name `Name`, it
should now be `Accessor`
# Expected complexity level and risk
1
# Testing
- [X] Build and tested locally
- [X] Ran regression tests locally
## Description of Changes
This PR primarily affects the `bindings-macro` and `schema` crates to
review:
### Core changes
1. Replaces the `name` macro with `accessor` for **Tables, Views,
Procedures, and Reducers** in Rust modules.
2. Extends `RawModuleDefV10` with a new section for:
* case conversion policies
* explicit names
New sections are not validated in this PR so not functional.
3. Updates index behavior:
* Index names are now always **system-generated** for clients. Which
will be fixed in follow-up PR when we start validating RawModuleDef with
explicit names.
* The `accessor` name for an index is used only inside the module.
## Breaking changes (API/ABI)
1. **Rust modules**
* The `name` macro must be replaced with `accessor`.
2. **Client bindings (all languages)**
* Index names are now system-generated instead of using explicitly
provided names.
**Complexity:** 3
A follow-up PR will reintroduce explicit names with support for case
conversion.
---------
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <bot@clockworklabs.com>
# Description of Changes
This PR implements support for the `spacetime.json` configuration file
that can be used to set up common `generate` and `publish` targets. An
example of `spacetime.json` could look like this:
```
{
"dev_run": "pnpm dev",
"generate": [
{ "out-dir": "./foobar", "module-path": "region-module", "language": "c-sharp" },
{ "out-dir": "./global", "module-path": "global-module", "language": "c-sharp" },
],
"publish": {
"database": "bitcraft",
"module-path": "spacetimedb",
"server": "local",
"children": [
{ "database": "region-1", "module-path": "region-module", server: "local" },
{ "database": "region-2", "module-path": "region-module", server: "local" }
]
}
}
```
With this config, running `spacetime generate` without any arguments
would generate bindings for two targets: `region-module` and
`global-module`. `spacetime publish` without any arguments would publish
three modules, starting from the parent: `bitcraft`, `region-1`, and
`region-2`. On top of that, the command `pnpm dev` would be executed
when using `spacetime dev`.
It is also possible to pass additional command line arguments when
calling the `publish` and `generate` commands, but there are certain
limitations. There is a special case when passing either a module path
to generate or a module name to publish. Doing that will filter out
entries in the config file that do not match. For example, running:
```
spacetime generate --project-path global-module
```
would only generate bindings for the second entry in the `generate`
list.
In a similar fashion, running:
```
spacetime publish region-1
```
would only publish the child database with the name `region-1`
Passing other existing arguments is also possible, but not all of the
arguments are available for multiple configs. For example, when running
`spacetime publish --server maincloud`, the publish command would be
applied to all of the modules listed in the config file, but the
`server` value from the command line arguments would take precedence.
Running with arguments like `--bin-path` would, however, would throw an
error as `--bin-path` makes sense only in a context of a specific
module, thus this wouldn't work: `spacetime publish --bin-path
spacetimedb/target/debug/bitcraft.wasm`. I will throw an error unless
there is only one entry to process, thus `spacetime publish --bin-path
spacetimedb/target/debug/bitcraft.wasm bitcraft` would work, as it
filters the publish targets to one entry.
# API and ABI breaking changes
None
# Expected complexity level and risk
3
The config file in itself is not overly complex, but when coupled with
the CLI it is somewhat tricky to get right. There are also some changes
that I had to make to how clap arguments are validated - because the
values can now come from both the config file and the clap config, we
can't use some of the built-in validations like `required`, or at least
I haven't found a clean way to do so.
# Testing
I've added some automated tests, but more tests and manual testing is
coming.
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: bradleyshep <148254416+bradleyshep@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: = <cloutiertyler@gmail.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Re-opened from #4289 which was accidentally merged into its base branch
(`rekhoff/csharp-module-defs-v10-no-et`) instead of `master`.
This is the same PR, now correctly targeting `master`.
---
Original description from #4289:
This PR migrates the C# SDK client path to WebSocket v2 semantics and
aligns behavior with the Rust SDK direction for 2.0, including follow-on
fixes in generation/tests.
See #4289 for full description.
---------
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
Set name for C# SDK network thread. Helpful when profiling and debugging
# API and ABI breaking changes
No breaking changes
# Expected complexity level and risk
1 - Trivial
# Testing
Tested in Unity Profiler
Co-authored-by: SteveGibsonX <stevegibsonxj@gmail.com>
Co-authored-by: Ryan <r.ekhoff@clockworklabs.io>
# Description of Changes
Standardizes the query builder API across all three language SDKs (Rust,
TypeScript, C#) for consistency.
**Rust:**
- Rename `Query` struct to `RawQuery`, make `Query` a trait with `fn
into_sql(self) -> String`
- All builder types (`Table`, `FromWhere`, `LeftSemiJoin`,
`RightSemiJoin`) implement `Query<T>` trait
- Views can return `-> impl Query<T>` instead of specifying exact
builder types
- The `#[view]` macro auto-detects `impl Query<T>` and rewrites to
`RawQuery<T>`
- Add `Not` variant to `BoolExpr` with `.not()` method
**TypeScript:**
- Add `ne()` to `ColumnExpression`
- Refactor `BooleanExpr` to `BoolExpr` class with chainable `.and()`,
`.or()`, `.not()` methods
- Make builders valid queries directly (`.build()` deprecated but still
works)
- Deprecate `from()` wrapper — use `tables.person.where(...)` directly
- Merge `query` export into `tables` so table refs are also query
builders
- Add subscription callback form: `subscribe(ctx =>
ctx.from.person.where(...))`
- Unify `useTable` with query builder syntax; deprecate `filter.ts`
**C#:**
- Add `Not()` method to `BoolExpr<TRow>`
- Add `IQuery<TRow>` interface implemented by all builder types
(`Table`, `FromWhere`, `LeftSemiJoin`, `RightSemiJoin`, `Query`)
- Add `ToSql()` to all builder types so `.Build()` is no longer required
- Update `AddQuery` to accept `IQuery<TRow>` instead of `Query<TRow>`
# API and ABI breaking changes
- Rust: `Query<T>` is now a trait (was a struct). The struct is renamed
to `RawQuery<T>`. This is a breaking change for any code that used
`Query<T>` as a type directly.
- TypeScript: `BooleanExpr` is now a `BoolExpr` class (was a
discriminated union type). The `query` export is deprecated in favor of
`tables`.
- C#: `AddQuery` now accepts `Func<QueryBuilder, IQuery<TRow>>` instead
of `Func<QueryBuilder, Query<TRow>>`. Existing `.Build()` calls still
work since `Query<TRow>` implements `IQuery<TRow>`.
# Expected complexity level and risk
3 — Changes touch multiple language SDKs and codegen, but each
individual change is straightforward. The Rust macro rewrite for `impl
Query<T>` detection is the most complex piece. All existing
`.build()`/`.Build()` calls continue to work.
# Testing
- [x] `cargo test -p spacetimedb-query-builder` — 16/16 tests pass
- [x] `cargo check -p spacetimedb` — clean, no warnings
- [x] `cargo check` on views-query, views-sql, views-basic,
views-trapped smoketest modules — all clean
- [x] `cargo test -p spacetimedb-codegen codegen_csharp` — snapshot
updated, passes
- [x] `npm test` (TypeScript) — 101/101 tests pass
- [x] C# QueryBuilder tests — new tests for `Not()`, `IQuery<T>`
interface
- [ ] CI passes
# Description of Changes
Update the Rust client SDK to use the new V2 WebSocket format, and
present the V2 user-facing API.
## Reducer events
### Remove on-reducer callbacks
It's no longer possible to observe reducers called by other clients by
registering callbacks with `ctx.reducers.on_{my_reducer}`. We no longer
code-generate those methods, or the associated
`ctx.reducers.remove_on_{my_reducer}`. Internal plumbing for storing and
invoking those callbacks is also removed.
### Add specific reducer invocation callbacks
In addition to the previous way to invoke reducers,
`ctx.reducers.{my_reducer}(args...)`, we add a method that registers a
callback to run after the reducer is finished. This method has the
suffix `_then`, as in `ctx.reducers.{my_reducer}_then(args...,
callback)`.
The callback will accept two arguments:
- `ctx: &ReducerEventContext`, the same context as was previously passed
to on-reducer callbacks.
- `status: Result<Result<(), String>, InternalError>`, denoting the
outcome of the reducer.
- `Ok(Ok(())` means the reducer committed. This corresponds to
`ReducerOutcome::Ok` or `ReducerOutcome::Okmpty` in the new WS format.
- `Ok(Err(message))` means the reducer returned an "expected" or "user"
error. This corresponds to `ReducerOutcome::Err` in the new WS format.
- `Err(internal_error)` means something went wrong with host execution.
This corresponds to `ReducerOutcome::InternalError` in the new WS
format.
Internally, the SDK stores the callbacks in its `ReducerCallbacks` map.
This is keyed on `request_id: u32`, a number that is generated for each
reducer call (from an `AtomicU32` that we increment each time), and
included in the `ClientMessage::CallReducer` request. The
`ServerMessage::ReducerResult` includes the same `request_id`, so the
SDK pops out of the `ReducerCallbacks` and invokes the appropriate
callback when processing that message.
These new callbacks are very similar to the existing procedure
callbacks.
### The `Event` exposed to row callbacks
Row callbacks caused by a reducer invoked by this client will see
`Event::Reducer`, the same as they would prior to this PR. These
callbacks will be the result of a `ServerMessage::ReducerResult` with
`ReducerOutcome::Ok`. In order to expose the reducer name and arguments
to this event, the client stores them in its `ReducerCallbacks` map,
alongside the callback for when the reducer is complete.
Row callbacks caused by any other reducer, or any non-reducer
transaction, are now indistinguishable to the client. These will see
`Event::Transaction`, which is renamed from the old
`Event::UnknownTransaction`.
### Less metadata in `ReducerEvent`
Some metadata is removed from `ReducerEvent`, as the V2 WebSocket format
no longer publishes it, even to the caller.
## `CallReducerFlags` are removed
All machinery for setting, storing and applying call reducer flags is
removed from the SDK, as the new WS format does not have any non-default
flags.
## Requesting rows in unsubscribe
When sending a `ClientMessage::Unsubscribe`, we always request that the
server include the matching rows in its response
`ServerMessage::UnsubscribeApplied`. This saves us having to update the
SDK to store query sets separately, at least for now. (We'll do that
later.)
## Handling rows
The new SDK does some additional parsing to wrangle rows in the new
WebSocket format into the same internal data structures as before,
rather than re-writing the client cache. (We'll do that later.)
Specifically, parsing of `DbUpdate` is changed so that:
- We parse raw `TransactionUpdate` into the generated `DbUpdate` type,
which requires an additional loop compared to the previous version, to
cope with the new WS format's dividing updates by query set. We define a
function `transaction_update_iter_table_updates` which encapsulates this
nested loop in an iterator.
- We have two new functions for parsing raw `QueryRows` into the
generated `DbUpdate` type, one for when they come from a
`SubscribeApplied`, and the other when they come from an
`UnsubscribeApplied`. `QueryRows` from `SubscribeApplied` translate to a
`DbUpdate` of all inserts, while one from `UnsubscribeApplied` will be
all deletes.
## Legacy subscriptions
"Legacy subscriptions" are removed. These were only used for
`subscribe_to_all_tables`, which as of now is stubbed. I will follow up
with a change to re-implement `subscribe_to_all_tables` by
code-generating a list of all known tables, and having it subscribe to
`select * from {table}` for every table in that list.
## `subscribe_to_all_tables` via a list
Previously, `subscribe_to_all_tables` worked by sending a legacy
subscription with the query `SELECT * FROM *`, which the host had
special handling to expand to subscribing to all tables. As legacy
subscriptions are no longer usable in V2 clients, this can't work.
Instead, we code-generate `SpacetimeModule::ALL_TABLE_NAMES`, a list of
all the known table names. `subscribe_to_all_tables` then maps across
this list to construct a list of queries in the form `SELECT * FROM
{table_name}`, and subscribes to all of those queries. This has the
upside that defining a new table in the module without regenerating
client bindings will no longer result in the client seeing rows of
tables it does not know about and cannot parse.
## Light mode removed
Light mode is no longer meaningful in the V2 WS format, so all code
related to it is removed.
## Internal changes
### Renamed WS messages
The SDK's internal code is updated to account for various renames:
- `QueryId` -> `QuerySetId`, `query_id` -> `query_set_id`.
- `SubscribeMulti` -> `Subscribe`, `UnsubscribeMulti` -> `Unsubscribe`.
## Incidental changes in this PR, not necessary for other client SDKs
### Don't filter out empty ranges in `RowSizeHint`
The Rust implementation of `RowSizeHint` in `BsatnRowList` got regressed
in the base branch to not work with zero-sized rows. This change fixes
that.
# API and ABI breaking changes
Boy howdy is it!
# Expected complexity level and risk
3? Changes ended up being less complicated than I feared, but we do have
some fiddly code here, and we have internal dependencies on the SDK.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Updated automated test suite.
- Known failures:
- [ ] `subscribe_all_select_star`, which is currently broken because
it's trying to subscribe to rows from private tables. #4241 will fix
this.
---------
Co-authored-by: Jeffrey Dallatezza <jeffreydallatezza@gmail.com>
Co-authored-by: = <cloutiertyler@gmail.com>
# Description of Changes
This adds the v2 websocket protocol and adds support on the server side.
For context on many of the changes/decisions, you can look at the
discussion on https://github.com/clockworklabs/SpacetimeDB/pull/4023.
To restate some of the key changes:
- The reducer event information is no longer sent with transaction
updates (because we don't want to broadcast reducer call information
anymore).
- If a client calls a reducer, they are sent a `ReducerResult` which
includes the outcome of the reducer call and and related row updates for
queries that the client is subscribed to.
- We no longer dedupe queries that appear in multiple query sets for the
same client. This is because we are moving toward per-query storage.
- Related to that, Unsubscribe requests have an option to send the
related rows. We need this for now, since clients don't have per-query
storage implemented yet.
- We don't have the json format in v2.
Notes for reviewers:
- This moves around the messages in
`crates/client-api-messages/src/websocket` (into `common`, `v1`, and
`v2`), and this renaming of existing messages adds a lot of noise to the
PR.
- In many places, I chose to duplicate a lot of code to have a v1
version and a v2 version. I went with this to make it easier to remove
the v1 version in the future (hopefully we can just fully delete most of
the v1 functions).
- `module_subscription_manager.rs` has probably has the biggest changes,
since we now track queries by query_set_id, and we get to remove some
complexity of v1's FormatSwitch.
<!-- Please describe your change, mention any related tickets, and so on
here. -->
# API and ABI breaking changes
The v1 protocol still works, though we won't send the reducer event info
for v10 modules.
# Expected complexity level and risk
4. This touches a lot of places.
# Testing
Unit testing is pretty minimal for the new code paths. I've done some
manual e2e testing with the typescript quickstart, and this has been
tested with a different branch implementing the v2 rust client.
---------
Co-authored-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: Jeffrey Dallatezza <jeffreydallatezza@gmail.com>
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
- Version upgrade to 2.0.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
1 - this is just a version bump
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] License file has been updated
- [x] CI passing
---------
Signed-off-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
Modules are like programs, databases are like processes. Your client
connects to a remote database, not a remote module.
<!-- Please describe your change, mention any related tickets, and so on
here. -->
# API and ABI breaking changes
Yep!
# Expected complexity level and risk
2? I made this change with find + replace, and it's possible that I
borked the edit, or missed some reference somehow. It seems unlikely,
however, that this change will have deeper implications we haven't
considered.
# Testing
N/a
# Description of Changes
Updated the codegen table/function iteration functions to take in a
parameter to check visibility in all locations for the supported
languages.
- Updated the util.rs functions for iterating tables/functions to check
for a CodegenVisibility enum (IncludePrivate, or OnlyPublic)
- Added a new CodegenOptions struct to pass around the CodegenVisibility
and future flags, defaulted visibility to OnlyPublic
- Updated the CLI to return a list of all private tables not included
(added a TODO to check the --include-private opt):
```bash
Optimising module with wasm-opt...
Build finished successfully.
Skipping private tables during codegen: secret_note, secret_order, secret_person.
Generate finished successfully.
```
# API and ABI breaking changes
Technically API breaking as the private tables will no longer be
available. (GitHub labels are not working at the moment)
# Expected complexity level and risk
1 - Simple change the testing took longer
# Testing
Turns out when you remove private tables you invalidate most of the
module_bindings across the system!
- [x] Rust test SDK for all languages
- [x] C# SDK tests
- [x] C# dotnet tests
- [x] Updated and checked snap files
- [x] Updated Blackholio (Unreal + Unity) module_bindings and tested
- [x] Ran Unreal SDK tests
---------
Signed-off-by: Jason Larabie <jason@clockworklabs.io>
NOTE: Cherry-picking
https://github.com/clockworklabs/SpacetimeDB/pull/4127 from the
`2.0-breaking-changes` branch.
## Original PR Description
This changes generated types in ts client bindings. We currently
generate a few different types of types: reducer args, procedure args,
rows, and user defined types. To avoid potential conflicts between these
types (for example, if a user defined a type called `FooRow`, and also
had a tabled named `foo`, we would end up with two types named
`FooRow`), this puts each set of types in a different file and
namespace. We also stopped exporting the `xxxRow` types, because there
is always another type generated for those. We now have a `types`
directory, which has an `index.ts` with user defined types, along with
`reducers.ts` and `procedures.ts` for the types generated for
reducer/procedure parameters.
```
import type * as Types from './module_bindings/types';
var currentMessages: Types.Message[] = [];
```
or
```
import { type Message } from './module_bindings/types';
var currentMessages: Message[] = [];
```
This has a couple other changes:
- For procedure and reducer types, this adds a suffix of `Args`, since
we may want types for the return values in the future.
- For all of the types, instead of exposing the schema object, we are
now giving the typescript type (e.g. `export type Message =
__Infer<typeof MessageRow>;`). I couldn't think of a reason for users to
want the schema object, so this should save users from needing to do all
of the `Infer` boilerplate.
This is a breaking change for v2.
2. This only changes typescript, and it should generally make thing
easier to use.
---------
Co-authored-by: Jeffrey Dallatezza <jeffreydallatezza@gmail.com>
# Description of Changes
Blocks procedures from requesting private ip ranges after dns
resolution.
Adds a new cargo feature to `spacetimedb-standalone` permitting loopback
http requests in test environments only.
# API and ABI breaking changes
None
# Expected complexity level and risk
2. I may have missed a range.
# Testing
- [x] Unit tests for IP address matching
- [x] Smoketests for blocking a private IP address
# Description of Changes
This is the implementation of the Typed Query Builder for C# modules
outlined in #3750.
This requires the changes from #4146 to be in place before it can
properly function.
This aligns the C# typed query builder SQL formatter with the Rust
implementation:
* Supports `And/Or` with same grouping styles as Rust.
* All comparison operators (`Eq/Neq/Lt/…`) wrap expressions in
parentheses so chained predicates produce byte-for-byte identical SQL.
# API and ABI breaking changes
Not breaking. Existing modules keep producing the same SQL semantics;
only redundant parentheses were added to match the Rust formatter.
# Expected complexity level and risk
2 — localized string-format updates plus parity harness tweaks. Low
functional risk; largest concern is ensuring every comparison path
gained parentheses.
# Testing
- [X] Tested against a local test harness validating output from these
changes match current Rust behavior.
---------
Signed-off-by: Ryan <r.ekhoff@clockworklabs.io>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
# Description of Changes
This adds C++ server bindings (/crate/bindings-cpp) to allow writing C++
20 modules.
- Emscripten WASM build system integration with CMake
- Macro-based code generation (SPACETIMEDB_TABLE, SPACETIMEDB_REDUCER,
etc)
- All SpacetimeDB types supported (primitives, Timestamp, Identity,
Uuid, etc)
- Product types via SPACETIMEDB_STRUCT
- Sum types via SPACETIMEDB_ENUM
- Constraints marked with FIELD* macros
# API and ABI breaking changes
None
# Expected complexity level and risk
2 - Doesn't heavily impact any other areas but is complex macro C++
structure to support a similar developer experience, did have a small
impact on init command
# Testing
- [x] modules/module-test-cpp - heavily tested every reducer
- [x] modules/benchmarks-cpp - tested through the standalone (~6x faster
than C#, ~6x slower than Rust)
- [x] modules/sdk-test-cpp
- [x] modules/sdk-test-procedure-cpp
- [x] modules/sdk-test-view-cpp
- [x] Wrote several test modules myself
- [x] Quickstart smoketest [Currently in progress]
- [ ] Write Blackholio C++ server module
---------
Signed-off-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: Ryan <r.ekhoff@clockworklabs.io>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
Jeff Rooks