# Description of Changes
Module launch now exposes a completion handle so callers can wait until
`st_module` is durable before cleaning up controldb bootstrap state.
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
Tests added in the corresponding private PR that deletes bootstrap state
from the controldb
## Compatibility note
This PR changes the generated TypeScript table/view handles for
snake_case module accessors to the target-language TypeScript accessor
casing, but keeps the old snake_case handles as deprecated aliases.
Generated TS clients now expose the intended TypeScript accessor casing:
- `ctx.db.databaseTree`
- `tables.loggedOutPlayer`
- `tables.myProgress`
The old snake_case handles continue to work and are marked deprecated in
the generated type surface:
- `ctx.db.database_tree` still works as an alias for
`ctx.db.databaseTree`
- `tables.logged_out_player` still works as an alias for
`tables.loggedOutPlayer`
- `tables.my_progress` still works as an alias for `tables.myProgress`
The database canonical names stay unchanged. For example, the generated
TypeScript handle is camelCase, but the table metadata still has `name:
'logged_out_player'`.
## Remaining API breakage risk
This should be non-breaking for normal generated TypeScript client
usage, including:
- direct table access through `conn.db.snake_case`
- callback contexts like `ctx.db.snake_case`
- exported query builders like `tables.snake_case`
- type inference for deprecated aliases
There are still a few edge cases where users may notice an API shape
change:
- Code that enumerates generated table handles with `Object.keys`,
`Object.entries`, `for...in`, or similar will now see both the
target-language handle and the deprecated snake_case alias.
- Code with pathological table/view accessor collisions may not get
every possible alias. The normal case is `logged_out_player` ->
generated handle `loggedOutPlayer` plus deprecated alias
`logged_out_player`. Pathological cases are shapes like:
- `foo_bar` and `fooBar`: both want the generated TypeScript handle
`fooBar`, so generated clients cannot provide two distinct
`tables.fooBar` entries.
- `foo_bar` and some other table/view whose target-language handle is
already `foo_bar`: the deprecated `foo_bar` alias for `foo_bar` ->
`fooBar` would shadow the other generated handle, so the generator skips
that alias.
- Code that compares the exact generated `index.ts` text, emitted
declaration text, or public type names will see new helper/types such as
`DbView`, `Tables`, and the alias metadata.
TypeScript modules themselves are not expected to break from this unless
they consume generated TypeScript client bindings or depend on exact
generated client object keys.
## Terminology
The casing proposal uses **canonical name** for the database/internal
name. That is language-independent. It uses **accessor name** for the
source/module/client-facing identifier that codegen derives
language-specific handles from.
This PR keeps database canonical names unchanged. It changes the
generated TypeScript accessor handles to match TypeScript casing while
retaining the old generated handles as deprecated aliases.
## Why
The TypeScript code generator was using `table.accessor_name` and
`view.accessor_name` directly as object keys in `tablesSchema`. That
preserved the raw module accessor spelling instead of applying the
target-language `Case::Camel` conversion for TypeScript handles. Per the
casing policy, client codegen should use the server accessor name as its
source and apply the target-language conversion.
## What changed
- Convert generated TypeScript table and view handle keys with
`Case::Camel`.
- Generate deprecated snake_case aliases for table/view handles when the
old generated handle differs from the target-language TypeScript handle.
- Keep runtime table metadata and database canonical names unchanged.
- Avoid duplicate runtime table definitions.
- Add a type-only aliased schema so callback contexts infer deprecated
aliases too.
- Add regression coverage for TypeScript-cased handles and deprecated
aliases.
- Update checked-in generated TS bindings and references that change
under this fix.
Generated code in this repo that changes as a result:
- `crates/bindings-typescript/test-app/src/module_bindings/index.ts`
-
`crates/bindings-typescript/test-react-router-app/src/module_bindings/index.ts`
-
`crates/bindings-typescript/test-solid-router/src/module_bindings/index.ts`
-
`crates/bindings-typescript/case-conversion-test-client/src/module_bindings/index.ts`
- `demo/Blackholio/client-ts/src/module_bindings/index.ts`
- `templates/hangman-react-ts/src/module_bindings/index.ts`
- `templates/money-exchange-react-ts/src/module_bindings/index.ts`
- `crates/codegen/tests/snapshots/codegen__codegen_typescript.snap`
I also updated the corresponding client/test references to the generated
TypeScript-cased handles.
## Verification
- `cargo fmt --all --check`
- `cargo test -p spacetimedb-codegen typescript`
- `pnpm --dir crates/bindings-typescript test --
tests/client_query.test.ts tests/db_connection.test.ts`
- `pnpm --dir crates/bindings-typescript exec vitest run
--typecheck.enabled tests/client_query.test.ts
tests/db_connection.test.ts`
- `git diff --check`
The explicit Vitest typecheck command still reports existing global
typecheck errors from unrelated files loaded by the test project,
including missing `spacetimesys@2.x` ambient modules and existing test
type issues, but the touched alias tests report `Type Errors no errors`
and the earlier generated-code `declare override` issue is gone.
---------
Signed-off-by: Ryan <r.ekhoff@clockworklabs.io>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Ryan <r.ekhoff@clockworklabs.io>
Addresses #5373. Hangs off the reconnect tracking issue #1936.
A SpacetimeDB web app commonly starts a user on an **anonymous**
connection and
then lets them **sign in** (e.g. via Google), at which point you want to
reconnect with the new token. The Svelte binding has no clean way to do
this: it
holds its connection in a **module-level singleton** reused across
remounts (so
even a `{#key}` swap keeps the old token), there's no reconnect path on
the
context value, and a `DbConnection`'s token is immutable after
`build()`. The
only thing that actually works today is `window.location.reload()`,
which throws
away all client state and flickers the UI on every sign-in/sign-out. The
Svelte
binding also lacks the exponential-backoff **auto-reconnect** that React
and
Solid already get from `ConnectionManager`.
This is two focused commits that close both gaps:
# Description of Changes
**1. `Add ConnectionManager.rebuild()` for token-swap reconnects.** The
shared
`ConnectionManager` (used by the React and Solid providers) can hold and
reference-count a connection but has no way to swap a live connection's
builder.
`rebuild(key, builder)` tears down the live connection and builds a
fresh one
from a new builder under the same ref count and listener set, so
framework hooks
(`useTable`, `useReducer`, …) re-bind automatically — swap the token in
the
builder, call `rebuild`, done. The old connection's callbacks are
detached
before it is closed (so its disconnect event can't leak into pool
state), any
deferred release or pending auto-reconnect is cancelled, and the backoff
counter
is reset. This is the `rebuild()` primitive from the open
multi-module-provider
PR #4887 (original design by @Ludv1gL), lifted into a focused change,
adapted to
the current post-#5185 reconnect machinery, and reusing the existing
`#buildManagedConnection` / `#detachCallbacks` helpers rather than that
PR's
`#install` / `#teardown` refactor.
**2. Put the Svelte binding on `ConnectionManager`; add `reconnect()`
for token
swaps.** `createSpacetimeDBProvider` is rebuilt on top of the shared
`ConnectionManager` — the same pool React and Solid already use —
replacing the
bespoke module-level singleton with the manager's reference counting and
deferred cleanup (which absorb HMR / `{#key}` remounts), and getting
exponential-backoff auto-reconnect for free. The context value gains
`reconnect(builder)` (backed by the new `ConnectionManager.rebuild`):
tear down
the current connection and reconnect with a fresh builder carrying a new
token,
no page reload. `useTable` / `useReducer` / `useSpacetimeDB` are
unchanged and
re-bind to the new connection automatically.
# API and ABI breaking changes
Minor, TypeScript-only: `ConnectionState.getConnection` on the Svelte
binding is
no longer generic — it returns `DbConnectionImpl<any> | null`, matching
the
React and Solid bindings. Call sites that wrote
`getConnection<MyConn>()` lose
the type argument; `getConnection()` is unchanged. The provider's return
type
stays `Writable<ConnectionState>`. `ConnectionManager.rebuild` is purely
additive.
# Expected complexity level and risk
2. Low. `rebuild()` is a small method reusing the same build/teardown
helpers as
`retain`/`release` and the auto-reconnect path; the only subtlety is
ordering
(detach-then-close + cancel pending timers so a stale disconnect or
scheduled
auto-reconnect can't race the new connection). The Svelte provider is
now a thin
adapter: it mirrors the manager's store into a Svelte store and wires up
`getConnection` / `reconnect` / `retain` / `release`. The
connection-lifecycle
logic that used to be bespoke here — the singleton, deferred cleanup,
reconnect
— now lives in the well-tested, shared `ConnectionManager` that React
and Solid
already depend on.
# Testing
- [x] Six new unit tests against the real `ConnectionManager` singleton
(`tests/connection_manager_reconnect.test.ts`): builder swap, ref-count
preservation, callback detachment, pending-reconnect cancellation +
backoff
reset, and both `null`-return cases.
- [x] Full vitest suite green; `tsc -p tsconfig.build.json` typechecks;
`build:js` emits the Svelte ESM/CJS + browser bundles cleanly; eslint +
prettier clean.
- [x] Manually smoke-tested end-to-end in a real Svelte app (anonymous →
Google sign-in → signed-in): the token swaps and the board state
survives
with **no page reload**, and sign-out reconnects anonymously the same
way.
- [ ] Reviewer: this package has no Svelte component-test harness today
(the
pre-existing binding had none), so the provider wiring is covered
indirectly
through the `ConnectionManager` unit tests. Worth a call on whether to
add
`@testing-library/svelte` coverage here. Vue has the same singleton
shape and
can follow this pattern as a follow-up.
# Description of Changes
Specifically:
- The number of objects read during replay
- The number of bytes read from disk during replay
- The time spent hashing objects during replay
# API and ABI breaking changes
Not considered breaking, but this patch does add a new label to a
pre-existing metric `spacetime_replay_snapshot_read_time_seconds`.
# Expected complexity level and risk
1.5
# Testing
Updates a pre-existing test
# Description of Changes
Uses `jsonwebtoken v10.4.0` instead. Important changes include:
**1. Token serialization**
Old tokens with `"exp": null` are still accepted, but new no-expiry
tokens now omit `exp` instead of serializing it as `"exp": null`.
**2. OIDC/JWKS validation**
Issuer extraction now uses `jsonwebtoken::dangerous::insecure_decode`
for key discovery only, not validation. And the old `spacetimedb-jwks`
crate required every JWK to have a `kid`, but this patch does not
preserve that limitation.
# API and ABI breaking changes
I don't believe this is considered breaking, but it bears repeating that
new no-expiry tokens now serialize without `exp` instead of `"exp":
null`.
# Expected complexity level and risk
2
# Testing
- [x] Verify a legacy no-expiry token serialized as `"exp": null` still
validates.
- [x] Verify a token with an expired `exp` is still rejected.
- [x] Verify OIDC/JWKS validation works when the JWKS keys omit the
optional `kid` field.
# Description of Changes
Fixes#5407.
A one-column prefix scan on a multi-column btree index — e.g.
`filter(1n)` on a
`[u64, string]` index — panicked at runtime with `TypeError:
serializeTerm is
not a function` inside `serializeRange`.
A bare scalar (or bare `Range`) has no `.length`, so in the multi-column
`filter`/`delete` accessor `range.length === numColumns` was `undefined
=== 2`
(falling into the range-scan branch), and inside `serializeRange`:
- `prefix_elems = range.length - 1` → `NaN` (prefix loop skipped),
- `serializeTerm = indexSerializers[range.length - 1]` →
`indexSerializers[NaN]`
→ `undefined`,
- `serializeTerm(writer, term)` → **`TypeError: serializeTerm is not a
function`**.
A bare scalar is the only *type-valid* way to express a one-column
prefix on a
multi-column index — `filter([1n])` is rejected by the generated types
and
`filter([1n, "x"])` is the full key — so there was previously **no**
working way
to do a one-column prefix scan.
The fix normalizes a non-array `range` to a single-element array at the
top of
the multi-column `filter`/`delete` before reading `.length`, so a bare
scalar or
`Range` is treated as a one-column prefix and takes the range-scan
branch
correctly. The full two-column key path is unchanged and still takes the
point-scan branch.
To regression-test the pure-JS index accessor under vitest,
`makeTableView` is
now exported from `src/server/runtime.ts` (it is **not** re-exported
from any
public entry point), the host-injected `spacetime:sys@*` virtual modules
are
stubbed, and aliased in `vitest.config.ts`.
# API and ABI breaking changes
None. `makeTableView` is newly exported from `src/server/runtime.ts` but
is not
re-exported from any public entry point. The host ABI is unchanged.
# Expected complexity level and risk
1 — a one-line input normalization on each of `filter`/`delete`, plus
test-only
scaffolding. No change to the full-key path or the host syscall
arguments.
# Testing
- [x] `npx vitest run tests/index_prefix_filter.test.ts` — bare scalar,
bare
`Range`, full two-column key, and `delete()` all scan without throwing
(4 passed).
- [x] `npx vitest run` — full suite, 223 passed.
- [x] `tsc -p tsconfig.build.json` and `prettier --check` clean.
- [x] Reviewer: confirm the prefix-scan semantics against a real
datastore
(the unit test uses a stubbed host iterator that yields no rows).
## Summary
Adding `#[unique]` or `#[primary_key]` to an existing column currently
triggers `AutoMigrateError::AddUniqueConstraint`, forcing a full
database clear to apply the schema change. This PR makes it a
non-breaking migration by validating existing data first:
- **If all values are unique**: constraint is added seamlessly
(non-breaking migration)
- **If duplicates exist**: migration fails with a detailed error listing
up to 10 duplicate groups
## Changes
- `auto_migrate.rs`: Replace hard `AddUniqueConstraint` error with
`CheckAddUniqueConstraintValid` precheck + `AddConstraint` migration
step
- `update.rs`: Implement precheck (full table scan, project constrained
columns, count duplicates) and `AddConstraint` step execution
- `relational_db.rs`: Expose `create_constraint()` (counterpart to
existing `drop_constraint()`)
- `traits.rs` / `datastore.rs`: Add `create_constraint_mut_tx` to
`MutTxDatastore` trait
- `mut_tx.rs`: Make `create_constraint` public
- `formatter.rs`: Format the new `AddConstraint` step
## Safety
- **Transaction safety**: Precheck and constraint creation run in the
same `MutTx` — no window for concurrent duplicate inserts
- **Index creation**: `auto_migrate_indexes()` already handles adding
the backing btree index (with `is_unique=true` from the new schema). The
constraint step only adds metadata.
- **Rollback**: If the precheck finds duplicates, the entire migration
aborts before any changes are applied
- **Error quality**: Duplicate error shows table name, column names, and
up to 10 example duplicate values with counts
## Example error output
```
Precheck failed: cannot add unique constraint 'Users_email_key' on table 'Users' column(s) [email]:
3 duplicate group(s) found.
- String("alice@example.com") appears 2 times
- String("bob@example.com") appears 3 times
- String("charlie@example.com") appears 2 times
```
## Test plan
- [x] All 12 `auto_migrate` tests pass
- [x] `cargo check` passes for `spacetimedb-schema` and
`spacetimedb-core`
- [x] Verified the previously-expected `AddUniqueConstraint` error test
is updated
- [x] Manual test: add `#[unique]` to existing column with clean data →
succeeds
- [x] Manual test: add `#[unique]` to existing column with duplicates →
fails with detailed error
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
In order to be able to suspend databases based on memory usage.
Reports memory usage for wasm linear memory as well as the v8 heap. Also
reports page-level memory for tables.
It does not report memory usage for indexes.
# API and ABI breaking changes
None
# Expected complexity level and risk
1.5
# Testing
Testing added in the private counterpart which handles enforcement of
memory limits.
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
- Bumps version to 2.7.0
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
- 1 - this is just a version bump
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Version number is correct (`2.7.0`)
- [x] BSL license file has been updated with the new date and version
number
# Description of Changes
Adds a Prometheus counter for client disconnects caused by the outgoing
WebSocket message queue reaching capacity.
The new metric is
`spacetime_client_outgoing_queue_disconnects_total{db=...}`. It
increments only on the `TrySendError::Full` path that kicks a client
after its bounded outgoing queue fills.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1. This is a narrow observability change: one metric definition and one
increment at the existing kick site.
# Testing
- [x] `cargo fmt --all`
- [x] `LC_ALL=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 LANG=en_US.UTF-8 cargo
check -p spacetimedb-core`
- [x] `git diff --check`
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
- Added a client-side query builder for the Unreal SDK
- Added typed query-builder subscriptions for Unreal C++ via
SubscriptionBuilder.AddQuery(...).Subscribe()
- Added generated Blueprint query-builder support with source query
nodes, column nodes, predicates, Where, AddQuery, and Subscribe
- Added Blueprint autocasts at the AddQuery boundary so source-specific
Blueprint queries can connect cleanly into the generic subscription
builder for better devex in Blueprint
- Synced the Unreal query-builder core with the C++ module
implementation
- Updated the copied Unreal core query-builder headers to match the
shared C++
- Moved Unreal-specific literal/type adapters out of the shared core
copy and into Unreal-specific expansion code
- Added Unreal SDK test coverage and documentation for the new
query-builder surface
- Added new test harnesses to start to match the View/ViewPk tests
- Added documentation for the Unreal query builder
- Refactor generation for modules with more than 255 reducers, required
to get the TestClient back in working order
# API and ABI breaking changes
- No intended API or ABI breaking changes to released Unreal SDK
behavior
- Adds a new public client-side query-builder API to the Unreal SDK for
C++ and Blueprint
- Raw SQL subscriptions remain available
# Expected complexity level and risk
3 - Adds new query-builder surface with large changes to the code-gen,
the risk here is keeping the C++ module core mirrored to the SDK in the
future
# Testing
What I've done so far:
- [x] Updated and ran the following tests:
- [x] TestClientEditor
- [x] TestViewClientEditor
- [x] TestViewPkClientEditor
- [x] Ran the Unreal harness suites covering the query-builder client
paths
- [x] Verified the generated Unreal query-builder surface in both C++
and Blueprint testing supported fields with a small sample project
Should be done at least once by a reviewer:
- [ ] Re-run the full `sdk-unreal-test-harness` as it's disabled in CI
- [ ] Play with the C++ and Blueprint versions in a small Unreal project
---------
Signed-off-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: Ryan <r.ekhoff@clockworklabs.io>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
# Description of Changes
Adds primary keys to procedural views in C++. This mirrors the work from
#5111, #5246, and #5327 adding the feature and the docs changes.
# API and ABI breaking changes
None
# Expected complexity level and risk
3
# Testing
- [x] Equivalent tests as were added in #5111 and #5246 for rust,
typescript, and C#
# Description of Changes
Was forgotten while migrating to the QueryBuilder i suppose.
Closes#5073
Maybe @clockwork-tien could lend a hand again in reviewing? ^^
# API and ABI breaking changes
None
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
1
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
Works in my project :>
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Type Error goes away when using .eq(uuid)
---------
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
# Description of Changes
Removes all references to `Promise.withResolvers` from the codebase
since it's not supported universally and replaces it with the classic
pre-ES2024 deferred pattern. See #5031 and #5342.
Also adds a lint to avoid re-introducing it in the future.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
...
# Description of Changes
Refactors the Rust smoketest helper surface so publish and subscribe
variants use builders instead of parallel helper methods.
- Replaces `publish_module*` helper variants with `Smoketest::publish()`
and fluent options for name, clear, current database, break clients,
stdin, replicas, organization, and source modules.
- Replaces `subscribe_*` / `subscribe_background*` variants with
`Smoketest::subscribe(...).expect_rows(...).confirmed(...).background()`.
- Updates smoketest call sites to the builder APIs.
# API and ABI breaking changes
Internal smoketest helper API change only. No product API or ABI
changes.
# Expected complexity level and risk
2
This touches many smoketest call sites, but the underlying CLI command
behavior remains centralized in the same helper internals.
# Testing
- [x] Existing CI passes
---------
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
# Description of Changes
This is an attempt at fixing a SIGABRT that sometimes happens when
running the standalone integration tests.
It's not clear exactly what causes it, however one thing that may have
contributed to it was that previously the tests did not initiate a clean
shutdown of the database.
Some modules schedule repeating work in `init`, and it's not obvious to
me that in flight operations will exit cleanly if we just drop all of
the database/runtime handles at once.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
1
# Testing
This is a testing fix.
# Description of Changes
- spacetime init --template without arg prints templates list and link
to website
# Screenshot
<img width="696" height="392" alt="screenshot"
src="https://github.com/user-attachments/assets/98e87537-554b-411b-96ab-3ceb9a6a9d45"
/>
<!-- Please describe your change, mention any related tickets, and so on
here. -->
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
1
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] I tested the changes
# Description of Changes
Closes#5250#4636 introduced a regression where the ProcedureContext used to include
the sender and connection_id from the caller while now it is always
empty (which is wrong)
Correct it.
Also fully migrate to `database_identity` which was forgotten about so i
deprecated it for procedures (since they are now stable) and just
changed it for HttpHandlers (because they are still unstable)
@gefjon since you did the lil woopsie (ugh pinging you again lol hope im
not annoying haha)
# API and ABI breaking changes
Breaking the HttpHandler function which is unstable.
Restoring behaviour of 2.3 which got lost with 2.4.
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
1. Trivial refactoring
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] The caller identity is there again for Procedures.
---------
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
# Description of Changes
Attempted to fix this test in #5343, but we're still getting SIGKILLS,
so ignoring for now. This will require more investigation to fix. I've
included what I believe is the reason in the help text and created
[this](#5362) tracker to unskip.
# API and ABI breaking changes
None
# Expected complexity level and risk
0
# Testing
N/A
# Description of Changes
Use an isolated server process per SDK test instead of a single process
for all of the tests. In addition to reducing the memory footprint of
each test run, this should also allow for more parallelism among the
individual tests.
# API and ABI breaking changes
None
# Expected complexity level and risk
1.5
# Testing
The SDK test suite should continue to work
# Description of Changes
Moves `RelationalDB` and related database code into a new
`spacetimedb-engine` crate.
The main motivation is to tighten dependency control around the engine
layer and isolate `RelationalDB`
behind a crate boundary.
- Majority of this PR is code-motion.
- Removes direct production dependence on `tokio` from
`spacetimedb-engine`.
- Keeps `tokio` only as a dev-dependency for test-only code in
`spacetimedb-engine`.
- This is intended to be a structural refactor only and should not
result in any functional change in
production.
- Adds a CI check to ensure `spacetimedb-engine` continues to compile in
simulation mode
# API and ABI breaking changes
NA
# Expected complexity level and risk
1.5.
# Testing
Existing tests should be enough.
---------
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
Fix TypeScript inference for generated object shapes containing
`Option<T>` fields.
Previously, generated TypeScript represented an optional SATS field as a
required object key whose value could be `undefined`:
```ts
{
foo: string | undefined;
}
```
This PR changes those generated shapes to make the key itself optional:
```ts
{
foo?: string | undefined;
}
```
The original issue was reported in #4516: generated TypeScript
reducer/procedure calls required users to pass explicit `undefined`
values for omitted optional arguments. This PR fixes that for reducer
and procedure params, and also applies the same optional-key inference
to generated row shapes.
This is a fresh bot-owned replacement for stalled PR #4518.
## Examples
A reducer with an optional argument now generates a TypeScript call
shape where the optional key may be omitted:
```rust
#[spacetimedb::reducer]
pub fn update_category(
_ctx: &spacetimedb::ReducerContext,
id: u64,
name: String,
button_text: Option<String>,
) {
// ...
}
```
Before this PR, generated TypeScript callers had to pass the optional
argument explicitly:
```ts
await conn.reducers.updateCategory({
id: 1n,
name: 'updated category name',
buttonText: undefined,
});
```
After this PR, generated TypeScript callers can omit the optional key:
```ts
await conn.reducers.updateCategory({
id: 1n,
name: 'updated category name',
});
```
A table row with an optional column also changes its generated
TypeScript row shape:
```rust
#[spacetimedb::table(name = player, public)]
pub struct Player {
#[primary_key]
pub id: u64,
pub display_name: String,
pub alias: Option<String>,
}
```
Before this PR, generated TypeScript treated `alias` as a required key:
```ts
type Player = {
id: bigint;
displayName: string;
alias: string | undefined;
};
```
After this PR, generated TypeScript treats `alias` as an optional key:
```ts
type Player = {
id: bigint;
displayName: string;
alias?: string | undefined;
};
```
# API and ABI breaking changes
This is a TypeScript source/API breaking change for generated type
shapes that contain `Option<T>` fields. It is not an ABI or wire-format
break: SATS encoding and decoding of `Option<T>` are unchanged.
The breaking edge is structural TypeScript code that requires optional
SATS fields to exist as object properties. For example, code like this
may stop compiling:
```ts
type RequiresAliasProperty = {
id: bigint;
displayName: string;
alias: string | undefined;
};
function renderPlayer(player: RequiresAliasProperty) {
return player.alias ?? player.displayName;
}
conn.db.player.onInsert((_ctx, player) => {
renderPlayer(player);
});
```
With this PR, the generated `player` row has `alias?: string |
undefined`, so it is not assignable to a type requiring an `alias`
property.
Code using `Required<GeneratedRow>`, explicit generated-row mirror
interfaces, or generic constraints that require optional SATS fields to
be present as object keys may need to loosen those keys to optional
properties.
Reducer and procedure params are primarily loosened by this change.
Existing calls that pass `undefined` explicitly should continue to
typecheck, while calls can now omit optional keys.
# Expected complexity level and risk
2.
The change is contained to TypeScript type inference and generated
TypeScript test coverage. The main risk is source compatibility for
TypeScript consumers that depend on the old required-key shape for
`Option<T>` fields.
# Testing
- [x] `pnpm --filter @clockworklabs/test-app build`
- [x] `pnpm test`
- [x] targeted `prettier --check`
- [x] `git diff --check`
Closes#4516.
---------
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
Fixes an OOM kill in the proptest `bsatn_invalid_wont_decode`.
`bsatn_invalid_wont_decode` generates arbitrary invalid bytes, proves
validation fails, then still calls full AlgebraicValue::decode. For
generated array-like types, decode reads a u32 length prefix, and the
generic array visitor then reserves that capacity. But because they're
random bytes, this could cause a huge initial allocation which could OOM
kill the test process.
Now the visitor reserves a smaller initial capacity instead of assuming
the binary input data is well formed.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
1
# Testing
This should fix the flaky `spacetimedb-sats` `Test Suite` failures that
occasionally end in a SIGKILL.
# Description of Changes
Review #5287 first.
This patch updates view backing table schemas to have a single private
column `arg_hash`. Previously sender-scoped views had a `sender` column
for the calling identity, however now that's been replaced with a single
unified `arg_hash` column that encodes the calling identity within it.
When we add parameterized views, the view args will also be encoded in
this hash and stored in this column.
This column exists for both anonymous and sender scoped views meaning
that the backing tables for all views now have the same number of
private columns - one.
This hash is now used as a runtime variable that the query engine uses
to evaluate view table scans.
In order to keep the diff small, this patch does update view read sets
with this new hash value. That change has a larger blast radius and will
be done in the next set of changes.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
2
# Testing
Existing coverage.
# Description of Changes
* Bump version to 2.6.0
# API and ABI breaking changes
None
# Expected complexity level and risk
* 1 - this is just a version bump
# Testing
- [X] Version number is correct (`2.6.0`)
- [X] BSL license file has been updated with the new date and version
number
# Description of Changes
Makes runtime parameters explicit in query plans (prerequisite for
parameterized views).
As part of this change, `sender` is no longer baked directly into query
plans as a literal value. Instead it is represented as a parameter in
the query plan. Values are supplied at runtime via a variable
environment called `ExecutionParams`.
Note, parameterized plans are still not shared across subscriptions yet.
That will be done in a follow up.
This is mostly a mechanical change. The majority of the diff is just
threading runtime params/variables through various call sites.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
...
# Testing
Existing coverage
# Description of Changes
Runtime specific changes from
https://github.com/clockworklabs/SpacetimeDB/pull/5113.
- `spacetimedb-runtime` enforces exactly one feature at compile time:
`tokio` or `simulation`. It rejects both-enabled and neither-enabled
builds explicitly.
- This also keeps `tokio::sync` re-exported from runtime, look at code
comment for reasoning.
# API and ABI breaking changes
NA
# Expected complexity level and risk
1
# Description of Changes
So as not to race with the manual snapshots in these tests.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
1
# Testing
Fixes a flaky test
# Description of Changes
Moves `test_index_scans` to its own job that uses the same runner as the
keynote benchmark.
This test had several issues:
1. It was a performance regression test that didn't run in an isolated
environment because it was just a test.
2. It measured timings by search the module log for `ns`(nanosecond) and
`us`(microsecond) suffixes
As a result it would occasionally flake.
Now it runs on dedicated hardware in an isolated environment, so we
shouldn't see anymore flakes.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
1
# Testing
N/A
# Description of Changes
Follow-up to #5288. Stacked on that branch; only the last commit is new.
#5288 fixed `drop_table` to delete a dropped event table's
`st_event_table` row, but databases which already dropped an event table
on an unfixed version still carry an orphaned `st_event_table` row in
their commitlogs and snapshots. This PR adds
`fixup_delete_orphaned_st_event_table_rows` to
`rebuild_state_after_replay`, following the existing
`fixup_delete_duplicate_system_sequence_rows` pattern: after replay,
delete any `st_event_table` row whose table id has no row in `st_table`.
This heals affected databases on their next restart, with no migration
or operator action required. The fixup is a no-op for databases without
orphans, and skips cleanly when `st_event_table` does not exist yet
(histories from before event tables were introduced, where
`migrate_system_tables` creates it later).
# API and ABI breaking changes
None.
# Expected complexity level and risk
1. Same shape as the existing system sequence fixup; runs once per
database open over two small system tables.
# Testing
- [x] New unit test `test_fixup_deletes_orphaned_st_event_table_rows`:
creates a real event table, replays an orphaned `st_event_table` row for
a nonexistent table id (simulating a commitlog written by an unfixed
version), runs the fixup, and asserts the orphan is deleted while the
extant event table's row is preserved.
- [x] Verified end-to-end against a data directory produced by stock
2.5.0 which dropped an event table (and was bricked by the replay bug):
opened with a binary built from this branch, the database replays, the
orphaned `st_event_table` row is gone, and user data is intact.
---------
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
# Description of Changes
- Added a query builder for C++ module bindings
- Added query-builder table/filter/join types
- Added semijoin support with compile-time checks for lookup-table and
indexed-field usage
- Added support for returning query-builder queries from C++ views
- Hooked query-builder metadata into the C++ table/view macros and V10
module-def path
- Added test coverage for the new C++ query-builder behavior
- Compile tests for pass/fail cases
- SQL tests for generated query output
- Added a C++ test module for view primary key coverage
- **Update:** Switched the core to pass the columns and index-columns
metadata with the table source for better client-side codegen to have
some shared code between server + client.
# API and ABI breaking changes
- No intended API or ABI breaking changes
- Adds a new public query-builder API to the C++ bindings
- C++ views can now return query-builder query types in addition to
materialized row results
# Expected complexity level and risk
3 - Mostly contained to C++ bindings, but it touches macros, view
registration/serialization, and module-def generation, so there are a
few places where the pieces need to stay in sync.
# Testing
I've done end to end testing of I think every type as well as built some
tests to confirm the SQL output.
- [x] Run the C++ query-builder SQL tests
[crates/bindings-cpp/tests/query-builder-compile/run_query_builder_compile_tests.sh]
- [x] Smoke test a generated C++ module using query-builder views
---------
Signed-off-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: Ryan <r.ekhoff@clockworklabs.io>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
# Description of Changes
Required before we add formal parameters to query plans which in turn is
required to support parameterized views.
Before this change, we cached two types of query representations for a
subscription - a physical plan and what we call a pipelined executor.
This was mainly an artifact of built up technical debt, so this change
removes the physical plan and only caches the actual executor for the
subscription, in preparation for adding bind variables that are
substituted by the executor at runtime.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
...
# Testing
Existing coverage
# Description of Changes
Fix TypeScript SDK React provider recovery when the underlying WebSocket
closes or reports an error while a `SpacetimeDBProvider` is still
mounted.
- Mark `DbConnection` inactive before emitting `disconnect` /
`connectError`, so lifecycle callbacks observe the closed state.
- Teach the React `ConnectionManager` to clear closed retained provider
connections and rebuild them after a short delay.
- Cancel pending reconnects during provider release so React StrictMode
cleanup does not create replacement connections.
- Detach manager callbacks from the old connection before reconnecting,
preventing stale closed connections from updating provider state.
- Use the latest same-key retained builder when a replacement connection
is needed.
- Add focused lifecycle/reconnect regression tests and document that
this recovery is provider-level only; direct `DbConnection` users still
own reconnection.
# API and ABI breaking changes
None. This does not remove or change public APIs. It changes React
provider lifecycle behavior so retained provider connections recover
after close/error.
# Expected complexity level and risk
3/5.
The change is localized to the TypeScript SDK, but it touches async
connection lifecycle behavior, timer cancellation, React provider
reference counting, and callback-visible connection state. Main risks
are duplicate reconnects, stale connection callbacks updating state,
using an outdated retained builder for replacement connections, or
StrictMode release/remount regressions. The implementation guards
against stale callbacks, detaches manager callbacks from closed
connections, and cancels reconnect timers during release.
# Testing
- [x] `pnpm --dir crates/bindings-typescript exec vitest run
tests/db_connection.test.ts tests/connection_manager.test.ts
tests/connection_manager_reconnect.test.ts`
- [x] `pnpm --dir crates/bindings-typescript lint`
- [x] `pnpm --dir crates/bindings-typescript build`
- [x] `pnpm --dir crates/bindings-typescript test`
# Description of Changes
Fixes two bugs around dropping event tables.
**Bug 1:** a deterministic commitlog replay failure introduced in #5269
that permanently prevents a database from restarting after an event
table is dropped. Observed in production on 2.5.0:
```
ERROR ... Failed to open database: DatastoreError: Error deleting row ProductValue { elements: [U32(4134), U16(0), String("account_id"), Array([13])] } from table "st_column" during transaction 26002 playback: ... TableError: Table with ID `4134` not found in `st_table`.
```
## Root cause
Dropping an event table (a `RemoveTable` automigration) deletes its
`st_table`, `st_column`, and `st_event_table` rows in a single
transaction. `prepare_tx_data_for_durability` sorts delete ops by
ascending table id, so replay applies them in this order:
1. The `st_table` (table id 1) delete is replayed first. The table is
recorded in `replay_table_dropped` and its `st_table` row is removed.
2. The `st_column` (table id 2) deletes are replayed next. The handling
added in #5269 calls `is_event_table_for_replay`, which still finds the
table's `st_event_table` (table id 17) row, since that delete has not
been replayed yet.
3. `st_column_changed` is invoked to refresh the table's layout. It
calls `find_st_table_row`, which fails because the `st_table` row is
already gone.
Production replay runs with `ErrorBehavior::FailFast`, so the replica
fails to launch on every restart. The commitlog itself is intact:
databases bricked by this bug recover with no data loss once opened with
this fix.
#5269 tested in-place event table reschemas (which keep the `st_table`
row alive) but not drops.
## Fix
Skip the layout refresh when the referenced table was dropped earlier in
the same transaction, as tracked by `replay_table_dropped`. That set is
the existing mechanism for exactly this replay ordering hazard, and the
`st_table` delete is always replayed before the `st_column` deletes due
to the table id sort, so the guard is reliable.
**Bug 2:** `drop_table` never deletes the table's `st_event_table` row.
It cleans up `st_table`, `st_column`, `st_index`, `st_sequence`,
`st_constraint`, the accessor tables, and `st_scheduled`, but
`st_event_table` was missed when event tables were introduced. Dropping
an event table therefore leaves an orphaned `st_event_table` row,
verified empirically:
```
> SELECT * FROM st_event_table -- after dropping the only event table
table_id
----------
4097
> SELECT table_id FROM st_table WHERE table_id = 4097
(no rows)
```
The orphan is latent today (`st_event_table` is only read via point
lookups by live table ids), but it is the precondition that exposed bug
1, and it is incorrect catalog state. `drop_table` now deletes the row.
Note that both fixes are required independently: the replay guard is
needed even with the `drop_table` fix, because the `st_event_table`
delete replays after the `st_column` deletes (table id 17 vs. 2), and
also because commitlogs already written by unfixed versions contain drop
transactions with no `st_event_table` delete at all. Databases which
already dropped an event table on an unfixed version keep their orphaned
row; cleaning those up retroactively is left for a follow-up (e.g. a
fixup in `rebuild_state_after_replay`, like
`fixup_delete_duplicate_system_sequence_rows`).
# API and ABI breaking changes
None.
# Expected complexity level and risk
1. A guard on an existing replay invariant, plus tests.
# Testing
- [x] New unit regression tests `replay_event_table_drop_no_snapshot`
and `replay_event_table_drop_after_snapshot` in
`crates/core/src/db/update.rs`, following the structure of the #5269
replay tests (create event table, write an event row, drop the table via
`RemoveTable` automigration, reopen to force replay). Verified both fail
with the production error signature before the fix and pass after:
```
---- db::update::test::replay_event_table_drop_no_snapshot stdout ----
Error: DatastoreError: Error deleting row ProductValue { elements:
[U32(4096), U16(0), String("payload"), Array([13])] } from table
"st_column" during transaction 3 playback
...
1: TableError: Table with ID `4096` not found in `st_table`.
```
- [x] The pre-existing #5269 replay tests
(`replay_event_table_schema_change_*`) still pass.
- [x] New smoketest
`automigrate_drop_event_table_replays_after_restart`: publishes a module
with an event table, emits an event, drops the table via automigration,
restarts the server, and verifies the database replays and serves reads
and writes. Passes locally.
- [x] Reproduced the production failure on a stock 2.5.0 standalone
(publish module with event table, republish without it, restart,
identical error), then opened the same bricked data directory with a
standalone built from this branch and verified the database replays and
serves queries with all data intact.
# Description of Changes
Previously, if the server closed the connection, it was not considered
an error. The reason for this was because `-n` did not always mean the
connection should receive **exactly** `n` updates. Sometimes it meant
that the connection should receive **at most** `n` updates. This was the
case for client disconnecting auto-migration tests.
However, as a consequence, if the server were to close the connection
for some other reason - let's say it crashed before broadcasting all `n`
updates - the cli would not report this as an error. If this happened
during a smoketest, it would fail on some later assertion, but it
wouldn't be immediately obvious why.
`spacetime subscribe -n N` now fails if the websocket closes before it
receives `N` transaction updates. `subscribe_background(..., n)` also
fails if the CLI exits successfully but produced fewer or more than `n`
JSON update lines, where previously it did not.
This change, makes it very clear when this happens that the connection
to the server was closed before the test received its expected number of
updates.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
The purpose of this change is to provide more diagnostic info for
smoketest failures.
---------
Signed-off-by: joshua-spacetime <josh@clockworklabs.io>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
# Description of Changes
Refactors index probes so that multi-column index keys are represented
as a single product value expression `PhysicalExpr::Product` instead of
being split across various fields and structs. As a result, this patch
also simplifies the index-scan and index-join query executor variants
now that index scans/probes share one physical shape.
This change is in preparation for adding formal parameters to query
plans. Since index probe values now have single unified representation
as a `PhysicalExpr`, a future parameterized plan can represent an index
scan or index join as follows:
```rust
IndexProbe::Point(PhysicalExpr::Product(vec![
PhysicalExpr::Param(sender_slot),
PhysicalExpr::Value(other_const),
]))
PhysicalExpr::Product(vec![
PhysicalExpr::Param(sender_slot),
PhysicalExpr::Field(lhs_join_field),
])
```
This will avoid having to duplicate parameter handling in a bunch of
different places or rules.
Note, a very nice consequence of this refactor is that it
removes/consolidates a significant amount of code as the diff shows.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
...
# Testing
Existing coverage
# Description of Changes
Makes C# consistent with the other module languages.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
1
# Testing
Unit test
# Description of Changes
This fixes a scheduler crash where a panic from a scheduled JS
reducer/procedure could unwind out of `SchedulerActor::handle_queued`.
The fix keeps the existing panic semantics through `ModuleHost`, so
`defer_on_unwind` still runs and poisons the failed module host, but the
scheduler now catches the unwind after that boundary, logs a warning,
and returns without rescheduling that queue item.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
1
# Testing
...
# Description of Changes
This commit adds special handling in automigrations to accept a broad
set of schema- and layout-altering automigrations on event tables,
including changes that we'd reject for non-event tables like removing or
reordering columns, or layout-incompatibly changing the types of
existing columns.
This is due to a change we want to make to the ControlDB schema, where
we have a product type which is used both as a table type, and as the
column type of an event table, and we want to add an element to that
product type.
I've added a simple smoketest of the new behavior,
`automigrate_reschema_event_table_arbitrarily`. Note that I have not
tested commitlog replay of a database which has undergone one of these
migrations, which has been a common place that bugs have appeared in
similar changes in the past.
I have done a pretty lazy job with the migration plan formatter for the
new step, just printing the name of the event table which is changing,
not any information about the specific changes or the columns. This is
in an effort to save time, as we'd like to release the ControlDB change
blocked by this.
# API and ABI breaking changes
N/a
# Expected complexity level and risk
3 at least: new categories of automigrations have a high risk to
introduce commitlog replay bugs, and it's also possible I have
misunderstood or mis-remembered some of the safety invariants of the
`table` crate.
# Testing
- [x] New (minimal) smoketest.
- [x] Manually tested replay:
- Copied event table from the before version new smoketest into
module-test.
- Published module-test.
- Replaced event table with after version, published, observed and
approved client-breaking automigration.
- Restarted SpacetimeDB.
- Called the procedure `return_value` in the migrated module to force
commitlog replay.
- [ ] Test the ControlDB change in staging, incl. restarting the control
node.
---------
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
# Description of Changes
It turns out that different templates have different behavior for
whether they get a major+minor version constraint, a major+minor+patch
version constraint, or, surprisingly, just a major version constraint.
See https://github.com/clockworklabs/SpacetimeDB/issues/5229 for a bit
more detail.
This PR brings them all in line to major+minor.
This fixes a bug where we could release a newer version of the
server+CLI, but not the crates, and that would cause the CLI to
initialize some templates to expect a version number that did not exist.
**I am not 100% sure that this doesn't have surprise consequences**,
since this is a weird situation in the first place.
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
spot tests, but more importantly some template smoketests have been
added to check that the version constraints are now `major.minor` on
some representative templates.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
Bump version to `2.5.0` for new release
# API and ABI breaking changes
N/A
# Expected complexity level and risk
1
# Testing
None
# Description of Changes
Adds primary keys to procedural views in C#. See for #5111 for the
equivalent feature in rust and C# as well as a more detailed
description.
# API and ABI breaking changes
None
# Expected complexity level and risk
3
# Testing
- [x] Equivalent tests as were added in #5111 for rust and typescript
# Description of Changes
Reviving a previous patch I wrote during our (internal) TPCC
experimentation. This has become important because, in addition to its
performance implications, it makes row insertion locations deterministic
regardless of datastore restarts, which previously they were not.
Previously, restarting the datastore would re-order the `non_full_pages`
list (i.e. sort it by increasing `PageIndex`, where normally it was not
sorted), meaning that which page a new row would be inserted into
depended on when the datastore was last restarted.
With this patch, that is not the case: the `non_full_pages` are always
kept in a deterministic order, so which page a new row goes into is also
deterministic.
Original commit message follows:
And sort them by number of available var-len granules. This prevents an
accidentally quadratic behavior where, for a table where the average row
contains many var-len granules, after inserting a large number of rows,
there would be a large number of pages in `non_full_pages` each of which
had enough space for at least one fixed-len row part, but insufficient
space for an actual row in practice due to insufficient var-len
granules. Each insertion would then do a linear scan over
`non_full_pages` before either inserting into the last page or
allocating a new page which went to the end.
Now, non-full pages are stored in a `BTreeSet` sorted by the number of
free var-len granules, and the search for a useable page is done with a
`BTreeSet::range` iterator for only the pages with enough granules. I
think there may still be an off-by-one-ish bug here, where a page may
have enough bytes in the gap that it could either store the fixed-len
part or the var-len granules, but not both, but this fix hopefully will
suffice for now.
# API and ABI breaking changes
N/a
# Expected complexity level and risk
2? Table code is a bit fiddly, and this path is performance-sensitive
when inserting rows.
# Testing
- [x] Passes table crate tests.
- [x] Was included in our internal TPCC experimentation, where it
significantly improved performance (due to that benchmark exercising the
accidentally-quadradic behavior this patch is designed to protect).
- [x] Joshua ran the keynote-2 benchmarks with this patch and did not
observe a decrease in throughput.
Makes the `call` command rewrite reducer arguments typed at `Identity`,
but given as just a (hex) string, into a JSON 1-tuple.
Proper deserialization on the server is more complicated to change, so
may be addressed separately.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Added a smoketest.
# Description of Changes
Prior to this commit, the metric `wasm_memory_bytes` had several
problems:
1. Despite its name, it was used for both Wasmtime and V8 modules. For
V8 modules, it was the same value as `v8_used_heap_size_bytes`.
2. It stored only the value for a single instance at any given time, so
it under-reported a database's memory usage.
3. The same row (set of label values) was written concurrently by all
instances of a particular database, with each one clobbering the
previously written value.
In this commit, we change the metric so that:
1. It is recorded only for Wasmtime instances, not V8 instances. For V8
instances, instead directly check `v8_used_heap_size_bytes`, or one of
the other V8 heap metrics. This change involved moving the recording of
this metric from `module_host_actor.rs` to
`wasmtime/wasm_instance_env.rs`
2. Similar to the V8 heap metrics, all the instances cooperatively share
the metric entry, updating it by incrementing and decrementing rather
than `set`ting.
Note that this metric is used for billing, and so we will need to update
our billing code (elsewhere) to account for the change. In particular,
our billing code should now charge for the sum of `wasm_memory_bytes`
and `v8_used_heap_size_bytes`. We also should expect with this change
for each database's recorded usage to increase, as we are now accurately
recording the usage for all instances, not just one.
# API and ABI breaking changes
Billing metric semantics changed.
# Expected complexity level and risk
3: billing metric semantics changed.
# Testing
I do not know how to test metrics.
## Summary
- Remove the clap-level requirement that forced `spacetime publish
--delete-data` to include a positional database name.
- Keep database-name validation in the merged command config, so
`spacetime.json` can still provide `database`.
- Add regression coverage for `publish -c=always` using a config-sourced
database.
Closes#5253.
## Tests
-
`PATH=/Users/clockworklabs/.rustup/toolchains/1.93.0-aarch64-apple-darwin/bin:$PATH
cargo fmt --all --check`
-
`PATH=/Users/clockworklabs/.rustup/toolchains/1.93.0-aarch64-apple-darwin/bin:$PATH
cargo test -p spacetimedb-cli subcommands::publish::tests`
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
Previously `assert_after_durable` assumed that `Poll::Pending` meant
confirmed reads was blocked on durability. That was not necessarily
true, because subscription messages pass through the subscription send
worker before reaching the client receiver. A `Poll::Pending` could also
mean the send worker had not delivered the message yet, so a single
immediate poll after marking durability was not guaranteed to be
`Poll::Ready`.
This patch removes `test_confirmed_reads` entirely since the guarantee
that client receivers do not receive messages before they are durable is
already covered by the `ClientConnectionReceiver` tests, where `Pending`
is unambiguous.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
1
# Testing
N/A
# Description of Changes
Removes a race(sleep) in
`local_durability_crashes_on_new_segment_if_not_enough_space()` where we
might not wait long enough for `fallocate` to fail and crash the
durability actor.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
1
# Testing
The test should no longer flake.
# Description of Changes
Graduate **procedures** (and the outgoing HTTP client used from
procedures) out of the `unstable` feature gate /
`SPACETIMEDB_UNSTABLE_FEATURES` across all module libraries. The
`procedure` macro, `ProcedureContext`, `with_tx`/`try_with_tx`,
scheduled procedures, and `ctx.http` are now available without opting
into `unstable`.
**Still gated** (unchanged): HTTP handlers/webhooks, views, RLS /
`client_visibility_filter`, and immediate-scheduling
(`volatile_nonatomic_schedule_immediate`).
Per library:
- **`bindings-sys`**: ungate the `procedure` host-call module + raw ABI
imports (`procedure_start/commit/abort_mut_tx`,
`procedure_http_request`) and `call_no_ret`. Scheduling ABI stays gated.
- **`bindings` (Rust)**: ungate the `procedure` macro,
`ProcedureContext`, `TxContext`/`with_tx`/`try_with_tx`,
`db_read_only`/`get_read_only`, the procedure traits,
`register_procedure`, `__call_procedure__`, and procedure RNG. The
`http` module (previously gated as a unit) now has fine-grained gating
so the outgoing `HttpClient` is exposed while
`HandlerContext`/`Router`/handler macros stay gated.
- **`bindings-csharp`**: drop `[Experimental("STDB_UNSTABLE")]` from
`ProcedureContext.WithTx/TryWithTx` (runtime + codegen) and the
generated `ProcedureTxContext`; FFI snapshots regenerated. Handler
context members + RLS attribute stay gated.
- **`bindings-cpp`**: ungate the procedure ABI (`abi.h`/`FFI.h`),
`tx_execution.h`, the outgoing HTTP client (`http.h`/`http_convert.h`),
and `procedure_context.h`.
`handler_context.h`/`router.h`/`http_handler_macros.h` still `#error`
without `SPACETIMEDB_UNSTABLE_FEATURES`.
- **docs**: remove the procedures beta notices (HTTP handlers stay
marked beta for a later release); regenerate `static/llms.md`.
- **`sdk-test-procedure`**: no longer needs `features = ["unstable"]`.
# API and ABI breaking changes
Not breaking. This is purely additive to the stable surface: procedures
become available **without** the `unstable` feature, while modules that
already opt into `unstable` are unaffected. The wasm host-ABI imports
were already implemented host-side and merely gated module-side, so
there is no ABI version change. (No breaking-change label needed.)
# Expected complexity level and risk
**3/5.** The diff itself is mostly removing
`#[cfg]`/`#ifdef`/`[Experimental]` guards, but the gate bundled
procedures + outgoing HTTP + handlers + the ABI crate together, so the
work was in *separating* procedures from the features that stay gated.
Areas reviewers may want to scrutinize:
- The Rust `http` module went from "gated as a whole" to fine-grained
per-item gating; the outgoing `HttpClient` is exposed while handler
types/macros stay behind `unstable`.
- The `unstable` cut reaches the ABI crate (`bindings-sys`), which is
the only way procedures can compile without the feature.
- C++ separation of the outgoing HTTP client from HTTP handlers across
several headers.
# Testing
- [x] Rust: `cargo check -p spacetimedb` passes in default, `--features
unstable`, and `--no-default-features --features unstable`.
- [x] Rust end-to-end: `sdk-test-procedure` (uses procedures,
`with_tx`/`try_with_tx`, `ctx.http.get`, `new_uuid_v7`, scheduled
procedures) builds **without** `unstable`.
- [x] C#: Codegen + Runtime build; `Codegen.Tests` pass (6/6) after FFI
snapshot regen.
- [x] C++: host syntax-check confirms procedures + `ctx.http.send()`
compile **without** the flag, full headers compile **with** it, and
`handler_context.h` still `#error`s without it. (Not built for the real
wasm/emscripten target locally — relying on CI.)
- [x] `cargo ci lint` components reproduced locally: rustfmt, clippy
(`-D warnings`, default + `unstable`), csharpier, and `cargo doc --deny
warnings`.
- [ ] Reviewer: confirm the wasm bindings + C#/C++ test suites pass in
CI (especially the C++ wasm build, which couldn't run locally).
---------
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
joshua-spacetime