# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
- Bumps version to 2.2.0
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
- 1 - this is just a version bump
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Version number is correct (`2.2.0`)
- [x] BSL license file has been updated with the new date and version
number
---------
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Tests for case conversion.
# API and ABI breaking changes
NA
# Expected complexity level and risk
1
---------
Co-authored-by: clockwork-labs-bot <bot@clockworklabs.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
1. This adds some basic tests of `LockedFile` (which would generally
have also passed before these changes).
2. `LockedFile` can now have contents, which are added to the errors if
we can't aquire the lock. The default metadata has the pid and a
timestamp.
I now see that there is a separate `Lockfile` that doesn't clean up
locks on a crash, so that should probably also be cleaned up (but in a
separate PR).
# Expected complexity level and risk
1.
# Testing
This has unit tests.
# Description of Changes
Revert the following PRs that have caused some breakage:
```
a32cffa76 Finish refactoring out replay (#4850)
d639be0af Replay: some code motion & reuse `ReplayCommittedState` (#4849)
78d6b6f7d Update NativeAOT-LLVM infrastructure to current ABI (#4515)
d5c1738c1 Better module backtraces for panics and whatnot (#577)
6f23b19f3 Wait for database update to become durable (#4846)
81c9eab86 Add `spacetime lock/unlock` to prevent accidental database deletion (#4502)
809aebd7c Move field `replay_table_updated` to `ReplayCommittedState` (#4807)
21b58ef99 Update axum (#2713)
b5cadff7a Extract replay stuff out of `CommittedState`, part 1 (#4804)
```
I also updated the Python smoketests for breakage introduced in
https://github.com/clockworklabs/SpacetimeDB/pull/4502. Reverting that
PR caused conflicts, so this fix is more straightforward.
# API and ABI breaking changes
Maybe kind of, but we haven't released any of these.
# Expected complexity level and risk
1
# Testing
Ask @bfops about testing
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
~~Axum now has what we need out of it for a websocket wrapper, so we no
longer need to duplicate `util/flat_csv.rs` and `util/websocket.rs`,
meaning we have less code to maintain.~~ Nevermind, we know send raw
frames, which axum's wrapper does not support. Makes this PR simpler.
# Expected complexity level and risk
2 - upgrading a dependency is a risk, but looking through the
[changelog](https://github.com/tokio-rs/axum/blob/main/axum/CHANGELOG.md#080)
there isn't anything that should affect us.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] tests pass <!-- maybe a test you want to do -->
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
## Summary
The `/v1/database/:name_or_identity/sql` endpoint now calls the module's
`client_connected` reducer before executing SQL, and
`client_disconnected` after. This allows module authors to accept or
reject SQL connections based on the caller's identity, matching the
existing behavior of the `/call` endpoint.
## Motivation
Previously, the `/sql` endpoint bypassed the module's `onConnect` hook
entirely, meaning module authors had no way to restrict who could run
SQL queries against their database. The `/call` endpoint already runs
the connect hook, so this brings `/sql` to parity.
## Changes
- `crates/client-api/src/routes/database.rs`: The `sql` handler now:
1. Generates a random connection ID
2. Calls `module.call_identity_connected()` before executing SQL
3. Executes the SQL query
4. Calls `module.call_identity_disconnected()` after
5. If `client_connected` rejects the connection, returns 403 Forbidden
without executing the query
- `sql_direct()` is unchanged since it is also used by the pgwire
server, which has its own connection lifecycle.
## Behavior
- If the module defines a `client_connected` reducer that throws/errors
for a given identity, the SQL request returns `403 Forbidden`
- If no `client_connected` reducer is defined, behavior is unchanged
- The connection is always cleaned up via `client_disconnected` after
the query completes
---------
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
The previous approaches would either:
- panic when the queue becomes full, as `append_tx` is run inside the
context of a `LocalSet`, which is basically a glorified current thread
runtime
- deadlock because the receiver runtime has no way of notifiying the
sender of freed capacity in the channel
`async-channel` handles wait queues and notifications internally, so can
be used freely from either blocking or async contexts.
This _may_ come at different performance characteristics, but I haven't
measured them.
---------
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
# Description of Changes
We only benchmark the typescript module and sdk now.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
N/A
# Description of Changes
While testing #4663, I discovered the server would crash from a V8 out
of memory error after processing many requests. Before #4663, this would
not happen. I theorized that because we now have a single JS worker that
can process an unbounded number of reducer calls over its lifetime, any
V8 heap retention that would previously have been spread across several
pooled isolates now accumulates in one isolate.
This patch now periodically collects heap statistics and forces GC or
replaces the isolate if memory cannot be reclaimed. This greatly reduces
the risk of hitting the V8 heap limit and crashing the server. It
doesn't remove the risk entirely however. But this risk was still
present before we switched to a single worker model in #4663. In order
to remove the risk of crashing the server entirely, we would need to run
V8 in a separate process.
# API and ABI breaking changes
None
# Expected complexity level and risk
3
# Testing
TBD
# Description of Changes
This PR adapts the Rust SDK test suite to work with the wasm version
added in https://github.com/clockworklabs/SpacetimeDB/pull/4089 (which
I've closed in favor of this PR).
Most of the changes revolve around wasm's different async semantics -
everything runs in one thread, so things that relied on background
threads didn't work directly. Several tests would lock up because
something in them blocked synchronously, which blocked any background
work from progressing.
We moved the test-clients contents into a `test_handlers.rs` so that it
could be called from both `main` (for native tests) and `lib` (for wasm
tests). To show what actually changed, use:
```bash
git diff --no-index -- <(git show origin/master:sdks/rust/tests/procedure-client/src/main.rs) sdks/rust/tests/procedure-client/src/test_handlers.rs
```
(or similar for other test-clients)
# API and ABI breaking changes
None, I think/hope.
# Expected complexity level and risk
2
# Testing
- [x] I've augmented the CI to also run the test suite with the `web`
feature
---------
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Thales R <thlsrmsdev@gmail.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Bump versions to 2.1.0
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
CI
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Adds `Deserialize::validate` and all the friends necessary to make that
work.
The point of this is to enable allocation-less validation of input in
some data format, in particular BSATN, against some expected type, in
particular an `AlgebraicValue` at an `AlgebraicType`.
This is extracted from
https://github.com/clockworklabs/SpacetimeDB/pull/4311 and is used there
to convert raw BSATN to `BytesKey` without constructing the composite
`AlgebraicValue` just for validation only to throw it away immediately.
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
TODO: tweak existing SATS roundtrip tests to also check validation.
---------
Co-authored-by: Noa <coolreader18@gmail.com>
# Description of Changes
This patch removes the dead legacy SQL query engine and the remaining
code that only existed to support it.
Removed:
- Old SQL compiler/type-checker and VM-based execution path in
spacetimedb-core
- `spacetimedb-vm` crate
- Dead vm specific error variants and compatibility code
- Obsolete tests, benchmarks, and config paths that still referenced the
legacy engine
Small pieces still used by the current engine were moved to their proper
homes instead of keeping the `vm` crate around. In particular,
`RelValue` was moved to `spacetimedb-execution`.
The `sqltest` crate was also updated to use the current engine. Notably
though these tests are not run in CI, however I've kept them around as
they may be beneficial as we look to expand our SQL support in the
future.
Requires codeowner review from @cloutiertyler due to the removal of the
`LICENSE` file in the (now removed) `vm` crate.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
None
---------
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
Update pgwire to 0.37
Make `DataRowEncoder` reusable as enabled in this release, make
introduce 3x performance on encoding.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
N/A
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [ ] <!-- maybe a test you want to do -->
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
---------
Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: joshua-spacetime <josh@clockworklabs.io>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
# Description of Changes
- Updated the Unreal SDK and generated Unreal bindings for the websocket
2.0 protocol/model
- Reworked DbConnectionBase to handle the updated message shapes
- Switched subscription handling over to new message types and
QuerySetId
- Updated reducer to ReducerResult, removal of callbacks, and set
reducer flags
- Added event table support
- Baked in multi-module support replacing [the old
PR](<https://github.com/clockworklabs/SpacetimeDB/pull/3417>)
- Added functionality to generate module support for multiple folders in
the Unreal project (add <module>.Build.cs, <module>.h, <module>.cpp)
using the --module-name
- Add new configuration option for spacetime generate to handle module
prefix
- Regenerated Unreal Blackholio/TestClient/QuickstartChat bindings
- Rebuilt Unreal Blackholio's consume entity to use event tables
- Updated migration documentation
- Updated the version bump tool to impact C++
# API and ABI breaking changes
- Unreal websocket/message handling updated to the new protocol
- Unreal generation now expects a real .uproject target and will stop
immediately if project
metadata is invalid instead of continuing past setup issues.
# Expected complexity level and risk
3 - A large set of changes to update the websocket/message handling
along with heavy codegen changes to handle multi-module support
# Testing
Test coverage of the Unreal SDK will need expansion in a future ticket
once our issues with flakiness on CI is resolved.
- [x] Updated Unreal Blackholio
- [x] Ran full Unreal SDK test suite
- [x] Built new test project using the new `--module-prefix`
- [x] Run through Unreal Blackholio (C++ and Blueprint)
- [x] Rebuilt Unreal Blackholio with multi-module, and duplicate
generated module testing side-by-side modules that would overlap
# Review Question(s)
- [x] Updates to `spacetime init` have made the tutorial a little
confusing with pathing for the Unreal Blackholio tutorial. To fix though
we'd have to update all the commands to be more explicit, or update the
tutorial `spacetime init` to use `--project-path .` to keep pathing
simpler, thoughts?
---------
Signed-off-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: Ryan <r.ekhoff@clockworklabs.io>
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
- Bumps version to 2.0.5
# API and ABI breaking changes
None - this is just a version bump
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
1 - just a version bump
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] License file has been properly updated, including the date change
- [x] CI is passing other than the internal check
Prior to #4549 the host type in `st_module` was always set to wasm. We
now correctly use the host type from the database, but the module may in
fact be a JS module. So if launching it as a wasm module fails, try JS
instead. If this succeeds, the module is definitely a JS module, so
attempt to repair `st_module` in this case.
# Expected complexity level and risk
2
# Testing
- [x] Added smoketest
# Description of Changes
Upgrades the rust SDK's prometheus dependency from 0.13 to 0.14.
Fixes https://github.com/clockworklabs/SpacetimeDB/issues/4597
# API and ABI breaking changes
[The prometheus
changelog](https://github.com/tikv/rust-prometheus/blob/master/CHANGELOG.md#0140)
claims that the MSRV for the new version is 1.82, but this project
doesn't seem to have an official MSRV, so I don't think that's an ABI
change.
I don't think depending on a different prometheus version is itself a
breaking change. Prometheus is exposed through the rust SDK, but in an
explicitly [unstable
module](https://github.com/clockworklabs/SpacetimeDB/blob/3f58b5951bf3c49971c51aecb526439597b9c044/sdks/rust/src/lib.rs#L69-L76)
which "may change incompatibly without a major version bump". Prometheus
structs are also exposed from several crates, but with the same
disclaimers about unstable interfaces.
# Expected complexity level and risk
1. This is a module bump with simple-looking changes.
# Testing
- [x] Just confirmed everything still compiles and the tests still pass
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
# Description of Changes
Query builder views return a subset of rows from a physical table. If
that table has a primary key, then so should the view. What this means
concretely is that the view should expose the same api as the table,
specifically as it relates to the primary key column.
With that in mind, this patch commits the following changes:
1. Annotates `ViewDef` with a `primary_key`
2. Updates the return type of query builder views in the raw module def
to a special product type
3. Adds an index for the primary key on the view's backing table
4. Updates the query planner to use this index
5. Updates rust client codegen to generate `on_update` for such views
# API and ABI breaking changes
None
Old `impl Query` views compiled with an older version of SpacetimeDB
will continue to work as they did before - without a primary key.
# Expected complexity level and risk
3
# Testing
- [x] New rust sdk integration suite exercising `on_update` for PK views
and semijoin scenarios
- [x] Smoketests for PK views and semijoin scenarios
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
- Bumps version to 2.0.4
# API and ABI breaking changes
None - this is just a version bump
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
1 - this is just a version bump
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] License file has been updated with correct version + time
---------
Signed-off-by: John Detter <4099508+jdetter@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
## Summary
Adds a version update check to the `spacetimedb-update` proxy, so users
are notified when a newer version of SpacetimeDB is available.
## Changes
Adds `crates/update/src/update_notice.rs` — a lightweight update check
that runs in the proxy path before exec'ing the CLI:
- **Cache-based**: Stores the last check result in
`~/.spacetime/.update_check_cache`. Only hits the network once every 24
hours.
- **Non-blocking on cache hit**: If the cache is fresh, it's a single
file read — no network, no delay.
- **Short timeout**: When the cache is stale, makes a single HTTP
request to GitHub releases API with a 5-second timeout. Uses the same
`SPACETIME_UPDATE_RELEASES_URL` env var as `spacetime version upgrade`.
- **Best-effort**: Any failure (network, parse, file I/O) is silently
ignored. The update check never interferes with the user's command.
- **Uses semver**: Proper version comparison via the `semver` crate
(already a dependency).
## Output
When a newer version is available:
```
A new version of SpacetimeDB is available: v2.1.0 (current: v2.0.0)
Run `spacetime version upgrade` to update.
```
# Testing
- [x] I get a warning if my local version is less than 2.0.2
- [x] If I have a cached update check, I get the same error even if I
have no network connection
- [x] if the cache is old, the next command checks again
- [x] if I'm not connected and the cache is stale, I'm still able to use
the CLI
---------
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <bot@clockworklabs.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
# Description of Changes
LLM benchmark updates for local development:
- **Local SDK paths**: Templates use relative paths to workspace crates
(`crates/bindings`, `crates/bindings-csharp`,
`crates/bindings-typescript`) instead of published packages, so the
bench runs against local SDK changes.
- **NODEJS_DIR support**: On Windows (e.g. nvm4w), if `pnpm` is not on
PATH, the bench uses `NODEJS_DIR` to locate `pnpm` and prepends it to
PATH for subprocesses.
- **Refactor**: Extracted `relative_to_workspace()` in `templates.rs`
and removed noisy `NODEJS_DIR` logging in `publishers.rs`.
- **Benchmark results**: Updated `docs/llms/llm-comparison-details.json`
and `docs/llms/llm-comparison-summary.json`.
# API and ABI breaking changes
None.
# Expected complexity level and risk
**2** — Local-only changes to the benchmark tool. Templates now require
local SDKs to be built (especially TypeScript: `pnpm build` in
`crates/bindings-typescript`). No impact on published SDKs or runtime.
# Testing
- [ ] Run `cargo llm run --lang rust --modes docs --providers openai`
from repo root
- [ ] Run TypeScript benchmarks with `pnpm build` in
`crates/bindings-typescript` first
- [ ] On Windows with nvm4w, set `NODEJS_DIR` if `pnpm` is not on PATH
and run TypeScript benchmarks
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: clockwork-labs-bot <bot@clockworklabs.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
- Version upgrade 2.0.3
# API and ABI breaking changes
- None, this is a version bump
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
- 1 - this is a version bump
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] License has been updated
- [x] Version number is correct (2.0.3)
## Problem
`spacetime generate --out-dir ../frontend-ts-src/module-bindings` was
resolving the path incorrectly, stripping the leading `..` component.
Reported in #4429 via a [user report on
Discord](https://discord.com/channels/1037340874172014652/1475935288919462072/1475935288919462072).
## Root Cause
`normalize_path_lexical()` in `spacetime_config.rs` used
`PathBuf::pop()` to handle `..` components, but `pop()` is a no-op on an
empty `PathBuf`. This silently dropped leading `..` segments:
- Input: `../frontend-ts-src/module-bindings`
- Output: `frontend-ts-src/module-bindings` (wrong — `..` was eaten)
## Fix
Replace the `PathBuf`-based normalization with a `Vec<Component>` stack
approach. `..` only cancels a preceding `Normal` component; otherwise it
is preserved. This correctly handles:
- `../foo` → `../foo` (leading `..` preserved)
- `../../a/b` → `../../a/b` (multiple leading `..` preserved)
- `a/b/../c` → `a/c` (inner `..` still resolves)
- `/home/user/project/../foo` → `/home/user/foo` (absolute paths work)
## Testing
Added `test_normalize_path_preserves_leading_dotdot` covering all edge
cases.
Closes#4429
---------
Co-authored-by: clockwork-labs-bot <bot@clockworklabs.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
Bumping all versions to 2.0.2 for the release.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
CI
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Just bumping versions to 2.0.1.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
- [ ] CI passes
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Update the Default casing policy to `snake_case` for `RawModuleDefV10`.
Messy PR contains changes at different places, so that CI can pass:
Here are the main changes as follows:
- `bindings-macro` & `bindings` crate: `name` macro in Indexes for
canonical name and supply it to `RawModuleDefV10` via `ExplicitNames`.
- `bindings-typescript`:
- Changes has been reviewed through this PR -
https://github.com/clockworklabs/SpacetimeDB/pull/4308.
- `binding-csharp`: a single line change to pass `sourceName` of index
instead of null.
- `codegen`:
- Changes has been merged from branch -
https://github.com/clockworklabs/SpacetimeDB/pull/4337.
- Except a fix in rust codegen to use canonical name in Query buillder
instead of accessor.
- `lib/db/raw_def`: Extends `RawDefModuleV10` structure to support case
conversion.
- `schema` crate:
- `validate/v9` - Nothing itself should change or changes in v9
validation logic but the file contains a `CoreValidator` which is shared
with `validate/v10`. No test have t be updated to `validate/v9` which
ensures we aren't regressing it.
- `validate/v10`: This is the main meat, look at the new tests added in
bottom to understand what it does.
- Rest of the files are either test updates or module bindings.
## Testing:
1. Extensive unit tests have been added to verify generated `ModuleDef`
is correct.
2. I have done some e2e testing to verify rust codegen with rust and
typescript modules.
3. I would have like to do more testing for other codegens , I am
continue doing .
I have removed `sql.py` smoketest, as that seems to be already migated
in new framework and was headache to update.
## Expected complexity level and risk
4, It could have side-effect which aren't easily visible.
- - -
---------
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <bot@clockworklabs.com>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
Co-authored-by: Noa <coolreader18@gmail.com>
Co-authored-by: = <cloutiertyler@gmail.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@clockworklabs.io>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
## Summary
- Fix `spacetime dev` failing on C# projects by watching the module
directory itself instead of hardcoded `spacetimedb_dir/src/` (which
doesn't exist for C# templates)
- Add layered file-watch filtering to avoid triggering rebuilds on build
artifacts:
1. Always-ignore dirs (`target/`, `bin/`, `obj/`, `node_modules/`, etc.)
2. Always-watch exceptions (`.env.local`, `spacetime.*.local.json`)
3. `.gitignore` rules from global, project, and module levels
- Suppress init's generic "Next steps" message when called from
`spacetime dev`, print a dev-appropriate `cd` hint instead when a new
project is created in a subdirectory
- Update 12 quickstart docs to tell users to open a new terminal and `cd
my-spacetime-app` before running CLI commands
## Test plan
- [x] `cargo build -p spacetimedb-cli` compiles successfully
- [ ] `spacetime dev --template basic-cs` no longer errors with "Input
watch path is neither a file nor a directory"
- [ ] Modifying a `.cs` file in `spacetimedb/` triggers a rebuild
- [ ] Build artifacts in `obj/`/`bin/` do not trigger rebuilds
- [ ] Rust projects (`spacetimedb/src/` exists) continue to work as
before
- [ ] `spacetime dev --template basic-cs` from a non-project directory
prints the "Tip: cd" hint, not the generic "Next steps"
- [ ] `spacetime init` standalone still prints "Next steps" as before
---------
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Basic tests for templates, the test tries to generate each of the
template, build and publish the SpacetimeDB part, and build or type
check the client side.
This PR also includes bindings for all of the clients as it's needed for
the tests to work and given the choices to either generate during the
test run or include the bindings in git we chose the latter. This makes
it consistent across the templates as some of them already included
bindings.
# Expected complexity level and risk
1
---------
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
# Description of Changes
This fixes the confusing error encountered by @gefjon:
```
$ spacetime publish -s local --module-path spacetimedb --no-config event-table-test-ts
Rolldown warning: Could not resolve 'spacetimedb/server' in spacetimedb/src/index.ts
Error: Your module doesn't import the `spacetimedb/server` package at all - this is likely a mistake, as your module will not be able to interface with the SpacetimeDB host.
```
After:
```
[UNRESOLVED_IMPORT] Error: Could not resolve 'spacetimedb/server' in src/index.ts
╭─[ src/index.ts:1:47 ]
│
1 │ import { schema, table, t, SenderError } from 'spacetimedb/server';
│ ──────────┬─────────
│ ╰─────────── Module not found.
│
│ Help: You may have forgotten to install dependencies with your package manager.
───╯
```
And also runs `tsc`, so that modules with type errors aren't silently
published. Perhaps we want a `--skip-tsc` so that people can run anyway
and get runtime errors?
# Expected complexity level and risk
2
# Testing
- [x] Ensured that tsc and rolldown errors are properly emitted.
# Description of Changes
This PR implements support for the `spacetime.json` configuration file
that can be used to set up common `generate` and `publish` targets. An
example of `spacetime.json` could look like this:
```
{
"dev_run": "pnpm dev",
"generate": [
{ "out-dir": "./foobar", "module-path": "region-module", "language": "c-sharp" },
{ "out-dir": "./global", "module-path": "global-module", "language": "c-sharp" },
],
"publish": {
"database": "bitcraft",
"module-path": "spacetimedb",
"server": "local",
"children": [
{ "database": "region-1", "module-path": "region-module", server: "local" },
{ "database": "region-2", "module-path": "region-module", server: "local" }
]
}
}
```
With this config, running `spacetime generate` without any arguments
would generate bindings for two targets: `region-module` and
`global-module`. `spacetime publish` without any arguments would publish
three modules, starting from the parent: `bitcraft`, `region-1`, and
`region-2`. On top of that, the command `pnpm dev` would be executed
when using `spacetime dev`.
It is also possible to pass additional command line arguments when
calling the `publish` and `generate` commands, but there are certain
limitations. There is a special case when passing either a module path
to generate or a module name to publish. Doing that will filter out
entries in the config file that do not match. For example, running:
```
spacetime generate --project-path global-module
```
would only generate bindings for the second entry in the `generate`
list.
In a similar fashion, running:
```
spacetime publish region-1
```
would only publish the child database with the name `region-1`
Passing other existing arguments is also possible, but not all of the
arguments are available for multiple configs. For example, when running
`spacetime publish --server maincloud`, the publish command would be
applied to all of the modules listed in the config file, but the
`server` value from the command line arguments would take precedence.
Running with arguments like `--bin-path` would, however, would throw an
error as `--bin-path` makes sense only in a context of a specific
module, thus this wouldn't work: `spacetime publish --bin-path
spacetimedb/target/debug/bitcraft.wasm`. I will throw an error unless
there is only one entry to process, thus `spacetime publish --bin-path
spacetimedb/target/debug/bitcraft.wasm bitcraft` would work, as it
filters the publish targets to one entry.
# API and ABI breaking changes
None
# Expected complexity level and risk
3
The config file in itself is not overly complex, but when coupled with
the CLI it is somewhat tricky to get right. There are also some changes
that I had to make to how clap arguments are validated - because the
values can now come from both the config file and the clap config, we
can't use some of the built-in validations like `required`, or at least
I haven't found a clean way to do so.
# Testing
I've added some automated tests, but more tests and manual testing is
coming.
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: bradleyshep <148254416+bradleyshep@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: = <cloutiertyler@gmail.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
# Description of Changes
This adds the v2 websocket protocol and adds support on the server side.
For context on many of the changes/decisions, you can look at the
discussion on https://github.com/clockworklabs/SpacetimeDB/pull/4023.
To restate some of the key changes:
- The reducer event information is no longer sent with transaction
updates (because we don't want to broadcast reducer call information
anymore).
- If a client calls a reducer, they are sent a `ReducerResult` which
includes the outcome of the reducer call and and related row updates for
queries that the client is subscribed to.
- We no longer dedupe queries that appear in multiple query sets for the
same client. This is because we are moving toward per-query storage.
- Related to that, Unsubscribe requests have an option to send the
related rows. We need this for now, since clients don't have per-query
storage implemented yet.
- We don't have the json format in v2.
Notes for reviewers:
- This moves around the messages in
`crates/client-api-messages/src/websocket` (into `common`, `v1`, and
`v2`), and this renaming of existing messages adds a lot of noise to the
PR.
- In many places, I chose to duplicate a lot of code to have a v1
version and a v2 version. I went with this to make it easier to remove
the v1 version in the future (hopefully we can just fully delete most of
the v1 functions).
- `module_subscription_manager.rs` has probably has the biggest changes,
since we now track queries by query_set_id, and we get to remove some
complexity of v1's FormatSwitch.
<!-- Please describe your change, mention any related tickets, and so on
here. -->
# API and ABI breaking changes
The v1 protocol still works, though we won't send the reducer event info
for v10 modules.
# Expected complexity level and risk
4. This touches a lot of places.
# Testing
Unit testing is pretty minimal for the new code paths. I've done some
manual e2e testing with the typescript quickstart, and this has been
tested with a different branch implementing the v2 rust client.
---------
Co-authored-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: Jeffrey Dallatezza <jeffreydallatezza@gmail.com>
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
- Version upgrade to 2.0.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
1 - this is just a version bump
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] License file has been updated
- [x] CI passing
---------
Signed-off-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
Rolldown likely [won't release 1.0 until after vite
8.0](https://voidzero.dev/posts/announcing-rolldown-rc), but I figure we
should keep up to date somewhat and switch from the beta to the rc.
# Expected complexity level and risk
1
# Testing
- [x] `spacetime build` with typescript modules still works.
---------
Signed-off-by: Noa <coolreader18@gmail.com>
# Description of Changes
Haven't changed `schema()` to accept an object yet, maybe that's for a
followup?
Now everything exported from the module must be exported from the
typescript module.
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [ ] <!-- maybe a test you want to do -->
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
# Description of Changes
Fixes#3926. This PR bundles in icudata (11MB) to the spacetime binary,
to allow i18n functions like `String.prototype.localeCompare()` to work.
The 2nd commit sets the default locale to [en-001, "International
English"](https://www.ctrl.blog/entry/en-001.html), in order to prevent
system locale from affecting the execution of reducers.
# Expected complexity level and risk
1
# Testing
- [x] Verify this fixes the issue.
# Description of Changes
This PR translates all of our Python smoketests into Rust tests which
can be run from `cargo run`
## Motivation
The purpose of this fivefold:
1. All developers on the team are familiar with Rust
2. It simplifies our devops because we can drop Python as a dependency
to run the tests
3. You can now run all tests in the repo through the single `cargo test`
interface
4. Because we use the `SpacetimeDbGuard` and `cargo test`/`cargo
nextest` we can easily parallelize the smoke tests
5. The smoketests can now use machinery imported from SpacetimeDB crates
(e.g. `bsatn` etc.)
IMPORTANT NOTE!
There are several ways to implement the smoke tests in Rust (none are
great):
1. A separate xtask specifically for the smoke tests
- This doesn't solve the problem of the CLI tests which also use the
`guard` crate
- Idiosyncratic way to run the smoke tests as opposed to cargo test
- Does NOT resolve the cargo within cargo problem because we still have
to build the test modules with cargo
2. A `build.rs` script in `guard` which first builds the executables as
a compile step for compiling guard
- Deadlocks on a cargo lock file conflict (Outer cargo compiles guard →
runs build.rs, inner cargo tries to acquire the build directory lock,
outer cargo holds the directory lock, deadlock)
- If you fix the deadlock by using different target dirs, it still looks
stuck on building guard because it's actually compiling all of
spacetimedb-standalone and spacetimedb-cli.
- Still technically runs cargo inside of cargo.
3. Add `spacetimedb-cli` and `spacetimedb-standalone` as an artifact
dependency of the guard crate
- Has good and clear output but requires +nightly when running the
smoketests and CLI tests, otherwise won't do the right thing. See
https://github.com/rust-lang/cargo/issues/9096
4. Compile the executables at runtime during the tests themselves where
the first test takes a lock while the executables are building using
cargo within cargo
- Makes the tests look like they're taking a long time when they're just
waiting for the build to complete
- Requires relatively complex locking machinery across
binaries/tests/processes
5. A two step solution where the developer has to build the binaries
before calling the smoke tests
- Very error prone
None of these are good. `xtask` is not bad, but doesn't enable us to run
other integration tests in other crates (e.g. the CLI)
(3) is the correct solution and has the best user experience, but it
requires nightly and I don't want to introduce that for all of our
tests.
I have chosen to do a combination of (1) and (4). You will now run the
smoketests with `cargo smoketest`. If you run `cargo test --all` (or use
`guard`) without doing `cargo smoketest` it will fall back to (4) which
compiles the executables at runtime. Running `cargo build` is the **only
way** to ensure that the executables are not stale because of the
internal fingerprint checking. Everything else is fragile not robust.
NOTE! There is no way to avoid cargo within cargo and have the smoke
tests be run as cargo tests because the modules under test must be
compiled with cargo.
# API and ABI breaking changes
Note that this is a BREAKING CHANGE to `cargo test --all`. The
smoketests are now part of `cargo test --all` unless you specifically
exclude them.
# Expected complexity level and risk
3, this is partially AI translated. We need to carefully review to
ensure the semantics have not regressed.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [ ] <!-- maybe a test you want to do -->
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
# Description of Changes
The goal of this PR is to:
- Make it easy to switch our backing implementation of identifiers, as
seen in the commit changing to `LeanString`.
- Improve type safety of our identifiers by having every identifier
start with `RawIdentifier` and then `Identifier` is just a wrapper with
validation on construction. `TableName` and `ReducerName` are then just
wrappers around `Identifier`.
- Reduce allocation in `InstanceEnv` by using the now clone-on-write
O(1) + SSO optimized `Identifier`.
- Reduce allocations in the query engine as a consequence of improving
`RawIdentifier` and `Identifier`.
- Have `&'static str` strings be further optimized by avoiding to
allocate even for long ones. This is supported by `impl From<&'static
str> for RawIdentifier` as well as switching to `LeanString`.
The PR results in a roughly 2k TPS improvement over master for WASM +
hash indices. This should also help V8, as this is VM-agnostic.
# API and ABI breaking changes
None
# Expected complexity level and risk
3? Mostly just small changes in many places, but in a sea of small
changes, mistakes can happen.
# Testing
Covered by existing tests.
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
Copied from: https://github.com/clockworklabs/SpacetimeDB/pull/4084
We originally reverted this because it was causing testsuite flakes in
private. Now we have solved the issue that was causing the flakes so
this should be safe to merge.
Version upgrade to `v1.12.0`.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
1 - this is just a version upgrade
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
The testsuite failures are fixed by
https://github.com/clockworklabs/SpacetimeDB/pull/4120
- [x] License has been properly updated including version number and
date
- [x] CI passes
---------
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
This PR shrinks `JsWorkerRequest` so that it is (almost) as small as the
call reducer request.
To do that, a bunch of trivial changes had to be done to auth code, that
mostly revolves around `String` -> `Box<str>`.
This should help the auth code, but that is incidental.
The main goal was to improve throughput through the request tx/rx
channel for V8, which is taking quite a bit of time in flamegraphs.
I also noticed while making this change that the wrong hash map was
being used in a bunch of places, so I fixed all of those.
A follow up PR will shrink the reply side to fit within a cache line.
Yet another follow up PR will change the channel to replace flume with
`fibre::spsc`.
# API and ABI breaking changes
None
# Expected complexity level and risk
2, fairly trivial changes.
# Testing
Covered by existing tests.
# Description of Changes
This reverts the version bump, since it seems to be causing test
flakiness somehow.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
- [ ] CI passes
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
The first commit defines a type `TableName` that is used in e.g.,
`TxData` and where determined profitable and necessary to do this
change.
`TableName` is backed by
[`ecow::EcoString`](https://docs.rs/ecow/0.2.6/ecow/string/struct.EcoString.html)
which affords O(1) clones and 15 bytes of inline storage and
`mem::size_of::<EcoString>() == 16`.
The second commit does the same for `ReducerName`. This is also used in
reducer execution.
Together, these commits increase TPS by around 5-7k TPS.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Covered by existing tests.
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
Version upgrade to `v1.12.0`.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
1 - this is just a version upgrade
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
The testsuite failures are fixed by
https://github.com/clockworklabs/SpacetimeDB/pull/4120
- [x] License has been properly updated including version number and
date
- [x] CI passes
---------
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
Defines `SmallHashMap` in `spacetimedb_data_structures`.
The data structure is a hybrid map that is backed by a vector and linear
scan for the first `M` elements.
Then it falls back into a normal hash map.
For the first `N` elements, where `N < M`, the vector is stored inline.
That is, the vector is a `SmallVec<[(K, V); N>`.
The structure is optimized for access patterns where the keys and values
are accessed together,
therefore, keys and values are not stored separately.
# API and ABI breaking changes
None, just additive.
# Expected complexity level and risk
1
# Testing
A comprehensive proptest suite is added for the datastructure.
# Description of Changes
Client Query builder for rust, as per proposal -
https://github.com/clockworklabs/SpacetimeDBPrivate/pull/2356.
1. Pach moves query builder to its separate crate, so that it can be
shared between module and sdk.
2. Implements `TypedSubscriptionBuilder` in `sdks/rust` as mentioned in
proposal
3. Modify codegen to extend types to support query builder as mentioned
in proposal
4. a test
# API and ABI breaking changes
NA, additive changes.
# Expected complexity level and risk
2
# Testing
Added a test.
---------
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
Kim Altintop