# Description of Changes
This reverts the version bump, since it seems to be causing test
flakiness somehow.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
- [ ] CI passes
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
The first commit defines a type `TableName` that is used in e.g.,
`TxData` and where determined profitable and necessary to do this
change.
`TableName` is backed by
[`ecow::EcoString`](https://docs.rs/ecow/0.2.6/ecow/string/struct.EcoString.html)
which affords O(1) clones and 15 bytes of inline storage and
`mem::size_of::<EcoString>() == 16`.
The second commit does the same for `ReducerName`. This is also used in
reducer execution.
Together, these commits increase TPS by around 5-7k TPS.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Covered by existing tests.
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
Version upgrade to `v1.12.0`.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
1 - this is just a version upgrade
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
The testsuite failures are fixed by
https://github.com/clockworklabs/SpacetimeDB/pull/4120
- [x] License has been properly updated including version number and
date
- [x] CI passes
---------
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
- Convert basic-rs template from crates.io deps to workspace deps
(spacetime init will convert them back for users)
- Add basic-rs template to workspace members in root Cargo.toml
- Rename basic-rs package to avoid collision with sdk-test-connect-disconnect
- Simplify default_module_clippy test to run both templates in place
- Add missing st_client table verification to client_disconnected test
# Description of Changes
Defines `SmallHashMap` in `spacetimedb_data_structures`.
The data structure is a hybrid map that is backed by a vector and linear
scan for the first `M` elements.
Then it falls back into a normal hash map.
For the first `N` elements, where `N < M`, the vector is stored inline.
That is, the vector is a `SmallVec<[(K, V); N>`.
The structure is optimized for access patterns where the keys and values
are accessed together,
therefore, keys and values are not stored separately.
# API and ABI breaking changes
None, just additive.
# Expected complexity level and risk
1
# Testing
A comprehensive proptest suite is added for the datastructure.
Extract static smoketest modules into a nested workspace at
crates/smoketests/modules/ that is pre-compiled during warmup.
This eliminates per-test WASM compilation overhead.
Key changes:
- Add 38 precompiled module crates in nested workspace
- Add module registry (src/modules.rs) for WASM path lookup
- Add precompiled_module() builder and use_precompiled_module() method
- Update xtask warmup to build nested workspace
- Migrate all static tests to use precompiled modules
- Tests using precompiled modules run in ~0.5-3s vs ~4-7s before
Tests that need runtime compilation (auto_migration, detect_wasm_bindgen,
intentionally-broken modules) continue to use module_code().
- Translate 4 more Python smoketests to Rust: auto_inc, describe,
module_nested_op, panic
- Simplify the call API by removing the generic call<T: Serialize>
method and renaming call_raw to call, since CLI args are strings
- Remove unused serde dependency
Create `crates/smoketests/` to translate Python smoketests to Rust:
- Add `Smoketest` struct with builder pattern for test setup
- Implement CLI helpers: `spacetime_cmd()`, `call()`, `sql()`, `logs()`, etc.
- Translate `smoketests/tests/sql.py` → `tests/sql.rs`
- Translate `smoketests/tests/call.py` → `tests/call.rs`
- Reuse `ensure_binaries_built()` from guard crate (now public)
Also fix Windows process cleanup in `SpacetimeDbGuard`:
- Use `taskkill /F /T /PID` to kill entire process tree
- Prevents orphaned `spacetimedb-standalone.exe` processes
# Description of Changes
Client Query builder for rust, as per proposal -
https://github.com/clockworklabs/SpacetimeDBPrivate/pull/2356.
1. Pach moves query builder to its separate crate, so that it can be
shared between module and sdk.
2. Implements `TypedSubscriptionBuilder` in `sdks/rust` as mentioned in
proposal
3. Modify codegen to extend types to support query builder as mentioned
in proposal
4. a test
# API and ABI breaking changes
NA, additive changes.
# Expected complexity level and risk
2
# Testing
Added a test.
---------
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
# Description of Changes
Major documentation overhaul focusing on tables, column types, and
indexes.
**Quickstart Guides:**
- Updated React, TypeScript, Rust, and C# quickstarts with table/reducer
examples
- Fixed CLI syntax (positional `--database` argument)
- Improved template consistency across languages
**Tables Documentation:**
- Added "Why Tables" section explaining table-oriented design philosophy
(tables as fundamental unit, system tables, data-oriented design
principles)
- Added "Physical and Logical Independence" section explaining how
subscription queries use the relational model independently of physical
storage
- Added brief sections linking to related pages (Visibility,
Constraints, Schedule Tables)
- Renamed "Scheduled Tables" to "Schedule Tables" throughout (tables
store schedules; reducers are scheduled)
**Column Types:**
- Split into dedicated page with unified type reference table
- Added "Representing Collections" section (Vec/Array vs table
tradeoffs)
- Added "Binary Data and Files" section for Vec<u8> storage patterns
- Added "Type Performance" section (smaller types, fixed-size types,
column ordering for alignment)
- Added complete example struct demonstrating all type categories
- Renamed "Structured" category to "Composite"
**Indexes:**
- Complete rewrite with textbook-style documentation
- Added "When to Use Indexes" guidance
- Documented single-column and multi-column index syntax (field-level
and table-level)
- Comprehensive range query examples with correct TypeScript `Range`
class syntax
- Explained multi-column index prefix matching semantics
- Added index-accelerated deletion examples
- Included index design guidelines
**Styling:**
- Added CSS for table border radius and row separators
- Created Check component for green checkmarks in tables
# API and ABI breaking changes
None. Documentation only.
# Expected complexity level and risk
1 - Documentation changes only, no code changes.
# Testing
- [ ] Verify docs build without errors
- [ ] Review rendered pages for formatting issues
- [ ] Confirm code examples are syntactically correct
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Signed-off-by: John Detter <4099508+jdetter@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
Add `impl MemoryUsage for TxState` and all the types below.
Extracted from https://github.com/clockworklabs/SpacetimeDB/pull/3831 to
reduce the diff to ease figuring out why its not helping perf.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
This is the second step to make in-memory-only databases not touch the
disk at all.
While at it, also make it so file-backed module logs are streamed in
constant memory where possible.
Depends-on: #3912
# Expected complexity level and risk
2
# Testing
Added some unit-level tests.
---------
Signed-off-by: Kim Altintop <kim@eagain.io>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
- Small fix for checking to see if a port is available on some given
interface.
updated:
The original implementation here used `bind` to try to discover if a
port is currently in use. This isn't reliable due to platform
differences - especially on windows where it's apparently acceptable to
have a service running on both `0.0.0.0:3000` and `127.0.0.1:3000`. This
would cause bind to return successfully when we wanted it to fail. Also:
binding on an ipv6 interface when a machine doesn't have ipv6 enabled
caused random errors and it was too unreliable to be useful.
This new implementation uses `get_socket_info` which returns info on all
sockets in use on the system. We can then look through this list to find
services which conflict with our requested port.
updated 1/14:
This PR now includes a fix for flaky CLI tests. Originally we were using
`find_free_port` to pick a random free port, but that was causing a race
condition which resulted in test flakes. This PR fixes this issue by
using `127.0.0.1:0` as the listen addr so the kernel will automatically
pick a free port for us.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
1 - this is a pretty isolated check, unlikely to introduce larger
issues.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
I tested on macos, windows and linux:
```
ALLOW
docusaurus is already running on 127.0.0.1:3000
SpacetimeDB then tries to start on 192.168.1.10:3000
ALLOW
docusaurus is already running on ::1:3000
SpacetimeDB then tries to start on 192.168.1.10:3000
DENY
docusaurus is already running on ::1:3000
SpacetimeDB then tries to start on 127.0.0.1:3000
DENY
docusaurus is already running on 0:0:0:0:0:0:0:0:3000
SpacetimeDB then tries to start on 0.0.0.0:3000
DENY
docusaurus is already running on 0:0:0:0:0:0:0:0:3000
SpacetimeDB then tries to start on 127.0.0.1:3000
DENY
docusaurus is already running on 0:0:0:0:0:0:0:0:3000
SpacetimeDB then tries to start on 192.168.1.10:3000
DENY
docusaurus is already running on 127.0.0.1:3000
SpacetimeDB then tries to start on 0:0:0:0:0:0:0:0:3000
DENY
docusaurus is already running on 192.168.1.10:3000
SpacetimeDB then tries to start on 0:0:0:0:0:0:0:0:3000
```
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
- Version bump to `v1.11.3` for just the CLI and rust
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
1
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] CLI version has been updated
- [x] Version + date in the license file has been updated
# Description of Changes
Fixes https://github.com/clockworklabs/SpacetimeDB/issues/1122.
Adds hash indices and exposes them through `#[index(hash)]` for Rust
modules,
with support for typescript and C# to come in follow ups.
On the client/sdk side, for now, any index is backed via a BTree/native
index as it is the most general.
A hash index may only be queried through point scan and never ranged
scans.
Attempting a ranged scan results in an error, with the mechanism
implemented in the previous PR
(https://github.com/clockworklabs/SpacetimeDB/pull/3974).
# API and ABI breaking changes
None
# Expected complexity level and risk
2?
# Testing
A test for ensuring that hash indices cannot be used for range scans is
added.
Tests exercising hash indices will come in the next PR.
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
- Version upgrade to `1.11.2`
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
- None, this is just a version bump
# Expected complexity level and risk
1 - no real changes here
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
- NA this is just a version bump, no functionality change here.
This is the first step to make in-memory only databases not touch the
disk at all. Pending is an in-memory only sink for module logs.
Responsibility for the lock file is transferred to `Durability`, which
means that only persistent databases opened for writing acquire the
lock.
As a consequence, the `Durability` trait gains a `close` method that
prevents further writes and drains the internal buffers, even when
multiple `Arc`-pointers to the `Durability` exist.
# Expected complexity level and risk
2
# Testing
Covered by existing tests.
# Description of Changes
Introduce a new **LLM benchmarking app** and supporting code.
* **CLI:** `llm` with subcommands `run`, `routes list`, `diff`,
`ci-check`.
* **Runner:** executes globally numbered tasks; filters by `--lang`,
`--categories`, `--tasks`, `--providers`, `--models`.
* **Providers/clients:** route layer (`provider:model`) with HTTP LLM
Vendor clients; env-driven keys/base URLs.
* **Evaluation:** deterministic scorers (hash/equality, JSON
shape/count, light schema/reducer parity) with clear failure messages.
* **Results:** stable JSON schema; single-file HTML viewer to
inspect/filter/export CSV.
* **Build & guards:** build script for compile-time setup;
* **Docs:** `DEVELOP.md` includes `cargo llm …` usage.
This PR is the initial addition of the app and its modules (runner,
config, routes, prompt/segmentation, scorers, schema/types,
defaults/constants/paths/hashing/combine, publishers, spacetime guard,
HTML stats viewer).
### How it works
1. **Pick what to run**
* Choose tasks (`--tasks 0,7,12`), or a language (`--lang rust|csharp`),
or categories (`--categories basics,schema`).
* Optionally limit vendors/models (`--providers …`, `--models …`).
2. **Resolve routes**
* Read env (API keys + base URLs) and build the active set (e.g.,
`openai:gpt-5`).
3. **Build context**
* Start Spacetime
* Publish golden answer modules
* Prepare prompts and send to LLM model
* Attempt to publish LLM module
4. **Execute calls**
* Run the selected tasks within each test against selected models and
languages.
5. **Score outputs**
* Apply deterministic scorers (hash/equality, JSON shape/count, simple
schema/reducer checks).
* Record the score and any short failure reason.
6. **Update results file**
* Write/update the single results JSON with task/route outcomes,
timings, and summaries.
# API and ABI breaking changes
None. New application and modules; no existing public APIs/ABIs altered.
# Expected complexity level and risk
**4/5.** New CLI, routing, evaluation, and artifact format.
* External model APIs may rate-limit/timeout; concurrency tunable via
`LLM_BENCH_CONCURRENCY` / `LLM_BENCH_ROUTE_CONCURRENCY`.
# Testing
I ran the full test matrix and generated results for every task against
every vendor, model, and language (rust + C#). I also tested the CI
check locally using [act](https://github.com/nektos/act).
**Please verify**
* [ ] `llm run --tasks 0,1,2` (explicit `run`)
* [ ] `llm run --lang rust --categories basics` (filters)
* [ ] `llm run --categories basics,schema` (multiple categories)
* [ ] `llm run --lang csharp` (language switch)
* [ ] `llm run --providers openai,anthropic --models "openai:gpt-5
anthropic:claude-sonnet-4-5"` (provider/model limits)
* [ ] `llm run --hash-only` (dry integrity)
* [ ] `llm run --goldens-only` (test goldens only)
* [ ] `llm run --force` (skip hash check)
* [ ] `llm ci-check`
* [ ] Stats viewer loads the JSON; filtering and CSV export work
* [ ] CI works as intended
---------
Signed-off-by: bradleyshep <148254416+bradleyshep@users.noreply.github.com>
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@aol.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: spacetimedb-bot <spacetimedb-bot@users.noreply.github.com>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
Closes
[#3290](https://github.com/clockworklabs/SpacetimeDB/issues/3290).
Adds a new "special" type to SATS, `UUID`, which is represented as the
product `{ __uuid__: u128 }`. Adds versions of this type to all of our
various languages' module bindings libraries and client SDKs, and
updates codegen to recognize it and output references to those named
library types. Adds methods for creating new UUIDs according to the V4
(all random) and V7 (timestamp, monotonic counter and random)
specifications.
# API and ABI breaking changes
We add a new type
# Expected complexity level and risk
2
it impacts all over the code
# Testing
- [x] Extends the Rust and Unreal SDK tests, and the associated
`module-test` modules in Rust, C# and TypeScript, with uses of UUIDs.
- [x] Extends the C# SDK regression tests with uses of UUIDs.
- [x] Extends the TypeScript test suite with tests with uses of UUIDs.
---------
Signed-off-by: Mario Montoya <mamcx@elmalabarista.com>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
- Bump version numbers to `1.11.1`
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
1
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
- [x] Verified that the license has been updated
- [x] `spacetime --version` on this commit is correct
There is also a corresponding private PR.
# Description of Changes
Make `cargo ci` work properly on Windows, in preparation for
https://github.com/clockworklabs/SpacetimeDB/pull/3702.
# API and ABI breaking changes
No. CI-only.
# Expected complexity level and risk
2. Not trivial, but not complicated.
# Testing
- [x] CI output seems to be genuinely running the tests, and it's
passing on Windows
- [x] Make a change to `crates/bindings-csharp` and see that `cargo ci
test` fails
- [x] I can manually run a minimal `cargo ci smoketests` invocation on a
Windows machine
---------
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Kasama <robertoaall@gmail.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Uses `with_host_stack` to provide a `StackCreator` that pools
`FiberStack`s.
This does not use the pooling instance allocator and is limited to just
stacks.
# API and ABI breaking changes
None
# Expected complexity level and risk
3? Some unsafe and wasmtime internals relied upon.
# Testing
Covered by existing tests.
# Description of Changes
This changes the ci runs to execute `cargo ci` instead of running
commands directly from the github workflow.
The goal here is to unify the commands under `cargo ci` so that it's
easier and more intuitive to run locally
# API and ABI breaking changes
There are no API/ABI changes.
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
Complexity: 1
It is not a complex change as it is mostly localized to the ci runs and
is easily reversible if something goes wrong. The biggest risk here is
to have future CI runs break, which can be remediated by reverting these
changes.
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] run `cargo ci` and its subcommands locally
- [x] run the github workflow against this branch to check if the CI
jobs are working properly.
---------
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Signed-off-by: Roberto Pommella Alegro <robertoaall@gmail.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
RFC 6455, Section 5.4 describes message fragmentation, and we can do
that with tungstenite.
It does seem to help getting control messages (ping, pong, close)
through without head-of-line blocking.
# Expected complexity level and risk
2 - Need to test with clients
# Testing
TBD - some more abstraction is needed due to the difficulty of
synthetically producing a large outgoing message.
# Description of Changes
Uses the `sourcemap` crate to map text locations in the bundle to text
locations in the original source code.
# Expected complexity level and risk
1 - essentially only related to diagnostics
# Testing
- [x] Manually tested
- [ ] Add an automated test for backtrace output
# Description of Changes
Fixes https://github.com/clockworklabs/SpacetimeDB/issues/3729
I genuinely don't know what came over me.
# API and ABI breaking changes
None
# Expected complexity level and risk
1.5 very straightforward but not strictly trivial
# Testing
Adds automated integration tests (written in Rust and run with `cargo
test`, although note this comment from @matklad about integration tests
for the future
https://internals.rust-lang.org/t/running-test-crates-in-parallel/15639/2):
- [x] Can publish an updated module if no migration is required
- [x] Can publish an updated module if auto-migration is required (with
the yes-break flag true/false)
- [x] Cannot publish if a manual migration is required
- [x] Can publish if a manual migration is required but the user
specified `--delete-data`
- [x] Can publish if a manual migration is required by the user
specified `--delete-data=on-conflict`
- [x] No data deletion occurs if no migration is required and
`--delete-data=on-conflict` is specified
---------
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
As discussed; could possibly improve performance/in general it's good to
keep up with patches, it's been over a year since we last bumped this.
# Expected complexity level and risk
2 - large, important dependency but wasmtime is very solid,
well-engineered software.
# Testing
- [x] smoketests run on wasmtime
# Description of Changes
Bumping versions to 1.11.0 in preparation for an upcoming release.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
- [x] Existing CI passes
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
This upgrades the SpacetimeDB version to 1.10.0.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
1
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
This is just a version bump - not tested.
# Description of Changes
Rust SDK test suite for views
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
This patch only adds tests, it does not change functionality.
# Description of Changes
There were mentions of `hashbrown` in the repo that did not go through
`spacetimedb_data_structures::map`.
This caused compile errors on master when running certain tests locally.
These have been replaced with the proper imports.
The PR also bump hashbrown to 0.16.1 and foldhash to 0.2.0.
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
Covered by existing tests.
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
Upgrade to version 1.9.0.
# API and ABI breaking changes
None - just a version upgrade.
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
1
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] I verified that the license has been updated
- [x] The version number looks correct (1.9.0)
---------
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Follow up to #3684. Moves `Error` and `Timeout` out of lib, so that we
don't have to implement `SpacetimeType` for them, and then removes the
http dependency altogether, so that `lib` can be leaner. I also got rid
of the separate `HttpValue` type, since it only really exists to mirror
the `http` crate and typescript won't make use of it.
# Expected complexity level and risk
1
# Testing
n/a - just code movement.
# Description of Changes
Closes#3517 .
With this PR, procedures (at least, those defined in Rust modules) can
perform HTTP requests! This is performed through a new field on the
`ProcedureContext`, `http: HttpClient`, which has a method `send` for
sending an `http::Request`, as well as a convenience wrapper `get`.
Internally, these methods hit the `procedure_http_request` ABI call /
host function, which uses reqwest to perform an HTTP request. The
request is run with a user-configurable timeout which defaults and is
clamped to 500 ms.
Rather than exposing the HTTP stream to modules, we download the entire
response body immediately, within the same timeout.
I've added an example usage of `get` to `module-test` which performs a
request against `localhost:3000` to read its own schema/moduledef.
This PR also makes all procedure-related definitions in the Rust module
bindings library `#[cfg(feature = "unstable")]`, as per #3644 . The
rename of the `/v1/database/:name/procedure/:name` route is not included
in this PR, so this does not close#3644 .
Left as TODOs are:
- Metrics for recording request and response size.
- Improving performance by stashing a long-lived `reqwest::Client`
someplace.
Currently we build a new `Client` for each request.
- Improving performance (possibly) by passing the request-future to the
global tokio executor
rather than running it on the single-threaded database executor.
# API and ABI breaking changes
Adds new APIs, which are marked as unstable. Adds a new ABI, which is
not unstable in any meaningful way (we can't really do that). Marks
unreleased APIs as unstable. Does not affect any pre-existing
already-released APIs or ABIs.
# Expected complexity level and risk
3 or so: networking is scary, and even though we impose a timeout which
prevents these connections from being truly long-lived, they're still
potentially long-lived on the scale of Tokio futures. It's possible that
running them on the database core is problematic in some way, and so
what I've left as a performance TODO could actually be a
concurrency-correctness issue.
# Testing
- [x] Manually wrote and executed some procedures which make HTTP
requests.
- [x] Added two automated tests to the `sdk-test` suite,
`procedure::http_ok` and `procedure::http_err`, which make successful
and failing requests respectively, then return its result. A client then
makes some assertions about the result.
---------
Co-authored-by: Noa <coolreader18@gmail.com>
# Description of Changes
This PR is a very large change to the workings of the TypeScript SDK and
as such requires a higher bar of testing than other PRs. However, it
does several important things:
1. Unifies the API of the server and client so they not only have the
same API, but they actually implement it with the same TypeScript types.
This fixes several inconsistencies between them and fixes several small
bugs as well.
2. Closes https://github.com/clockworklabs/SpacetimeDB/issues/3365
3. Closes https://github.com/clockworklabs/SpacetimeDB/issues/3431
4. Closes https://github.com/clockworklabs/SpacetimeDB/issues/3435
5. Subsumes the work done in
https://github.com/clockworklabs/SpacetimeDB/pull/3447
6. Derives all type information on the client from a single
`RemoteModule` type which vastly cleans up the correctness of type
checking on the client and helped me to find several small bugs
It accomplishes this by changing code generation of TypeScript on the
client to code generation approximately what a developer would manually
write in their module. The ultimate goal would be to allow the developer
to use the types and functions that they define on in their module
directly on the client without needing to do any code generation at all,
provided they are using TypeScript on the server and client.
https://github.com/clockworklabs/SpacetimeDB/issues/3365 is resolved by
`.build()`ing the `DbConnection` inside a React `useEffect` rather than
doing it directly in line with the render of the provider. In order to
do that we needed to not expose the `DbConnection` directly to
developers by returning a different type from `useSpacetimeDB`.
`useSpacetimeDB` now returns a `ConnectionState` object which is stored
as React state and updates when any of the fields change. This change
also resolves https://github.com/clockworklabs/SpacetimeDB/issues/3431.
https://github.com/clockworklabs/SpacetimeDB/issues/3435 was the issue
that initially lead me down the rabbit hole of unifying the server and
the client because it was nearly impossible to track down all the
various type functions and how they connect to the values that we code
generate on the server. After several hours of attempting this, I
decided to clean up the types a bit to be more uniform.
Implementing the unification between the client and the server also
necessitated fully implemented parts of the API that were fully
implemented on the server, but were broken or missing on the client.
# API and ABI breaking changes
[Unification] -> Means that this is breaking behavior for the client
SDK, but that the new behavior is identical to the server's existing
behavior
## Breaking changes:
- Table accessor names and index accessor names are converted to
camelCase on the `ctx`, so `ctx.db.foo_bar` is now `ctx.db.fooBar`
- [Unification] On the client `my_table.iter()` returns
`IterableIterator` instead of an `Array`
- [Unification] `module_bindings` now export `TypeBuilder`s for all
types instead of a `type MyType` and object `MyType`, so instead of
using `MyType` as a type directly, you need to infer the type `MyType`
-> `Infer<typeof MyType>`.
- [Unification] We no longer generate and export `MyTypeVariants` for
sum types (these are now accessed by `Infer<typeof
MyType.variants.myVariant>`)
- [Unification] `MyType.getTypeScriptAlgebraicType()` has been replaced
with `MyType.algebraicType`
- `useSpacetimeDB()` no longer takes type parameters
- `useTable()` now takes a `TableDef` parameter and type params are
inferred
- `useTable()` now just returns an `Array` directly instead of a object
with `{ rows }`
- [Unification] `ctx.reducers.createPlayer(argA, argB)` ->
`ctx.reducers.createPlayer({ argA, argB })`
- [Unification] `ctx.reducers.onCreatePlayer(ctx, argA, argB)` ->
`ctx.reducers.onCreatePlayer(ctx, { argA, argB })`
- [Unification] `ctx.reducers.removeOnCreatePlayer(ctx, argA, argB)` ->
`ctx.reducers.removeOnCreatePlayer(ctx, { argA, argB })`
- [Unification] `myTable.count(): number` -> `myTable.count(): bigint`
## Additive changes:
- `Infer<>` now also does `InferTypeOfRow<>` if applicable
- Added a `useReducer()` React hook
- `module_bindings` now exports a `tables` object with references to all
the `TableDef`s
- `module_bindings` now exports a `reducers` object with references to
all the `ReducerDef`s
- Added a new `MyType.create('MyVariant', ...)` function in addition to
the `MyType.MyVariant(...)` constructors (this is private)
## Notable things that did not change:
- `MyType.serialize(writer: BinaryWriter, value: Infer<typeof MyType>)`
and `MyType.deserialize(reader: BinaryReader): Infer<typeof MyType>` are
still supported exactly as before.
- The `MyType.MyVariant(...)` constructor function on sum types is still
present, but implemented with the private `MyType.create('MyVariant',
...)`. We could choose to move away from this API later if we didn't
like the variants polluting the namespace
# Expected complexity level and risk
4 - This is a deep reaching an complex change for the SDK. For the
server, it is much less deep reaching since it reuses much of the same
machinery, although it does require thorough testing there as some of
the code was modified.
This change is fully localized to TypeScript and does not touch the host
(or other languages) at all, and therefore only impacts a beta aspect of
SpacetimeDB.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [ ] Added regression test for
https://github.com/clockworklabs/SpacetimeDB/issues/3435
- [x] Manually tested `test-app` and `test-react-router-app`
- [ ] Add test cases for camelCase-ing
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: Noa <coolreader18@gmail.com>
# Description of Changes
This makes us go from 3 threads to 2.
The next step is to core pin the V8 worker thread.
# API and ABI breaking changes
None
# Expected complexity level and risk
4
# Testing
Existing tests should cover this.
---------
Co-authored-by: Noa <coolreader18@gmail.com>
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
- This upgrades the versions of all SDKs, the CLI, etc. to 1.8.0
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
1
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] I verified that all versions seem to be updated including the BSL
license update <!-- maybe a test you want to do -->
We have 1 `1.7.0` that didn't get upgraded automatically because it is
part of the module bindings for a template:
```
crates/cli/.templates/parent_parent_crates_bindings-typescript_examples_quickstart-chat/src/module_bindings/index.ts: cliVersion: '1.7.0',
```
A case could possibly be made for bumping the template but it shouldn't
cause any issues as the module bindings directory should just get
regenerated by the user. @cloutiertyler should we be bumping module
bindings for templates when we upgrade versions?
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
This adds authorization to the relevant API endpoints, updates the CLI
commands and adds smoketests for the teams feature.
**Note**: Authorizing SQL (incl. subscriptions) is a bit more involved,
and submitted as a separate PR in the series.
Depends-on: https://github.com/clockworklabs/SpacetimeDB/pull/3519
So far, the `--clear-database` option to `publish` has simply dropped
and then re-created the database (if it did exist).
This will no longer work when databases can have "children": because
dropping and re-creating is not atomic, children would either become
orphans, or be dropped as well.
To solve this, `reset_database` is introduced as a separate action that:
- shuts down all replicas
- if a `program_bytes` is supplied, replaces the database's initial
program
- if a `host_type` is supplied, replaces the database's host type
- starts `num_replicas` or the previous number of replicas, which
initialize themselves as normal
As this could be its own CLI command, the action is provided as its own
API endpoint (undocumented).
However, since `publish` has no way of knowing whether the database it
operates on actually exists, the `publish_database` handler will just
invoke the `reset_database` handler if `clear=true` and the database
exists, and return its result. This is to avoid starting the transfer of
the program in the request body, only to receive a redirect.
Some refactoring was necessary to dissect and understand the flow.
# API and ABI breaking changes
Introduces a new, undocumented API endpoint.
We may want to nest it under `/unstable`.
# Expected complexity level and risk
2
# Testing
From the outside, the observed behavior should be as before,
so smoketests should cover it.
# Description of Changes
This commit adds support to the Rust client SDK for calling procedures.
Similar to reducers, each `DbContext` implementor has a `pub procedures:
RemoteProcedures` field, with methods provided by extension traits for
each procedure.
Unlike reducers, the provided methods are invoke and invoke-then.
Invoke-then takes a `FnOnce` callback to run
when the SDK is notified of the procedure's termination status, while
invoke ignores that notification.
No mechanism is provided for observing procedures invoked by other
clients.
Procedure callbacks are implemented by storing a map from `request_id`
to `ProcedureCallback`, with the callback closure internally knowing how
to deserialize the return value. It's mildly unfortunate to deserialize
within the callback instead of on the preprocess background task, but it
saves significant complexity.
This commit also adds a new sdk-test module, `sdk-test-procedures`, and
a new Rust test client, `procedure-client`.
Together, these are used in two tests of invoking and observing
procedures. I've left TODOs for other tests that we should write as we
implement additional procedure features.
I also had to fix a few minor bugs in the Rust codegen which were not
strictly related to procedures: we previously assumed that the sets of
reducers and of tables were non-empty, which led to wonky invalid
codegen on modules which did not define any reducers or which did not
define any tables.
I'm sneaking a change to the Nix flake into this PR as well - when
initially writing it I had included `cargoArtifacts` (the pre-built and
cached dependencies of our actual builds) in its `packages`, but that
was neither necessary or useful, and just made building the shell take a
long time after dependency changes.
# API and ABI breaking changes
Breaks the internal interface between the Rust client SDK and codegen,
so users will have to re-run `spacetime generate`.
# Expected complexity level and risk
2-ish? Pretty simple change to the Rust SDK overall.
# Testing
- [x] Added new automated integration tests exercising the new
functionality.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
When a new commitlog segment is created, allocate disk space for it up
to the maximum segment size. Also do this when resuming writes to an
existing segment, such that segments created without preallocation will
allocate as well when the database is opened.
Preallocation is gated behind the feature "fallocate", because it is not
always desirable to preallocate, e.g. for local `standalone` users.
The feature can only be enabled on Linux targets, because allocation is
done using the Linux-specific `fallocate(2)` system call.
Unlike `ftruncate(2)` or the portable `posix_fallocate(3)`,
`fallocate(2)`
supports allocating disk space without zeroing. This is currently
required, because the commitlog format does not handle padding bytes.
If not enough space can be allocated, the commitlog refuses writes. For
commitlogs that were created without preallocation, this means that the
commitlog cannot even be opened in this situation.
The local durability impl will crash if it detects that the commitlog is
unable to allocate enough space.
This means that a database will eventually crash and be unable to start
in
an out-of-space situation.
Allocated space is not included in the reported size of the commitlog.
Instead, allocated blocks are reported separately.
# Expected complexity level and risk
3 - Disk size monitoring may need to be adjusted.
# Testing
- [x] Adds a test that demonstrates the crash behavior of
[`spacetimedb_durability::Local`]
when there is insufficient space. The test performs I/O against a loop
device.
- [x] Modified the `repo::Memory` impl so that it can run out of space.
No test currently
utilizes this, but existing tests assuming infinite space still pass.
# Description of Changes
Host implementation to invoke `call_view` method.
I also covers:
1. API `MutTxId::is_materialized`to check if existing view exisits and
updated.
2. Update in readsets logic to remove stale views.
3. sql caller implmentation.
# API and ABI breaking changes
NA
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
3
# Description of Changes
This reverts commit #3496.
# API and ABI breaking changes
Technically maybe yes? But definitely nothing is using the new code yet.
# Expected complexity level and risk
1
# Testing
CI only
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
So far, the `--clear-database` option to `publish` has simply dropped
and then re-created the database (if it did exist).
This will no longer work when databases can have "children": because
dropping and re-creating is not atomic, children would either become
orphans, or be dropped as well.
To solve this, `reset_database` is introduced as a separate action that:
- shuts down all replicas
- if a `program_bytes` is supplied, replaces the database's initial
program
- if a `host_type` is supplied, replaces the database's host type
- starts `num_replicas` or the previous number of replicas, which
initialize themselves as normal
As this could be its own CLI command, the action is provided as its own
API endpoint (undocumented).
However, since `publish` has no way of knowing whether the database it
operates on actually exists, the `publish_database` handler will just
invoke the `reset_database` handler if `clear=true` and the database
exists, and return its result. This is to avoid starting the transfer of
the program in the request body, only to receive a redirect.
Some refactoring was necessary to dissect and understand the flow.
# API and ABI breaking changes
Introduces a new, undocumented API endpoint.
We may want to nest it under `/unstable`.
# Expected complexity level and risk
2
# Testing
From the outside, the observed behavior should be as before,
so smoketests should cover it.
Zeke Foppa