# Description of Changes
Review #5287 first.
This patch updates view backing table schemas to have a single private
column `arg_hash`. Previously sender-scoped views had a `sender` column
for the calling identity, however now that's been replaced with a single
unified `arg_hash` column that encodes the calling identity within it.
When we add parameterized views, the view args will also be encoded in
this hash and stored in this column.
This column exists for both anonymous and sender scoped views meaning
that the backing tables for all views now have the same number of
private columns - one.
This hash is now used as a runtime variable that the query engine uses
to evaluate view table scans.
In order to keep the diff small, this patch does update view read sets
with this new hash value. That change has a larger blast radius and will
be done in the next set of changes.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
2
# Testing
Existing coverage.
# Description of Changes
Adds support for primary keys to procedural views in rust and typescript
modules. Now clients can receive update events when subscribed to such
views.
```rust
#[spacetimedb::view(accessor = my_players, public, primary_key = id)]
pub fn my_players(ctx: &spacetimedb::ViewContext) -> Vec<Player> {
ctx.db.players().owner().filter(ctx.sender()).collect()
}
```
```ts
const Player = t.row('Player', {
id: t.u64().primaryKey(),
owner: t.identity().index('btree'),
name: t.string(),
});
export const my_players = spacetimedb.view(
{ public: true },
t.array(players.rowType),
ctx => Array.from(ctx.db.players.owner.filter(ctx.sender))
);
```
In rust, a view's primary key is declared within the `#[view]` macro,
whereas in typescript it is declared at the column/row level as it is
for tables. I could have made it a view level attribute for typescript
as well, but since primary keys are already row-level attributes in
typescript, I just kept that as is.
Note, care must be taken to ensure that the view never returns duplicate
primary keys, or else it will fail which will currently result in the
transaction that triggered the view refresh to be rolled back. Better
error handling for this exact scenario will be added in a separate
follow up patch.
## Alternative Considered: #[primary_key] attribute on `SpacetimeType`s
in Rust
This would be equivalent to what we do in typescript.
# API and ABI breaking changes
None, although adding a primary key to an existing view will require a
client update.
# Expected complexity level and risk
3
# Testing
Compile-time failure scenarios:
- [x] Primary key on non-existent column
- [x] Primary key doesn't reference custom accessor name (rust)
- [x] Multiple primary key columns specified (typescript)
- [x] Primary keys must be `FilterableValue`s
SDK tests (rust sdk, rust and ts modules):
- [x] `OnUpdate`
- [x] `OnUpdate` is sender-scoped
- [X] Can join on primary key columns
Smoketests:
- [x] Modifying primary key columns breaks clients (auto-migrations)
## Summary
- repoint selected BSL crate `LICENSE` symlinks to
`licenses/apache2.txt`
- add the missing Unreal SDK `LICENSE` symlink to `licenses/apache2.txt`
Requested crates/SDK:
- bindings-macro
- bindings-sys
- bindings
- lib
- primitives
- query-builder
- sats
- unreal SDK
Note: `query-builder` was already pointing at `licenses/apache2.txt`, so
it has no diff.
## Testing
- Verified all requested `LICENSE` paths resolve to
`../../licenses/apache2.txt`
- `cargo run -p check-license-symlinks` does not complete cleanly in
this checkout because it also scans unrelated existing copied template
files and `node_modules` license files that are not symlinks
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
Adds support to Rust modules and the SpacetimeDB host for defining HTTP
handlers and registering them to routes.
## User-facing API
In a Rust module, users can annotate functions with the new macro
`#[spacetimedb::http::handler]`. A function annotated this way must
accept exactly two arguments, of types `&mut
spacetimedb::http::HandlerContext` and `spacetimedb::http::Request`
(which is a type alias for `http::Request<spacetimedb::http::Body>`. It
must also return `spacetimedb::http::Response` (which is a type alias
for `http::Response<spacetimedb::http::Body>`).
Once the user has defined an HTTP handler, they can register it to a
route by annotating a function with `#[spacetimedb::http::router]`. Such
a function must take no arguments and return a
`spacetimedb::http::Router`. (The original design put this annotation on
a `static` variable rather than a function, but that turned out to be
undesirable because it required that constructing a `Router` be
`const`.) `Router` exposes various methods for registering handlers to
routes.
All of a database's user-defined routes are exposed under
`/v1/database/:name_or_identity/route/{*path}`.
## Example
See [the new
smoketest](https://github.com/clockworklabs/SpacetimeDB/blob/phoebe/http-handlers-webhooks/crates/smoketests/tests/smoketests/http_routes.rs)
for a more exhaustive example.
A simpler example, which stores arbitrary byte data in a table via a
`POST` request, returns an ID, and then retrieves that same data via a
`GET` request with a query parameter:
```rust
#[spacetimedb::table(accessor = data)]
struct Data {
#[primary_key]
#[auto_inc]
id: u64,
body: Vec<u8>,
}
#[spacetimedb::http::handler]
fn insert(ctx: &mut HandlerContext, request: Request) -> Response {
let body: Vec<u8> = request.into_body().into_bytes().into();
let id = ctx.with_tx(|tx| tx.db.data().insert(Data { id: 0, body: body.clone() }).id);
Response::new(Body::from_bytes(format!("{id}")))
}
#[spacetimedb::http::handler]
fn retrieve(ctx: &mut HandlerContext, request: Request) -> Response {
let id = request
.uri()
.query()
.and_then(|query| query.strip_prefix("id="))
.and_then(|id| u64::from_str(id).ok())
.unwrap();
let body = ctx.with_tx(|tx| tx.db.data().id().find(id).map(|data| data.body));
if let Some(body) = body {
Response::new(Body::from_bytes(body))
} else {
Response::builder().status(404).body(Body::empty()).unwrap()
}
}
#[spacetimedb::http::router]
fn router() -> Router {
Router::new().post("/insert", insert).get("/retrieve", retrieve)
}
```
## Design and implementation notes
- As mentioned above, the router is registered via a function, not a
`static` or `const` item. This is because `static` or `const`
initializers must be `const`, and it turns out to be a pain to make all
of the `Router` constructors be `const fn`s.
- The `#[handler]` macro clobbers the original function name with a
`const` variable of type `HttpHandler`. This is unfortunate, but AFAICT
necessary, 'cause we need to pass the string identifier for the handler
to the `Router`, not the function pointer, and Rust allows no (stable
and reliable) way to get a unique string identifier out of a function
item/value, nor to attach data or implement traits for function
items/values. The alternative(s) would involve changing the signature of
the `Router` methods to have uglier and more complex callsites, e.g.
like `.get("/retrieve", retrieve::handler())`, `.get("/retrieve",
handler!(retrieve))` or `.get::<retrieve>("/retrieve")`. I believe that
registering handlers will be much more common than calling their
functions, so I've chosen to make it so that registering them gets the
convenient syntax, even though the inability to call them directly will
be somewhat surprising.
- I haven't wired up energy handling or timing metrics for handler
execution to anywhere. Procedures are still in the same boat.
- HTTP requests to user-defined routes bypass the usual SpacetimeDB auth
middleware, meaning that the host does not validate (or inspect in any
way) `Authorization` headers in requests before invoking the
user-defined handler. This is required to allow arbitrary
user-programmable handling of `Authorization` headers, including those
in formations which SpacetimeDB would reject. As a result of this,
`HandlerContext` doesn't expose a `sender` or `sender_connection_id`.
- HTTP route paths may consist only of a very restrictive set of
characters. I've chosen this set to keep our options open in the future
to add additional syntax to routes, like for registering wildcard
segments and path parameters:
- ASCII digits.
- ASCII letters.
- `-_~/`.
- The internal data structure that represents a `Router` is currently a
`Vec<Route>`, meaning that resolving a request to a route is
`O(num_routes)`. Registering a route checks against each previous route
for uniqueness, meaning that constructing a router is `O(num_routes ^
2)`. There are TODO comments to use a trie, but I think this can wait,
as I expect most databases to register few routes.
- Commit 999a7c317 contains a fix to a mostly-unrelated bug where a few
bindings introduced by the SATS derive macros were unhygienic and not in
a reserved namespace, leading to name conflicts. I discovered this
'cause I tried writing an HTTP handler named `index` to serve the
index/root of a website and it broke.
## Still TODO
- [x] Resolve various TODO comments in the diff.
- [x] Documentation.
- [x] C# bindings support.
- [x] C++ bindings support.
- [x] V8 host support.
- [x] TypeScript bindings support.
# API and ABI breaking changes
New APIs, currently flagged as `unstable`, which will eventually need
stability guarantees. No (intentional) breaking changes, or changes to
existing APIs at all.
# Expected complexity level and risk
3? Changes to our HTTP routing to support the user-defined routes.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] New smoketest of the behavior!
- [x] I dunno, maybe try hosting a simple webpage and see how it works?
- [x] Build a test app with Stripe integration.
- @aasoni did this.
---------
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
# Description of Changes
Revert recursive module mounts in schema def. This appears to be causing
unintended issues for language bindings.
The intention was for this to not cause any issues, but it appears to be
problematic when adding new enum veriants to the def sections (e.g. for
procedures).
# API and ABI breaking changes
No
# Expected complexity level and risk
Reverting to a previous version
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [ ] <!-- maybe a test you want to do -->
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
Closes#4989
We were not using `arbitrary_precision` in the workspace: I only found
the workspace dependency declaration itself plus a normal
`serde_json::Number::from(3u8)` call in
`crates/cli/src/spacetime_config.rs`, which does not require the
feature.
## Summary
- remove `serde_json/arbitrary_precision` from the workspace dependency
declaration
- keep `raw_value`, which is the only serde_json feature the workspace
is clearly using here
- avoid forcing a non-additive serde_json feature onto downstream
consumers via feature unification
## Validation
- `cargo tree -e features -p serde_json`
- `cargo check -p spacetimedb-sdk -p spacetimedb-lib -p spacetimedb-sats
-p spacetimedb-schema`
---------
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Add a recursive `mounts` fields to ModuleDef that references submodules.
# API and ABI breaking changes
This should not be a breaking change. However new V10 modules that are
built with this change will not be publishable to older hosts as the new
field won't be recognized.
# Expected complexity level and risk
2
# Testing
Mostly tested using ci tests. Will be tested more thoroughly in the
typescript PR that will use the new mounts field.
# Description of Changes
Make Timestamp a FilterableValue in Rust, C#, and Typescript. I'm not
sure this is changing all the places because we have the server and the
client in those 3 languages.
# API and ABI breaking changes
It's an additive change.
# Expected complexity level and risk
3. There are some designs decisions, like comparing timestamp to
strings/numbers.
# Testing
Added unit tests for the 3 languages.
# Description of Changes
Query builder views return a subset of rows from a physical table. If
that table has a primary key, then so should the view. What this means
concretely is that the view should expose the same api as the table,
specifically as it relates to the primary key column.
With that in mind, this patch commits the following changes:
1. Annotates `ViewDef` with a `primary_key`
2. Updates the return type of query builder views in the raw module def
to a special product type
3. Adds an index for the primary key on the view's backing table
4. Updates the query planner to use this index
5. Updates rust client codegen to generate `on_update` for such views
# API and ABI breaking changes
None
Old `impl Query` views compiled with an older version of SpacetimeDB
will continue to work as they did before - without a primary key.
# Expected complexity level and risk
3
# Testing
- [x] New rust sdk integration suite exercising `on_update` for PK views
and semijoin scenarios
- [x] Smoketests for PK views and semijoin scenarios
## Summary
Adds a cross-language `#[spacetimedb::settings]` mechanism for declaring
module-level settings. The first supported setting is
`CASE_CONVERSION_POLICY`, which controls how identifiers are
case-converted.
### Rust
```rust
use spacetimedb::CaseConversionPolicy;
#[spacetimedb::settings]
const CASE_CONVERSION_POLICY: CaseConversionPolicy = CaseConversionPolicy::None;
```
- Proc macro validates const name against `KNOWN_SETTINGS`
- Generates `__preinit__` export for registration
- Duplicate settings produce a linker error (duplicate symbol)
### TypeScript
```ts
export const moduleSettings: ModuleSettings = {
caseConversionPolicy: CaseConversionPolicy.None,
};
```
- `ModuleSettings` interface with typed fields
- `caseConversionPolicy` section pushed in `rawModuleDefV10()`
## Stacked on
- `shub/case-conversion` (base)
- PR #4364 (internal opt-out API)
This PR adds the proper user-facing API on top of the internal mechanism
in #4364.
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
Update the Default casing policy to `snake_case` for `RawModuleDefV10`.
Messy PR contains changes at different places, so that CI can pass:
Here are the main changes as follows:
- `bindings-macro` & `bindings` crate: `name` macro in Indexes for
canonical name and supply it to `RawModuleDefV10` via `ExplicitNames`.
- `bindings-typescript`:
- Changes has been reviewed through this PR -
https://github.com/clockworklabs/SpacetimeDB/pull/4308.
- `binding-csharp`: a single line change to pass `sourceName` of index
instead of null.
- `codegen`:
- Changes has been merged from branch -
https://github.com/clockworklabs/SpacetimeDB/pull/4337.
- Except a fix in rust codegen to use canonical name in Query buillder
instead of accessor.
- `lib/db/raw_def`: Extends `RawDefModuleV10` structure to support case
conversion.
- `schema` crate:
- `validate/v9` - Nothing itself should change or changes in v9
validation logic but the file contains a `CoreValidator` which is shared
with `validate/v10`. No test have t be updated to `validate/v9` which
ensures we aren't regressing it.
- `validate/v10`: This is the main meat, look at the new tests added in
bottom to understand what it does.
- Rest of the files are either test updates or module bindings.
## Testing:
1. Extensive unit tests have been added to verify generated `ModuleDef`
is correct.
2. I have done some e2e testing to verify rust codegen with rust and
typescript modules.
3. I would have like to do more testing for other codegens , I am
continue doing .
I have removed `sql.py` smoketest, as that seems to be already migated
in new framework and was headache to update.
## Expected complexity level and risk
4, It could have side-effect which aren't easily visible.
- - -
---------
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <bot@clockworklabs.com>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
Co-authored-by: Noa <coolreader18@gmail.com>
Co-authored-by: = <cloutiertyler@gmail.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@clockworklabs.io>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
## Description of Changes
This PR primarily affects the `bindings-macro` and `schema` crates to
review:
### Core changes
1. Replaces the `name` macro with `accessor` for **Tables, Views,
Procedures, and Reducers** in Rust modules.
2. Extends `RawModuleDefV10` with a new section for:
* case conversion policies
* explicit names
New sections are not validated in this PR so not functional.
3. Updates index behavior:
* Index names are now always **system-generated** for clients. Which
will be fixed in follow-up PR when we start validating RawModuleDef with
explicit names.
* The `accessor` name for an index is used only inside the module.
## Breaking changes (API/ABI)
1. **Rust modules**
* The `name` macro must be replaced with `accessor`.
2. **Client bindings (all languages)**
* Index names are now system-generated instead of using explicitly
provided names.
**Complexity:** 3
A follow-up PR will reintroduce explicit names with support for case
conversion.
---------
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <bot@clockworklabs.com>
# Description of Changes
Case conversion addtion to `RawModuleDefV10`.
Chunk has been taken from -
https://github.com/clockworklabs/SpacetimeDB/pull/4264 to unblock
csharp, and ts modules.
# API and ABI breaking changes
NA
# Description of Changes
Cherry-picks the core event table implementation (`fa83d6d40`,
`07d57a552`) and adds:
1. **Bootstrap fix**: Register `st_event_table` in
`bootstrap_system_tables()` — it was missing from the cherry-picked
commit, causing event table creation to fail at runtime when inserting
into `ST_EVENT_TABLE_ID`.
2. **4 datastore unit tests** covering event table behaviors:
- Insert+delete in the same tx cancels out (no TxData entry, no
committed state)
- Update yields only the final row in TxData (old row replaced, not
accumulated)
- Bare insert records in TxData but not in committed state
- `replay_insert` is a no-op for event tables (commitlog replay doesn't
rebuild state)
3. **Migration validation**: `ChangeTableEventFlag` error in
`auto_migrate.rs` prevents changing a table's event flag
(event→non-event or vice versa) via auto-migration. Includes tests for
both directions and a positive test that identical flags are accepted.
4. **Test fixture updates**: Bootstrap test expectations updated for the
new `st_event_table` system table (table row, column, index ID 23,
constraint ID 19). Enabled `spacetimedb-schema/test` feature in
datastore dev-deps to fix pre-existing test compilation issue with
`for_test` methods.
# API and ABI breaking changes
None. The `ChangeTableEventFlag` error variant is added to the internal
`AutoMigrateError` enum but this is not a public API.
# Expected complexity level and risk
1 — Straightforward additions. The bootstrap fix is a one-liner that was
simply missing from the cherry-pick. The tests exercise existing
behavior without changing it. The migration guard follows the exact same
pattern as the existing `ChangeTableType` check.
# Testing
- [x] `cargo test -p spacetimedb-datastore` — 77 tests pass (73 existing
+ 4 new)
- [x] `cargo test -p spacetimedb-schema --features test --lib` — 98
tests pass (96 existing + 2 new)
- [x] `cargo check --workspace --exclude view-client` — compiles clean
(`view-client` has a pre-existing issue with `CanBeLookupTable` from the
cherry-pick)
- [ ] Reviewer: confirm that the 4 event table test names/assertions
match the behaviors described in review feedback
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
# Description of Changes
Title.
# API and ABI breaking changes
N/a
# Expected complexity level and risk
1
# Testing
- [ ] Get @coolreader18 to confirm what I've written here is correct.
# Description of Changes
V8 Modules to expose `RawModuleDefV10`
# API and ABI breaking changes
It breaks client, hence should be merged in `2.0-breaking-changes`
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [ ] Existing smoketests with run on new module def to verify if its
correct.
---------
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: Noa <coolreader18@gmail.com>
# Description of Changes
- Codegen to skip creating reducer and procedure file if `visibility ==
FunctionVisiblity::Internal`
- `UpdateHost` to check for visibility before excuting reducers and
procedures.
# API and ABI breaking changes
NA, master is not exposing new raw module version yet.
# Expected complexity level and risk
1. While the current patch is simple, could there be other places where
visibility should also be applied?
# Testing
Smoketest can be in `2.0-breaking-changes` branch once we merge this.
Exising test should cover for regression
# Description of Changes
The goal of this PR is to:
- Make it easy to switch our backing implementation of identifiers, as
seen in the commit changing to `LeanString`.
- Improve type safety of our identifiers by having every identifier
start with `RawIdentifier` and then `Identifier` is just a wrapper with
validation on construction. `TableName` and `ReducerName` are then just
wrappers around `Identifier`.
- Reduce allocation in `InstanceEnv` by using the now clone-on-write
O(1) + SSO optimized `Identifier`.
- Reduce allocations in the query engine as a consequence of improving
`RawIdentifier` and `Identifier`.
- Have `&'static str` strings be further optimized by avoiding to
allocate even for long ones. This is supported by `impl From<&'static
str> for RawIdentifier` as well as switching to `LeanString`.
The PR results in a roughly 2k TPS improvement over master for WASM +
hash indices. This should also help V8, as this is VM-agnostic.
# API and ABI breaking changes
None
# Expected complexity level and risk
3? Mostly just small changes in many places, but in a sea of small
changes, mistakes can happen.
# Testing
Covered by existing tests.
# Description of Changes
Keep RLS in `RawModuleDefV10` for now, revisit replacing it with Views
before 2.0 release.
<!-- Please describe your change, mention any related tickets, and so on
here. -->
# API and ABI breaking changes
NA
# Expected complexity level and risk1
1
# Description of Changes
Host side changes for Wasm to work with `RawModuleDefV10` along with
prior versions.
Based on #https://github.com/clockworklabs/SpacetimeDB/pull/4098
# API and ABI breaking changes
NA
# Expected complexity level and risk
1.5
# Testing
Existing test will cover for any regression.
Not yet functional for `RawModuleDefV10`, so can not be tested now.
# Description of Changes
Patch:
1. `crates/lib/src/db/raw_def/v10.rs` - a definition as per
https://github.com/clockworklabs/SpacetimeDBPrivate/issues/2412.
2. Refactors `RawModuleDefV9` validation code to dedup some of the core
validation logic, No functionality should change there.
3. Validation logic for new `RawModuleDefV10`.
4. Modify `crates::schema::def::ModuleDef` :
- To include `raw_module_def_version`
- `visibility` field to existing `RedcuerDef` and `ProcedureDef`
5. It deprecates RLS in favour of Views, If anyoine feel otherwise, let
me know.
Lot of code is duplicated from `RawModuleDefV9` includng tests.
# API and ABI breaking changes
NA, `RawModuleDefV10` is not yet exported by modules.
# Expected complexity level and risk
3? close analyses of structure is important to ensure future
extensibility.
# Testing
For code motion in `RawModuleDefV9` validation: Existing unit tests
seems to cover any functionality change.
For `RawModuleDefV10`: V9 test has been repeated with more `assert`s to
check on `visibility`
# Description of Changes
`Uuid` addad in the last update is not filterable even though it is
`Copy` and (imo) obviously _very_ suited to filter.
Promote it to a special `FilterableValue` in the same way as `Identity`
and adjust the docs.
# API and ABI breaking changes
None
# Expected complexity level and risk
1. Only copies the same strategy as `Identity` which is also a newtype
wrapper around a integer like `Uuid`
# Testing
Tested on my project.
After the change:
<img width="586" height="298" alt="image"
src="https://github.com/user-attachments/assets/98e97c16-906b-4e66-b132-676ece40462e"
/>
Before the change:
<img width="1495" height="547" alt="image"
src="https://github.com/user-attachments/assets/da74e49a-1dbd-43ae-9b13-39dcc39c81dc"
/>
You can also test this yourself when adding:
`spacetimedb = { version = "1.11.2", git =
"https://github.com/kistz/SpacetimeDB.git", branch = "uuid-as-filter" }`
to your cargo.toml project
- [x] Tested before and after. Before it was a compiler error afterwards
not.
# Description of Changes
Client Query builder for rust, as per proposal -
https://github.com/clockworklabs/SpacetimeDBPrivate/pull/2356.
1. Pach moves query builder to its separate crate, so that it can be
shared between module and sdk.
2. Implements `TypedSubscriptionBuilder` in `sdks/rust` as mentioned in
proposal
3. Modify codegen to extend types to support query builder as mentioned
in proposal
4. a test
# API and ABI breaking changes
NA, additive changes.
# Expected complexity level and risk
2
# Testing
Added a test.
---------
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
# Description of Changes
Add `impl MemoryUsage for TxState` and all the types below.
Extracted from https://github.com/clockworklabs/SpacetimeDB/pull/3831 to
reduce the diff to ease figuring out why its not helping perf.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Description of Changes
We've received a number of community pull requests which add `impl
FilterableValue` for new column types, without implementing the features
which `FilterableValue` is intended to describe. This PR extends the doc
comments on `FilterableValue` to explicitly write what the requirements
are to add a new implementation.
# API and ABI breaking changes
N/a
# Expected complexity level and risk
N/a
# Testing
N/a
# Description of Changes
Despecializes direct indices for `u64` keys that have too large values
and convert them into a B-Tree index instead.
Otherwise, we are exposed to OOM aborts.
Also hardens typed index tests to cover more combinations.
This is how the bug was caught.
Some index code is also simplified.
# API and ABI breaking changes
None
# Expected complexity level and risk
2?
# Testing
Proptests were generalized to catch the bug and to then verify that it
was fixed.
# Description of Changes
Fixes https://github.com/clockworklabs/SpacetimeDB/issues/1122.
Adds hash indices and exposes them through `#[index(hash)]` for Rust
modules,
with support for typescript and C# to come in follow ups.
On the client/sdk side, for now, any index is backed via a BTree/native
index as it is the most general.
A hash index may only be queried through point scan and never ranged
scans.
Attempting a ranged scan results in an error, with the mechanism
implemented in the previous PR
(https://github.com/clockworklabs/SpacetimeDB/pull/3974).
# API and ABI breaking changes
None
# Expected complexity level and risk
2?
# Testing
A test for ensuring that hash indices cannot be used for range scans is
added.
Tests exercising hash indices will come in the next PR.
# Description of Changes
Closes#3673
*NOTE*: C++ part will be in another PR
# Expected complexity level and risk
2
Adding a new type touch everywhere
# Testing
- [x] Adding smoke and unit test
---------
Signed-off-by: Ryan <r.ekhoff@clockworklabs.io>
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
# Description of Changes
Closes
[#3290](https://github.com/clockworklabs/SpacetimeDB/issues/3290).
Adds a new "special" type to SATS, `UUID`, which is represented as the
product `{ __uuid__: u128 }`. Adds versions of this type to all of our
various languages' module bindings libraries and client SDKs, and
updates codegen to recognize it and output references to those named
library types. Adds methods for creating new UUIDs according to the V4
(all random) and V7 (timestamp, monotonic counter and random)
specifications.
# API and ABI breaking changes
We add a new type
# Expected complexity level and risk
2
it impacts all over the code
# Testing
- [x] Extends the Rust and Unreal SDK tests, and the associated
`module-test` modules in Rust, C# and TypeScript, with uses of UUIDs.
- [x] Extends the C# SDK regression tests with uses of UUIDs.
- [x] Extends the TypeScript test suite with tests with uses of UUIDs.
---------
Signed-off-by: Mario Montoya <mamcx@elmalabarista.com>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
Based on #3887 . Review starting from commit 233b48cc4.
We've encountered a commitlog which includes inserts into `st_table`,
`st_column`, &c of the rows which describe `st_view`, `st_view_param`,
&c. This caused replay to fail, as those rows were already inserted
during bootstrapping,
so we got set-semantic duplicate errors. With this commit, we ignore
set-semantic duplicate errors when replaying a commitlog specifically
for rows in system tables which describe system tables.
We also have to do an additional fixup for sequences. This is described
in-depth in comments added at the relevant locations.
# API and ABI breaking changes
N/a
# Expected complexity level and risk
1 - I was careful not to swallow any errors which aren't obviously safe.
# Testing
- [x] Manually replayed commitlog which includes the above mentioned
inserts, got error prior to this commit, no error with this commit.
# Description of Changes
Provides new WASM ABIs:
- `datastore_index_scan_point_bsatn`
- `datastore_delete_by_index_scan_point_bsatn`
These are then used where applicable to speed up `.find(_)` and friends.
Point scans are also used more internally where applicable.
What remains after this is use in C# module bindings and to expose this
in TS as well.
The PR makes TPS go from roughly 36k to 38k TPS on my machine and also
makes a difference in flamegraphs where the time spent in some index
scans are substantially decreased.
# API and ABI breaking changes
None
# Expected complexity level and risk
3? This touches the datastore an how we expose it to modules.
# Testing
Some existing tests now exercise the new ABIs by changing what
`.find(_)` and friends do.
---------
Signed-off-by: Mazdak Farrokhzad <twingoow@gmail.com>
# Description of Changes
This adds some changes for how we return data from view functions.
Originally, we interpreted the output of a view function as a bsatn
encoding of an array of rows. Since we also want to be able to return
queries from view functions, we need to be able to return different
types too. At this point, this is effectively not a functional change,
since we don't use the new format, and we don't actually try to parse
the new format.
This introduces a new format for view returns, which is a
`ViewResultHeader`, potentially followed by additional data. For
example, if a view were returning rows directly, it would write a
`ViewResultHeader::RowData`, followed by an array of rows. Note that we
could have given that object a byte array with the rows instead of using
a header an a separate object, but that would force us create an extra
copy when encoding and when decoding.
To make this backward compatible with existing views, we have a
different way to return the new format. For v8 views, if they return a
byte array, we assume it is the old format. If they return an object, we
expect the `data` field of that object to be the actual return data.
For wasm views, we interpret a return code of 2 to mean that it uses the
new format.
On the host side, we handle this naively: we will perform the query, and
we will act as though the view has a read dependency on the tables in
the query. In follow up PRs we can make this more efficient.
# API and ABI breaking changes
This is not a breaking change, but it does make the ABI more complicated
(specifically to avoid breaking it).
# Expected complexity level and risk
1.5. This should not affect the existing return style.
# Testing
I've done manual testing of this with a version of the typescript
bindings that returns queries.
# Description of Changes
This commit adds several new metrics to `DB_METRICS` for tracking
procedures' HTTP requests:
- `procedure_http_request_size_bytes`.
- `procedure_http_response_size_bytes`.
- `procedure_num_http_requests`.
- `procedure_num_successful_http_requests`.
- `procedure_num_failed_http_requests`.
- `procedure_num_timeout_http_requests`.
- `procedure_num_in_progress_http_requests`.
See help strings in `crates/datastore/src/db_metrics/mod.rs` for details
on what each of these tracks.
Closes#3712 .
# API and ABI breaking changes
N/a - I don't think we count metrics as a stable API.
# Expected complexity level and risk
2, I guess? If we intend to use these for billing, some of the choices
I've made about tracking may impact our business.
# Testing
None; I don't know how to test Prometheus metrics.
Co-authored-by: Noa <coolreader18@gmail.com>
# Description of Changes
Follow up to #3684. Moves `Error` and `Timeout` out of lib, so that we
don't have to implement `SpacetimeType` for them, and then removes the
http dependency altogether, so that `lib` can be leaner. I also got rid
of the separate `HttpValue` type, since it only really exists to mirror
the `http` crate and typescript won't make use of it.
# Expected complexity level and risk
1
# Testing
n/a - just code movement.
# Description of Changes
Closes#3517 .
With this PR, procedures (at least, those defined in Rust modules) can
perform HTTP requests! This is performed through a new field on the
`ProcedureContext`, `http: HttpClient`, which has a method `send` for
sending an `http::Request`, as well as a convenience wrapper `get`.
Internally, these methods hit the `procedure_http_request` ABI call /
host function, which uses reqwest to perform an HTTP request. The
request is run with a user-configurable timeout which defaults and is
clamped to 500 ms.
Rather than exposing the HTTP stream to modules, we download the entire
response body immediately, within the same timeout.
I've added an example usage of `get` to `module-test` which performs a
request against `localhost:3000` to read its own schema/moduledef.
This PR also makes all procedure-related definitions in the Rust module
bindings library `#[cfg(feature = "unstable")]`, as per #3644 . The
rename of the `/v1/database/:name/procedure/:name` route is not included
in this PR, so this does not close#3644 .
Left as TODOs are:
- Metrics for recording request and response size.
- Improving performance by stashing a long-lived `reqwest::Client`
someplace.
Currently we build a new `Client` for each request.
- Improving performance (possibly) by passing the request-future to the
global tokio executor
rather than running it on the single-threaded database executor.
# API and ABI breaking changes
Adds new APIs, which are marked as unstable. Adds a new ABI, which is
not unstable in any meaningful way (we can't really do that). Marks
unreleased APIs as unstable. Does not affect any pre-existing
already-released APIs or ABIs.
# Expected complexity level and risk
3 or so: networking is scary, and even though we impose a timeout which
prevents these connections from being truly long-lived, they're still
potentially long-lived on the scale of Tokio futures. It's possible that
running them on the database core is problematic in some way, and so
what I've left as a performance TODO could actually be a
concurrency-correctness issue.
# Testing
- [x] Manually wrote and executed some procedures which make HTTP
requests.
- [x] Added two automated tests to the `sdk-test` suite,
`procedure::http_ok` and `procedure::http_err`, which make successful
and failing requests respectively, then return its result. A client then
makes some assertions about the result.
---------
Co-authored-by: Noa <coolreader18@gmail.com>
# Description of Changes
Fixes a subscription plan caching bug related to client-specific views.
Before this fix, you could define a client-specific view:
```rust
fn my_view(ctx: &ViewContext) -> Option<Player> {
ctx.db.player().identity().find(ctx.sender)
}
```
And subscribe to it as follows:
```sql
SELECT * FROM my_view
```
Note this view is implicitly parameterized by `:sender`, however when
generating a query hash for this subscription, this fact would not be
taken into account which would result in this query being cached and
reused for all callers.
After this fix, a query hash is generated for this subscription as
though it were given as:
```sql
SELECT * FROM my_view WHERE identity = :sender
```
# API and ABI breaking changes
None
**Note for CLI code owners:**
I had to touch the `subscribe` cli command file. No updates to the api.
It just needed to be updated to look for views in the module def.
# Expected complexity level and risk
1
# Testing
- [x] Added a regression smoketest
Permissions for evaluating SQL/DML are not generally "actions", but more
a set of permissions that are checked during evaluation.
To make this work with the teams feature, this patch extends `AuthCtx`
to allow checking a set of permissions as mandated by the spec. This set
is a bit more fine-grained than "is owner", so as to avoid baking in the
concept of teams/collaborators, or assumptions about what a role might
entail. Both are likely to evolve in the future, so evaluation of
permissions / capabilities should be confined to the impl of the
`Authorization` trait.
Unlike "actions", the `AuthCtx` must be able to evaluate permission
checks quickly and without side-effects, nor can it enter an `async`
context. In that sense, it is precomputed (if you will), and stored as a
closure in the `AuthCtx` for external authorization.
A challenge posed is how to thread through the constructed `AuthCtx` for
subscriptions.
A tempting approach would have been to equip the `HostController` with
the ability to summon an `AuthCtx`. That, however, would have created a
gnarly circular dependency, because the `HostController` also controls
the controldb, which itself demands an `AuthCtx`.
Instead, the `AuthCtx` is obtained in the endpoint handler and passed to
each method call that requires one. That's less pretty, but more
effective.
---------
Signed-off-by: Kim Altintop <kim@eagain.io>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
# Description of Changes
`call_view` and `call_view_anon` maintains separate `Vec` list for view
inside Module but `ModuleDef` used to have single `Vec`. Hence, Views
index is not same across `ModuleDef` and module itself.
Added a `index` field inside `ViewDef` which stores it index inside
module.
# API and ABI breaking changes
NA
How complicated do you think these changes are? Grade on a scale from 1
to 5,
2,
there could be some existing code relying on poistion in `Vec` and may
not appear as compiler error
# Testing
Future smoketests should cover this.
# Description of Changes
When building under Nix, Git metadata is not available within the
sandbox, as we use `lib.cleanSource` on our source directory. This is
important because it avoids spurious rebuilds and/or determinism
hazards.
The build was broken due to our new `spacetime init` template system
accessing Git metadata in the CLI's build.rs
to filter out non-git-tracked files from the templates. The Flake
sandbox does this automatically (even without `lib.cleanSource`!), so
when building under Nix it's unnecessary to do twice. (I remain
unconvinced that it's necessary to do in non-Nix builds either, as CI
builds should have a clean checkout and local dev builds don't need
clean templates, but the behavior was already in master and I didn't
feel comfortable removing it.)
As an enhancement, I've also found a Nix-ey way to embed our Git commit
hash in builds. Previously, builds under Nix had the empty string
instead of a commit hash, because we included the `git` CLI tool but
scrubbed the necessary metadata. Now, we inject an environment variable
from the Nix flake, and don't make the `git` CLI tool available at all.
This has the convenient upside of allowing Nix builds to reference
`dirtyRev` in builds with a dirty worktree, which should reduce
confusion.
# API and ABI breaking changes
N/a
# Expected complexity level and risk
3? I didn't have a strong understanding of what the CLI build script was
doing, and to what extent it was doing things intentionally versus for
convenience. As such, it's possible that I've inadvertently damaged
something load-bearing.
# Testing
- [x] Built with `nix build`, ran `spacetime init`, chose the
`basic-rust` template, and got a reasonable-looking template
instantiation.
- [ ] Hopefully we have automated tests for this?
# Description of Changes
Adds a bit more docs to `JwtClaims` and `AuthCtx`.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Just adds docs.
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
Adds type definitions for `RawViewDefV9` and `ViewDef`. Also validates
these module defs and populates the system tables:
- `st_table`
- `st_column`
- `st_view`
- `st_view_param`
- `st_view_column`
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
2
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
Integration tests to follow when views are added to
`__describe_module__`
---------
Signed-off-by: joshua-spacetime <josh@clockworklabs.io>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
# Description of Changes
This commit extends various schema and schema-adjacent structures to
describe procedures, a new kind of database function which are allowed
to perform side effects.
This includes extending `RawModuleDefV9` with a way to register
`RawProcedureDefV9`s in the `misc_exports`, preserving compatibility
with modules that predate procedures.
The module validation path is reorganized somewhat to validate various
properties related to procedures while preserving code clarity and
maintainability.
Additionally, the `ArgsTuple` machinery for ser/de-ing reducer arguments
using the argument type as a seed is extended to also support procedure
arguments.
All of this is currently unused.
# API and ABI breaking changes
Additive and backwards-compatible additions to `RawModuleDefV9` and
friends.
# Expected complexity level and risk
2 - some minor complexity in schema validation which may have gotten
borked in a merge at some point.
# Testing
Unsure what tests would be useful, open to suggestions from reviewers.
- [ ] <!-- maybe a test you want to do -->
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
# Description of Changes
Necessary for pulling in rolldown.
# API and ABI breaking changes
None
# Expected complexity level and risk
1, with the caveat that this updates the Rust version and therefore
touches all the code.
# Testing
- [ ] Just the automated testing
# Description of Changes
This obscured other used code, so let's remove it as its dead.
# API and ABI breaking changes
None
# Expected complexity level and risk
-1
# Testing
Not applicable.
# Description of Changes
This adds auto migration support to the `schema` crate.
Additional changes are needed to fully implement this feature: see
https://github.com/clockworklabs/SpacetimeDB/issues/2912
# API and ABI breaking changes
None
# Expected complexity level and risk
1, at this level the change is fairly straightforward. The integration
will be more difficult.
# Testing
Added `schema` unit tests for success/failure cases.
---------
Co-authored-by: Shubham Mishra <shivam828787@gmail.com>
# Description of Changes
Defines a building piece `call_describe_module` that can be used to call
the expected ABI function `describe_module` in JS.
There might be further changes to this but this is a first working pass
at it.
# API and ABI breaking changes
None
# Expected complexity level and risk
2, this does not integrate with existing code reachable in production,
but the exception handling required some googling/reading docs.
# Testing
A test `call_describe_module_works` is added that tests returning an
empty `RawModuleDef` from JS.
# Description of Changes
We recently merged several repos together. This PR clarifies the license
terms for several subdirectories, as well as the relationship between
the licenses.
The licenses in our subdirectories have become symbolic links to
licenses in our toplevel `licenses` directory. For any particular
subdirectory's license file in the diff, you can click `... -> View
file` and then click on the text that says "Symbolic Link" on that page.
This will take you to the license file that it links to.
I have also updated the `tools/upgrade-version` script to update the
change date in the new `licenses/BSL.txt` file.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
None. Only changes to license files.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
joshua-spacetime