# Description of Changes
Host side changes for Wasm to work with `RawModuleDefV10` along with
prior versions.
Based on #https://github.com/clockworklabs/SpacetimeDB/pull/4098
# API and ABI breaking changes
NA
# Expected complexity level and risk
1.5
# Testing
Existing test will cover for any regression.
Not yet functional for `RawModuleDefV10`, so can not be tested now.
# Description of Changes
Patch:
1. `crates/lib/src/db/raw_def/v10.rs` - a definition as per
https://github.com/clockworklabs/SpacetimeDBPrivate/issues/2412.
2. Refactors `RawModuleDefV9` validation code to dedup some of the core
validation logic, No functionality should change there.
3. Validation logic for new `RawModuleDefV10`.
4. Modify `crates::schema::def::ModuleDef` :
- To include `raw_module_def_version`
- `visibility` field to existing `RedcuerDef` and `ProcedureDef`
5. It deprecates RLS in favour of Views, If anyoine feel otherwise, let
me know.
Lot of code is duplicated from `RawModuleDefV9` includng tests.
# API and ABI breaking changes
NA, `RawModuleDefV10` is not yet exported by modules.
# Expected complexity level and risk
3? close analyses of structure is important to ensure future
extensibility.
# Testing
For code motion in `RawModuleDefV9` validation: Existing unit tests
seems to cover any functionality change.
For `RawModuleDefV10`: V9 test has been repeated with more `assert`s to
check on `visibility`
# Description of Changes
`Uuid` addad in the last update is not filterable even though it is
`Copy` and (imo) obviously _very_ suited to filter.
Promote it to a special `FilterableValue` in the same way as `Identity`
and adjust the docs.
# API and ABI breaking changes
None
# Expected complexity level and risk
1. Only copies the same strategy as `Identity` which is also a newtype
wrapper around a integer like `Uuid`
# Testing
Tested on my project.
After the change:
<img width="586" height="298" alt="image"
src="https://github.com/user-attachments/assets/98e97c16-906b-4e66-b132-676ece40462e"
/>
Before the change:
<img width="1495" height="547" alt="image"
src="https://github.com/user-attachments/assets/da74e49a-1dbd-43ae-9b13-39dcc39c81dc"
/>
You can also test this yourself when adding:
`spacetimedb = { version = "1.11.2", git =
"https://github.com/kistz/SpacetimeDB.git", branch = "uuid-as-filter" }`
to your cargo.toml project
- [x] Tested before and after. Before it was a compiler error afterwards
not.
# Description of Changes
Client Query builder for rust, as per proposal -
https://github.com/clockworklabs/SpacetimeDBPrivate/pull/2356.
1. Pach moves query builder to its separate crate, so that it can be
shared between module and sdk.
2. Implements `TypedSubscriptionBuilder` in `sdks/rust` as mentioned in
proposal
3. Modify codegen to extend types to support query builder as mentioned
in proposal
4. a test
# API and ABI breaking changes
NA, additive changes.
# Expected complexity level and risk
2
# Testing
Added a test.
---------
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
# Description of Changes
Add `impl MemoryUsage for TxState` and all the types below.
Extracted from https://github.com/clockworklabs/SpacetimeDB/pull/3831 to
reduce the diff to ease figuring out why its not helping perf.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Description of Changes
We've received a number of community pull requests which add `impl
FilterableValue` for new column types, without implementing the features
which `FilterableValue` is intended to describe. This PR extends the doc
comments on `FilterableValue` to explicitly write what the requirements
are to add a new implementation.
# API and ABI breaking changes
N/a
# Expected complexity level and risk
N/a
# Testing
N/a
# Description of Changes
Despecializes direct indices for `u64` keys that have too large values
and convert them into a B-Tree index instead.
Otherwise, we are exposed to OOM aborts.
Also hardens typed index tests to cover more combinations.
This is how the bug was caught.
Some index code is also simplified.
# API and ABI breaking changes
None
# Expected complexity level and risk
2?
# Testing
Proptests were generalized to catch the bug and to then verify that it
was fixed.
# Description of Changes
Fixes https://github.com/clockworklabs/SpacetimeDB/issues/1122.
Adds hash indices and exposes them through `#[index(hash)]` for Rust
modules,
with support for typescript and C# to come in follow ups.
On the client/sdk side, for now, any index is backed via a BTree/native
index as it is the most general.
A hash index may only be queried through point scan and never ranged
scans.
Attempting a ranged scan results in an error, with the mechanism
implemented in the previous PR
(https://github.com/clockworklabs/SpacetimeDB/pull/3974).
# API and ABI breaking changes
None
# Expected complexity level and risk
2?
# Testing
A test for ensuring that hash indices cannot be used for range scans is
added.
Tests exercising hash indices will come in the next PR.
# Description of Changes
Closes#3673
*NOTE*: C++ part will be in another PR
# Expected complexity level and risk
2
Adding a new type touch everywhere
# Testing
- [x] Adding smoke and unit test
---------
Signed-off-by: Ryan <r.ekhoff@clockworklabs.io>
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
# Description of Changes
Closes
[#3290](https://github.com/clockworklabs/SpacetimeDB/issues/3290).
Adds a new "special" type to SATS, `UUID`, which is represented as the
product `{ __uuid__: u128 }`. Adds versions of this type to all of our
various languages' module bindings libraries and client SDKs, and
updates codegen to recognize it and output references to those named
library types. Adds methods for creating new UUIDs according to the V4
(all random) and V7 (timestamp, monotonic counter and random)
specifications.
# API and ABI breaking changes
We add a new type
# Expected complexity level and risk
2
it impacts all over the code
# Testing
- [x] Extends the Rust and Unreal SDK tests, and the associated
`module-test` modules in Rust, C# and TypeScript, with uses of UUIDs.
- [x] Extends the C# SDK regression tests with uses of UUIDs.
- [x] Extends the TypeScript test suite with tests with uses of UUIDs.
---------
Signed-off-by: Mario Montoya <mamcx@elmalabarista.com>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
Based on #3887 . Review starting from commit 233b48cc4.
We've encountered a commitlog which includes inserts into `st_table`,
`st_column`, &c of the rows which describe `st_view`, `st_view_param`,
&c. This caused replay to fail, as those rows were already inserted
during bootstrapping,
so we got set-semantic duplicate errors. With this commit, we ignore
set-semantic duplicate errors when replaying a commitlog specifically
for rows in system tables which describe system tables.
We also have to do an additional fixup for sequences. This is described
in-depth in comments added at the relevant locations.
# API and ABI breaking changes
N/a
# Expected complexity level and risk
1 - I was careful not to swallow any errors which aren't obviously safe.
# Testing
- [x] Manually replayed commitlog which includes the above mentioned
inserts, got error prior to this commit, no error with this commit.
# Description of Changes
Provides new WASM ABIs:
- `datastore_index_scan_point_bsatn`
- `datastore_delete_by_index_scan_point_bsatn`
These are then used where applicable to speed up `.find(_)` and friends.
Point scans are also used more internally where applicable.
What remains after this is use in C# module bindings and to expose this
in TS as well.
The PR makes TPS go from roughly 36k to 38k TPS on my machine and also
makes a difference in flamegraphs where the time spent in some index
scans are substantially decreased.
# API and ABI breaking changes
None
# Expected complexity level and risk
3? This touches the datastore an how we expose it to modules.
# Testing
Some existing tests now exercise the new ABIs by changing what
`.find(_)` and friends do.
---------
Signed-off-by: Mazdak Farrokhzad <twingoow@gmail.com>
# Description of Changes
This adds some changes for how we return data from view functions.
Originally, we interpreted the output of a view function as a bsatn
encoding of an array of rows. Since we also want to be able to return
queries from view functions, we need to be able to return different
types too. At this point, this is effectively not a functional change,
since we don't use the new format, and we don't actually try to parse
the new format.
This introduces a new format for view returns, which is a
`ViewResultHeader`, potentially followed by additional data. For
example, if a view were returning rows directly, it would write a
`ViewResultHeader::RowData`, followed by an array of rows. Note that we
could have given that object a byte array with the rows instead of using
a header an a separate object, but that would force us create an extra
copy when encoding and when decoding.
To make this backward compatible with existing views, we have a
different way to return the new format. For v8 views, if they return a
byte array, we assume it is the old format. If they return an object, we
expect the `data` field of that object to be the actual return data.
For wasm views, we interpret a return code of 2 to mean that it uses the
new format.
On the host side, we handle this naively: we will perform the query, and
we will act as though the view has a read dependency on the tables in
the query. In follow up PRs we can make this more efficient.
# API and ABI breaking changes
This is not a breaking change, but it does make the ABI more complicated
(specifically to avoid breaking it).
# Expected complexity level and risk
1.5. This should not affect the existing return style.
# Testing
I've done manual testing of this with a version of the typescript
bindings that returns queries.
# Description of Changes
This commit adds several new metrics to `DB_METRICS` for tracking
procedures' HTTP requests:
- `procedure_http_request_size_bytes`.
- `procedure_http_response_size_bytes`.
- `procedure_num_http_requests`.
- `procedure_num_successful_http_requests`.
- `procedure_num_failed_http_requests`.
- `procedure_num_timeout_http_requests`.
- `procedure_num_in_progress_http_requests`.
See help strings in `crates/datastore/src/db_metrics/mod.rs` for details
on what each of these tracks.
Closes#3712 .
# API and ABI breaking changes
N/a - I don't think we count metrics as a stable API.
# Expected complexity level and risk
2, I guess? If we intend to use these for billing, some of the choices
I've made about tracking may impact our business.
# Testing
None; I don't know how to test Prometheus metrics.
Co-authored-by: Noa <coolreader18@gmail.com>
# Description of Changes
Follow up to #3684. Moves `Error` and `Timeout` out of lib, so that we
don't have to implement `SpacetimeType` for them, and then removes the
http dependency altogether, so that `lib` can be leaner. I also got rid
of the separate `HttpValue` type, since it only really exists to mirror
the `http` crate and typescript won't make use of it.
# Expected complexity level and risk
1
# Testing
n/a - just code movement.
# Description of Changes
Closes#3517 .
With this PR, procedures (at least, those defined in Rust modules) can
perform HTTP requests! This is performed through a new field on the
`ProcedureContext`, `http: HttpClient`, which has a method `send` for
sending an `http::Request`, as well as a convenience wrapper `get`.
Internally, these methods hit the `procedure_http_request` ABI call /
host function, which uses reqwest to perform an HTTP request. The
request is run with a user-configurable timeout which defaults and is
clamped to 500 ms.
Rather than exposing the HTTP stream to modules, we download the entire
response body immediately, within the same timeout.
I've added an example usage of `get` to `module-test` which performs a
request against `localhost:3000` to read its own schema/moduledef.
This PR also makes all procedure-related definitions in the Rust module
bindings library `#[cfg(feature = "unstable")]`, as per #3644 . The
rename of the `/v1/database/:name/procedure/:name` route is not included
in this PR, so this does not close#3644 .
Left as TODOs are:
- Metrics for recording request and response size.
- Improving performance by stashing a long-lived `reqwest::Client`
someplace.
Currently we build a new `Client` for each request.
- Improving performance (possibly) by passing the request-future to the
global tokio executor
rather than running it on the single-threaded database executor.
# API and ABI breaking changes
Adds new APIs, which are marked as unstable. Adds a new ABI, which is
not unstable in any meaningful way (we can't really do that). Marks
unreleased APIs as unstable. Does not affect any pre-existing
already-released APIs or ABIs.
# Expected complexity level and risk
3 or so: networking is scary, and even though we impose a timeout which
prevents these connections from being truly long-lived, they're still
potentially long-lived on the scale of Tokio futures. It's possible that
running them on the database core is problematic in some way, and so
what I've left as a performance TODO could actually be a
concurrency-correctness issue.
# Testing
- [x] Manually wrote and executed some procedures which make HTTP
requests.
- [x] Added two automated tests to the `sdk-test` suite,
`procedure::http_ok` and `procedure::http_err`, which make successful
and failing requests respectively, then return its result. A client then
makes some assertions about the result.
---------
Co-authored-by: Noa <coolreader18@gmail.com>
# Description of Changes
Fixes a subscription plan caching bug related to client-specific views.
Before this fix, you could define a client-specific view:
```rust
fn my_view(ctx: &ViewContext) -> Option<Player> {
ctx.db.player().identity().find(ctx.sender)
}
```
And subscribe to it as follows:
```sql
SELECT * FROM my_view
```
Note this view is implicitly parameterized by `:sender`, however when
generating a query hash for this subscription, this fact would not be
taken into account which would result in this query being cached and
reused for all callers.
After this fix, a query hash is generated for this subscription as
though it were given as:
```sql
SELECT * FROM my_view WHERE identity = :sender
```
# API and ABI breaking changes
None
**Note for CLI code owners:**
I had to touch the `subscribe` cli command file. No updates to the api.
It just needed to be updated to look for views in the module def.
# Expected complexity level and risk
1
# Testing
- [x] Added a regression smoketest
Permissions for evaluating SQL/DML are not generally "actions", but more
a set of permissions that are checked during evaluation.
To make this work with the teams feature, this patch extends `AuthCtx`
to allow checking a set of permissions as mandated by the spec. This set
is a bit more fine-grained than "is owner", so as to avoid baking in the
concept of teams/collaborators, or assumptions about what a role might
entail. Both are likely to evolve in the future, so evaluation of
permissions / capabilities should be confined to the impl of the
`Authorization` trait.
Unlike "actions", the `AuthCtx` must be able to evaluate permission
checks quickly and without side-effects, nor can it enter an `async`
context. In that sense, it is precomputed (if you will), and stored as a
closure in the `AuthCtx` for external authorization.
A challenge posed is how to thread through the constructed `AuthCtx` for
subscriptions.
A tempting approach would have been to equip the `HostController` with
the ability to summon an `AuthCtx`. That, however, would have created a
gnarly circular dependency, because the `HostController` also controls
the controldb, which itself demands an `AuthCtx`.
Instead, the `AuthCtx` is obtained in the endpoint handler and passed to
each method call that requires one. That's less pretty, but more
effective.
---------
Signed-off-by: Kim Altintop <kim@eagain.io>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
# Description of Changes
`call_view` and `call_view_anon` maintains separate `Vec` list for view
inside Module but `ModuleDef` used to have single `Vec`. Hence, Views
index is not same across `ModuleDef` and module itself.
Added a `index` field inside `ViewDef` which stores it index inside
module.
# API and ABI breaking changes
NA
How complicated do you think these changes are? Grade on a scale from 1
to 5,
2,
there could be some existing code relying on poistion in `Vec` and may
not appear as compiler error
# Testing
Future smoketests should cover this.
# Description of Changes
When building under Nix, Git metadata is not available within the
sandbox, as we use `lib.cleanSource` on our source directory. This is
important because it avoids spurious rebuilds and/or determinism
hazards.
The build was broken due to our new `spacetime init` template system
accessing Git metadata in the CLI's build.rs
to filter out non-git-tracked files from the templates. The Flake
sandbox does this automatically (even without `lib.cleanSource`!), so
when building under Nix it's unnecessary to do twice. (I remain
unconvinced that it's necessary to do in non-Nix builds either, as CI
builds should have a clean checkout and local dev builds don't need
clean templates, but the behavior was already in master and I didn't
feel comfortable removing it.)
As an enhancement, I've also found a Nix-ey way to embed our Git commit
hash in builds. Previously, builds under Nix had the empty string
instead of a commit hash, because we included the `git` CLI tool but
scrubbed the necessary metadata. Now, we inject an environment variable
from the Nix flake, and don't make the `git` CLI tool available at all.
This has the convenient upside of allowing Nix builds to reference
`dirtyRev` in builds with a dirty worktree, which should reduce
confusion.
# API and ABI breaking changes
N/a
# Expected complexity level and risk
3? I didn't have a strong understanding of what the CLI build script was
doing, and to what extent it was doing things intentionally versus for
convenience. As such, it's possible that I've inadvertently damaged
something load-bearing.
# Testing
- [x] Built with `nix build`, ran `spacetime init`, chose the
`basic-rust` template, and got a reasonable-looking template
instantiation.
- [ ] Hopefully we have automated tests for this?
# Description of Changes
Adds a bit more docs to `JwtClaims` and `AuthCtx`.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Just adds docs.
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
Adds type definitions for `RawViewDefV9` and `ViewDef`. Also validates
these module defs and populates the system tables:
- `st_table`
- `st_column`
- `st_view`
- `st_view_param`
- `st_view_column`
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
2
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
Integration tests to follow when views are added to
`__describe_module__`
---------
Signed-off-by: joshua-spacetime <josh@clockworklabs.io>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
# Description of Changes
This commit extends various schema and schema-adjacent structures to
describe procedures, a new kind of database function which are allowed
to perform side effects.
This includes extending `RawModuleDefV9` with a way to register
`RawProcedureDefV9`s in the `misc_exports`, preserving compatibility
with modules that predate procedures.
The module validation path is reorganized somewhat to validate various
properties related to procedures while preserving code clarity and
maintainability.
Additionally, the `ArgsTuple` machinery for ser/de-ing reducer arguments
using the argument type as a seed is extended to also support procedure
arguments.
All of this is currently unused.
# API and ABI breaking changes
Additive and backwards-compatible additions to `RawModuleDefV9` and
friends.
# Expected complexity level and risk
2 - some minor complexity in schema validation which may have gotten
borked in a merge at some point.
# Testing
Unsure what tests would be useful, open to suggestions from reviewers.
- [ ] <!-- maybe a test you want to do -->
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
# Description of Changes
Necessary for pulling in rolldown.
# API and ABI breaking changes
None
# Expected complexity level and risk
1, with the caveat that this updates the Rust version and therefore
touches all the code.
# Testing
- [ ] Just the automated testing
# Description of Changes
This obscured other used code, so let's remove it as its dead.
# API and ABI breaking changes
None
# Expected complexity level and risk
-1
# Testing
Not applicable.
# Description of Changes
This adds auto migration support to the `schema` crate.
Additional changes are needed to fully implement this feature: see
https://github.com/clockworklabs/SpacetimeDB/issues/2912
# API and ABI breaking changes
None
# Expected complexity level and risk
1, at this level the change is fairly straightforward. The integration
will be more difficult.
# Testing
Added `schema` unit tests for success/failure cases.
---------
Co-authored-by: Shubham Mishra <shivam828787@gmail.com>
# Description of Changes
Defines a building piece `call_describe_module` that can be used to call
the expected ABI function `describe_module` in JS.
There might be further changes to this but this is a first working pass
at it.
# API and ABI breaking changes
None
# Expected complexity level and risk
2, this does not integrate with existing code reachable in production,
but the exception handling required some googling/reading docs.
# Testing
A test `call_describe_module_works` is added that tests returning an
empty `RawModuleDef` from JS.
# Description of Changes
We recently merged several repos together. This PR clarifies the license
terms for several subdirectories, as well as the relationship between
the licenses.
The licenses in our subdirectories have become symbolic links to
licenses in our toplevel `licenses` directory. For any particular
subdirectory's license file in the diff, you can click `... -> View
file` and then click on the text that says "Symbolic Link" on that page.
This will take you to the license file that it links to.
I have also updated the `tools/upgrade-version` script to update the
change date in the new `licenses/BSL.txt` file.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
None. Only changes to license files.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Shubham Mishra