# Description of Changes
Patch:
1. `crates/lib/src/db/raw_def/v10.rs` - a definition as per
https://github.com/clockworklabs/SpacetimeDBPrivate/issues/2412.
2. Refactors `RawModuleDefV9` validation code to dedup some of the core
validation logic, No functionality should change there.
3. Validation logic for new `RawModuleDefV10`.
4. Modify `crates::schema::def::ModuleDef` :
- To include `raw_module_def_version`
- `visibility` field to existing `RedcuerDef` and `ProcedureDef`
5. It deprecates RLS in favour of Views, If anyoine feel otherwise, let
me know.
Lot of code is duplicated from `RawModuleDefV9` includng tests.
# API and ABI breaking changes
NA, `RawModuleDefV10` is not yet exported by modules.
# Expected complexity level and risk
3? close analyses of structure is important to ensure future
extensibility.
# Testing
For code motion in `RawModuleDefV9` validation: Existing unit tests
seems to cover any functionality change.
For `RawModuleDefV10`: V9 test has been repeated with more `assert`s to
check on `visibility`
# Description of Changes
`Uuid` addad in the last update is not filterable even though it is
`Copy` and (imo) obviously _very_ suited to filter.
Promote it to a special `FilterableValue` in the same way as `Identity`
and adjust the docs.
# API and ABI breaking changes
None
# Expected complexity level and risk
1. Only copies the same strategy as `Identity` which is also a newtype
wrapper around a integer like `Uuid`
# Testing
Tested on my project.
After the change:
<img width="586" height="298" alt="image"
src="https://github.com/user-attachments/assets/98e97c16-906b-4e66-b132-676ece40462e"
/>
Before the change:
<img width="1495" height="547" alt="image"
src="https://github.com/user-attachments/assets/da74e49a-1dbd-43ae-9b13-39dcc39c81dc"
/>
You can also test this yourself when adding:
`spacetimedb = { version = "1.11.2", git =
"https://github.com/kistz/SpacetimeDB.git", branch = "uuid-as-filter" }`
to your cargo.toml project
- [x] Tested before and after. Before it was a compiler error afterwards
not.
# Description of Changes
Client Query builder for rust, as per proposal -
https://github.com/clockworklabs/SpacetimeDBPrivate/pull/2356.
1. Pach moves query builder to its separate crate, so that it can be
shared between module and sdk.
2. Implements `TypedSubscriptionBuilder` in `sdks/rust` as mentioned in
proposal
3. Modify codegen to extend types to support query builder as mentioned
in proposal
4. a test
# API and ABI breaking changes
NA, additive changes.
# Expected complexity level and risk
2
# Testing
Added a test.
---------
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
# Description of Changes
This pr reexports `PointIndex` and `PointIndexReadOnly` because
currently the new `Hash` indices dont compile when used in projects.
This is because the table macro cant find those two exports.
Maybe @Centril could take a look? :>
# API and ABI breaking changes
None
# Expected complexity level and risk
1. Only fixes a compiler error.
# Testing
- [x] Project compiles now.
Before:
<img width="1454" height="337" alt="image"
src="https://github.com/user-attachments/assets/42df164c-dd09-42b9-97df-a1df719dc3b4"
/>
After:
<img width="1404" height="172" alt="image"
src="https://github.com/user-attachments/assets/0d11158c-daae-44b6-b001-ccccfe0940fd"
/>
# Description of Changes
Fixes https://github.com/clockworklabs/SpacetimeDB/issues/1122.
Adds hash indices and exposes them through `#[index(hash)]` for Rust
modules,
with support for typescript and C# to come in follow ups.
On the client/sdk side, for now, any index is backed via a BTree/native
index as it is the most general.
A hash index may only be queried through point scan and never ranged
scans.
Attempting a ranged scan results in an error, with the mechanism
implemented in the previous PR
(https://github.com/clockworklabs/SpacetimeDB/pull/3974).
# API and ABI breaking changes
None
# Expected complexity level and risk
2?
# Testing
A test for ensuring that hash indices cannot be used for range scans is
added.
Tests exercising hash indices will come in the next PR.
# Description of Changes
This PR does several small things:
1. It removes the explicit `h1` tags on every page, and either uses the
side bar title directly, or puts it in the frontmatter
2. It merges what are currently called quickstarts into a single Chat
App Tutorial
3. It creates new quickstarts which just use `spacetime dev --template`
to get you up and running quickly
4. It adds a "The Zen of SpacetimeDB" page much like the Zen of Python
which goes over the 5 key principles of SpacetimeDB
5. It reorders all Tabs groups so that the ordering is `TypeScript`,
`C#`, `Rust`, `Unreal`, `C++`, `Blueprints` (order of decreasing
popularity).
6. It improves the sidebar navigation by having categories act as
overview pages, and also fixes the breadcrumbs
7. It fixes various small typos and issues
8. Closes#3610 and adds cursor rules files generally
9. It fixes general styling on the docs page by bring it inline with the
UI design:
Old:
<img width="1678" height="958" alt="image"
src="https://github.com/user-attachments/assets/f36efee6-b81a-4463-a179-da68b3a7152e"
/>
New:
<img width="1678" height="957" alt="image"
src="https://github.com/user-attachments/assets/f430f77d-0663-47f2-9727-45cbfe10e4c7"
/>
https://github.com/user-attachments/assets/adc5a78a-ada8-45b5-8078-a45cb81477a3
# API and ABI breaking changes
This PR does NOT change any old links. It does add new pages though.
# Expected complexity level and risk
3 - it's a large change. I manually tested the TypeScript Chat App
Tutorial but I have not gone through the Rust and C# quickstarts.
However, we have testing on the quickstarts and this is text only so can
be carefully reviewed.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Ran through each step of the Chat App TypeScript tutorial to
ensure it is working
- [x] Ran and tested the styles and the functionality of the side bar
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: spacetimedb-bot <spacetimedb-bot@users.noreply.github.com>
Co-authored-by: clockworklabs-bot <clockworklabs-bot@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
Closes
[#3290](https://github.com/clockworklabs/SpacetimeDB/issues/3290).
Adds a new "special" type to SATS, `UUID`, which is represented as the
product `{ __uuid__: u128 }`. Adds versions of this type to all of our
various languages' module bindings libraries and client SDKs, and
updates codegen to recognize it and output references to those named
library types. Adds methods for creating new UUIDs according to the V4
(all random) and V7 (timestamp, monotonic counter and random)
specifications.
# API and ABI breaking changes
We add a new type
# Expected complexity level and risk
2
it impacts all over the code
# Testing
- [x] Extends the Rust and Unreal SDK tests, and the associated
`module-test` modules in Rust, C# and TypeScript, with uses of UUIDs.
- [x] Extends the C# SDK regression tests with uses of UUIDs.
- [x] Extends the TypeScript test suite with tests with uses of UUIDs.
---------
Signed-off-by: Mario Montoya <mamcx@elmalabarista.com>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
Provides new WASM ABIs:
- `datastore_index_scan_point_bsatn`
- `datastore_delete_by_index_scan_point_bsatn`
These are then used where applicable to speed up `.find(_)` and friends.
Point scans are also used more internally where applicable.
What remains after this is use in C# module bindings and to expose this
in TS as well.
The PR makes TPS go from roughly 36k to 38k TPS on my machine and also
makes a difference in flamegraphs where the time spent in some index
scans are substantially decreased.
# API and ABI breaking changes
None
# Expected complexity level and risk
3? This touches the datastore an how we expose it to modules.
# Testing
Some existing tests now exercise the new ABIs by changing what
`.find(_)` and friends do.
---------
Signed-off-by: Mazdak Farrokhzad <twingoow@gmail.com>
# Description of Changes
As title.
# API and ABI breaking changes
incremental binding.
# Expected complexity level and risk
# Testing
Unit and smoketest
# Description of Changes
Fixes#3827
Adding lte and gte support for query_builder in rust.
# API and ABI breaking changes
Incremental change to bindings.
# Expected complexity level and risk
1
# Testing
unit test and smoketest
# Description of Changes
As title
# API and ABI breaking changes
NA
# Expected complexity level and risk
0
---------
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
# Description of Changes
This reapplies the patch from #3704, and fixes the issues that were
causing it to deadlock.
The reason it was deadlocking was that it allowed for the following
sequence of events:
* `SchedulerActor::handle_queued()` begins mutable tx
* `ModuleHost::disconnect_client()` submits call to `call_reducer(tx:
None)`
* scheduler submits call to `call_reducer(tx: Some)`
* `WasmModuleInstance::disconnect_client` now has to try to take tx
lock, but the scheduler's call_reducer already holds it and is behind it
in the queue
So, I moved most of the logic from `handle_queued` back to being
executed in the module worker thread, but kept the code in
`scheduler.rs` so that it can all be reasoned about locally.
Fixes#3645. Should I uncomment the implementation of
`ExportFunctionForScheduledTable for F: Procedure` now?
# Expected complexity level and risk
2 - there's a chance that this patch hasn't fully fixed the deadlock
issue from #3704, but I'm quite confident.
# Testing
- [x] Manually verified that deadlock no longer occurs - previously,
`while true; do python -m smoketests schedule_reducer -k
test_scheduled_table_subscription; done` would freeze up in only 2 or 3
iterations, but now it can run for 10 minutes without issues.
# Description of Changes
Patch contain few different things, required to run query builder end to
end.
Git history seems messed up but it can be reviewed commitwise from -
69c1907b3e.
1. Decouples generic type `T` from table name, as type represents the
`struct` and not table, and there couple be multiple tables implemented
from same struct.
2. modify table macro implementaiton to generate code supporting
`traits` and types required for query builder,
3. Disable aliasing in sql queries for now, as it was causing
semantically wrong query when we were doing joins on `FromWhere` (as
original `Expr` were referencing to original table names).
4. smoketests.
# API and ABI breaking changes
Add `Query<T>` type as Views return type.
# Expected complexity level and risk
2, changes are simple but not splitted up well.
# Testing
1. Added smoketest.
TODO:
ui test.
---------
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
# Description of Changes
Updates the rust module bindings to use new view abi
(`ViewResultHeader`) and updates the return codes for the
`__call_view__` and `__call_view_anon__` module exports. This is in
preparation for `Query` builder support.
# API and ABI breaking changes
Not breaking. Existing modules will continue to use the old abi. New
modules will use the new abi. However previous host versions will not
support modules built using this version of the bindings.
# Expected complexity level and risk
2
# Testing
This is an internal refactor. All existing tests should continue to
pass. The only tests that needed updating were the negative module
compilation tests because the error messages produced by rustc changed.
# Description of Changes
Implements a query builder for rust modules according to the
[spec](https://github.com/clockworklabs/SpacetimeDBPrivate/blob/master/proposals/0030-views.md#5-query-builder-for-incremental-evaluation).
Look at the tests in `query_builder.rs`.
TODO: Extend return type for views to include `Query` and update the
`#[table]` macro to generate query builder bindings.
# API and ABI breaking changes
This is an additive change. Current modules do not need to be updated
unless they wish to use the query builder.
# Expected complexity level and risk
1.5, Will appreciate comments to better structure code.
# Testing
[x]: unit tests.
I have also done type safety checks manually from module.
---------
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
# Description of Changes
The trait has already been implemented for ReducerContext and TxContext,
but implementation for AnonymousViewContext and ViewContext is still
missing.
## Background why I need this change
In SpacetimeDSL, I currently have the ReducerContext hard-coded as a
prerequisite for creating a DSL instance (see
https://github.com/tamaro-skaljic/SpacetimeDSL/blob/main/src/lib.rs#L9-L33)
.
With the addition of
- ViewContext,
- AnonymousViewContext,
- ProcedureContext, and
- TxContext,
I am now faced with the dilemma that I also have to support SpacetimeDSL
with the other contexts.
I hope to be able to use `spacetimedb::DbContext<DbView =
spacetimedb::Local>` and `spacetimedb::DbContext<DbView =
spacetimedb::LocalReadOnly>` as a common denominator.
(I signed the CLA)
# API and ABI breaking changes
No
# Expected complexity level and risk
1
Signed-off-by: Tamaro Skaljic <49238587+tamaro-skaljic@users.noreply.github.com>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
# Description of Changes
We've been unable to get scheduled procedures into the initial release
of procedures due to a nondeterministic hang in some of our tests which
we don't have time to debug. As such, this commit disables scheduled
procedures by:
- Changing the `RawModuleDevV9` validator to reject scheduled
procedures.
- Changing the Rust bindings library's typechecking magic so that
scheduled table annotations referencing procedures will cause a type
error.
- Removes some docs from the Rust bindings library which reference
scheduled procedures.
# API and ABI breaking changes
N/a
# Expected complexity level and risk
1
# Testing
Automated tests should be sufficient.
Prior to merging #3774 , CI for this branch merged with master failed in
the intended way: modules which attempted to define scheduled procedures
failed to typecheck.
# Description of Changes
Follow up to #3684. Moves `Error` and `Timeout` out of lib, so that we
don't have to implement `SpacetimeType` for them, and then removes the
http dependency altogether, so that `lib` can be leaner. I also got rid
of the separate `HttpValue` type, since it only really exists to mirror
the `http` crate and typescript won't make use of it.
# Expected complexity level and risk
1
# Testing
n/a - just code movement.
# Description of Changes
Closes#3517 .
With this PR, procedures (at least, those defined in Rust modules) can
perform HTTP requests! This is performed through a new field on the
`ProcedureContext`, `http: HttpClient`, which has a method `send` for
sending an `http::Request`, as well as a convenience wrapper `get`.
Internally, these methods hit the `procedure_http_request` ABI call /
host function, which uses reqwest to perform an HTTP request. The
request is run with a user-configurable timeout which defaults and is
clamped to 500 ms.
Rather than exposing the HTTP stream to modules, we download the entire
response body immediately, within the same timeout.
I've added an example usage of `get` to `module-test` which performs a
request against `localhost:3000` to read its own schema/moduledef.
This PR also makes all procedure-related definitions in the Rust module
bindings library `#[cfg(feature = "unstable")]`, as per #3644 . The
rename of the `/v1/database/:name/procedure/:name` route is not included
in this PR, so this does not close#3644 .
Left as TODOs are:
- Metrics for recording request and response size.
- Improving performance by stashing a long-lived `reqwest::Client`
someplace.
Currently we build a new `Client` for each request.
- Improving performance (possibly) by passing the request-future to the
global tokio executor
rather than running it on the single-threaded database executor.
# API and ABI breaking changes
Adds new APIs, which are marked as unstable. Adds a new ABI, which is
not unstable in any meaningful way (we can't really do that). Marks
unreleased APIs as unstable. Does not affect any pre-existing
already-released APIs or ABIs.
# Expected complexity level and risk
3 or so: networking is scary, and even though we impose a timeout which
prevents these connections from being truly long-lived, they're still
potentially long-lived on the scale of Tokio futures. It's possible that
running them on the database core is problematic in some way, and so
what I've left as a performance TODO could actually be a
concurrency-correctness issue.
# Testing
- [x] Manually wrote and executed some procedures which make HTTP
requests.
- [x] Added two automated tests to the `sdk-test` suite,
`procedure::http_ok` and `procedure::http_err`, which make successful
and failing requests respectively, then return its result. A client then
makes some assertions about the result.
---------
Co-authored-by: Noa <coolreader18@gmail.com>
# Description of Changes
Adds `ProcedureContext::{with_tx, try_with_tx}`.
Fixes https://github.com/clockworklabs/SpacetimeDB/issues/3515.
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
An integration test `test_calling_with_tx` is added.
# Description of Changes
This PR modifies the `--delete-data` flag on `spacetime publish` and
adds the `--delete-data` flag on `spacetime dev`.
In particular instead of `--delete-data` being a boolean, it is now a an
enum:
- `always` -> corresponds to the old value of `true`
- `never` -> corresponds to the old value of `false`
- `on-conflict` -> clears the database, but only if publishing would
have required a manual migration
This flag does NOT change any behavior about prompting users to confirm
if they want to delete the data. Users will still be prompted to confirm
UNLESS they pass the separate `--yes` flag.
`spacetime dev` gets the same `--delete-data` flag. The default value of
`never` is equivalent to the existing behavior. `spacetime dev`
continues to publish with `--yes` just as before. This behavior is
unchanged.
# API and ABI breaking changes
Adds the flags specified above. This is NOT a breaking change to the
CLI. Passing `--delete-data` is the equivalent of
`--delete-data=always`.
This IS technically a breaking change to the `pre_publish` route. As far
as I'm aware this is *only* used by our CLI however.
> IMPORTANT SIDE NOTE: I would argue that `--break-clients` should
really be renamed to `--yes-break-clients` because it actually behaves
like the `--yes` force flag, but only for a subset of the user prompts.
I have not made this change because it would be a breaking change, but
if the reviewers agree, I will make this change.
# Expected complexity level and risk
2, Very small change, but if we get it wrong users could accidentally
lose data. I would ask reviewers to think about ways that users might
accidentally pass `--delete-data --yes`.
# Testing
- [ ] I have not yet tested manually.
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
Includes Rust, C#, and TypeScript.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Just docs.
# Description of Changes
Disables parameterized views in rust until we can pass arguments from
sql. Note parameters are already disabled in C# and typescript.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Needed to update some tests. Will revert these changes once we re-enable
parameters.
# Description of Changes
`call_view` and `call_view_anon` maintains separate `Vec` list for view
inside Module but `ModuleDef` used to have single `Vec`. Hence, Views
index is not same across `ModuleDef` and module itself.
Added a `index` field inside `ViewDef` which stores it index inside
module.
# API and ABI breaking changes
NA
How complicated do you think these changes are? Grade on a scale from 1
to 5,
2,
there could be some existing code relying on poistion in `Vec` and may
not appear as compiler error
# Testing
Future smoketests should cover this.
# Description of Changes
The `aud` claim is required in the OIDC spec, but the server currently
allows it to be missing, and the spacetime auth tokens we use for the
website don't have an audience.
Previously the module bindings would throw an error if there were no
`aud` claim in a jwt payload (if someone used the `audience` within a
reducer), but this change makes us treat a missing audience as an empty
list.
This also renames the `authCtx` fields to `senderAuth` in the typescript
and csharp module APIs, so they match rust.
# API and ABI breaking changes
This doesn't break any ABIs.
This changes the ReducerContext APIs in typescript and rust, but only by
renaming a field that hasn't been released yet.
# Expected complexity level and risk
1.
# Testing
I've tested accessing the `audience` within a reducer for a token
missing an `aud` claim in Typescript and Rust.
# Description of Changes
This commit builds support for executing procedures in WASM modules.
This includes an HTTP endpoint,
`/v1/database/:name_or_address/procedure/:name POST`, as well as an
extension to the WS protocol. These new APIs are not wired up to the CLI
or SDKs, but I have manually tested the HTTP endpoint via `curl`. The
new WS extensions are completely untested.
Several TODOs are scattered throughout the new code, most notably for
sensibly tracking procedure execution time in the metrics.
I also expect that we will want to remove the `procedure_sleep_until`
syscall and the `ProcedureContext::sleep_until` method prior to release.
# API and ABI breaking changes
Adds new APIs and ABIs.
# Expected complexity level and risk
3? 4? Unlikely to break existing stuff, 'cause it's mostly additive, but
adds plenty of potentially-fragile new stuff. Notably is the first time
we're doing anything actually `async`hronous on a database core Tokio
worker, and we don't yet have strong evidence of how that will affect
reducer execution.
# Testing
- [x] Manually published `modules/module-test` and executed procedures
with the following `curl` invocations:
- `curl -X POST -H "Content-Type:application/json" -d '[]'
http://localhost:3000/v1/database/module-test/procedure/sleep_one_second`
- `curl -X POST -H "Content-Type:application/json" -d '[1223]'
http://localhost:3000/v1/database/module-test/procedure/return_value`
- [ ] Need to write automated tests.
---------
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
This is a draft of the new functionality for `spacetime init`. In order
to run it with built-in templates you have to set the path to the config
file:
```
export SPACETIMEDB_CLI_TEMPLATES_FILE=crates/cli/.init-templates.json
```
In the future it will fetch the list from GH.
A few notes:
* the previous functionality of `spacetime init` does not work at the
moment
* the code needs a bit more cleanup and tests before merging
* there is a bit of a mix in how we generate empty server and client
projects. For Rust we use the existing way of generating. For TypeScript
we clone an empty project from the repo. I wanted to play with both ways
of doing things, and I'm still not sure which is better. Generation in
Rust means that the generated code will match the CLI version and not
necessarily whatever is in Git. On the other hand, for the builtin
templates we will be fetching the newest version from GH, which I guess
might also not what we want, ie. we probably want only stable templates.
More discussion is needed here
* we use `spacetimedb` directory for the server files
* I don't particularly like the inability to disable interactive mode
easily. We discussed disabling it by default if all of the required
arguments are passed, but I don't think it's feature proof. For example,
if someone relies on a non-interactive mode, and we add a new required
argument, instead of printing a message `missing --foo`, we will
automatically launch interactive mode, which is harder to debug. That's
why I think I'd prefer to implement `--non-interactive` argument
* it's kind of hard to keep the legacy behaviour. If you don't pass any
arguments, we go into interactive mode. In the legacy version, we would
print required arguments. If someone passes `--lang` or `--project-path`
explicitly, I guess we could run the legacy workflow, but not sure if
it's worth it, as the command was marked as unstable anyway
* the project path defaults to the project name, but I think we should
probably replace change whitespaces to dashes, or at least ask for the
project path with the project name being the default (or both)
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Signed-off-by: John Detter <4099508+jdetter@users.noreply.github.com>
Co-authored-by: = <cloutiertyler@gmail.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@aol.com>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
Adds a bit more docs to `JwtClaims` and `AuthCtx`.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Just adds docs.
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
Makes `name` a required parameter of the `#[view]` macro. This is to be
consistent with the `#[table]` macro since views should expose the same
interface as tables.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
1
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Added new compile tests
- [x] Updated existing compile tests
# Description of Changes
This commit adds a macro attribute `#[procedure]` which applies to
functions, and various in-WASM machinery for defining, calling and
running procedures.
A simple example of a procedure, included in `modules/module-test`:
```rust
fn sleep_one_second(ctx: &mut ProcedureContext) {
let prev_time = ctx.timestamp;
let target = prev_time + Duration::from_secs(1);
ctx.sleep_until(target);
let new_time = ctx.timestamp;
let actual_delta = new_time.duration_since(prev_time).unwrap();
log::info!("Slept from {prev_time} to {new_time}, a total of {actual_delta:?}");
}
```
We intend eventually to make procedures be `async` functions (with the
trivial `now_or_never` executor from `future-util`), but I found that
making the types work for this was giving me a lot of trouble, and
decided to put it off in the interest of unblocking more parallelizable
work.
Host-side infrastructure for executing procedures is not included in
this commit. I have a prototype working, but cleaning it up for review
and merge will come a bit later.
One item of complexity in this PR is enabling scheduled tables to
specify either reducers or procedures, while still providing
compile-time diagnostics for ill-typed scheduled functions (as opposed
to publish-time). I had to rewrite the previous
`schedule_reducer_typecheck` into a more complex `schedule_typecheck`
with a trait `ExportFunctionForScheduledTable`, which takes a "tacit
trait parameter" encoding reducer-ness or procedure-ness, as described
in https://willcrichton.net/notes/defeating-coherence-rust/ .
The trait name `ExportFunctionForScheduledTable` is user-facing in the
sense that it will appear in compiler diagnostics in ill-typed modules.
As such, I am open to bikeshedding.
# API and ABI breaking changes
Adds a new user-facing API, which we intend to change before releasing.
# Expected complexity level and risk
2? Mostly pretty mechanical changes to macros and bindings.
# Testing
- [x] Added a procedure definition to `module-test`, saw that it
typechecks.
- [x] Executed same procedure definition using #3390 , the prototype
implementation this PR draws from.
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
Removing `connection_id` from `ViewContext` for performance reasons. We
intend to wait to hear from clients before assuming that it's necessary
to include.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None (unreleased feature)
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
0
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
n/a
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
Adds the `#[view]` procedural macro and module describers for views.
```rust
#[view(public)]
fn player(ctx: &ViewContext) -> Vec<Player> {
ctx.db.player().identity().find(ctx.sender).into_iter().collect()
}
#[view(public)]
fn player(ctx: &AnonymousViewContext, level: u32) -> Vec<Player> {
ctx.db.player().level().filter(level).collect()
}
```
Note, this deviates from the proposal in that views may only return
`Vec<T>` or `Option<T>`. They can't return an arbitrary `SpacetimeType`.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
This technically isn't a breaking change, but it's worth mentioning that
this patch refactors `ReducerInfo` so that we can use it for views as
well.
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
2
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Negative compile tests
- [x] Negative publish (module validation) tests
- [x] Test system tables are updated accurately
# Description of Changes
This exposes JWT claims for csharp modules, similar to how they are
exposed to rust modules in
https://github.com/clockworklabs/SpacetimeDB/pull/3288.
This adds the new types `AuthCtx` and `JwtClaims`, and adds an `AuthCtx`
to the `ReducerContext`.
`AuthCtx` represents the credentials associated with the request, and
`JwtClaims` represents a jwt token.
One difference from the rust version is that I didn't create helpers to
build an `AuthCtx` from a jwt payload. The reason is that we would need
to be able to compute the identity from the payload claims, which
requires a blake3 hash implementation. The first two c# libraries I
found had issues at runtime
([Blake3](https://www.nuget.org/packages/Blake3) is wrapping a rust
implementation, and
[HashifyNet](https://github.com/Deskasoft/HashifyNET/tree/main/HashifyNet/Algorithms/Blake3)
seems to be broken by our trimming because it uses reflection heavily).
I can look into taking the implementation from `HashifyNet`, since it is
MIT licensed, but I don't think we need to block merging on that.
# API and ABI breaking changes
This adds the new types `AuthCtx` and `JwtClaims`, and adds an `AuthCtx`
to the `ReducerContext`.
This also adds a csharp wrapper for the get_jwt ABI function added in
https://github.com/clockworklabs/SpacetimeDB/pull/3288.
# Expected complexity level and risk
2.
# Testing
This has a very minimal unit test of JwtClaims.
I manually tested using this locally with the csharp quickstart, and I
was able to print jwt tokens inside the module.
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
Adds `rust-src` to the toolchain and updates the expected output of the
`trybuild` tests.
`rust-src` places the entire source code of the rust stdlib under the
local toolchain directory. `rust-analyzer` uses it for "go to
definition" (among other things) in the stdlib. When included in the
toolchain, rustc can provide code frames that it can't otherwise.
The reason for its addition is that if anyone includes it in their local
toolchain, the error messages produced by rustc when running the `ui`
tests will be slightly different (have some more information) than when
run in CI. Hence those tests will fail locally but not in CI. Rather
than removing `rust-src` locally so that the tests will pass, I've opted
to just add it to the toolchain so that everyone runs with it by
default, including in CI.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
0
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Updated rustc tests
# Description of Changes
This exposes client credentials in reducer calls for rust.
# API and ABI breaking changes
API Changes:
The main API change is the addition of `AuthCtx` and the `sender_auth`
in `ReducerContext`. This also adds JwtClaims, which has some helpers
for getting commonly used claims.
ABI Changes:
This adds one new functions `get_jwt`. This uses
`st_connection_credentials` to look up the credentials associated with a
connection id.
This adds ABI version 10.2.
# Expected complexity level and risk
2. This adds new ABI functions
# Testing
I've done some manual testing with modified versions of the quickstart.
We should add some examples that use the new API.
# Description of Changes
This commit extends various schema and schema-adjacent structures to
describe procedures, a new kind of database function which are allowed
to perform side effects.
This includes extending `RawModuleDefV9` with a way to register
`RawProcedureDefV9`s in the `misc_exports`, preserving compatibility
with modules that predate procedures.
The module validation path is reorganized somewhat to validate various
properties related to procedures while preserving code clarity and
maintainability.
Additionally, the `ArgsTuple` machinery for ser/de-ing reducer arguments
using the argument type as a seed is extended to also support procedure
arguments.
All of this is currently unused.
# API and ABI breaking changes
Additive and backwards-compatible additions to `RawModuleDefV9` and
friends.
# Expected complexity level and risk
2 - some minor complexity in schema validation which may have gotten
borked in a merge at some point.
# Testing
Unsure what tests would be useful, open to suggestions from reviewers.
- [ ] <!-- maybe a test you want to do -->
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
# Description of Changes
Host-side changes extracted from #3327
I added AUTO_INC_OVERFLOW even though we don't currently ever return it,
in order to future-proof so it's already there when we start emitting
it.
Prepublish was failing because it was expecting a wasm module
unconditionally, so now it takes ?host_type.
I tweaked JS deser to accept null/undefined when the unit type or an
option type is expected.
I switched to bsatn, because the native sats->js translator wasn't
matching what js was expecting.
I renamed the sys module: my thinking is that `spacetime:` as a scheme
will help disambiguate it, and maybe it could also be used for IMC in
the future or something? And I believe we had discussed wanting this to
be versioned, similar to wasm imports.
Trying to get a borrowed str from deserialize_js doesn't work, because
v8 strings don't store utf8.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] All this was done in the course of getting an actual typescript
module to successfully publish.
# Description of Changes
Necessary for pulling in rolldown.
# API and ABI breaking changes
None
# Expected complexity level and risk
1, with the caveat that this updates the Rust version and therefore
touches all the code.
# Testing
- [ ] Just the automated testing
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
The `#[table]` macro now generates read-only table and index handles.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
1
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] positive and negative test cases using
`ReducerContext::as_read_only()`
# Description of Changes
Each reducer gets its arguments through an `ArgSource`, a Unix-file-like
abstraction for streams of bytes. Prior to this commit, we had an ABI
designed as if it could support other args sources, but it actually
hardcoded the ID of the reducer args source, and errored elsewhere.
This commit extends the `BytesSource` infrastructure to support other
bytes sources. This will be useful for exposing JWT payloads and HTTP
responses. No other `BytesSource` uses are actually included in this
commit, only the infrastructure.
This commit also defines a new host call,
`bytes_source_remaining_length`. This is intended to allow callers to
pre-allocate a buffer correctly sized to read the entire `BytesSource`
all at once. The new host function is added to a new ABI minor version,
10.1, so that old SpacetimeDB hosts can detect and reject too-new
compiled modules. I have added uses of this new function to
`__call_reducer__` in both Rust and C#, even though it's not strictly
necessary,
and I haven't removed the loop which repeatedly calls
`bytes_source_read` and grows the buffer.
# API and ABI breaking changes
Adds a new ABI minor version, `spacetime_10.1`. This means that old
SpacetimeDB hosts will reject new compiled modules.
# Expected complexity level and risk
2-ish? WASM ABI code is always fiddly, but this is a pretty simple case.
# Testing
- [x] New behavior and new host function are both hit through existing
tests that instantiate modules and call reducers against them, so I
believe automated testing is sufficient.
---------
Signed-off-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
# Description of Changes
Implements `Serialize` and `Deserialize` for tuples.
Extracted from https://github.com/clockworklabs/SpacetimeDB/pull/3276.
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
A test `roundtrip_tuples_in_different_data_formats` is added.
# Description of Changes
PR introduces support for column-level default values via a new
`#[default(...)]` attribute.
It also validates, `default` macro is not used along with `primary_key`,
`unique` or `auto_inc`.
# API and ABI breaking changes
NA
# Expected complexity level and risk
2
# Testing
Start using macro in `module-test`.
---------
Co-authored-by: James Gilles <jameshgilles@gmail.com>
# Description of Changes
This adds auto migration support to the `schema` crate.
Additional changes are needed to fully implement this feature: see
https://github.com/clockworklabs/SpacetimeDB/issues/2912
# API and ABI breaking changes
None
# Expected complexity level and risk
1, at this level the change is fairly straightforward. The integration
will be more difficult.
# Testing
Added `schema` unit tests for success/failure cases.
---------
Co-authored-by: Shubham Mishra <shivam828787@gmail.com>
# Description of Changes
We recently merged several repos together. This PR clarifies the license
terms for several subdirectories, as well as the relationship between
the licenses.
The licenses in our subdirectories have become symbolic links to
licenses in our toplevel `licenses` directory. For any particular
subdirectory's license file in the diff, you can click `... -> View
file` and then click on the text that says "Symbolic Link" on that page.
This will take you to the license file that it links to.
I have also updated the `tools/upgrade-version` script to update the
change date in the new `licenses/BSL.txt` file.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
None. Only changes to license files.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Shubham Mishra