# Description of Changes
Adds a non-repeating scheduled reducer integration test to the existing
Rust and C# integration tests, in order to address #3213
# API and ABI breaking changes
No
# Expected complexity level and risk
1
# Testing
- [X] Ran `cargo test -p spacetimedb-testing` and all tests passed
---------
Signed-off-by: Ryan <r.ekhoff@clockworklabs.io>
# Description of Changes
Fixes#4584. Now we check, fully recursively, whether compound
`AlgebraicTypeUse`s contain a ref.
# Expected complexity level and risk
1
# Testing
- [x] Added a test to ensure correct codegen
---------
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
# Description of Changes
This PR adapts the Rust SDK test suite to work with the wasm version
added in https://github.com/clockworklabs/SpacetimeDB/pull/4089 (which
I've closed in favor of this PR).
Most of the changes revolve around wasm's different async semantics -
everything runs in one thread, so things that relied on background
threads didn't work directly. Several tests would lock up because
something in them blocked synchronously, which blocked any background
work from progressing.
We moved the test-clients contents into a `test_handlers.rs` so that it
could be called from both `main` (for native tests) and `lib` (for wasm
tests). To show what actually changed, use:
```bash
git diff --no-index -- <(git show origin/master:sdks/rust/tests/procedure-client/src/main.rs) sdks/rust/tests/procedure-client/src/test_handlers.rs
```
(or similar for other test-clients)
# API and ABI breaking changes
None, I think/hope.
# Expected complexity level and risk
2
# Testing
- [x] I've augmented the CI to also run the test suite with the `web`
feature
---------
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Thales R <thlsrmsdev@gmail.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
In the Rust client SDK, with this PR, doing
`.with_debug_to_file("path.txt")` on the connection builder will result
in the SDK logging additional verbose info to `path.txt`.
# API and ABI breaking changes
Adds a new user-facing-ish API to the Rust client SDK.
# Expected complexity level and risk
2? Some possibility of deadlock due to adding a new mutex, but this is
explicitly not for production use.
# Testing
- [x] Ran `chat-console-rs` locally with this enabled and got some debug
logs out of it.
# Description of Changes
Update the Rust client SDK to use the new V2 WebSocket format, and
present the V2 user-facing API.
## Reducer events
### Remove on-reducer callbacks
It's no longer possible to observe reducers called by other clients by
registering callbacks with `ctx.reducers.on_{my_reducer}`. We no longer
code-generate those methods, or the associated
`ctx.reducers.remove_on_{my_reducer}`. Internal plumbing for storing and
invoking those callbacks is also removed.
### Add specific reducer invocation callbacks
In addition to the previous way to invoke reducers,
`ctx.reducers.{my_reducer}(args...)`, we add a method that registers a
callback to run after the reducer is finished. This method has the
suffix `_then`, as in `ctx.reducers.{my_reducer}_then(args...,
callback)`.
The callback will accept two arguments:
- `ctx: &ReducerEventContext`, the same context as was previously passed
to on-reducer callbacks.
- `status: Result<Result<(), String>, InternalError>`, denoting the
outcome of the reducer.
- `Ok(Ok(())` means the reducer committed. This corresponds to
`ReducerOutcome::Ok` or `ReducerOutcome::Okmpty` in the new WS format.
- `Ok(Err(message))` means the reducer returned an "expected" or "user"
error. This corresponds to `ReducerOutcome::Err` in the new WS format.
- `Err(internal_error)` means something went wrong with host execution.
This corresponds to `ReducerOutcome::InternalError` in the new WS
format.
Internally, the SDK stores the callbacks in its `ReducerCallbacks` map.
This is keyed on `request_id: u32`, a number that is generated for each
reducer call (from an `AtomicU32` that we increment each time), and
included in the `ClientMessage::CallReducer` request. The
`ServerMessage::ReducerResult` includes the same `request_id`, so the
SDK pops out of the `ReducerCallbacks` and invokes the appropriate
callback when processing that message.
These new callbacks are very similar to the existing procedure
callbacks.
### The `Event` exposed to row callbacks
Row callbacks caused by a reducer invoked by this client will see
`Event::Reducer`, the same as they would prior to this PR. These
callbacks will be the result of a `ServerMessage::ReducerResult` with
`ReducerOutcome::Ok`. In order to expose the reducer name and arguments
to this event, the client stores them in its `ReducerCallbacks` map,
alongside the callback for when the reducer is complete.
Row callbacks caused by any other reducer, or any non-reducer
transaction, are now indistinguishable to the client. These will see
`Event::Transaction`, which is renamed from the old
`Event::UnknownTransaction`.
### Less metadata in `ReducerEvent`
Some metadata is removed from `ReducerEvent`, as the V2 WebSocket format
no longer publishes it, even to the caller.
## `CallReducerFlags` are removed
All machinery for setting, storing and applying call reducer flags is
removed from the SDK, as the new WS format does not have any non-default
flags.
## Requesting rows in unsubscribe
When sending a `ClientMessage::Unsubscribe`, we always request that the
server include the matching rows in its response
`ServerMessage::UnsubscribeApplied`. This saves us having to update the
SDK to store query sets separately, at least for now. (We'll do that
later.)
## Handling rows
The new SDK does some additional parsing to wrangle rows in the new
WebSocket format into the same internal data structures as before,
rather than re-writing the client cache. (We'll do that later.)
Specifically, parsing of `DbUpdate` is changed so that:
- We parse raw `TransactionUpdate` into the generated `DbUpdate` type,
which requires an additional loop compared to the previous version, to
cope with the new WS format's dividing updates by query set. We define a
function `transaction_update_iter_table_updates` which encapsulates this
nested loop in an iterator.
- We have two new functions for parsing raw `QueryRows` into the
generated `DbUpdate` type, one for when they come from a
`SubscribeApplied`, and the other when they come from an
`UnsubscribeApplied`. `QueryRows` from `SubscribeApplied` translate to a
`DbUpdate` of all inserts, while one from `UnsubscribeApplied` will be
all deletes.
## Legacy subscriptions
"Legacy subscriptions" are removed. These were only used for
`subscribe_to_all_tables`, which as of now is stubbed. I will follow up
with a change to re-implement `subscribe_to_all_tables` by
code-generating a list of all known tables, and having it subscribe to
`select * from {table}` for every table in that list.
## `subscribe_to_all_tables` via a list
Previously, `subscribe_to_all_tables` worked by sending a legacy
subscription with the query `SELECT * FROM *`, which the host had
special handling to expand to subscribing to all tables. As legacy
subscriptions are no longer usable in V2 clients, this can't work.
Instead, we code-generate `SpacetimeModule::ALL_TABLE_NAMES`, a list of
all the known table names. `subscribe_to_all_tables` then maps across
this list to construct a list of queries in the form `SELECT * FROM
{table_name}`, and subscribes to all of those queries. This has the
upside that defining a new table in the module without regenerating
client bindings will no longer result in the client seeing rows of
tables it does not know about and cannot parse.
## Light mode removed
Light mode is no longer meaningful in the V2 WS format, so all code
related to it is removed.
## Internal changes
### Renamed WS messages
The SDK's internal code is updated to account for various renames:
- `QueryId` -> `QuerySetId`, `query_id` -> `query_set_id`.
- `SubscribeMulti` -> `Subscribe`, `UnsubscribeMulti` -> `Unsubscribe`.
## Incidental changes in this PR, not necessary for other client SDKs
### Don't filter out empty ranges in `RowSizeHint`
The Rust implementation of `RowSizeHint` in `BsatnRowList` got regressed
in the base branch to not work with zero-sized rows. This change fixes
that.
# API and ABI breaking changes
Boy howdy is it!
# Expected complexity level and risk
3? Changes ended up being less complicated than I feared, but we do have
some fiddly code here, and we have internal dependencies on the SDK.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Updated automated test suite.
- Known failures:
- [ ] `subscribe_all_select_star`, which is currently broken because
it's trying to subscribe to rows from private tables. #4241 will fix
this.
---------
Co-authored-by: Jeffrey Dallatezza <jeffreydallatezza@gmail.com>
Co-authored-by: = <cloutiertyler@gmail.com>
# Description of Changes
Updated the codegen table/function iteration functions to take in a
parameter to check visibility in all locations for the supported
languages.
- Updated the util.rs functions for iterating tables/functions to check
for a CodegenVisibility enum (IncludePrivate, or OnlyPublic)
- Added a new CodegenOptions struct to pass around the CodegenVisibility
and future flags, defaulted visibility to OnlyPublic
- Updated the CLI to return a list of all private tables not included
(added a TODO to check the --include-private opt):
```bash
Optimising module with wasm-opt...
Build finished successfully.
Skipping private tables during codegen: secret_note, secret_order, secret_person.
Generate finished successfully.
```
# API and ABI breaking changes
Technically API breaking as the private tables will no longer be
available. (GitHub labels are not working at the moment)
# Expected complexity level and risk
1 - Simple change the testing took longer
# Testing
Turns out when you remove private tables you invalidate most of the
module_bindings across the system!
- [x] Rust test SDK for all languages
- [x] C# SDK tests
- [x] C# dotnet tests
- [x] Updated and checked snap files
- [x] Updated Blackholio (Unreal + Unity) module_bindings and tested
- [x] Ran Unreal SDK tests
---------
Signed-off-by: Jason Larabie <jason@clockworklabs.io>
# Description of Changes
Switch Wasm module to `__describe_module_v10__` from
`__describe_module__` to expose `RawModuleDefV10`.
Note: Originally merged into the 2.0 breaking changes branch
https://github.com/clockworklabs/SpacetimeDB/pull/4115.
# API and ABI breaking changes
2.0 breaking change.
# Expected complexity level and risk
1
# Testing
Exising smoketests should be enough.
---------
Co-authored-by: Shubham Mishra <shivam828787@gmail.com>
# Description of Changes
Client Query builder for rust, as per proposal -
https://github.com/clockworklabs/SpacetimeDBPrivate/pull/2356.
1. Pach moves query builder to its separate crate, so that it can be
shared between module and sdk.
2. Implements `TypedSubscriptionBuilder` in `sdks/rust` as mentioned in
proposal
3. Modify codegen to extend types to support query builder as mentioned
in proposal
4. a test
# API and ABI breaking changes
NA, additive changes.
# Expected complexity level and risk
2
# Testing
Added a test.
---------
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
# Description of Changes
This reapplies the patch from #3704, and fixes the issues that were
causing it to deadlock.
The reason it was deadlocking was that it allowed for the following
sequence of events:
* `SchedulerActor::handle_queued()` begins mutable tx
* `ModuleHost::disconnect_client()` submits call to `call_reducer(tx:
None)`
* scheduler submits call to `call_reducer(tx: Some)`
* `WasmModuleInstance::disconnect_client` now has to try to take tx
lock, but the scheduler's call_reducer already holds it and is behind it
in the queue
So, I moved most of the logic from `handle_queued` back to being
executed in the module worker thread, but kept the code in
`scheduler.rs` so that it can all be reasoned about locally.
Fixes#3645. Should I uncomment the implementation of
`ExportFunctionForScheduledTable for F: Procedure` now?
# Expected complexity level and risk
2 - there's a chance that this patch hasn't fully fixed the deadlock
issue from #3704, but I'm quite confident.
# Testing
- [x] Manually verified that deadlock no longer occurs - previously,
`while true; do python -m smoketests schedule_reducer -k
test_scheduled_table_subscription; done` would freeze up in only 2 or 3
iterations, but now it can run for 10 minutes without issues.
# Description of Changes
Fixes https://github.com/clockworklabs/SpacetimeDB/issues/3729
I genuinely don't know what came over me.
# API and ABI breaking changes
None
# Expected complexity level and risk
1.5 very straightforward but not strictly trivial
# Testing
Adds automated integration tests (written in Rust and run with `cargo
test`, although note this comment from @matklad about integration tests
for the future
https://internals.rust-lang.org/t/running-test-crates-in-parallel/15639/2):
- [x] Can publish an updated module if no migration is required
- [x] Can publish an updated module if auto-migration is required (with
the yes-break flag true/false)
- [x] Cannot publish if a manual migration is required
- [x] Can publish if a manual migration is required but the user
specified `--delete-data`
- [x] Can publish if a manual migration is required by the user
specified `--delete-data=on-conflict`
- [x] No data deletion occurs if no migration is required and
`--delete-data=on-conflict` is specified
---------
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
This reverts commit b2e37e8008.
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
Reverts #3704 which I'm pretty sure contains some sort of bug which is
causing the smoketests to hang.
# API and ABI breaking changes
None
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
1
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] CI passing again
# Description of Changes
Reworks how `SchedulerActor::handle_queued` works so that it first
determines the parameters of the call to a reducer or the parameters of
the call to the procedure. This also enables the removal of the special
case `call_scheduled_reducer`.
Fixes#3645.
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
A test `schedule_procedure` is added.
---------
Co-authored-by: Noa <coolreader18@gmail.com>
Co-authored-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
# Description of Changes
Closes#3517 .
With this PR, procedures (at least, those defined in Rust modules) can
perform HTTP requests! This is performed through a new field on the
`ProcedureContext`, `http: HttpClient`, which has a method `send` for
sending an `http::Request`, as well as a convenience wrapper `get`.
Internally, these methods hit the `procedure_http_request` ABI call /
host function, which uses reqwest to perform an HTTP request. The
request is run with a user-configurable timeout which defaults and is
clamped to 500 ms.
Rather than exposing the HTTP stream to modules, we download the entire
response body immediately, within the same timeout.
I've added an example usage of `get` to `module-test` which performs a
request against `localhost:3000` to read its own schema/moduledef.
This PR also makes all procedure-related definitions in the Rust module
bindings library `#[cfg(feature = "unstable")]`, as per #3644 . The
rename of the `/v1/database/:name/procedure/:name` route is not included
in this PR, so this does not close#3644 .
Left as TODOs are:
- Metrics for recording request and response size.
- Improving performance by stashing a long-lived `reqwest::Client`
someplace.
Currently we build a new `Client` for each request.
- Improving performance (possibly) by passing the request-future to the
global tokio executor
rather than running it on the single-threaded database executor.
# API and ABI breaking changes
Adds new APIs, which are marked as unstable. Adds a new ABI, which is
not unstable in any meaningful way (we can't really do that). Marks
unreleased APIs as unstable. Does not affect any pre-existing
already-released APIs or ABIs.
# Expected complexity level and risk
3 or so: networking is scary, and even though we impose a timeout which
prevents these connections from being truly long-lived, they're still
potentially long-lived on the scale of Tokio futures. It's possible that
running them on the database core is problematic in some way, and so
what I've left as a performance TODO could actually be a
concurrency-correctness issue.
# Testing
- [x] Manually wrote and executed some procedures which make HTTP
requests.
- [x] Added two automated tests to the `sdk-test` suite,
`procedure::http_ok` and `procedure::http_err`, which make successful
and failing requests respectively, then return its result. A client then
makes some assertions about the result.
---------
Co-authored-by: Noa <coolreader18@gmail.com>
# Description of Changes
Adds `ProcedureContext::{with_tx, try_with_tx}`.
Fixes https://github.com/clockworklabs/SpacetimeDB/issues/3515.
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
An integration test `test_calling_with_tx` is added.
# Description of Changes
Fixes client codegen for views.
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
I'm not sure what tests we have for C++.
- [x] Updated client snapshots for rust, C#, and typescript
- [ ] Rust sdk test
---------
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
# Description of Changes
This commit adds support to the Rust client SDK for calling procedures.
Similar to reducers, each `DbContext` implementor has a `pub procedures:
RemoteProcedures` field, with methods provided by extension traits for
each procedure.
Unlike reducers, the provided methods are invoke and invoke-then.
Invoke-then takes a `FnOnce` callback to run
when the SDK is notified of the procedure's termination status, while
invoke ignores that notification.
No mechanism is provided for observing procedures invoked by other
clients.
Procedure callbacks are implemented by storing a map from `request_id`
to `ProcedureCallback`, with the callback closure internally knowing how
to deserialize the return value. It's mildly unfortunate to deserialize
within the callback instead of on the preprocess background task, but it
saves significant complexity.
This commit also adds a new sdk-test module, `sdk-test-procedures`, and
a new Rust test client, `procedure-client`.
Together, these are used in two tests of invoking and observing
procedures. I've left TODOs for other tests that we should write as we
implement additional procedure features.
I also had to fix a few minor bugs in the Rust codegen which were not
strictly related to procedures: we previously assumed that the sets of
reducers and of tables were non-empty, which led to wonky invalid
codegen on modules which did not define any reducers or which did not
define any tables.
I'm sneaking a change to the Nix flake into this PR as well - when
initially writing it I had included `cargoArtifacts` (the pre-built and
cached dependencies of our actual builds) in its `packages`, but that
was neither necessary or useful, and just made building the shell take a
long time after dependency changes.
# API and ABI breaking changes
Breaks the internal interface between the Rust client SDK and codegen,
so users will have to re-run `spacetime generate`.
# Expected complexity level and risk
2-ish? Pretty simple change to the Rust SDK overall.
# Testing
- [x] Added new automated integration tests exercising the new
functionality.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Specifically generate client table handles for views.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Added a `#[view]` to `module-test` and generated new client snapshots
# Description of Changes
Closes https://github.com/clockworklabs/SpacetimeDBPrivate/issues/1987
# API and ABI breaking changes
This does not affect any APIs but it does affect the code generated
# Expected complexity level and risk
1
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [ ] Snap tests
# Description of Changes
This changes the behavior on the rust client, so that we let the server
generate the connection id if the client hasn't called the unstable
method to set a connection id.
This is awkward for the `connection_id` function, since it now panics if
the connection id hasn't been received from the server. We should
deprecate this function in favor of a `try` version.
This also changes the behavior of reusing the connection id if a client
reconnects.
The corresponding private PR is
https://github.com/clockworklabs/SpacetimeDBPrivate/pull/1921.
# API and ABI breaking changes
Technically not changing any API signatures, but this is
behavior-changing, since the `connection_id` function can now panic.
# Expected complexity level and risk
2. The risk here is people relying on that behavior.
# Testing
I think some tests need to be updated.
Ryan