# Description of Changes
Closes#5250#4636 introduced a regression where the ProcedureContext used to include
the sender and connection_id from the caller while now it is always
empty (which is wrong)
Correct it.
Also fully migrate to `database_identity` which was forgotten about so i
deprecated it for procedures (since they are now stable) and just
changed it for HttpHandlers (because they are still unstable)
@gefjon since you did the lil woopsie (ugh pinging you again lol hope im
not annoying haha)
# API and ABI breaking changes
Breaking the HttpHandler function which is unstable.
Restoring behaviour of 2.3 which got lost with 2.4.
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
1. Trivial refactoring
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] The caller identity is there again for Procedures.
---------
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
# Description of Changes
Use an isolated server process per SDK test instead of a single process
for all of the tests. In addition to reducing the memory footprint of
each test run, this should also allow for more parallelism among the
individual tests.
# API and ABI breaking changes
None
# Expected complexity level and risk
1.5
# Testing
The SDK test suite should continue to work
# Description of Changes
Graduate **procedures** (and the outgoing HTTP client used from
procedures) out of the `unstable` feature gate /
`SPACETIMEDB_UNSTABLE_FEATURES` across all module libraries. The
`procedure` macro, `ProcedureContext`, `with_tx`/`try_with_tx`,
scheduled procedures, and `ctx.http` are now available without opting
into `unstable`.
**Still gated** (unchanged): HTTP handlers/webhooks, views, RLS /
`client_visibility_filter`, and immediate-scheduling
(`volatile_nonatomic_schedule_immediate`).
Per library:
- **`bindings-sys`**: ungate the `procedure` host-call module + raw ABI
imports (`procedure_start/commit/abort_mut_tx`,
`procedure_http_request`) and `call_no_ret`. Scheduling ABI stays gated.
- **`bindings` (Rust)**: ungate the `procedure` macro,
`ProcedureContext`, `TxContext`/`with_tx`/`try_with_tx`,
`db_read_only`/`get_read_only`, the procedure traits,
`register_procedure`, `__call_procedure__`, and procedure RNG. The
`http` module (previously gated as a unit) now has fine-grained gating
so the outgoing `HttpClient` is exposed while
`HandlerContext`/`Router`/handler macros stay gated.
- **`bindings-csharp`**: drop `[Experimental("STDB_UNSTABLE")]` from
`ProcedureContext.WithTx/TryWithTx` (runtime + codegen) and the
generated `ProcedureTxContext`; FFI snapshots regenerated. Handler
context members + RLS attribute stay gated.
- **`bindings-cpp`**: ungate the procedure ABI (`abi.h`/`FFI.h`),
`tx_execution.h`, the outgoing HTTP client (`http.h`/`http_convert.h`),
and `procedure_context.h`.
`handler_context.h`/`router.h`/`http_handler_macros.h` still `#error`
without `SPACETIMEDB_UNSTABLE_FEATURES`.
- **docs**: remove the procedures beta notices (HTTP handlers stay
marked beta for a later release); regenerate `static/llms.md`.
- **`sdk-test-procedure`**: no longer needs `features = ["unstable"]`.
# API and ABI breaking changes
Not breaking. This is purely additive to the stable surface: procedures
become available **without** the `unstable` feature, while modules that
already opt into `unstable` are unaffected. The wasm host-ABI imports
were already implemented host-side and merely gated module-side, so
there is no ABI version change. (No breaking-change label needed.)
# Expected complexity level and risk
**3/5.** The diff itself is mostly removing
`#[cfg]`/`#ifdef`/`[Experimental]` guards, but the gate bundled
procedures + outgoing HTTP + handlers + the ABI crate together, so the
work was in *separating* procedures from the features that stay gated.
Areas reviewers may want to scrutinize:
- The Rust `http` module went from "gated as a whole" to fine-grained
per-item gating; the outgoing `HttpClient` is exposed while handler
types/macros stay behind `unstable`.
- The `unstable` cut reaches the ABI crate (`bindings-sys`), which is
the only way procedures can compile without the feature.
- C++ separation of the outgoing HTTP client from HTTP handlers across
several headers.
# Testing
- [x] Rust: `cargo check -p spacetimedb` passes in default, `--features
unstable`, and `--no-default-features --features unstable`.
- [x] Rust end-to-end: `sdk-test-procedure` (uses procedures,
`with_tx`/`try_with_tx`, `ctx.http.get`, `new_uuid_v7`, scheduled
procedures) builds **without** `unstable`.
- [x] C#: Codegen + Runtime build; `Codegen.Tests` pass (6/6) after FFI
snapshot regen.
- [x] C++: host syntax-check confirms procedures + `ctx.http.send()`
compile **without** the flag, full headers compile **with** it, and
`handler_context.h` still `#error`s without it. (Not built for the real
wasm/emscripten target locally — relying on CI.)
- [x] `cargo ci lint` components reproduced locally: rustfmt, clippy
(`-D warnings`, default + `unstable`), csharpier, and `cargo doc --deny
warnings`.
- [ ] Reviewer: confirm the wasm bindings + C#/C++ test suites pass in
CI (especially the C++ wasm build, which couldn't run locally).
---------
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
## Description of Changes
This PR primarily affects the `bindings-macro` and `schema` crates to
review:
### Core changes
1. Replaces the `name` macro with `accessor` for **Tables, Views,
Procedures, and Reducers** in Rust modules.
2. Extends `RawModuleDefV10` with a new section for:
* case conversion policies
* explicit names
New sections are not validated in this PR so not functional.
3. Updates index behavior:
* Index names are now always **system-generated** for clients. Which
will be fixed in follow-up PR when we start validating RawModuleDef with
explicit names.
* The `accessor` name for an index is used only inside the module.
## Breaking changes (API/ABI)
1. **Rust modules**
* The `name` macro must be replaced with `accessor`.
2. **Client bindings (all languages)**
* Index names are now system-generated instead of using explicitly
provided names.
**Complexity:** 3
A follow-up PR will reintroduce explicit names with support for case
conversion.
---------
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <bot@clockworklabs.com>
# Description of Changes
Closes
[#3290](https://github.com/clockworklabs/SpacetimeDB/issues/3290).
Adds a new "special" type to SATS, `UUID`, which is represented as the
product `{ __uuid__: u128 }`. Adds versions of this type to all of our
various languages' module bindings libraries and client SDKs, and
updates codegen to recognize it and output references to those named
library types. Adds methods for creating new UUIDs according to the V4
(all random) and V7 (timestamp, monotonic counter and random)
specifications.
# API and ABI breaking changes
We add a new type
# Expected complexity level and risk
2
it impacts all over the code
# Testing
- [x] Extends the Rust and Unreal SDK tests, and the associated
`module-test` modules in Rust, C# and TypeScript, with uses of UUIDs.
- [x] Extends the C# SDK regression tests with uses of UUIDs.
- [x] Extends the TypeScript test suite with tests with uses of UUIDs.
---------
Signed-off-by: Mario Montoya <mamcx@elmalabarista.com>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
This reapplies the patch from #3704, and fixes the issues that were
causing it to deadlock.
The reason it was deadlocking was that it allowed for the following
sequence of events:
* `SchedulerActor::handle_queued()` begins mutable tx
* `ModuleHost::disconnect_client()` submits call to `call_reducer(tx:
None)`
* scheduler submits call to `call_reducer(tx: Some)`
* `WasmModuleInstance::disconnect_client` now has to try to take tx
lock, but the scheduler's call_reducer already holds it and is behind it
in the queue
So, I moved most of the logic from `handle_queued` back to being
executed in the module worker thread, but kept the code in
`scheduler.rs` so that it can all be reasoned about locally.
Fixes#3645. Should I uncomment the implementation of
`ExportFunctionForScheduledTable for F: Procedure` now?
# Expected complexity level and risk
2 - there's a chance that this patch hasn't fully fixed the deadlock
issue from #3704, but I'm quite confident.
# Testing
- [x] Manually verified that deadlock no longer occurs - previously,
`while true; do python -m smoketests schedule_reducer -k
test_scheduled_table_subscription; done` would freeze up in only 2 or 3
iterations, but now it can run for 10 minutes without issues.
This reverts commit b2e37e8008.
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
Reverts #3704 which I'm pretty sure contains some sort of bug which is
causing the smoketests to hang.
# API and ABI breaking changes
None
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
# Expected complexity level and risk
1
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] CI passing again
# Description of Changes
Reworks how `SchedulerActor::handle_queued` works so that it first
determines the parameters of the call to a reducer or the parameters of
the call to the procedure. This also enables the removal of the special
case `call_scheduled_reducer`.
Fixes#3645.
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
A test `schedule_procedure` is added.
---------
Co-authored-by: Noa <coolreader18@gmail.com>
Co-authored-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
# Description of Changes
Closes#3517 .
With this PR, procedures (at least, those defined in Rust modules) can
perform HTTP requests! This is performed through a new field on the
`ProcedureContext`, `http: HttpClient`, which has a method `send` for
sending an `http::Request`, as well as a convenience wrapper `get`.
Internally, these methods hit the `procedure_http_request` ABI call /
host function, which uses reqwest to perform an HTTP request. The
request is run with a user-configurable timeout which defaults and is
clamped to 500 ms.
Rather than exposing the HTTP stream to modules, we download the entire
response body immediately, within the same timeout.
I've added an example usage of `get` to `module-test` which performs a
request against `localhost:3000` to read its own schema/moduledef.
This PR also makes all procedure-related definitions in the Rust module
bindings library `#[cfg(feature = "unstable")]`, as per #3644 . The
rename of the `/v1/database/:name/procedure/:name` route is not included
in this PR, so this does not close#3644 .
Left as TODOs are:
- Metrics for recording request and response size.
- Improving performance by stashing a long-lived `reqwest::Client`
someplace.
Currently we build a new `Client` for each request.
- Improving performance (possibly) by passing the request-future to the
global tokio executor
rather than running it on the single-threaded database executor.
# API and ABI breaking changes
Adds new APIs, which are marked as unstable. Adds a new ABI, which is
not unstable in any meaningful way (we can't really do that). Marks
unreleased APIs as unstable. Does not affect any pre-existing
already-released APIs or ABIs.
# Expected complexity level and risk
3 or so: networking is scary, and even though we impose a timeout which
prevents these connections from being truly long-lived, they're still
potentially long-lived on the scale of Tokio futures. It's possible that
running them on the database core is problematic in some way, and so
what I've left as a performance TODO could actually be a
concurrency-correctness issue.
# Testing
- [x] Manually wrote and executed some procedures which make HTTP
requests.
- [x] Added two automated tests to the `sdk-test` suite,
`procedure::http_ok` and `procedure::http_err`, which make successful
and failing requests respectively, then return its result. A client then
makes some assertions about the result.
---------
Co-authored-by: Noa <coolreader18@gmail.com>
# Description of Changes
Adds `ProcedureContext::{with_tx, try_with_tx}`.
Fixes https://github.com/clockworklabs/SpacetimeDB/issues/3515.
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
An integration test `test_calling_with_tx` is added.
# Description of Changes
This commit adds support to the Rust client SDK for calling procedures.
Similar to reducers, each `DbContext` implementor has a `pub procedures:
RemoteProcedures` field, with methods provided by extension traits for
each procedure.
Unlike reducers, the provided methods are invoke and invoke-then.
Invoke-then takes a `FnOnce` callback to run
when the SDK is notified of the procedure's termination status, while
invoke ignores that notification.
No mechanism is provided for observing procedures invoked by other
clients.
Procedure callbacks are implemented by storing a map from `request_id`
to `ProcedureCallback`, with the callback closure internally knowing how
to deserialize the return value. It's mildly unfortunate to deserialize
within the callback instead of on the preprocess background task, but it
saves significant complexity.
This commit also adds a new sdk-test module, `sdk-test-procedures`, and
a new Rust test client, `procedure-client`.
Together, these are used in two tests of invoking and observing
procedures. I've left TODOs for other tests that we should write as we
implement additional procedure features.
I also had to fix a few minor bugs in the Rust codegen which were not
strictly related to procedures: we previously assumed that the sets of
reducers and of tables were non-empty, which led to wonky invalid
codegen on modules which did not define any reducers or which did not
define any tables.
I'm sneaking a change to the Nix flake into this PR as well - when
initially writing it I had included `cargoArtifacts` (the pre-built and
cached dependencies of our actual builds) in its `packages`, but that
was neither necessary or useful, and just made building the shell take a
long time after dependency changes.
# API and ABI breaking changes
Breaks the internal interface between the Rust client SDK and codegen,
so users will have to re-run `spacetime generate`.
# Expected complexity level and risk
2-ish? Pretty simple change to the Rust SDK overall.
# Testing
- [x] Added new automated integration tests exercising the new
functionality.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Kilian Strunz