# Description of Changes
In order to be able to suspend databases based on memory usage.
Reports memory usage for wasm linear memory as well as the v8 heap. Also
reports page-level memory for tables.
It does not report memory usage for indexes.
# API and ABI breaking changes
None
# Expected complexity level and risk
1.5
# Testing
Testing added in the private counterpart which handles enforcement of
memory limits.
# Description of Changes
Moves `RelationalDB` and related database code into a new
`spacetimedb-engine` crate.
The main motivation is to tighten dependency control around the engine
layer and isolate `RelationalDB`
behind a crate boundary.
- Majority of this PR is code-motion.
- Removes direct production dependence on `tokio` from
`spacetimedb-engine`.
- Keeps `tokio` only as a dev-dependency for test-only code in
`spacetimedb-engine`.
- This is intended to be a structural refactor only and should not
result in any functional change in
production.
- Adds a CI check to ensure `spacetimedb-engine` continues to compile in
simulation mode
# API and ABI breaking changes
NA
# Expected complexity level and risk
1.5.
# Testing
Existing tests should be enough.
---------
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
Adds the commitlog knobs `max_segment_size`, `write_buffer_size`, and
`preallocate_segments` to the server's `config.toml`. There are other
seemingly more advanced knobs that I did not add at this time.
This patch also increases the `DEFAULT_WRITE_BUFFER_SIZE` from `8KiB` to
`128KiB` to optimize high throughput workloads like the keynote-2
benchmark.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Manual
# Description of Changes
The core motivation for this change is simple: avoid cross-thread
handoffs and synchronization on the main execution path.
Before this change, the ingress task for each websocket connection would
wait for a completion response on each request before submitting the
next request to the database. This was mainly used to guarantee that we
delivered message responses in receive-order per connection. However it
also meant that for every request, we notified a waiting Tokio task,
which potentially incurred kernel-assisted wakeup and scheduler
overhead.
Note this design existed mainly for historical reasons. Before the
database had a dedicated job thread, requests were not serialized
through a single queue. The module instance was gated behind a semaphore
which guaranteed mutual exclusion, but it did not guarantee FIFO
ordering. Awaiting the completion of each request in `ws_recv_task` was
therefore the mechanism that enforced per-connection receive-order
semantics. However it now serves primarily as a source of overhead.
Procedures are the important exception. They are not serialized through
the main worker queue. Instead they use their own instance pool so as to
be able to run concurrently with other requests. However procedures may
be composed of multiple transactions and they may effectively yield
between transactions. This means that before this change, if a procedure
were to yield, it would effectively block all subsequent requests from
that client until it returned which is quite undesirable.
So with this change, procedures may execute out of order with other
operations received on the same WebSocket. Hence if this is not a
desirable property, clients must enforce ordering themselves by waiting
for a response before submitting the next request.
## What changed?
### 1. Different instance managers for procedures and everything else
Procedures use a bounded instance pool where each instance is backed by
an isolate running in a thread. Reducers and all other operations are
serialized through an mpsc queue that feeds a single isolate running in
a thread.
Trapped isolates are replaced inline. Only a fatal error within one of
the instance threads results in the `ModuleHost` and all its connections
being dropped. The host controller will recreate a new `ModuleHost`
lazily on the next request.
### 2. New enqueue-only `ModuleHost` interface
`ClientConnection` now calls enqueue-only methods on `ModuleHost` which
return immediately after enqueuing on the main instance lane or in the
case of a procedure, checking out an available instance and starting the
operation.
### 3. Separate `ModuleHost` interfaces for scheduled reducers and
scheduled procedures
Scheduled reducers now target the main js instance/worker, while
scheduled procedures go through the pool. The scheduler now
distinguishes between reducers and procedures and calls the appropriate
method.
Note, the scheduler does not pipeline its operations. It waits for each
one to complete before scheduling the next operation. This means that a
long running procedure will block all other operations from being
scheduled. This will need to be fixed at some point, but this patch
doesn't change the current behavior.
### 4. Misc
This patch also names the main js worker thread for better diagnostics.
It also disables core pinning by default and makes it an explicit
opt-in.
This last one is pretty important. The current architecture reduces
thread and context switching significantly such that naive core pinning
may perform worse than just deferring to the OS scheduler on certain
platforms. As it stands, the main motivation which led us to our
original core pinning strategy no longer exists, so we should probably
just defer to the OS until we've designed a proper scheduler that suits
our needs.
# API and ABI breaking changes
As mentioned above, with this change, procedures may execute out of
order with other operations received on the same WebSocket. Hence if
this is not a desirable property, clients must enforce ordering
themselves by waiting for a response before submitting the next request.
# Expected complexity level and risk
4
# Testing
This is mainly a performance oriented refactor, so no additional
correctness tests were added. However this patch does touch a lot of
code that could probably use more coverage in general. Benchmarks were
run to verify expected performance characteristics.
---------
Signed-off-by: joshua-spacetime <josh@clockworklabs.io>
Co-authored-by: Noa <coolreader18@gmail.com>
# Description of Changes
Fix the `spacetime start --listen-addr` help text to match the actual
default value.
The flag already defaults to `0.0.0.0:3000`, but the help text
incorrectly said port 80.
# API and ABI breaking changes
None.
# Expected complexity level and risk
Complexity 1.
# Testing
- [x] `cargo fmt --all --check`
- [x] `cargo check -p spacetimedb-standalone`
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
# Description of Changes
Revert the following PRs that have caused some breakage:
```
a32cffa76 Finish refactoring out replay (#4850)
d639be0af Replay: some code motion & reuse `ReplayCommittedState` (#4849)
78d6b6f7d Update NativeAOT-LLVM infrastructure to current ABI (#4515)
d5c1738c1 Better module backtraces for panics and whatnot (#577)
6f23b19f3 Wait for database update to become durable (#4846)
81c9eab86 Add `spacetime lock/unlock` to prevent accidental database deletion (#4502)
809aebd7c Move field `replay_table_updated` to `ReplayCommittedState` (#4807)
21b58ef99 Update axum (#2713)
b5cadff7a Extract replay stuff out of `CommittedState`, part 1 (#4804)
```
I also updated the Python smoketests for breakage introduced in
https://github.com/clockworklabs/SpacetimeDB/pull/4502. Reverting that
PR caused conflicts, so this fix is more straightforward.
# API and ABI breaking changes
Maybe kind of, but we haven't released any of these.
# Expected complexity level and risk
1
# Testing
Ask @bfops about testing
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Uses `Isolate::add_near_heap_limit_callback` to prevent unbounded heap
growth. Upon nearing the heap limit, we will now:
1. Call `Isolate::terminate_execution`.
2. Request that v8 double the heap limit.
Then, upon finishing a function call, we lower the heap limit back down.
This should hopefully fix the issue where v8 hits the heap limit and
crashes the whole process.
Also improves the way termination requests are checked for and
processed.
# Expected complexity level and risk
2
# Testing
- [ ] Manual testing with memory-leaky modules
## Motivation
Feature request: "Is there any way we can lock a module to prevent it
from being deleted? A bit concerned about some fat finger risk of
accidentally deleting prod."
## Solution
Adds a database lock mechanism. A locked database cannot be deleted
until explicitly unlocked.
### New CLI Commands
```bash
# Lock a database to prevent deletion
spacetime lock my-database
# Attempt to delete a locked database (fails with 403)
spacetime delete my-database
# Error: Database is locked and cannot be deleted. Run \`spacetime unlock\` first.
# Unlock when you actually need to delete
spacetime unlock my-database
spacetime delete my-database
```
Both commands support `--server` and `--no-config` flags, and resolve
the database from `spacetime.json` when no argument is given (same as
`spacetime delete`).
### New HTTP API
- `POST /v1/database/:name_or_identity/lock` -- Lock a database
- `POST /v1/database/:name_or_identity/unlock` -- Unlock a database
Both require the same authorization as `DELETE` (owner only).
### Implementation
- Lock state stored in a separate `database_locks` sled tree in the
standalone control DB (avoids changing the `Database` struct and needing
a data migration)
- `ControlStateReadAccess::is_database_locked()` and
`ControlStateWriteAccess::set_database_lock()` added to the trait
- `delete_database` route checks lock state before proceeding; returns
`403 Forbidden` with a descriptive message if locked
- Locking is idempotent (locking an already-locked database is a no-op,
same for unlock)
- Lock only prevents deletion, not publishing updates
### What is NOT locked
- `spacetime publish` (updating module code) still works on locked
databases
- Only `spacetime delete` is blocked
This matches the intent: protect prod from accidental destruction while
allowing normal deployments.
---------
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Require a runtime handle and identifier for `spawn_tokio_stats`, so
multiple runtimes can be instrumented.
# Expected complexity level and risk
1
# Testing
n/a
# Description of Changes
Introduces types `TypedIndexKey` and `IndexKey` to free the table index
code on its direct dependency on `AlgebraicValue`, allowing index scans
by BSATN, `RowRef`s as well without first going through
`AlgebraicValue`.
This also has the effect of optimizing string scans by avoiding
allocating in `AlgebraicValue::String`.
This also adds a byte-array based future optimization for
all-primitive-multi-column indexes.
Also in the future, this will enable optimizing `iter_by_col_eq`, which
is used by the frequent connect/disconnect logic.
# API and ABI breaking changes
None
# Expected complexity level and risk
3? Unsafe code and very load bearing code.
# Testing
Should be covered by existing tests.
# Description of Changes
While testing #4663, I discovered the server would crash from a V8 out
of memory error after processing many requests. Before #4663, this would
not happen. I theorized that because we now have a single JS worker that
can process an unbounded number of reducer calls over its lifetime, any
V8 heap retention that would previously have been spread across several
pooled isolates now accumulates in one isolate.
This patch now periodically collects heap statistics and forces GC or
replaces the isolate if memory cannot be reclaimed. This greatly reduces
the risk of hitting the V8 heap limit and crashing the server. It
doesn't remove the risk entirely however. But this risk was still
present before we switched to a single worker model in #4663. In order
to remove the risk of crashing the server entirely, we would need to run
V8 in a separate process.
# API and ABI breaking changes
None
# Expected complexity level and risk
3
# Testing
TBD
Fixes two issues that would prevent updating a database while also
changing the host type:
- We never actually updated the `ModuleKind` in `st_module` (hardcoded
to wasm)
- We never actually honored the value from `st_module` when
instantiating a module
To do so, the `Program` type from the datastore crate now carries the
`ModuleKind`, forcing call sites to make a decision.
Small adjustments to the smoketests / guard crate where made when
writing the test for this.
# Expected complexity level and risk
1
# Testing
- [x] Added smoketest
# Description of Changes
Update the Default casing policy to `snake_case` for `RawModuleDefV10`.
Messy PR contains changes at different places, so that CI can pass:
Here are the main changes as follows:
- `bindings-macro` & `bindings` crate: `name` macro in Indexes for
canonical name and supply it to `RawModuleDefV10` via `ExplicitNames`.
- `bindings-typescript`:
- Changes has been reviewed through this PR -
https://github.com/clockworklabs/SpacetimeDB/pull/4308.
- `binding-csharp`: a single line change to pass `sourceName` of index
instead of null.
- `codegen`:
- Changes has been merged from branch -
https://github.com/clockworklabs/SpacetimeDB/pull/4337.
- Except a fix in rust codegen to use canonical name in Query buillder
instead of accessor.
- `lib/db/raw_def`: Extends `RawDefModuleV10` structure to support case
conversion.
- `schema` crate:
- `validate/v9` - Nothing itself should change or changes in v9
validation logic but the file contains a `CoreValidator` which is shared
with `validate/v10`. No test have t be updated to `validate/v9` which
ensures we aren't regressing it.
- `validate/v10`: This is the main meat, look at the new tests added in
bottom to understand what it does.
- Rest of the files are either test updates or module bindings.
## Testing:
1. Extensive unit tests have been added to verify generated `ModuleDef`
is correct.
2. I have done some e2e testing to verify rust codegen with rust and
typescript modules.
3. I would have like to do more testing for other codegens , I am
continue doing .
I have removed `sql.py` smoketest, as that seems to be already migated
in new framework and was headache to update.
## Expected complexity level and risk
4, It could have side-effect which aren't easily visible.
- - -
---------
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <bot@clockworklabs.com>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
Co-authored-by: Noa <coolreader18@gmail.com>
Co-authored-by: = <cloutiertyler@gmail.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@clockworklabs.io>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
# Description of Changes
Blocks procedures from requesting private ip ranges after dns
resolution.
Adds a new cargo feature to `spacetimedb-standalone` permitting loopback
http requests in test environments only.
# API and ABI breaking changes
None
# Expected complexity level and risk
2. I may have missed a range.
# Testing
- [x] Unit tests for IP address matching
- [x] Smoketests for blocking a private IP address
# Description of Changes
Required in order to bump rolldown to its latest version, which I'm
planning to do afterwards. I also figure we may as well do this before
releasing a major version.
# Expected complexity level and risk
2
# Testing
n/a
# Description of Changes
This got a little bigger than I had hoped, but I think it's still pretty
manageable. This PR partially reverts back to before #3263: cores are
re-balanced with a `watch::Receiver<CoreId>` that each database thread
will listen for updates on in order to repin itself, and multiple OS
threads (each matched to a database) can be pinned to one core. As I
understand it, that second part is something Phoebe was trying to avoid,
but given that there's no way to asyncify a JS module, it's kind of
necessary.
JS is single-threaded, and uses cooperative rather than preemptive
multitasking (callbacks/async, not green threads). That means that if a
JS function has an infinite loop, no other event handlers would be able
to run unless that loop were to exit. Coupled with the fact that we
can't `Send` a v8 isolate across threads, it makes more sense to keep
the module host on one thread and repin that thread as needed. An
alternative option, as was brought up, would be to deconstruct and
reconstruct the module onto a different thread when needed, since
load-balancing won't be happening often anyway.
# Expected complexity level and risk
3 - reworks the threadpool that databases run on, and so could lead to
deadlocks or other concurrency bugs. However, that seems unlikely, since
this separates databases each onto their own thread, and as such
decreases the likelihood of them interacting poorly with each other.
# Testing
Not sure if there's anything specific I should do, since this doesn't
change behavior.
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
---------
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Implements the client-api changes for the organizations feature.
The interesting part of this PR are the `teams` smoketests.
There is one subtlety with deletion.
Note that I haven't (yet) covered the interaction between collaborators
and organization existing for the same database.
# Description of Changes
This PR shrinks `JsWorkerRequest` so that it is (almost) as small as the
call reducer request.
To do that, a bunch of trivial changes had to be done to auth code, that
mostly revolves around `String` -> `Box<str>`.
This should help the auth code, but that is incidental.
The main goal was to improve throughput through the request tx/rx
channel for V8, which is taking quite a bit of time in flamegraphs.
I also noticed while making this change that the wrong hash map was
being used in a bunch of places, so I fixed all of those.
A follow up PR will shrink the reply side to fit within a cache line.
Yet another follow up PR will change the channel to replace flume with
`fibre::spsc`.
# API and ABI breaking changes
None
# Expected complexity level and risk
2, fairly trivial changes.
# Testing
Covered by existing tests.
This is the second step to make in-memory-only databases not touch the
disk at all.
While at it, also make it so file-backed module logs are streamed in
constant memory where possible.
Depends-on: #3912
# Expected complexity level and risk
2
# Testing
Added some unit-level tests.
---------
Signed-off-by: Kim Altintop <kim@eagain.io>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
# Description of Changes
Anther knob for benchmarking only without job core pinning but keeping
other core pinning (tokio background, rayon, etc.).
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
No semantic changes.
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
- Small fix for checking to see if a port is available on some given
interface.
updated:
The original implementation here used `bind` to try to discover if a
port is currently in use. This isn't reliable due to platform
differences - especially on windows where it's apparently acceptable to
have a service running on both `0.0.0.0:3000` and `127.0.0.1:3000`. This
would cause bind to return successfully when we wanted it to fail. Also:
binding on an ipv6 interface when a machine doesn't have ipv6 enabled
caused random errors and it was too unreliable to be useful.
This new implementation uses `get_socket_info` which returns info on all
sockets in use on the system. We can then look through this list to find
services which conflict with our requested port.
updated 1/14:
This PR now includes a fix for flaky CLI tests. Originally we were using
`find_free_port` to pick a random free port, but that was causing a race
condition which resulted in test flakes. This PR fixes this issue by
using `127.0.0.1:0` as the listen addr so the kernel will automatically
pick a free port for us.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
1 - this is a pretty isolated check, unlikely to introduce larger
issues.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
I tested on macos, windows and linux:
```
ALLOW
docusaurus is already running on 127.0.0.1:3000
SpacetimeDB then tries to start on 192.168.1.10:3000
ALLOW
docusaurus is already running on ::1:3000
SpacetimeDB then tries to start on 192.168.1.10:3000
DENY
docusaurus is already running on ::1:3000
SpacetimeDB then tries to start on 127.0.0.1:3000
DENY
docusaurus is already running on 0:0:0:0:0:0:0:0:3000
SpacetimeDB then tries to start on 0.0.0.0:3000
DENY
docusaurus is already running on 0:0:0:0:0:0:0:0:3000
SpacetimeDB then tries to start on 127.0.0.1:3000
DENY
docusaurus is already running on 0:0:0:0:0:0:0:0:3000
SpacetimeDB then tries to start on 192.168.1.10:3000
DENY
docusaurus is already running on 127.0.0.1:3000
SpacetimeDB then tries to start on 0:0:0:0:0:0:0:0:3000
DENY
docusaurus is already running on 192.168.1.10:3000
SpacetimeDB then tries to start on 0:0:0:0:0:0:0:0:3000
```
It is possible that, under pathological conditions, a database has a
huge transaction backlog, or that there is some bug that prevents
progress on draining this backlog upon shutdown.
In order to avoid piling up `exit_module_host` tasks (which we would not
notice), impose a timeout to be specified after which `exit_module_host`
will drop resources without waiting for the shutdown to complete
gracefully.
# Description of Changes
Adds a feature `no-core-pinning` that disables core pinning.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Covered by existing tests.
# Description of Changes
This PR does several small things:
1. It removes the explicit `h1` tags on every page, and either uses the
side bar title directly, or puts it in the frontmatter
2. It merges what are currently called quickstarts into a single Chat
App Tutorial
3. It creates new quickstarts which just use `spacetime dev --template`
to get you up and running quickly
4. It adds a "The Zen of SpacetimeDB" page much like the Zen of Python
which goes over the 5 key principles of SpacetimeDB
5. It reorders all Tabs groups so that the ordering is `TypeScript`,
`C#`, `Rust`, `Unreal`, `C++`, `Blueprints` (order of decreasing
popularity).
6. It improves the sidebar navigation by having categories act as
overview pages, and also fixes the breadcrumbs
7. It fixes various small typos and issues
8. Closes#3610 and adds cursor rules files generally
9. It fixes general styling on the docs page by bring it inline with the
UI design:
Old:
<img width="1678" height="958" alt="image"
src="https://github.com/user-attachments/assets/f36efee6-b81a-4463-a179-da68b3a7152e"
/>
New:
<img width="1678" height="957" alt="image"
src="https://github.com/user-attachments/assets/f430f77d-0663-47f2-9727-45cbfe10e4c7"
/>
https://github.com/user-attachments/assets/adc5a78a-ada8-45b5-8078-a45cb81477a3
# API and ABI breaking changes
This PR does NOT change any old links. It does add new pages though.
# Expected complexity level and risk
3 - it's a large change. I manually tested the TypeScript Chat App
Tutorial but I have not gone through the Rust and C# quickstarts.
However, we have testing on the quickstarts and this is text only so can
be carefully reviewed.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Ran through each step of the Chat App TypeScript tutorial to
ensure it is working
- [x] Ran and tested the styles and the functionality of the side bar
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: spacetimedb-bot <spacetimedb-bot@users.noreply.github.com>
Co-authored-by: clockworklabs-bot <clockworklabs-bot@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
# Description of Changes
Fixes https://github.com/clockworklabs/SpacetimeDB/issues/2824.
Defines a global pool `BsatnRowListBuilderPool` which reclaims the
buffers of a `ServerMessage<BsatnFormat>` and which is then used when
building new `ServerMessage<BsatnFormat>`s.
Notes:
1. The new pool `BsatnRowListBuilderPool` reports the same kind of
metrics to prometheus as `PagePool` does.
2. `BsatnRowListBuilder` now works in terms of `BytesMut`.
3. The trait method `fn to_bsatn_extend` is redefined to be capable of
dealing with `BytesMut` as well as `Vec<u8>`.
4. A trait `ConsumeEachBuffer` is defined from
`ServerMessage<BsatnFormat>` and down to extract buffers.
`<ServerMessage<_> as ConsumeEachBuffer>::consume_each_buffer(...)` is
then called in `messages::serialize(...)` just after bsatn-encoding the
entire message and before any compression is done. This is the place
where the pool reclaims buffers.
# Benchmarks
Benchmark numbers vs. master using `cargo bench --bench subscription --
--baseline subs` on i7-7700K, 64GB RAM:
```
footprint-scan time: [21.607 ms 21.873 ms 22.187 ms]
change: [-62.090% -61.438% -60.787%] (p = 0.00 < 0.05)
Performance has improved.
full-scan time: [22.185 ms 22.245 ms 22.324 ms]
change: [-36.884% -36.497% -36.166%] (p = 0.00 < 0.05)
Performance has improved.
```
The improvements in `footprint-scan` are mostly thanks to
https://github.com/clockworklabs/SpacetimeDB/pull/2918, but 7 ms of the
improvements here are thanks to the pool. The improvements to
`full-scan` should be only thanks to the pool.
# API and ABI breaking changes
None
# Expected complexity level and risk
2?
# Testing
- Tests for `Pool<T>` also apply to `BsatnRowListBuilderPool`.
# Description of Changes
Provides new WASM ABIs:
- `datastore_index_scan_point_bsatn`
- `datastore_delete_by_index_scan_point_bsatn`
These are then used where applicable to speed up `.find(_)` and friends.
Point scans are also used more internally where applicable.
What remains after this is use in C# module bindings and to expose this
in TS as well.
The PR makes TPS go from roughly 36k to 38k TPS on my machine and also
makes a difference in flamegraphs where the time spent in some index
scans are substantially decreased.
# API and ABI breaking changes
None
# Expected complexity level and risk
3? This touches the datastore an how we expose it to modules.
# Testing
Some existing tests now exercise the new ABIs by changing what
`.find(_)` and friends do.
---------
Signed-off-by: Mazdak Farrokhzad <twingoow@gmail.com>
Historically, controldb reads could be treated as just a datastructure,
but that became a lie when reconnections were introduced.
Some tricks were employed to enter the async context when needed, but
those always bear the risk of introducing a deadlock somewhere.
So, just make it async.
# Expected complexity level and risk
2
It may or may not be safe to use sled in an async context. We did
already for
the write path, but if it's a problem it'll show now.
# Testing
Not a functional change.
Introduces a "routes struct" for the `/identity` endpoints, much like
the `DatabaseRoutes`.
This is useful for overriding individual handlers.
See companion for motivation.
Depends-on: #3525
Permissions for evaluating SQL/DML are not generally "actions", but more
a set of permissions that are checked during evaluation.
To make this work with the teams feature, this patch extends `AuthCtx`
to allow checking a set of permissions as mandated by the spec. This set
is a bit more fine-grained than "is owner", so as to avoid baking in the
concept of teams/collaborators, or assumptions about what a role might
entail. Both are likely to evolve in the future, so evaluation of
permissions / capabilities should be confined to the impl of the
`Authorization` trait.
Unlike "actions", the `AuthCtx` must be able to evaluate permission
checks quickly and without side-effects, nor can it enter an `async`
context. In that sense, it is precomputed (if you will), and stored as a
closure in the `AuthCtx` for external authorization.
A challenge posed is how to thread through the constructed `AuthCtx` for
subscriptions.
A tempting approach would have been to equip the `HostController` with
the ability to summon an `AuthCtx`. That, however, would have created a
gnarly circular dependency, because the `HostController` also controls
the controldb, which itself demands an `AuthCtx`.
Instead, the `AuthCtx` is obtained in the endpoint handler and passed to
each method call that requires one. That's less pretty, but more
effective.
---------
Signed-off-by: Kim Altintop <kim@eagain.io>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
This adds authorization to the relevant API endpoints, updates the CLI
commands and adds smoketests for the teams feature.
**Note**: Authorizing SQL (incl. subscriptions) is a bit more involved,
and submitted as a separate PR in the series.
Depends-on: https://github.com/clockworklabs/SpacetimeDB/pull/3519
So far, the `--clear-database` option to `publish` has simply dropped
and then re-created the database (if it did exist).
This will no longer work when databases can have "children": because
dropping and re-creating is not atomic, children would either become
orphans, or be dropped as well.
To solve this, `reset_database` is introduced as a separate action that:
- shuts down all replicas
- if a `program_bytes` is supplied, replaces the database's initial
program
- if a `host_type` is supplied, replaces the database's host type
- starts `num_replicas` or the previous number of replicas, which
initialize themselves as normal
As this could be its own CLI command, the action is provided as its own
API endpoint (undocumented).
However, since `publish` has no way of knowing whether the database it
operates on actually exists, the `publish_database` handler will just
invoke the `reset_database` handler if `clear=true` and the database
exists, and return its result. This is to avoid starting the transfer of
the program in the request body, only to receive a redirect.
Some refactoring was necessary to dissect and understand the flow.
# API and ABI breaking changes
Introduces a new, undocumented API endpoint.
We may want to nest it under `/unstable`.
# Expected complexity level and risk
2
# Testing
From the outside, the observed behavior should be as before,
so smoketests should cover it.
# Description of Changes
This reverts commit #3496.
# API and ABI breaking changes
Technically maybe yes? But definitely nothing is using the new code yet.
# Expected complexity level and risk
1
# Testing
CI only
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
So far, the `--clear-database` option to `publish` has simply dropped
and then re-created the database (if it did exist).
This will no longer work when databases can have "children": because
dropping and re-creating is not atomic, children would either become
orphans, or be dropped as well.
To solve this, `reset_database` is introduced as a separate action that:
- shuts down all replicas
- if a `program_bytes` is supplied, replaces the database's initial
program
- if a `host_type` is supplied, replaces the database's host type
- starts `num_replicas` or the previous number of replicas, which
initialize themselves as normal
As this could be its own CLI command, the action is provided as its own
API endpoint (undocumented).
However, since `publish` has no way of knowing whether the database it
operates on actually exists, the `publish_database` handler will just
invoke the `reset_database` handler if `clear=true` and the database
exists, and return its result. This is to avoid starting the transfer of
the program in the request body, only to receive a redirect.
Some refactoring was necessary to dissect and understand the flow.
# API and ABI breaking changes
Introduces a new, undocumented API endpoint.
We may want to nest it under `/unstable`.
# Expected complexity level and risk
2
# Testing
From the outside, the observed behavior should be as before,
so smoketests should cover it.
# Description of Changes
This exposes client credentials in reducer calls for rust.
# API and ABI breaking changes
API Changes:
The main API change is the addition of `AuthCtx` and the `sender_auth`
in `ReducerContext`. This also adds JwtClaims, which has some helpers
for getting commonly used claims.
ABI Changes:
This adds one new functions `get_jwt`. This uses
`st_connection_credentials` to look up the credentials associated with a
connection id.
This adds ABI version 10.2.
# Expected complexity level and risk
2. This adds new ABI functions
# Testing
I've done some manual testing with modified versions of the quickstart.
We should add some examples that use the new API.
# Description of Changes
Currently based on #3361
Implements most of the TS module API (not yet a function for type
aliases).
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Extremely basic module stuff works
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
---------
Signed-off-by: Noa <coolreader18@gmail.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@aol.com>
Co-authored-by: = <cloutiertyler@gmail.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Necessary for pulling in rolldown.
# API and ABI breaking changes
None
# Expected complexity level and risk
1, with the caveat that this updates the Rust version and therefore
touches all the code.
# Testing
- [ ] Just the automated testing
# Description of Changes
In service of adding procedures, which will need to execute WASM code in
an `async` environment so as to suspend execution while making HTTP
requests and whatnot.
Prior to this PR, `JobCores` worked by spawning an OS thread for each
database. These threads would then each be pinned to a specific core,
and in multi-tenant deployments multiple threads could be pinned to the
same core. Now, instead, we spawn one thread per available core at
startup. Each of these threads runs a single-threaded Tokio executor.
Each database is assigned to one of these executors, and runs tasks on
it via `tokio::spawn`.
When we run without core pinning (usually due to having too few hardware
cores), we won't spawn any additional threads or Tokio runtimes at all;
instead we will run database jobs on the "global" Tokio executor. These
jobs may block Tokio worker threads, which might be an issue if a very
core-constrained device runs multiple databases with very long-running
reducers. If this is an issue, we could in this case instead build a
second Tokio runtime only for running database jobs, and let the OS
scheduler figure things out like it did previously.
Previously, we implemented load-balancing among the database cores by
occasionally instructing per-database threads to re-pin themselves. Now,
we instead periodically send the database a new
`wasmtime::runtime::Handle`, which they will `spawn` future jobs into.
Previously, it was possible for a database thread to become canceled,
most likely as a result of `ModuleHost::exit`, after which calls would
fail with `NoSuchModule`. Cancellation is no longer meaningful, as the
database holds a `Handle` to a long-lived `tokio::runtime::Runtime`,
which should always outlive the `ModuleHost`. I have added an
`AtomicBool` flag to `ModuleHost` which is flipped by `exit` and checked
by calls to maintain the previous functionality.
Within this PR, the jobs run on the database-execution Tokio tasks are
not actually asynchronous; they will never yield. This is important
because these jobs may (will) hold a transaction open, and attempting to
swap execution to another job which wants a transaction on the same
database would be undesirable.
Note that this may regress our multi-tenant performance / fairness:
previously, in multi-tenant environments, the OS scheduler would divide
the database cores' time between the per-database threads, potentially
causing one high-load database to be interrupted in the middle of a
reducer in order to run other databases pinned to the same core. Now, a
high-load database will instead run its entire reducer to completion
before any other database gets to run.
We could, in the future, change this by instructing Wasmtime to yield
periodically, either via [epoch
interruption](https://docs.wasmtime.dev/api/wasmtime/struct.Store.html#method.epoch_deadline_async_yield_and_update)
or
[fuel](https://docs.wasmtime.dev/api/wasmtime/struct.Store.html#method.fuel_async_yield_interval),
both of which we're already configuring Wasmtime to track. We'd need (or
at least want) to (re-)introduce a queue s.t. we only attempt to run one
job for each database at a time. I have chosen not to do so within this
patch because I felt the changeset was complex enough already, and we
have so far not treated fairness in multi-tenant environments as a high
priority.
I have also reworked our module host machinery to no longer use dynamic
dispatch and trait polymorphism to manage modules and their instances,
and instead introduced `enum Module` and `enum Instance`, each of which
has a variant for Wasm and another for V8.
During this rewrite, I reworked `AutoReplacingModuleInstance`, which
previously used type-erased trait generics in a way that was brittle and
hard to re-use in the new `async` context. (Specifically, the module
instance no longer lives on the job thread, rather, the database grabs
the instance and sends it to the job thread, then gets it back when the
job exits. This is necessary to allow the re-worked load balancing
described above, as we can't have a single long-lived async task.) While
refactoring, I replaced it with `ModuleInstanceManager`, which can now
maintain multiple instances of the same module. This is not yet useful,
but will become necessary with procedures, as each concurrent procedure
will need its own instance. Relatedly, I changed
`ModuleHost::on_module_thread` (used by one-off and initial subscription
queries) to no longer acquire the/an instance. I discussed this with the
team, and consensus was that "locking" the module instance in that path
was not a useful behavior, just an artifact of the previous
implementation.
I have also switched our Wasmtime configuration to set
`async_support(true)`. This causes a variety of methods, notably
`InstancePre::instantiate` and `TypedFunc::call`, to panic, and requires
that we instead call their `_async` variants. As mentioned above, I have
not yet introduced any actual asynchronicity or concurrency, so these
methods should never yield. Rather than `.await`ing their futures, I
have defined a degenerate `async` executor, `poll_once_executor`, which
polls a future exactly once, failing if it does not return
`Poll::Ready`. This means that we will panic if one of these futures
returns `Poll::Pending` unexpectedly.
The previous `trait Module` had a method `initial_instances`. `Module`
is now a concrete type, and I gave it this method, but it appears to be
unused. This is causing lints to fail. I am unsure what, if anything,
that method was for.
The previous `AutoReplacingModuleInstance` called `create_instance` on
the job thread. I am unsure if this was intentional, or just an artifact
of the previous implementation, where the `AutoReplacingModuleInstance`
lived on the job thread. I have written the new `ModuleInstanceManager`
to call `create_instance` on the calling thread, but it would be easy to
move that call into the job executor if that behavior is desired.
# API and ABI breaking changes
None user-facing
# Expected complexity level and risk
4: significant rewrite of performance-sensitive fiddly concurrency code.
Note specifically in above description:
- Running database jobs on the global Tokio runtime when not using core
pinning.
- Multi-tenant fairness issue: no longer possible to interrupt a
performance-intensive database mid-reducer to run another database
pinned to the same core.
- Unused method `module_instances`.
- Running `create_instance` on the calling thread rather than the
database thread.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Will arrange for a bot test.
- [ ] Determine to what extent we can run with real or synthetic
multi-tenant load in a test or staging environment.
---------
Signed-off-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
# Description of Changes
Update:
This PR did all of the below but was split. Now it just does:
1. Exposes V8/JS modules via the `unstable` feature flag on the host. To
publish a JS module, `--js-path path/to/module.js`
This PR:
1. Exposes V8/JS modules via the `unstable` feature flag on the host. To
publish a JS module, `--js-path path/to/module.js` needs to be used.
2. Bumps V8 to 140.2.
3. Shares more logic with WASM and makes some minor refactorings to
energy/budget logic.
4. Moves logic from `WasmInstanceEnv` to `InstanceEnv` and friends.
5. Makes JS modules actually work in terms of `create_instance`,
`make_actor`,
6. Fleshes out `call_reducer` with timeouts and long-running logs added
as well.
7. Adds all the syscalls with associated documentation as well.
# API and ABI breaking changes
None
# Expected complexity level and risk
2? It's only available on unstable and mostly touches V8 stuff.
# Testing
Follow up PRs will add unit tests for parts.
We'll also need to add integration tests for whole modules.
# Description of Changes
PR contains:
* CLI changes for the `pre_publish` endpoint when publishing a module.
* The regular `--yes` flag will not bypasses the *break clients* warning
prompt — an extra confirmation is now required. For CI, a hidden flag
`--break-clients` is added.
* Added smoketest.
* Some trivial naming changes in `client-api-*` crates for consistency
reasons.
* `pre_publish` route to accept similar Body size limit as `publish`
route.
# API and ABI breaking changes
an additive API change, does not break anything.
# Expected complexity level and risk
2
# Testing
- Existing smoketests passing for backward compatibility.
- New smoketest for add columns
---------
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
The `DurabilityProvider` trait was introduced to enable the
`HostController` to procure an alternative `Durability` impl from an
external source.
It is also useful to be able to instantiate a `SnapshotWorker`
externally, in order to subscribe to snapshot creation events without
access to the `RelationalDB` instance it is operating on.
At a later stage, we may also use it to control the snapshot frequency
externally.
This patch thus reframes the trait as `PersistenceProvider`, whose job
is to provide persistence-related services.
Also separates snapshot creation and compression of older snapshots, and
adds instrumentation to gather timing information for both.
# Description of Changes
Re-submit of #3281 (reverted by #3293), with only the intended changes.
# Expected complexity level and risk
1.5
# Testing
No functional changes.
# Description of Changes
As the title says
# Expected complexity level and risk
1
# Testing
- [x] Manually check the error launching a local pg instance
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
# Description of Changes
As the title says.
Add `--pg-port NUM` to the `start` command
# API and ABI breaking changes
Before this, it was set to `5432` unconditionally.
Docs updated at https://github.com/clockworklabs/SpacetimeDB/pull/3302.
# Expected complexity level and risk
1
# Testing
- [x] Run smoke tests
This reverts commit 2b61190d4d.
An accident happened, and the patch contains changes that were intended
for a separate PR.
Perhaps better to start over.
The `DurabilityProvider` trait was introduced to enable the
`HostController` to procure an alternative `Durability` impl from an
external source.
It is also useful to be able to instantiate a `SnapshotWorker`
externally, in order to subscribe to snapshot creation events without
access to the `RelationalDB` instance it is operating on.
At a later stage, we may also use it to control the snapshot frequency
externally.
This patch thus reframes the trait as `PersistenceProvider`, whose job
is to provide persistence-related services.
Also separates snapshot creation and compression of older snapshots, and
adds instrumentation to gather timing information for both.
# Expected complexity level and risk
1.5
# Testing
Not a functional change, existing tests should cover that.
joshua-spacetime