# Description of Changes
Notify user in CLI that js/ts is beta quality.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
N/A
# Description of Changes
Currently based on #3361
Implements most of the TS module API (not yet a function for type
aliases).
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Extremely basic module stuff works
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
---------
Signed-off-by: Noa <coolreader18@gmail.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@aol.com>
Co-authored-by: = <cloutiertyler@gmail.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Bumping versions to 1.6.0 in preparation for upcoming release.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
- [x] Existing CI only
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
TableIndex merge logic to was missing variants.
# Description of Changes
This pull request fixes an issue where the TableIndex::can_merge
function was missing match variants.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1 / 5 (Trivial)
This is a very low-risk change. It simply adds a few lines to a match
statement
# Testing
Manual testing confirm that creating two tables with indices with the
for the missing variants no longer causes a panic.
# Description of Changes
We've run into a problem on Maincloud caused by a database that was
writing a relatively small number of very large transactions. This was
accruing many commitlog segments consuming hundreds of gigabytes of
disk, but had not ever taken a snapshot, or compressed or archived any
data, as the database had not progressed past one million transactions.
With this PR, we take a snapshot every time the commitlog segment
rotates. We still also snapshot every million transactions.
One BitCraft database we looked at had 2.5 million transactions per
commitlog segment, meaning that this change will not meaningfully affect
the frequency of snapshots. The offending Maincloud database, however,
had only 50 transactions per segment!
# API and ABI breaking changes
N/a
# Expected complexity level and risk
3: Hastily made changes to finnicky code across several crates.
# Testing
I am unsure how to test these changes.
- [ ] <!-- maybe a test you want to do -->
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
# Description of Changes
See the inline comments for the motivation. This was originally
introduced to our Windows CI in #3351. This PR moves it from CI to
general Windows target builds, since it seems like Windows builds are
now generally having this issue.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
- [x] Windows CI still passes
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Removes the use of the Derived Data Cache during CI, will in
# API and ABI breaking changes
None
# Expected complexity level and risk
1 - Small change for CI
# Testing
- [x] Re-ran tests on both Linux + Windows with the change
# Description of Changes
- Fixed logic issue around Option<Vec<Option<>>> applying the wrong
types for primitives and dropping Optional
- Fixed an issue with enum vs enum variants wrapped in Option<>
producing the incorrect Unreal type
- Removed unnecessary and incorrect header bindings
- Type fix in the tests around the Optional Int32
# API and ABI breaking changes
No breaking changes
# Expected complexity level and risk
2 - Reworked incorrect optional lookups which can happen recursively
# Testing
I built out a few simple and complex objects in a Rust module to
triple-check possible cases beyond what the test framework calls out.
- [x] Tested many combinations from a Rust module to an Unreal project
- [x] Ran and updated Unreal test cases as necessary
# Description of Changes
`console.log` debugging statements accidentally made it into the
release.
# API and ABI breaking changes
None
# Expected complexity level and risk
1, trivial
# Testing
- [x] Automated testing only
# Description of Changes
Python is funny, if a file `token.py` is created and another script run
on the same dir, it will cause python to block:
```python
python3-3.13.7/lib/python3.13/tokenize.py", line 35, in <module>
from token import *
File "/Users/mamcx/token.py", line 3, in <module>
text = sys.stdin.read()
```
By coincidence the docs on `pg wire` use this name. Changed to one that
don't cause the trouble.
# Expected complexity level and risk
1
# Testing
- [x] Created another script and run it, see it blocks because this...
# Description of Changes
Host-side changes extracted from #3327
I added AUTO_INC_OVERFLOW even though we don't currently ever return it,
in order to future-proof so it's already there when we start emitting
it.
Prepublish was failing because it was expecting a wasm module
unconditionally, so now it takes ?host_type.
I tweaked JS deser to accept null/undefined when the unit type or an
option type is expected.
I switched to bsatn, because the native sats->js translator wasn't
matching what js was expecting.
I renamed the sys module: my thinking is that `spacetime:` as a scheme
will help disambiguate it, and maybe it could also be used for IMC in
the future or something? And I believe we had discussed wanting this to
be versioned, similar to wasm imports.
Trying to get a borrowed str from deserialize_js doesn't work, because
v8 strings don't store utf8.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] All this was done in the course of getting an actual typescript
module to successfully publish.
# Description of Changes
* Make sure the user provides at least one of `--rust-and-cli`,
`--typescript`, or `--csharp`, since providing none of them is a no-op
as of #3308
* Do a semver-parsing of the arg before doing anything, and use that
parsed version everywhere
* Consolidate some version strings that we were computing in a few
places
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
- [x] Running `cargo bump-versions 1.5.0 --typescript --rust-and-cli
--csharp` only shows a diff in the change dates
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Check that our generated C# files are up-to-date in our CI.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
- [x] CI all passes
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
* Small cleanups in `tools/check-diff.sh`
* Use `tools/check-diff.sh` wherever appropriate
* Simplify the `sdks/csharp/tools~/gen-*.sh` files after the repo merge
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
- [x] CI still passes
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Necessary for pulling in rolldown.
# API and ABI breaking changes
None
# Expected complexity level and risk
1, with the caveat that this updates the Rust version and therefore
touches all the code.
# Testing
- [ ] Just the automated testing
# Description of Changes
Tweaks V8 module support to use JS modules for `spacetimedb_sys` and the
user module rather than using scripts and the global object.
To this end, the code is also made more modular so that it e.g., cares
less avoid global vs. modules.
An `Object` with the functions is e.g., enough for the lowest level
`call_call_reducer`.
Some `.unwrap()`s are also removed.
Also, `run_timeout_and_cb_every` is disabled, as it currently leads to
UB due to bugs in the `v8` crate.
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
Future work.
# Description of Changes
In the React Integration tutorial
- Replace the staging URL by the production one
- Add useAutoSignin to automatically redirect users to the login page
- Add post_logout_redirect_uri to redirect users back to the application
after login
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
Local test
# Description of Changes
I changed some variables related to caching in the TypeScript test CI,
since it was failing on master due to suspected cache issues.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
- [x] CI passes on this PR when it didn't on master
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Pin the `temporal_rs` and `timezone_provider` versions to `0.0.11`,
because future versions such as `0.0.16` are incompatible, but their
version constraints are not correct so we keep getting
auto-rolled-forward to build-breaking versions.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
- [x] CI passes
- [x] If I `rm Cargo.lock && cargo clippy`, the build still passes
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
In service of adding procedures, which will need to execute WASM code in
an `async` environment so as to suspend execution while making HTTP
requests and whatnot.
Prior to this PR, `JobCores` worked by spawning an OS thread for each
database. These threads would then each be pinned to a specific core,
and in multi-tenant deployments multiple threads could be pinned to the
same core. Now, instead, we spawn one thread per available core at
startup. Each of these threads runs a single-threaded Tokio executor.
Each database is assigned to one of these executors, and runs tasks on
it via `tokio::spawn`.
When we run without core pinning (usually due to having too few hardware
cores), we won't spawn any additional threads or Tokio runtimes at all;
instead we will run database jobs on the "global" Tokio executor. These
jobs may block Tokio worker threads, which might be an issue if a very
core-constrained device runs multiple databases with very long-running
reducers. If this is an issue, we could in this case instead build a
second Tokio runtime only for running database jobs, and let the OS
scheduler figure things out like it did previously.
Previously, we implemented load-balancing among the database cores by
occasionally instructing per-database threads to re-pin themselves. Now,
we instead periodically send the database a new
`wasmtime::runtime::Handle`, which they will `spawn` future jobs into.
Previously, it was possible for a database thread to become canceled,
most likely as a result of `ModuleHost::exit`, after which calls would
fail with `NoSuchModule`. Cancellation is no longer meaningful, as the
database holds a `Handle` to a long-lived `tokio::runtime::Runtime`,
which should always outlive the `ModuleHost`. I have added an
`AtomicBool` flag to `ModuleHost` which is flipped by `exit` and checked
by calls to maintain the previous functionality.
Within this PR, the jobs run on the database-execution Tokio tasks are
not actually asynchronous; they will never yield. This is important
because these jobs may (will) hold a transaction open, and attempting to
swap execution to another job which wants a transaction on the same
database would be undesirable.
Note that this may regress our multi-tenant performance / fairness:
previously, in multi-tenant environments, the OS scheduler would divide
the database cores' time between the per-database threads, potentially
causing one high-load database to be interrupted in the middle of a
reducer in order to run other databases pinned to the same core. Now, a
high-load database will instead run its entire reducer to completion
before any other database gets to run.
We could, in the future, change this by instructing Wasmtime to yield
periodically, either via [epoch
interruption](https://docs.wasmtime.dev/api/wasmtime/struct.Store.html#method.epoch_deadline_async_yield_and_update)
or
[fuel](https://docs.wasmtime.dev/api/wasmtime/struct.Store.html#method.fuel_async_yield_interval),
both of which we're already configuring Wasmtime to track. We'd need (or
at least want) to (re-)introduce a queue s.t. we only attempt to run one
job for each database at a time. I have chosen not to do so within this
patch because I felt the changeset was complex enough already, and we
have so far not treated fairness in multi-tenant environments as a high
priority.
I have also reworked our module host machinery to no longer use dynamic
dispatch and trait polymorphism to manage modules and their instances,
and instead introduced `enum Module` and `enum Instance`, each of which
has a variant for Wasm and another for V8.
During this rewrite, I reworked `AutoReplacingModuleInstance`, which
previously used type-erased trait generics in a way that was brittle and
hard to re-use in the new `async` context. (Specifically, the module
instance no longer lives on the job thread, rather, the database grabs
the instance and sends it to the job thread, then gets it back when the
job exits. This is necessary to allow the re-worked load balancing
described above, as we can't have a single long-lived async task.) While
refactoring, I replaced it with `ModuleInstanceManager`, which can now
maintain multiple instances of the same module. This is not yet useful,
but will become necessary with procedures, as each concurrent procedure
will need its own instance. Relatedly, I changed
`ModuleHost::on_module_thread` (used by one-off and initial subscription
queries) to no longer acquire the/an instance. I discussed this with the
team, and consensus was that "locking" the module instance in that path
was not a useful behavior, just an artifact of the previous
implementation.
I have also switched our Wasmtime configuration to set
`async_support(true)`. This causes a variety of methods, notably
`InstancePre::instantiate` and `TypedFunc::call`, to panic, and requires
that we instead call their `_async` variants. As mentioned above, I have
not yet introduced any actual asynchronicity or concurrency, so these
methods should never yield. Rather than `.await`ing their futures, I
have defined a degenerate `async` executor, `poll_once_executor`, which
polls a future exactly once, failing if it does not return
`Poll::Ready`. This means that we will panic if one of these futures
returns `Poll::Pending` unexpectedly.
The previous `trait Module` had a method `initial_instances`. `Module`
is now a concrete type, and I gave it this method, but it appears to be
unused. This is causing lints to fail. I am unsure what, if anything,
that method was for.
The previous `AutoReplacingModuleInstance` called `create_instance` on
the job thread. I am unsure if this was intentional, or just an artifact
of the previous implementation, where the `AutoReplacingModuleInstance`
lived on the job thread. I have written the new `ModuleInstanceManager`
to call `create_instance` on the calling thread, but it would be easy to
move that call into the job executor if that behavior is desired.
# API and ABI breaking changes
None user-facing
# Expected complexity level and risk
4: significant rewrite of performance-sensitive fiddly concurrency code.
Note specifically in above description:
- Running database jobs on the global Tokio runtime when not using core
pinning.
- Multi-tenant fairness issue: no longer possible to interrupt a
performance-intensive database mid-reducer to run another database
pinned to the same core.
- Unused method `module_instances`.
- Running `create_instance` on the calling thread rather than the
database thread.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Will arrange for a bot test.
- [ ] Determine to what extent we can run with real or synthetic
multi-tenant load in a test or staging environment.
---------
Signed-off-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
The `#[table]` macro now generates read-only table and index handles.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
1
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] positive and negative test cases using
`ReducerContext::as_read_only()`
# Description of Changes
It turns out that cargo automatically uses the latest semver-compatible
versions of dependencies, which is not what we expected. tl;dr
specifying `1.5.0` actually means `>=1.5.0 <2.0.0`, but we actually
intend `1.5.*`.
This PR updates our `upgrade-version` tool, and re-runs it to fix the
dep versions.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
- [x] I ran `cargo bump-versions 1.5.0 --rust-and-cli` to regenerate the
other committed files.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
This resolves an issue reported in Discord:
https://discordapp.com/channels/1037340874172014652/1398209084699709492/1423784670402842766
> note the name is Counter. in the typescript bindings, my useTable only
gives me the option for counter => useTable<DbConnection,
Counter>('counter');
> I switched my name in the c# module to [SpacetimeDB.Table(Public =
true, Name = "counter")] and it just works.
> Capitalization and whatnot strikes again
The diff seems large because of the mechanical changes to codegen. You
only really need to review code gen'd tables and `index.ts`. All the
other files just have a new `__TableHandle` type import.
# API and ABI breaking changes
Technically API breaking, but it fixes a bug where the old API doesn't
work
# Expected complexity level and risk
2, relatively straight-foward changes to the TS SDK and TS codegen
# Testing
- [x] I have tested with a Rust module that has an `UpperCase` table,
that that name is passed correctly down to the TypeScript client through
the codegen and that it works in `useTable`
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
# Description of Changes
Update:
This PR did all of the below but was split. Now it just does:
1. Exposes V8/JS modules via the `unstable` feature flag on the host. To
publish a JS module, `--js-path path/to/module.js`
This PR:
1. Exposes V8/JS modules via the `unstable` feature flag on the host. To
publish a JS module, `--js-path path/to/module.js` needs to be used.
2. Bumps V8 to 140.2.
3. Shares more logic with WASM and makes some minor refactorings to
energy/budget logic.
4. Moves logic from `WasmInstanceEnv` to `InstanceEnv` and friends.
5. Makes JS modules actually work in terms of `create_instance`,
`make_actor`,
6. Fleshes out `call_reducer` with timeouts and long-running logs added
as well.
7. Adds all the syscalls with associated documentation as well.
# API and ABI breaking changes
None
# Expected complexity level and risk
2? It's only available on unstable and mostly touches V8 stuff.
# Testing
Follow up PRs will add unit tests for parts.
We'll also need to add integration tests for whole modules.
# Description of Changes
This is the implementation of issue #3191. This adds a Default attribute
to C# module fields.
**Note**: In C#, attribute arguments must be compile-time constants,
which means you can't directly use non-constant expressions like new
expressions, method calls, or dynamic values in attribute constructors.
(Ref:
https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/language-specification/attributes#2324-attribute-parameter-types)
For this reason, these default values are limited to primitive types,
enums, and strings.
This includes (shown as `C# Type` (`BSATN type`):
* `bool` (`Bool`)
* `sbyte` (`I8`)
* `byte` (`U8`)
* `short` (`I16`)
* `ushort` (`U16`)
* `int` (`I32`)
* `unit` (`U32`)
* `long` (`I64`)
* `ulong` (`U64`)
* `float` (`F32`)
* `double` (`F64`)
* `enum` (`Enum`)
* `string` (`String`)
* `null` (`RefOption`) <- Nullable type
Because of C# limitations, for nullable and complex data types, such as
a struct, can take use `[Default(null)]` to populate values with null
defaults. This allows things like structs to workaround the non-constant
expressions in attribute constructors limitation by allowing these
complex types to still be able to be added as new column tables.
The `int` type can also be in the form of Hex or Binary literals, such
as `[Default(0x2A)]` or `[Default(0b00101010)]`
Both Decimal (like `[Default(3.14m)]`) and Char (like `[Default('A')]`)
are unsupported types in BSATN and will still return `BSATN0001` errors.
# API and ABI breaking changes
Not API breaking.
This change only adds the `[Default(value)]` attribute logic.
Using the `[Default(value)]` attribute with older versions SpacetimeDB
C# modules will result in an error.
# Expected complexity level and risk
2
# Testing
Local testing of this requires use of CLI changes in #3278
- [x] Regression test of functionality added.
# Description of Changes
v8: use fast static strings for known strings
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Covered by existing.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Added support for wss:// protocol in Unreal SDK. The SDK previously
forced all connections to use ws://, preventing connections to servers
behind SSL/TLS proxies. The fix detects if the URI already includes a
protocol (ws:// or wss://) and preserves it; otherwise defaults to ws://
for backward compatibility. This enables connections to SpacetimeDB
servers hosted with Cloudflare Tunnels or other secure proxies.
# API and ABI breaking changes
This change is fully backwards compatible. URIs without a protocol
prefix will continue to work as before, with ws:// being automatically
prepended. No API signatures or existing behavior has changed.
# Expected complexity level and risk
Simple string prefix check before URL construction. The core WebSocket
connection logic remains unchanged, and the default behavior is
preserved when no protocol is specified.
# Testing
Successfully tested with a wss:// connection to my self-hosted
SpacetimeDB server behind Cloudflare Tunnel. Verified backward
compatibility with protocol-less URIs when directly connecting to the
server. Currently running without issues.
---------
Signed-off-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
# Description of Changes
Tiny fix to update the size scripts for the `spacetimedb` TypeScript
package
# API and ABI breaking changes
none
# Expected complexity level and risk
1
# Testing
- [ ] I verified the size script do be workin.
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
# Description of Changes
- Shares more logic with WASM and makes some minor refactorings to
energy/budget logic.
- Moves logic from `WasmInstanceEnv` to `InstanceEnv` and friends.
- Makes JS modules actually work in terms of `create_instance`,
`make_actor`,
- Fleshes out `call_reducer` with timeouts and long-running logs added
as well.
- Adds all the syscalls with associated documentation as well.
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
Future work.
# Description of Changes
- Extract from `wasm_instance_env.rs`
- `console_timer_end`: use Noop backtrace.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
No semantic changes.
# Description of Changes
Fix bindgen tests (due to crate `timezone_provider`)
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Fixes CI tests.
# Description of Changes
Extract `InstanceEnv::console_timer_end`.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Just code motion.
# Description of Changes
Prefix of https://github.com/clockworklabs/SpacetimeDB/pull/3276 to
bisect a problem in smoketests.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Future work.
Make it so the `SnapshotWorker` can be re-configured with a new
committed state. This allows event subscriptions to remain valid while a
replica transitions from leader to follower and vice versa.
This is considerably simpler than keeping the lifetimes of database and
persistence services strictly in-sync, at the expense of an idle task
per replica.
# Expected complexity level and risk
1.5
---------
Signed-off-by: Kim Altintop <kim@eagain.io>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
# Description of Changes
Fix various issues on the React Integration page:
- Invalid links (Closes#3332)
- Fix broken syntax highlight
- Update the page title (React -> `React integration`)
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
Tested using a local version of the website
Zeke Foppa