# Description of Changes
Python is funny, if a file `token.py` is created and another script run
on the same dir, it will cause python to block:
```python
python3-3.13.7/lib/python3.13/tokenize.py", line 35, in <module>
from token import *
File "/Users/mamcx/token.py", line 3, in <module>
text = sys.stdin.read()
```
By coincidence the docs on `pg wire` use this name. Changed to one that
don't cause the trouble.
# Expected complexity level and risk
1
# Testing
- [x] Created another script and run it, see it blocks because this...
# Description of Changes
Host-side changes extracted from #3327
I added AUTO_INC_OVERFLOW even though we don't currently ever return it,
in order to future-proof so it's already there when we start emitting
it.
Prepublish was failing because it was expecting a wasm module
unconditionally, so now it takes ?host_type.
I tweaked JS deser to accept null/undefined when the unit type or an
option type is expected.
I switched to bsatn, because the native sats->js translator wasn't
matching what js was expecting.
I renamed the sys module: my thinking is that `spacetime:` as a scheme
will help disambiguate it, and maybe it could also be used for IMC in
the future or something? And I believe we had discussed wanting this to
be versioned, similar to wasm imports.
Trying to get a borrowed str from deserialize_js doesn't work, because
v8 strings don't store utf8.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] All this was done in the course of getting an actual typescript
module to successfully publish.
# Description of Changes
* Make sure the user provides at least one of `--rust-and-cli`,
`--typescript`, or `--csharp`, since providing none of them is a no-op
as of #3308
* Do a semver-parsing of the arg before doing anything, and use that
parsed version everywhere
* Consolidate some version strings that we were computing in a few
places
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
- [x] Running `cargo bump-versions 1.5.0 --typescript --rust-and-cli
--csharp` only shows a diff in the change dates
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Check that our generated C# files are up-to-date in our CI.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
- [x] CI all passes
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
* Small cleanups in `tools/check-diff.sh`
* Use `tools/check-diff.sh` wherever appropriate
* Simplify the `sdks/csharp/tools~/gen-*.sh` files after the repo merge
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
- [x] CI still passes
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Necessary for pulling in rolldown.
# API and ABI breaking changes
None
# Expected complexity level and risk
1, with the caveat that this updates the Rust version and therefore
touches all the code.
# Testing
- [ ] Just the automated testing
# Description of Changes
Tweaks V8 module support to use JS modules for `spacetimedb_sys` and the
user module rather than using scripts and the global object.
To this end, the code is also made more modular so that it e.g., cares
less avoid global vs. modules.
An `Object` with the functions is e.g., enough for the lowest level
`call_call_reducer`.
Some `.unwrap()`s are also removed.
Also, `run_timeout_and_cb_every` is disabled, as it currently leads to
UB due to bugs in the `v8` crate.
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
Future work.
# Description of Changes
In the React Integration tutorial
- Replace the staging URL by the production one
- Add useAutoSignin to automatically redirect users to the login page
- Add post_logout_redirect_uri to redirect users back to the application
after login
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
Local test
# Description of Changes
I changed some variables related to caching in the TypeScript test CI,
since it was failing on master due to suspected cache issues.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
- [x] CI passes on this PR when it didn't on master
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Pin the `temporal_rs` and `timezone_provider` versions to `0.0.11`,
because future versions such as `0.0.16` are incompatible, but their
version constraints are not correct so we keep getting
auto-rolled-forward to build-breaking versions.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
- [x] CI passes
- [x] If I `rm Cargo.lock && cargo clippy`, the build still passes
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
In service of adding procedures, which will need to execute WASM code in
an `async` environment so as to suspend execution while making HTTP
requests and whatnot.
Prior to this PR, `JobCores` worked by spawning an OS thread for each
database. These threads would then each be pinned to a specific core,
and in multi-tenant deployments multiple threads could be pinned to the
same core. Now, instead, we spawn one thread per available core at
startup. Each of these threads runs a single-threaded Tokio executor.
Each database is assigned to one of these executors, and runs tasks on
it via `tokio::spawn`.
When we run without core pinning (usually due to having too few hardware
cores), we won't spawn any additional threads or Tokio runtimes at all;
instead we will run database jobs on the "global" Tokio executor. These
jobs may block Tokio worker threads, which might be an issue if a very
core-constrained device runs multiple databases with very long-running
reducers. If this is an issue, we could in this case instead build a
second Tokio runtime only for running database jobs, and let the OS
scheduler figure things out like it did previously.
Previously, we implemented load-balancing among the database cores by
occasionally instructing per-database threads to re-pin themselves. Now,
we instead periodically send the database a new
`wasmtime::runtime::Handle`, which they will `spawn` future jobs into.
Previously, it was possible for a database thread to become canceled,
most likely as a result of `ModuleHost::exit`, after which calls would
fail with `NoSuchModule`. Cancellation is no longer meaningful, as the
database holds a `Handle` to a long-lived `tokio::runtime::Runtime`,
which should always outlive the `ModuleHost`. I have added an
`AtomicBool` flag to `ModuleHost` which is flipped by `exit` and checked
by calls to maintain the previous functionality.
Within this PR, the jobs run on the database-execution Tokio tasks are
not actually asynchronous; they will never yield. This is important
because these jobs may (will) hold a transaction open, and attempting to
swap execution to another job which wants a transaction on the same
database would be undesirable.
Note that this may regress our multi-tenant performance / fairness:
previously, in multi-tenant environments, the OS scheduler would divide
the database cores' time between the per-database threads, potentially
causing one high-load database to be interrupted in the middle of a
reducer in order to run other databases pinned to the same core. Now, a
high-load database will instead run its entire reducer to completion
before any other database gets to run.
We could, in the future, change this by instructing Wasmtime to yield
periodically, either via [epoch
interruption](https://docs.wasmtime.dev/api/wasmtime/struct.Store.html#method.epoch_deadline_async_yield_and_update)
or
[fuel](https://docs.wasmtime.dev/api/wasmtime/struct.Store.html#method.fuel_async_yield_interval),
both of which we're already configuring Wasmtime to track. We'd need (or
at least want) to (re-)introduce a queue s.t. we only attempt to run one
job for each database at a time. I have chosen not to do so within this
patch because I felt the changeset was complex enough already, and we
have so far not treated fairness in multi-tenant environments as a high
priority.
I have also reworked our module host machinery to no longer use dynamic
dispatch and trait polymorphism to manage modules and their instances,
and instead introduced `enum Module` and `enum Instance`, each of which
has a variant for Wasm and another for V8.
During this rewrite, I reworked `AutoReplacingModuleInstance`, which
previously used type-erased trait generics in a way that was brittle and
hard to re-use in the new `async` context. (Specifically, the module
instance no longer lives on the job thread, rather, the database grabs
the instance and sends it to the job thread, then gets it back when the
job exits. This is necessary to allow the re-worked load balancing
described above, as we can't have a single long-lived async task.) While
refactoring, I replaced it with `ModuleInstanceManager`, which can now
maintain multiple instances of the same module. This is not yet useful,
but will become necessary with procedures, as each concurrent procedure
will need its own instance. Relatedly, I changed
`ModuleHost::on_module_thread` (used by one-off and initial subscription
queries) to no longer acquire the/an instance. I discussed this with the
team, and consensus was that "locking" the module instance in that path
was not a useful behavior, just an artifact of the previous
implementation.
I have also switched our Wasmtime configuration to set
`async_support(true)`. This causes a variety of methods, notably
`InstancePre::instantiate` and `TypedFunc::call`, to panic, and requires
that we instead call their `_async` variants. As mentioned above, I have
not yet introduced any actual asynchronicity or concurrency, so these
methods should never yield. Rather than `.await`ing their futures, I
have defined a degenerate `async` executor, `poll_once_executor`, which
polls a future exactly once, failing if it does not return
`Poll::Ready`. This means that we will panic if one of these futures
returns `Poll::Pending` unexpectedly.
The previous `trait Module` had a method `initial_instances`. `Module`
is now a concrete type, and I gave it this method, but it appears to be
unused. This is causing lints to fail. I am unsure what, if anything,
that method was for.
The previous `AutoReplacingModuleInstance` called `create_instance` on
the job thread. I am unsure if this was intentional, or just an artifact
of the previous implementation, where the `AutoReplacingModuleInstance`
lived on the job thread. I have written the new `ModuleInstanceManager`
to call `create_instance` on the calling thread, but it would be easy to
move that call into the job executor if that behavior is desired.
# API and ABI breaking changes
None user-facing
# Expected complexity level and risk
4: significant rewrite of performance-sensitive fiddly concurrency code.
Note specifically in above description:
- Running database jobs on the global Tokio runtime when not using core
pinning.
- Multi-tenant fairness issue: no longer possible to interrupt a
performance-intensive database mid-reducer to run another database
pinned to the same core.
- Unused method `module_instances`.
- Running `create_instance` on the calling thread rather than the
database thread.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Will arrange for a bot test.
- [ ] Determine to what extent we can run with real or synthetic
multi-tenant load in a test or staging environment.
---------
Signed-off-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
The `#[table]` macro now generates read-only table and index handles.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
1
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] positive and negative test cases using
`ReducerContext::as_read_only()`
# Description of Changes
It turns out that cargo automatically uses the latest semver-compatible
versions of dependencies, which is not what we expected. tl;dr
specifying `1.5.0` actually means `>=1.5.0 <2.0.0`, but we actually
intend `1.5.*`.
This PR updates our `upgrade-version` tool, and re-runs it to fix the
dep versions.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
- [x] I ran `cargo bump-versions 1.5.0 --rust-and-cli` to regenerate the
other committed files.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
This resolves an issue reported in Discord:
https://discordapp.com/channels/1037340874172014652/1398209084699709492/1423784670402842766
> note the name is Counter. in the typescript bindings, my useTable only
gives me the option for counter => useTable<DbConnection,
Counter>('counter');
> I switched my name in the c# module to [SpacetimeDB.Table(Public =
true, Name = "counter")] and it just works.
> Capitalization and whatnot strikes again
The diff seems large because of the mechanical changes to codegen. You
only really need to review code gen'd tables and `index.ts`. All the
other files just have a new `__TableHandle` type import.
# API and ABI breaking changes
Technically API breaking, but it fixes a bug where the old API doesn't
work
# Expected complexity level and risk
2, relatively straight-foward changes to the TS SDK and TS codegen
# Testing
- [x] I have tested with a Rust module that has an `UpperCase` table,
that that name is passed correctly down to the TypeScript client through
the codegen and that it works in `useTable`
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
# Description of Changes
Update:
This PR did all of the below but was split. Now it just does:
1. Exposes V8/JS modules via the `unstable` feature flag on the host. To
publish a JS module, `--js-path path/to/module.js`
This PR:
1. Exposes V8/JS modules via the `unstable` feature flag on the host. To
publish a JS module, `--js-path path/to/module.js` needs to be used.
2. Bumps V8 to 140.2.
3. Shares more logic with WASM and makes some minor refactorings to
energy/budget logic.
4. Moves logic from `WasmInstanceEnv` to `InstanceEnv` and friends.
5. Makes JS modules actually work in terms of `create_instance`,
`make_actor`,
6. Fleshes out `call_reducer` with timeouts and long-running logs added
as well.
7. Adds all the syscalls with associated documentation as well.
# API and ABI breaking changes
None
# Expected complexity level and risk
2? It's only available on unstable and mostly touches V8 stuff.
# Testing
Follow up PRs will add unit tests for parts.
We'll also need to add integration tests for whole modules.
# Description of Changes
This is the implementation of issue #3191. This adds a Default attribute
to C# module fields.
**Note**: In C#, attribute arguments must be compile-time constants,
which means you can't directly use non-constant expressions like new
expressions, method calls, or dynamic values in attribute constructors.
(Ref:
https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/language-specification/attributes#2324-attribute-parameter-types)
For this reason, these default values are limited to primitive types,
enums, and strings.
This includes (shown as `C# Type` (`BSATN type`):
* `bool` (`Bool`)
* `sbyte` (`I8`)
* `byte` (`U8`)
* `short` (`I16`)
* `ushort` (`U16`)
* `int` (`I32`)
* `unit` (`U32`)
* `long` (`I64`)
* `ulong` (`U64`)
* `float` (`F32`)
* `double` (`F64`)
* `enum` (`Enum`)
* `string` (`String`)
* `null` (`RefOption`) <- Nullable type
Because of C# limitations, for nullable and complex data types, such as
a struct, can take use `[Default(null)]` to populate values with null
defaults. This allows things like structs to workaround the non-constant
expressions in attribute constructors limitation by allowing these
complex types to still be able to be added as new column tables.
The `int` type can also be in the form of Hex or Binary literals, such
as `[Default(0x2A)]` or `[Default(0b00101010)]`
Both Decimal (like `[Default(3.14m)]`) and Char (like `[Default('A')]`)
are unsupported types in BSATN and will still return `BSATN0001` errors.
# API and ABI breaking changes
Not API breaking.
This change only adds the `[Default(value)]` attribute logic.
Using the `[Default(value)]` attribute with older versions SpacetimeDB
C# modules will result in an error.
# Expected complexity level and risk
2
# Testing
Local testing of this requires use of CLI changes in #3278
- [x] Regression test of functionality added.
# Description of Changes
v8: use fast static strings for known strings
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Covered by existing.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Added support for wss:// protocol in Unreal SDK. The SDK previously
forced all connections to use ws://, preventing connections to servers
behind SSL/TLS proxies. The fix detects if the URI already includes a
protocol (ws:// or wss://) and preserves it; otherwise defaults to ws://
for backward compatibility. This enables connections to SpacetimeDB
servers hosted with Cloudflare Tunnels or other secure proxies.
# API and ABI breaking changes
This change is fully backwards compatible. URIs without a protocol
prefix will continue to work as before, with ws:// being automatically
prepended. No API signatures or existing behavior has changed.
# Expected complexity level and risk
Simple string prefix check before URL construction. The core WebSocket
connection logic remains unchanged, and the default behavior is
preserved when no protocol is specified.
# Testing
Successfully tested with a wss:// connection to my self-hosted
SpacetimeDB server behind Cloudflare Tunnel. Verified backward
compatibility with protocol-less URIs when directly connecting to the
server. Currently running without issues.
---------
Signed-off-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
# Description of Changes
Tiny fix to update the size scripts for the `spacetimedb` TypeScript
package
# API and ABI breaking changes
none
# Expected complexity level and risk
1
# Testing
- [ ] I verified the size script do be workin.
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Noa