# Description of Changes
This PR removes the `@clockworklabs/typescript-sdk` from the repository
and retains only `spacetimedb` in the `crates/bindings-typescript`
directory. Some files are migrated to `spacetimedb`. I have also updated
the appropriate READMEs.
In addition I have symlinked the old `sdks/typescript` directory to
point to `crates/bindings-typescript`.
# API and ABI breaking changes
This is not technically a breaking change of any kind, although it does
orphan and deprecate the
[@clockworklabs/spacetimedb-sdk](https://www.npmjs.com/package/@clockworklabs/spacetimedb-sdk)
npm package. This package will no longer work with SpacetimeDB.
Users should now install and use the `spacetimedb` package.
# Expected complexity level and risk
2, it's a straightforward change but affects many files.
# Testing
- [ ] I ran `pnpm test` in the `spacetimedb` package
- [ ] I ran the quickstart app
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
This adds `react` as an optional peer dependency. If the TypeScript SDK
is imported into a library that uses React (of the appropriate version)
then they will have access to two new React hooks:
```ts
useSpacetimeDB<DbConnection>();
useTable<DbConnection, MyTable>('my_table');
```
This PR also updates the TypeScript `quickstart-chat` tutorial in the
docs to use the new React hooks. I will split that tutorial into
separate `React` and `TypeScript` tutorials in a future PR.
# API and ABI breaking changes
This is a purely additive change to the SDK
# Expected complexity level and risk
2 because it changes how the SDK is built a bit. Namely it makes the
`spacetimedb` library an external dependency.
# Testing
TODO: I am not through testing yet.
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: Julien Lavocat <JulienLavocat@users.noreply.github.com>
# Description of Changes
Closes#3219
This adds the Unreal SDK, the new Unreal test cases, updates the test
runner to handle Unreal, codegen updates for Unreal, and a QuickStart
Chat.
# API and ABI breaking changes
No breaking changes.
# Expected complexity level and risk
2 - This impacts the subcommand generate.rs to include unrealcpp and
crates/testing to expand for Unreal
# Testing
- [x] Run the new Unreal tests
- [x] Run any previous automation testing - with all the changes to
generate/testing I'm uncertain if there is an impact
- [x] Review the new CLI generate documentation changes
---------
Co-authored-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
This PR moves most of the contents of `@clockworklabs/spacetimedb-sdk`
into the `spacetimedb` module. The `spacetimedb` module now exports
`sdk` and `server` as separate subpaths where `sdk` contains the code
which was previously in `@clockworklabs/spacetimedb-sdk`.
In particular it makes the following moves:
- `/sdks/typescript/packages/sdk` -> `/sdks/typescript`
- most of the contents of `/sdks/typescript/packages/sdk` ->
`crates/bindings-typescript`
- `/sdks/typescript/packages/test-app` ->
`crates/bindings-typescript/test-app`
The following packags was NOT moved:
`/sdks/typescript/examples/quickstart-chat`
## Motivation
In accordance with
https://github.com/clockworklabs/SpacetimeDB/issues/3250, we would like
to consolidate `@clockworklabs/spacetimedb-sdk` into a single
`spacetimedb` package so that users can import the different things they
need from a single package.
### Pros:
- allow users to install a single package with subpaths `spacetimedb`,
`spacetimedb/react`, `spacetimedb/sdk`, `spacetimedb/server`, etc.
- Is much simpler for bundling, etc.
- Is backwards compatible with `@clockworklabs/spacetimedb-sdk` which
now becomes a thin wrapper
- eventually allow us to break up the `spacetimedb` package into other
packages if we want to split them up (e.g. `@spacetimedb/lib`,
`@spacetimedb/sdk`, etc.) and we can solve the build complexity that
introduces when we get to it
- eventually allow us to move `bindings-csharp` out of the crates
directory where it probably doesn't belong anyway
- organizes all TypeScript packages into the packages directory where
you'd normally expect them, with the possible exception of
`/sdks/typescript` if we wanted to leave that separate
### Cons:
- The `sdk` directory is now a bit of a ruse as to where the code
actually lives since it's just a thin wrapper. If it eventually becomes
its own independent package, we'll also have to break up spacetimedb
into `@spacetimedb/lib` and `@spacetimedb/server` so that
`@clockworklabs/spacetimedb-sdk` can depend on `@spacetimedb/lib` while
being a dependency of `spacetimedb`.
Ideally this change would have been made later, however, it became
necessary for the following **heinously disastrous chain of forcing
moves**:
1. Adding `react` support necessitated shipping react as an optional
peer dependency under `@clockworklabs/spacetimedb-sdk/react`
2. This required adding a new build target/export/bundle
3. Previously `@clockworklabs/spacetimedb-sdk` was configured to have
`noExternal` for `spacetimedb` meaning it would collect the library into
the sdk bundle. I attempted to continue this for react support but...
4. Creating a new `react` bundle which also pulled in `spacetimedb`
caused their to be nominal type conflicts between classes in the
duplicate `spacetimedb` bundles.
5. Changing `spacetimedb` to be included as `external` caused compile
errors because `@clockworklabs/spacetimedb-sdk` is configured in
`tsconfig.json` to fail on unused variables and it was now including the
source of `spacetimedb` which is not configured to error on unused
variables and has "unused" private variables which are actually used by
us secretly, but not exposed to the clients.
> SIDE NOTE: The unused variables settings cannot be turned off on a
line by line basis, so it has to be turned off entirely, but in order to
maintain the linting checks we had I used `eslint` to enforce the rule
so that we could disable it line by line. (This caused me to discover
quite a lot of things that were broken that were caught by `eslint`
being applied to the entire project. `eslint` was previously only
applied to the `quickstart-chat` and the `crates/bindings-typescript`
library)
6. Changing the build to be external, now requires `spacetimedb` to also
be published to npm as its own module which
`@clockworklabs/spacetimedb-sdk` now imports, which requires that we add
`tsup` config to `spacetimedb` to publish a built version of the
library.
7. The only way to avoid that is to move the `sdk` and `react` code from
`@clockworklabs/spacetimedb-sdk` into the existing `spacetimedb` package
to avoid the duplicate import problem on step 4 and change
`@clockworklabs/spacetimedb-sdk` back to again use `noExternal` for its
`spacetimedb` dependency.
And here we are. I chose not to move `/crates/bindings-typescript` even
though that's probably not a great place long term. It would be better
to have it in `/packages/spacetimedb` or `/npm-packages/spacetimedb` or
`/ts-packages/spacetimedb` or something, and move all our TypeScript
packages in there. But that is a different matter.
The net result however is that we have a new `spacetimedb` package which
exports the different parts of the API under:
- `spacetimedb`
- `spacetimedb/server`
- `spacetimedb/sdk`
- `spacetimedb/react`
while still not breaking the existing deploy process, nor any
users/developers who are currently using
`@clockworklabs/spacetimedb-sdk`.
I think long term should we ever decide to split `spacetimedb` up into
multiple packages or if we have additional unrelated packages, we should
publish them to the `@spacetimedb` org which I reserved for us here:
https://www.npmjs.com/org/spacetimedb
> NOTE: `spacetimedb` is a package and `@spacetimedb` is an org.
`spacetimedb/sdk` is not a separate package, it's a subpath export of
the `spacetimedb` package, whereas `@spacetimedb/sdk` would be (and
would need to be) it's own separate package. You can certainly have both
`spacetimedb/sdk` and `@spacetimedb/sdk`. We could for example host the
code for the sdk at `@spacetimedb/sdk` and just reexport it from
`spacetimedb` under the `spacetimedb/sdk` subpath.
# API and ABI breaking changes
This should not change or modify the API or ABI in any way. If it does
so accidentally it is a bug, although I carefully went through the
exports.
# Expected complexity level and risk
3 because it changes how the SDK is built a bit and rearranges a lot of
paths.
# Testing
- [x] All of the CI passes
- [x] I also ran quickstart-chat to confirm that it is not broken
- [x] I also ran test-app
# Description of Changes
This adds a new system table to store the jwt payloads of connected
clients. I'm planning to use this system table to expose client claims
to modules in subsequent PRs.
The new table is called `st_connection_credentials`. It is a **private**
system table which stores a mapping from `connection_id` to
`jwt_payload`. Note that a jwt payload is a json representation of the
clients claims, not a fully signed token.
The times when we need to insert and delete these rows closely mirrors
that of the existing `st_client` table, with 1.5 exceptions:
1. We weren't previously inserting to `st_client` until after the
`OnConnect` reducer ran (even though it was in the same transaction). We
want `st_connection_credentials` to be populated before calling the
reducer, so that the reducer can use it get the credentials, so I made a
change to insert to `st_client` and `st_connection_credentials` before
calling the reducer.
2. This difference has not actualized, but when clients start sending
refresh tokens, we will probably need to update the credentials stored
in this table.
This also enforces uniqueness of connection ids. A duplicate connection
id will now make the on-connect reducer fail (since it will violate
uniqueness when trying to insert to `st_connection_credentials`).
# Expected complexity level and risk
2.5
Adding a system table is a bit risky. This is almost rollback safe, with
one annoying case that is worth calling out:
If a database is created with this system table, opening it with an
older version of spacetimedb will only work if there is a snapshot of
the database. If we try to load a table without a snapshot, replaying
will fail on the first row for that table. This is because we don't
write the table schema information to the commit log when creating a
database. In practice, this is unlikely to be an issue, because new
databases asynchronously trigger a snapshot immediately after creation.
Migrating existing databases will be fine. On startup this will detect
that there is a missing system table, and add it in a way that writes it
to the commit log. Since it is in the commit log, we can open the
database with an older version and still understand the data for that
table.
# Testing
There are unit tests that cover opening a database created with an older
version (which doesn't have this table).
I manually tested opening a migrated database with an older version of
spacetimedb.
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
In order to facilitate commitlog and snapshot archival, this patch adds
channel params to `snapshot_watching_commitlog_compressor`
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
1
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
The method is no longer used.
Note that this patch must merge **after** the private companion.
# API and ABI breaking changes
Internal API change.
# Expected complexity level and risk
1
# Testing
In private.
# Description of Changes
PR introduces support for column-level default values via a new
`#[default(...)]` attribute.
It also validates, `default` macro is not used along with `primary_key`,
`unique` or `auto_inc`.
# API and ABI breaking changes
NA
# Expected complexity level and risk
2
# Testing
Start using macro in `module-test`.
---------
Co-authored-by: James Gilles <jameshgilles@gmail.com>
# Description of Changes
Closes
[#2686](https://github.com/clockworklabs/SpacetimeDB/issues/2686).
Add support for listening using the [PG wire
protocol](https://www.postgresql.org/docs/current/protocol.html) so `pg`
clients could be used against the database.
# API and ABI breaking changes
The output of `duration` is changed to `rfc3339`, instead of the way is
made with `sats` because is what is done in `pg`, see note below.
# Expected complexity level and risk
2
~~There is open questions that are in the [ticket
#2686](https://github.com/clockworklabs/SpacetimeDB/issues/2686). Also
the crate used here require `RustTls`, so it could be good idea to
decide if~~:
* ~~Rewrite a big chunk of code to use `OpenSSL`~~
* ~~Move to `RustTls`
https://github.com/clockworklabs/SpacetimeDB/pull/1700~~
* ~~Pay for the extra compilation cost~~.
I open another port(`5433`) to listen for `pg` connections using `ssl`.
Need to be decided if this is the way or instead try to multi-plex the
current port for both protocols.
# Testing
Only manual testing so far. Solving the above questions allow me to
implement some unit tests. Also, not yet integrated into cloud for the
same reasons.
- [x] Adding some test for the binary encoding of special and primitive
types
- [x] Smoke test using `psql` that connect to the db instance and run
some queries
- [x] Manually inspect using a UI database explorer how infer the types,
some of this tools generate special widgets when displaying `json,
duration, etc`
---------
Co-authored-by: Noa <coolreader18@gmail.com>
# Description of Changes
- Adds endpoint for for pretty printing migration plan.
- It also changes current `publish` endpoint to optionally provide
`MigrationToken` and `MigrationPolicy` to allow migration with breaking
clients.
# API and ABI breaking changes
Backward compatible change to existing API and new Api
# Expected complexity level and risk
2
# Testing
- Existing smoketest should cover changes for `publish` endpoint.
- For pretty print endpoint, smoketests can be written only after cli
changes.
---------
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
Co-authored-by: James Gilles <jameshgilles@gmail.com>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
# Description of Changes
Implements [subscribing to durable
commits](https://github.com/clockworklabs/SpacetimeDBPrivate/issues/1594).
The setting works on a per-connection level, and essentially just delays
sending transaction updates until the transaction is reported as durable
by the database.
For connectionless SQL operations, the setting works per-request. No SQL
syntax is provided by this patch to toggle the configuration.
After some deliberation, I opted to obtain the offset when a transaction
commits (as opposed to when it starts). This creates some mild
inconvenience, because we prevent the transaction from committing until
the corresponding subscription updates are enqueued.
The strategy is, however, more correct should we ever support weaker
isolation levels, and it is easier to document.
Follow-ups include:
- Provide SQL syntax (`SET synchronous_commit = ON` or something)
- C# and TypeScript SDKs
- Reference docs?
# API and ABI breaking changes
Not breaking, but adds a parameter to the subscribe and sql endpoints.
# Expected complexity level and risk
4
To the author's understanding, ordering of outbound messages is not
changed by this patch, even if there are messages that don't have a
transaction offset (such as error messages). I.e. while waiting for the
transaction offset of a message to become durable, no message enqueued
after that message will be delivered. This may not be desirable in some
cases.
The patch may contain concurrency bugs, e.g. awaiting futures that may
never resolve.
# Testing
- [x] Implemented a new test in the `module_subscription_actor` module
- [x] Added unit tests for the core logic in `ClientConnectionReceiver`
It would be desirable to also have integration-level tests, but I'm
currently unsure how to write those without being able to control if and
when the database reports an offset as durable.
---------
Signed-off-by: Kim Altintop <kim@eagain.io>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
# Description of Changes
This is a purely additive change to introduce, `TypeBuilder`s,
`ColumnBuilder`s, and a `t` factory export that has methods for creating
`TypeBuilder`s.
There are derived types from `TypeBuilder`s for each variant of
`AlgebraicType`. `TypeBuilder`s can be converted into a `ColumnBuilder`
which supports creating additional metadata specifying whether a column
is a primary key or should be unique. `ColumnBuilder`s are not allowed
within `TypeBuilder`s, but the opposite is allowed (for composite
types).
`ColumnBuilder`s can only be used at the top level and contain a
`TypeBuilder`.
`TypeBuilder` stores three pieces of information:
1. A phantom typescript type that the SATS type should be interpreted as
in TS
2. A phantom typescript type which is the type of the SATS type
3. A runtime AlgebraicType which stores the information to be reported
to host.
`ColumnBuilder` stores three pieces of information:
1. A `TypeBuilder`
2. A phantom typescript type with the column metadata
3. A runtime `ColumnMetadata` which stores the information to be
reported to the host.
Additionally, it is only possible to add metadata to `ColumnBuilder` or
`TypeBuilder` types which are compatible with that metadata. e.g. It is
a compile-time error to add `isAutoIncrementing` to a `String`
type/column.
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] I've created a set of type tests in `type_builders.test-d.ts`
- [x] I also created a set of runtime tests in `index.test.ts`
- [x] I have modified CI so that these tests run on every PR
# Description of Changes
This simplifies system table bootstrapping logic by removing
auto-incremented ids for indexes, constraints, and sequences in system
tables. By defining them all up front, we avoid the need for
`reset_system_table_schemas`, and make it easier to add new things to
system tables.
This change should not change the initial system table rows written
during bootstrapping, and the `load_1_2_*` tests in `relational_db.rs`
should verify that by loading data written by a previous version with
and without a snapshot.
I also added some sequence-related changes. The sequence schema previous
included the `allocated` field, which tracked where to start generating
new values after a restart. This made some checks awkward, since the
'schema' was changing as rows were inserted. I moved this outside of the
`schema`, so those should no longer change unless something actually
changes the shape of the tables.
As part of that, I also changed where we begin generating on startup to
be `allocated` instead of `allocated + 1`. The code that generates
values always stopped early enough to allow the next restart to use
`allocated`, so we were just being overly defensive. We also had some
issues if sequences wrapped over, or if we had sequences with negative
increments. Those were supported by the schema, but didn't really work
with the generation/allocation code. That should have better testing
now.
# Expected complexity level and risk
2. The bootstrapped system tables should match, since we have tests with
older data. The changes to sequences include some more invariant
checking, which has some risk of causing problems if previous versions
wrote values that I didn't consider.
# Testing
This has existing tests covering bootstrapping and restoring from a
snapshot.
# Description of Changes
Previously all TypeScript generated code had a `// @ts-ignore` directive
at the top to suppress unused variable errors that users may have run
into with their generated code. This worked to suppress unused, but it
was too broad and suppressed actual errors with our generated code. This
PR removes that directive and fixes the latent errors.
Most notably this fixes the issue in the quickstart where we were
getting `serialize` is not a function.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] This PR increases the test surface only by now typechecking the
typescript generated code within our repository
# Description of Changes
Wire `call_reducer` to `call_reducer_with_tx` and leave some TODOs for
future work.
Also fixes some small misc.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Not in a testable state yet, future PRs will add tests.
# Description of Changes
Closes https://github.com/clockworklabs/SpacetimeDBPrivate/issues/1987
# API and ABI breaking changes
This does not affect any APIs but it does affect the code generated
# Expected complexity level and risk
1
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [ ] Snap tests
# Description of Changes
Adds utilities for marking and deleting snapshot directories that have
been archived
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
1
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
Testing will be handled by the patch that adds archival
# Description of Changes
This PR:
- standardizes the prettier config across all TypeScript projects
- adds a root level package.json
- standardizes all `pnpm` commands to be the same
- updates documentation accordingly
- adds some additional typescript testing for serialization and
deserialization
**IMPORTANT!** Once this PR merges we will need to change the
`compile-and-test` required check to `build-and-test`
# API and ABI breaking changes
No breaking changes.
# Expected complexity level and risk
2 - It in principle doesn't change any code, but could affect deploy
processes.
# Testing
- [x] Just the automated testing that we had previously
- [x] I added additional automated tests
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Please note, much of the code changed in this PR is generated code.
This change updates the TypeScript SDK to use a new `spacetimedb`
TypeScript library which lives under the `/crates/bindings-typescript`
folder alongside `/crates/bindings-csharp`. Just like with the C#
bindings library, the types for `AlgebraicType` and `RawModuleDef` are
now code generated with a script in `/crates/codegen/examples`.
Pulling this out into a library allows us to use the same types and
serialization code on both the server and the client for TypeScript
modules.
In the process of making this change I also found and fixed several
issues with the TypeScript code generation. Namely an issue with
recursive types and an issue with the `never` type. I also removed any
use of `namespace`s since those are a TypeScript only feature, and we
want to have JavaScript + types so that we can use the generated code
with ESBuild without TSC.
I have also improved the npm/pnpm scripts to be able to generate
TypeScript code for us automatically by running `pnpm generate` for any
place that we have to generate typescript code. Namely:
- Quickstart module bindings
- AlgebraicType/ModuleDef for TypeScript module library
- Client API messages for the TypeScript API
- TestApp module bindings
# API and ABI breaking changes
IMPORTANT! This is an API breaking change for clients, as such it should
be a major version change. However, I am going to see if I can shim in
the old API as best as possible to make it compatible.
Notably, we were previously exporting APIs that end users do not need,
and I don't think it would ever occur to them to use, namely the whole
`AlgebraicValue` API and also the `AlgebraicType` API. In principle, no
one should have a need for these, although it was technically possible
for them to use it.
Indeed, we could potentially even just remove AlgebraicType completely
from the API by directly code generating the serialization code.
Listed below are all of the **BREAKING** changes to the API and their
effect:
- `AlgebraicType` is now a structural union literal type instead of a
class (nominal type), this is a consequence of generating the type with
our code gen. Users did not have a reason to use `AlgebraicType`
directly.
- The `AlgebraicValue` type has been removed entirely. This was
previously a class that was exported from the SDK, but very unlikely to
be used by users.
- The `ProductValue` type has been removed.
- The `ReducerArgsAdapter` and `ValueAdapter` types, which were used
with AlgebraicValues have been removed.
- Generated code has changed incompatibly so users will have to
regenerate code when upgrading to this version. Technically a breaking
change, but pretty easy to fix.
Listed below are the non-breaking API changes:
- I am now exporting generated product types as the default export in
addition to how I was exporting them before
- For generated sum type `T`, I am now exporting a `TVariants` namespace
which has the types of all the variants for `T`. This was previously
exposed on the namespace `T`, but was inaccessible in modules other than
the one it was defined in because I also export a type `T` in addition
to namespace `T` and in the type position `T` was being interpreted as a
type rather than a namespace. It's unclear why TypeScript resolved it as
the `T` namespace within the module in which is was defined previously.
Anyway, since the old types were apparently unobservable to users, I've
replaced them with the types in `TVariants`. (Open to other names for
this namespace).
- I fixed a bug where never types (sum types with no variants) were not
correctly generated.
# Expected complexity level and risk
3. The most complex thing about the PR are the potential impacts to the
API. I am reasonably certain, but not 100% certain that I have accounted
for everything above.
# Testing
- [x] I ran `pnpm test` which runs all the tests in the repo including
an integration test which tests the connection to SpacetimeDB. I also
fixed broken tests.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
For V8 modules:
- Add `call_call_reducer`
- Add stack traces for traps.
- Wire `update_database`
It remains to wire up `call_call_reducer`, but this shouldn't be that
much work, but I'll do that in a follow up.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Tests for `call_call_reducer` are added.
# Description of Changes
Title. When validating column type changes in `Table::change_columns_to`
and comparing layouts in `RowTypeLayout::is_compatible_with`, return
structured error objects which describe the mismatch.
I made this change while debugging the issue that led to #3203 , where
`change_columns_to` returned an error during replay of an automigration
which had succeeded the first time. The original error contained enough
information to debug, but it was presented in a noisy and unhelpful way,
so I wrote this patch to get better diagnostics.
Per @Centril 's comments, these errors are for internal assertions, not
user-facing error reporting, so we value fail-fast rather than error
hygiene and choose not to use the `ErrorStream` combinator.
Also note that I sprinkled `Box`es around some large-ish error types to
quiet
https://rust-lang.github.io/rust-clippy/master/index.html#result_large_err
.
# API and ABI breaking changes
N/a
# Expected complexity level and risk
1.
# Testing
Manual testing with BitCraft. As this is purely altering the error
reporting for non-user-facing assertions, I don't believe any further
testing is necessary.
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
- Simple patch version bump. Because we forgot to bump the version
number for 1.3.1 this is bumping from 1.3.0 => 1.3.2
# API and ABI breaking changes
0
# Testing
- Verified that the version number in the license has been updated
- Verified the output of `spacetime --version`:
```
$ spacetime --version
spacetime Path: C:\Users\boppy\AppData\Local\SpacetimeDB\bin\current\spacetimedb-cli.exe
Commit: 0027d094e1
spacetimedb tool version 1.3.2; spacetimedb-lib version 1.3.2;
```
---------
Signed-off-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
In `st_column_changed`, `iter_st_column_for_table` returns rows in
physical storage order. In simple cases (without unrelated deletes) this
will be the same as insertion order, and therefore also the same as
sorted order, but in cases with multiple column-changing automigrations
the physical storage order of the rows diverges from the correct sorted
order. Because this isn't a hot path, we can just call `.sort()` and not
worry about it.
# API and ABI breaking changes
N/a
# Expected complexity level and risk
1
# Testing
None yet. Needs manual testing with BitCraft update.
---------
Signed-off-by: Mazdak Farrokhzad <twingoow@gmail.com>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Co-authored-by: Shubham Mishra <shubham@clockworklabs.io>
# Description of Changes
Fixes
https://github.com/orgs/clockworklabs/projects/22?pane=issue&itemId=102392974&issue=clockworklabs%7Ccom.clockworklabs.spacetimedbsdk%7C276
by renaming `internal` `static` serializer fields so that they do not
overlap with user-provided names.
Note, however, that some field names still will not work: `ReadFields`,
`WriteFields`, `Equals`, and `GetHashCode`.
This would require a separate fix since the error would happen in a
different place. In this case we would need to change the name of the
generated member to something like `ReadFields_` or `Equals_`, which I'm
not sure is a good idea.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
0
# Testing
SDK tests and `dotnet-verify` tests are passing.
---------
Signed-off-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
# Description of Changes
First commit reverts disabling adding enum variants
(https://github.com/clockworklabs/SpacetimeDB/pull/3178).
Second commit makes replay recognize inserts / deletes to `st_column`
and triggers a refresh of the in-memory table that was referenced in the
`st_column` change.
# API and ABI breaking changes
None
# Expected complexity level and risk
3, small code bug "deep".
# Testing
I've confirmed manually that a module that couldn't be restarted before
this PR can now be restarted.
We should probably follow up this PR with a smoketest.
# Description of Changes
`fn alter_table_row_type` now writes the new row type to `st_column`.
This is not a fix for the adding-variants problem, but it does get us
closer.
# API and ABI breaking changes
None
# Expected complexity level and risk
2, system tables stuff is always risky.
# Testing
`test_alter_table_row_type_is_transactional` is amended with assertions.
# Description of Changes
Refactors the machinery of call_reducer and update_database to
a) have smaller pieces that make up the whole so that the code becomes
clearer
b) extract all the VM-independent stuff so that it can be reused by V8
modules.
This is best reviewed commit by commit.
# API and ABI breaking changes
None.
# Expected complexity level and risk
2, it's an important place, but this is doing just code motion.
# Testing
No semantic changes, just code motion.
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
Disables automigrations for column type changes
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
0
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Updated smoketests
# Description of Changes
Iterating over a HashMap does not guarantee any ordering of the items.
To ensure consistent and predictable pretty-printing, explicitly sort
entries.
# API and ABI breaking changes
NA
# Expected complexity level and risk
0
# Description of Changes
This obscured other used code, so let's remove it as its dead.
# API and ABI breaking changes
None
# Expected complexity level and risk
-1
# Testing
Not applicable.
# Description of Changes
I'm moving `crates/sdk` to `sdks/rust` to be more in line with where the
rest of our SDKs are listed. I updated the corresponding paths etc. that
pointed to the previous location.
This PR is based on
https://github.com/clockworklabs/SpacetimeDB/pull/3185, because if we
merge this without that, our release scripts will be broken.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
- [x] Existing CI passes
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
This adds auto migration support to the `schema` crate.
Additional changes are needed to fully implement this feature: see
https://github.com/clockworklabs/SpacetimeDB/issues/2912
# API and ABI breaking changes
None
# Expected complexity level and risk
1, at this level the change is fairly straightforward. The integration
will be more difficult.
# Testing
Added `schema` unit tests for success/failure cases.
---------
Co-authored-by: Shubham Mishra <shivam828787@gmail.com>
# Description of Changes
fixes#3174 .
During initialization, entries were added to the `DelayQueue` but not to
`key_map`.
### Detailed Explanation:
1. `DelayQueue` is **not set-semantic**, so we track uniqueness with a
`key_map: FxHashMap` but that wasn't updated during initiliaziation.
2. `Scheduler::schedule` is **not transactional**: it enqueues reducers
even if the DB transaction later fails (abort, duplicate row, etc.). On
yield, `SchedulerActor` checks the DB before execution.
**Combined Effect**: A transaction that does not actually change a
scheduled entry but still calls `Scheduler::schedule` after a module
update will cause a duplicate entry in the `DelayQueue`, since `key_map`
does not yet contain that entry.
**Why It Didn’t Show Earlier**:
When a repeating reducer executes, we re-schedule it by updating both
`DelayQueue` and `key_map` correctly.
The bug only appears in the window after updating module but before the
first execution, if a transaction calls schedule without actually
modifying the DB row.
Which was indeed happening as per discord chat:
> but yeah most likely order of event was modue was updated
> and then update_scheduled_timers_from_static_data was called
window between update module and first execution is 1 hour for this
case.
## Repo steps:
1. publish this module, it makes `send_scheduled_message` reducer to be
called every 10 secs.
```rust
#[spacetimedb::table(name = scheduled_message, public, scheduled(send_scheduled_message))]
pub struct ScheduledMessage {
#[primary_key]
#[auto_inc]
scheduled_id: u64,
scheduled_at: ScheduleAt,
}
#[spacetimedb::reducer]
fn send_scheduled_message(ctx: &ReducerContext, sched: ScheduledMessage) -> Result<(), String> {
info!("Sending scheduled message: {:?}", ctx.timestamp);
Ok(())
}
#[spacetimedb::reducer(init)]
pub fn init(ctx: &ReducerContext) {
ctx.db.scheduled_message().insert(ScheduledMessage {
scheduled_id: 0,
scheduled_at: Duration::from_secs(10).into(),
});
}
#[spacetimedb::reducer]
pub fn update_timer(ctx: &ReducerContext) {
for mut timer in ctx.db.scheduled_message().iter() {
timer.scheduled_at = Duration::from_secs(10).into();
ctx.db.scheduled_message().scheduled_id().update(timer);
log::info!("building decay agent timer was updated");
}
}
```
2. Update module to support automigration (add a table) and re-publish
it.
3. Call reducer `update_timer` and do it before first execution of
`send_scheduled_message` after updating module.
4. As `update_timer` doesn't change the existing scheduler but calls
`Scheduler::schedule` it will cause duplicate entry in `DelayQueue`.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
1, pretty obvious fix.
# Testing
manually.
The code fix is straightforward, but the issue only becomes visible
under specific conditions.
# Description of Changes
This change adds the following `System.TimeSpan`-style static
construction methods to `SpacetimeDB.TimeDuration`:
* `static TimeDuration FromMilliseconds(double milliseconds)`
* `static TimeDuration FromSeconds(double seconds)`
* `static TimeDuration FromMinutes(double minutes)`
* `static TimeDuration FromHours(double hours)`
* `static TimeDuration FromDays(double days)`
These mirror the equivalently named static methods on `System.TimeSpan`
and dramatically improve usability and familiarity for experienced C#
users with no more overhead than the user performing the multiplication
themselves.
Wish I'd thought to do this before v1.1.2 got released. Ah well.
# API and ABI breaking changes
None. Convenience methods added in bindings only.
# Expected complexity level and risk
1 (potentially up to a low 2 if cleanup is desired elsewhere in the
bindings to leverage these new methods).
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Ensure the changes build. <!-- maybe a test you want to do -->
- [ ] New contributor check! Review to make sure repo style & substance
standards are complied with. <!-- maybe a test you want a reviewer to
do, so they can check it off when they're satisfied. -->
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
We stopped incrementing the incoming queue length metric. This patch
increments it again and adds a regression test.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
1
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Regression test
# Description of Changes
Closes#3170 .
Commit messages:
### Increase the default incoming-queue-length limit
2048 turned out to be too low a value for BitCraft, as their world
upload process requests on the order of 6000 reducers very rapidly. We
still feel that having a limit is valuable to prevent malicious or
misguided clients from taking an arbitrarily large amount of host
memory, so we bump the value to give us a wide safety error for
BitCraft's needs but don't remove the limit entirely.
### Add log at `warn` when the host disconnects a client due to too many
requests
# API and ABI breaking changes
N/a
# Expected complexity level and risk
1
# Testing
- [x] @mamcx to run a BitCraft bot test.
# Description of Changes
Defines a building piece `call_describe_module` that can be used to call
the expected ABI function `describe_module` in JS.
There might be further changes to this but this is a first working pass
at it.
# API and ABI breaking changes
None
# Expected complexity level and risk
2, this does not integrate with existing code reachable in production,
but the exception handling required some googling/reading docs.
# Testing
A test `call_describe_module_works` is added that tests returning an
empty `RawModuleDef` from JS.
The `Clone` impl for `ClientConnection` would create an independent
instance that could not observe module hotswapping. This would result in
methods called on a replaced `ModuleHost` to fail, because that host
exited already.
Fix by reading the `ModuleHost` from the watch channel directly, instead
of maintaining a redundant copy.
Also fix `watch_module_host` to properly mark the current module host as
seen.
# Expected complexity level and risk
2
# Testing
- [x] test suite passes
- [x] ran @bfops repro script
# Description of Changes
We recently merged several repos together. This PR clarifies the license
terms for several subdirectories, as well as the relationship between
the licenses.
The licenses in our subdirectories have become symbolic links to
licenses in our toplevel `licenses` directory. For any particular
subdirectory's license file in the diff, you can click `... -> View
file` and then click on the text that says "Symbolic Link" on that page.
This will take you to the license file that it links to.
I have also updated the `tools/upgrade-version` script to update the
change date in the new `licenses/BSL.txt` file.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
None. Only changes to license files.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
This does 2 things:
1. simplifies the deserialization of optional sums so that they are
treated as regular sums.
2. adds higher level v8 ser/de interfaces and privatizes what can be
private.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Existing ser/de tests are adjusted to use the new interfaces.
# Description of Changes
As requested by @gefjon
Most of this is:
- `'a` -> `'this`
- `'s` -> `'scope`
# API and ABI breaking changes
None
# Expected complexity level and risk
0
# Testing
No semantic changes.
# Description of Changes
On databases with many already-compressed snapshots, this was leading to
log spam without providing any useful information.
# API and ABI breaking changes
N/a
# Expected complexity level and risk
1
# Testing
N/a
Adds methods and free-standing functions to allow folds to stop at an
upper
bound, by passing a range instead of only a start offset.
# Expected complexity level and risk
1
# Testing
# Description of Changes
Currently with the function maybe_log_error to log errors we hide the
real source line where the error happened. So to still have the same
logic but with more accurate source line, I moved it to a macro to have
the same abstraction for logging.
# API and ABI breaking changes
-
# Expected complexity level and risk
1
# Testing
- [ ] I have tested it my application and it now show the better source
line of the log message
---------
Signed-off-by: ResuBaka <ResuBaka@users.noreply.github.com>
Co-authored-by: Phoebe Goldman <phoebe@goldman-tribe.org>
# Description of Changes
This changes the behavior on the rust client, so that we let the server
generate the connection id if the client hasn't called the unstable
method to set a connection id.
This is awkward for the `connection_id` function, since it now panics if
the connection id hasn't been received from the server. We should
deprecate this function in favor of a `try` version.
This also changes the behavior of reusing the connection id if a client
reconnects.
The corresponding private PR is
https://github.com/clockworklabs/SpacetimeDBPrivate/pull/1921.
# API and ABI breaking changes
Technically not changing any API signatures, but this is
behavior-changing, since the `connection_id` function can now panic.
# Expected complexity level and risk
2. The risk here is people relying on that behavior.
# Testing
I think some tests need to be updated.
# Description of Changes
Avoid building a temporary `Vec` in `execute_plan` by exposing a
list-building interface instead.
- The old `fn encode_list` is rewritten in terms of this list-building
interface.
- The `BsatnRowList` and `BsatnRowListBuilder` types are split into two
entirely separate types. The latter now tries to recognize the case
where there isn't a known static layout, but where the BSATN lengths
happen to be the same for all rows anyways. In those cases, the
allocation of `RowSizeHint::RowOffsets` is avoided in favor of just
storing the found length in bytes. This is in particular useful for
small table updates as statistically, the fewer rows, the more chance of
the lengths being all equal. In the case of a single row, the chance is
notably 100%. It is also good for the case of when we don't have
`RelValue::Row` or `Row::Ptr` but where the underlying table that
actually has a static layout.
In the future, we might want to avoid these lists in incremental as
well.
# Benchmarks
Benchmark numbers vs. master using `cargo bench --bench subscription --
--baseline subs` on i7-7700K, 64GB RAM:
```
footprint-scan time: [28.731 ms 28.924 ms 29.171 ms]
change: [-49.728% -49.006% -48.388%] (p = 0.00 < 0.05)
Performance has improved.
```
Performance goes from roughly 56.721 ms to 28.795 ms.
# API and ABI breaking changes
None
# Expected complexity level and risk
2, fairly local change to just subscriptions.
# Testing
Covered by existing tests.
# Description of Changes
Bumped all versions to 1.3.0 in preparation for an upcoming minor
release.
# API and ABI breaking changes
No breaking changes.
# Expected complexity level and risk
1
# Testing
None
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Tyler Cloutier