# Description of Changes
The `AutoMigrateStep::DisconnectAllUsers` step is implemented as
follows:
1. The `spacetimedb::db::update::update_database` function returns a
response of type
`UpdateDatabaseResult::UpdatePerformedWithClientDisconnect`.
2. Upon receiving this response, the `host_controller::update_module`
proceeds to drop the `watch::Sender<ModuleHost>` field within the
`core::host_controller::Host` and disconnect clients.
# API and ABI breaking changes
NA
# Expected complexity level and risk
3.
Diff code is simple but It depends on the subcription logic to behave
correctly.
# Testing
Manually.
---------
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Report more metrics about snapshot compression, namely:
- time to compress a single snapshot (histogram)
- for each compression pass:
- number of snapshots found to be already compressed (gauge)
- number of snapshots compressed (gauge)
- cumulative number of objects compressed (gauge)
- cumulative number of objects hardlinked (gauge)
Those metrics are collected from the `spacetimedb-snapshot` crate
without imposing a prometheus dependency on it, i.e. they can be
observed by the caller as ordinary Rust types.
This is exploited to avoid scanning the entire snapshot repository on
each pass -- only the range `(last_compressed + 1)..newest_snapshot` is
visited (note that the `compress_snapshots` method now short-circuits on
errors).
Lastly, the snapshot worker can now be configured to disable
compression. This greatly simplifies implementation of alternative
post-processing strategies, e.g. involving archival, for which a more
coarse-grained compression strategy may be more appropriate.
Subcribers are notified of a new snapshot _after_ compression, such that
any filesystem locks should be released.
# Expected complexity level and risk
2
# Testing
May need some, I'm pondering.
The `DurabilityProvider` trait was introduced to enable the
`HostController` to procure an alternative `Durability` impl from an
external source.
It is also useful to be able to instantiate a `SnapshotWorker`
externally, in order to subscribe to snapshot creation events without
access to the `RelationalDB` instance it is operating on.
At a later stage, we may also use it to control the snapshot frequency
externally.
This patch thus reframes the trait as `PersistenceProvider`, whose job
is to provide persistence-related services.
Also separates snapshot creation and compression of older snapshots, and
adds instrumentation to gather timing information for both.
# Description of Changes
Re-submit of #3281 (reverted by #3293), with only the intended changes.
# Expected complexity level and risk
1.5
# Testing
No functional changes.
# Description of Changes
As the title says
# Expected complexity level and risk
1
# Testing
- [x] Manually check the error launching a local pg instance
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
# Description of Changes
I chose to regenerate all of the bindings in
https://github.com/clockworklabs/SpacetimeDB/pull/3310, but that caused
the snapshots to change. This PR follows up to update them.
I also updated the outdated script `tools~/gen-quickstart-chat.sh`
(which was generating from a different module than the CI was), and
updated the CI to use it instead of manually running a similar command.
# API and ABI breaking changes
Test-only changes.
# Expected complexity level and risk
1
# Testing
- [x] unity-testsuite passes on this PR (which it does not on `master`).
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
We had places that weren't passing `-y`. I assume these were only
working because they were in environments where the things were already
installed, so there wasn't any confirmation dialog. For some reason, the
arm runner now newly needs to install packages, so CI invocations began
failing.
# API and ABI breaking changes
None. CI only.
# Expected complexity level and risk
1
# Testing
- [x] CI passes
- [ ] the "Test spacetimedb-update" flow passes on arm, which is not
currently true elsewhere.
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Move the unity testsuite to `spacetimedb-runner` since it otherwise runs
out of disk space.
# API and ABI breaking changes
None. CI only.
# Expected complexity level and risk
1
# Testing
- [x] Unity CI passes now.
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Move the global.json file to the repo root, which should make `dotnet`
resolve the correct SDK regardless of the directory it is run in.
Where a copy of the global.json file is needed, symlink it to the one at
the root.
Should fix "C#/Unity - Test Suite".
# Description of Changes
As the title says.
Add `--pg-port NUM` to the `start` command
# API and ABI breaking changes
Before this, it was set to `5432` unconditionally.
Docs updated at https://github.com/clockworklabs/SpacetimeDB/pull/3302.
# Expected complexity level and risk
1
# Testing
- [x] Run smoke tests
# Description of Changes
Each reducer gets its arguments through an `ArgSource`, a Unix-file-like
abstraction for streams of bytes. Prior to this commit, we had an ABI
designed as if it could support other args sources, but it actually
hardcoded the ID of the reducer args source, and errored elsewhere.
This commit extends the `BytesSource` infrastructure to support other
bytes sources. This will be useful for exposing JWT payloads and HTTP
responses. No other `BytesSource` uses are actually included in this
commit, only the infrastructure.
This commit also defines a new host call,
`bytes_source_remaining_length`. This is intended to allow callers to
pre-allocate a buffer correctly sized to read the entire `BytesSource`
all at once. The new host function is added to a new ABI minor version,
10.1, so that old SpacetimeDB hosts can detect and reject too-new
compiled modules. I have added uses of this new function to
`__call_reducer__` in both Rust and C#, even though it's not strictly
necessary,
and I haven't removed the loop which repeatedly calls
`bytes_source_read` and grows the buffer.
# API and ABI breaking changes
Adds a new ABI minor version, `spacetime_10.1`. This means that old
SpacetimeDB hosts will reject new compiled modules.
# Expected complexity level and risk
2-ish? WASM ABI code is always fiddly, but this is a pretty simple case.
# Testing
- [x] New behavior and new host function are both hit through existing
tests that instantiate modules and call reducers against them, so I
believe automated testing is sufficient.
---------
Signed-off-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
# Description of Changes
Implements `Serialize` and `Deserialize` for tuples.
Extracted from https://github.com/clockworklabs/SpacetimeDB/pull/3276.
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
A test `roundtrip_tuples_in_different_data_formats` is added.
This reverts commit 2b61190d4d.
An accident happened, and the patch contains changes that were intended
for a separate PR.
Perhaps better to start over.
The `DurabilityProvider` trait was introduced to enable the
`HostController` to procure an alternative `Durability` impl from an
external source.
It is also useful to be able to instantiate a `SnapshotWorker`
externally, in order to subscribe to snapshot creation events without
access to the `RelationalDB` instance it is operating on.
At a later stage, we may also use it to control the snapshot frequency
externally.
This patch thus reframes the trait as `PersistenceProvider`, whose job
is to provide persistence-related services.
Also separates snapshot creation and compression of older snapshots, and
adds instrumentation to gather timing information for both.
# Expected complexity level and risk
1.5
# Testing
Not a functional change, existing tests should cover that.
# Description of Changes
Adds a way to configure the DbConnection to use confirmed reads.
# Expected complexity level and risk
1
# Testing
Still figuring out...
# Description of Changes
Inspired by a question @Shubham8287 asked me: how long does it take to
replay real production databases, e.g. a BitCraft region module? I did
not have a good answer, and didn't know how I could get an answer.
Add 5 new metrics:
- `replay_total_time_seconds(db: Identity)`, total time to read and
restore snapshot and replay commitlog.
- `replay_snapshot_read_time_seconds(db: Identity)`, time to read
snapshot from disk.
- `replay_snapshot_restore_time_seconds(db: Identity)`, time to restore
snapshot once it's already in memory.
- `replay_commitlog_time_seconds(db: Identity)`, time to replay
commitlog once snapshot is restored.
- `replay_commitlog_num_commits(db: Identity)`, number of commits
visited during commitlog replay, i.e. the length of the suffix after the
most recent usable snapshot.
All of these are `Gauge`s (or `IntGauge` for the `num_commits` metric),
as we expect observations to be very infrequent
(approx. once per label value per SpacetimeDB process).
# API and ABI breaking changes
N/a
# Expected complexity level and risk
1
# Testing
I remain unsure how to test metrics. This won't break anything, though,
so I'm not worried.
# Description of Changes
In environments without dotnet installed, these tests were the only
thing preventing `cargo test --all -- --skip csharp` from completing
successfully.
I also included `rust` in the name, though running tests in an
environment without cargo/rustc installed seems less likely to work.
Extracted from #3263 at request of @Centril .
# API and ABI breaking changes
N/a
# Expected complexity level and risk
1
# Testing
N/a
# Description of Changes
Finish the work started in #3227, which switched a hardcoded `127.0.0.1`
to a dynamic server address.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
- [x] Smoketests still pass
- [ ] `Internal Tests` pass under this PR (they don't on master)
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
- `add_columns_to_table` api to handle `AutoMigrateStep::AddColumns` --
Look at doc comment for more Info.
depends on: https://github.com/clockworklabs/SpacetimeDB/pull/3261
TODO: handle `AutoMigrateStep::DisconnectAllUsers`.
# API and ABI breaking changes
N/A.
# Expected complexity level and risk
2? Changes are not in hotpath but a bug in migration can corrupt tables.
# Testing
a test to exercise the API.
---------
Signed-off-by: Mazdak Farrokhzad <twingoow@gmail.com>
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
# Description of Changes
Extends log records with a new field, `function`, which stores the name
of the reducer being executed when the log was produced. I have chosen
to name this field `function` rather than `reducer` because we will soon
be adding procedures, which are not reducers but will also be valid
values for this field.
While making this change, I noticed inconsistent values for injected
logs. Previously, we injected logs in three places, with different
values for the record fields:
1. `SystemLogger` (used when publishing and updating) set `filename:
Some("spacetimedb")` and `target: None`.
2. `log_reducer_error` (used for reducer error returns) set `filename:
None` and `target: Some(reducer)`, with `reducer` being the name of the
reducer.
3. `ModuleHost::inject_logs` (used for calls to nonexistent reducers and
calls with ill-typed arguments) set `filename: Some("external")` and
`target: None`.
With this change, I have decided that injected logs universally have
`filename: Some("__spacetimedb__")` and `target:
Some("__spacetimedb__")`. I have chosen the double-underscore convention
for reserved names to avoid confusion should a user name a source file
or reducer `spacetimedb`. I am not terribly attached to using
`spacetimedb` here, and would be happy to change to some other string,
like `internal` or `system`.
I have further decided that `log_reducer_error` and `inject_logs` both
have access to a sensible non-`None` value for `function` and so should
set it to the name of the reducer. The `target` field is not used for
this. Note that in cases where a client attempts to call a non-existent
reducer, the `function` field of the log record will be set to that
client-supplied non-existent reducer name.
# API and ABI breaking changes
Changes our log record format. This is at least backwards-compatible (as
in, newer consumers can read older logs) because the new field is
optional. I am unsure if it is forwards-compatible (as in, older
consumers reading newer logs) because I don't know if `serde_json` will
ignore unknown fields or will error.
# Expected complexity level and risk
1
# Testing
- [x] Manually ran `quickstart-chat` (modified to log messages and name
changes) and saw sensible output.
# Description of Changes
Adds support for truncation in the datastore.
The commitlog format already supported it.
A table is considered truncated in a transaction when there are deletes
and when the row count post-deletes is 0.
# API and ABI breaking changes
None
# Expected complexity level and risk
3?
# Testing
Some existing datastore tests have been amended to check whether
`TxData` contain truncation or not.
---------
Signed-off-by: Mazdak Farrokhzad <twingoow@gmail.com>
Co-authored-by: Shubham Mishra <shubham@clockworklabs.io>
Co-authored-by: Shubham Mishra <shivam828787@gmail.com>
# Description of Changes
Aternative to and closes
https://github.com/clockworklabs/SpacetimeDB/pull/3210.
This version relies on `pending_schema_changes`.
The first commit adds `clear_table` to the datastore that's efficient
and can be exposed to the module ABI in a follow up.
The second commit fixes `drop_table`.
# API and ABI breaking changes
None
# Expected complexity level and risk
3?
# Testing
`test_drop_table_is_transactional` is amended to check `TxData`.
---------
Signed-off-by: Mazdak Farrokhzad <twingoow@gmail.com>
Co-authored-by: Shubham Mishra <shubham@clockworklabs.io>
Co-authored-by: Shubham Mishra <shivam828787@gmail.com>
# Description of Changes
Bumping remaining files so that everything is at 1.4.0.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
None
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
This PR removes the `@clockworklabs/typescript-sdk` from the repository
and retains only `spacetimedb` in the `crates/bindings-typescript`
directory. Some files are migrated to `spacetimedb`. I have also updated
the appropriate READMEs.
In addition I have symlinked the old `sdks/typescript` directory to
point to `crates/bindings-typescript`.
# API and ABI breaking changes
This is not technically a breaking change of any kind, although it does
orphan and deprecate the
[@clockworklabs/spacetimedb-sdk](https://www.npmjs.com/package/@clockworklabs/spacetimedb-sdk)
npm package. This package will no longer work with SpacetimeDB.
Users should now install and use the `spacetimedb` package.
# Expected complexity level and risk
2, it's a straightforward change but affects many files.
# Testing
- [ ] I ran `pnpm test` in the `spacetimedb` package
- [ ] I ran the quickstart app
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Closes: https://github.com/clockworklabs/SpacetimeDBPrivate/issues/1925
This adds Unreal Blackholio to the demo/Blackholio folder matching the
Unity version using server-rust as the default. Includes:
- Leaderboard
- Circle Material - wavy border
- Split/Sucide/Input Lock
- Parallax Background
- Camera movement based on total mass
# API and ABI breaking changes
No changes
# Expected complexity level and risk
1 - Only adds the demo has no impact elsewise
# Testing
I've tested this backwards and forwards against the server-rust version
and played alongside the Unity client.
- [ ] Run it locally for review
---------
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
This adds `react` as an optional peer dependency. If the TypeScript SDK
is imported into a library that uses React (of the appropriate version)
then they will have access to two new React hooks:
```ts
useSpacetimeDB<DbConnection>();
useTable<DbConnection, MyTable>('my_table');
```
This PR also updates the TypeScript `quickstart-chat` tutorial in the
docs to use the new React hooks. I will split that tutorial into
separate `React` and `TypeScript` tutorials in a future PR.
# API and ABI breaking changes
This is a purely additive change to the SDK
# Expected complexity level and risk
2 because it changes how the SDK is built a bit. Namely it makes the
`spacetimedb` library an external dependency.
# Testing
TODO: I am not through testing yet.
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: Julien Lavocat <JulienLavocat@users.noreply.github.com>
# Description of Changes
Closes#3219
This adds the Unreal SDK, the new Unreal test cases, updates the test
runner to handle Unreal, codegen updates for Unreal, and a QuickStart
Chat.
# API and ABI breaking changes
No breaking changes.
# Expected complexity level and risk
2 - This impacts the subcommand generate.rs to include unrealcpp and
crates/testing to expand for Unreal
# Testing
- [x] Run the new Unreal tests
- [x] Run any previous automation testing - with all the changes to
generate/testing I'm uncertain if there is an impact
- [x] Review the new CLI generate documentation changes
---------
Co-authored-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
This PR moves most of the contents of `@clockworklabs/spacetimedb-sdk`
into the `spacetimedb` module. The `spacetimedb` module now exports
`sdk` and `server` as separate subpaths where `sdk` contains the code
which was previously in `@clockworklabs/spacetimedb-sdk`.
In particular it makes the following moves:
- `/sdks/typescript/packages/sdk` -> `/sdks/typescript`
- most of the contents of `/sdks/typescript/packages/sdk` ->
`crates/bindings-typescript`
- `/sdks/typescript/packages/test-app` ->
`crates/bindings-typescript/test-app`
The following packags was NOT moved:
`/sdks/typescript/examples/quickstart-chat`
## Motivation
In accordance with
https://github.com/clockworklabs/SpacetimeDB/issues/3250, we would like
to consolidate `@clockworklabs/spacetimedb-sdk` into a single
`spacetimedb` package so that users can import the different things they
need from a single package.
### Pros:
- allow users to install a single package with subpaths `spacetimedb`,
`spacetimedb/react`, `spacetimedb/sdk`, `spacetimedb/server`, etc.
- Is much simpler for bundling, etc.
- Is backwards compatible with `@clockworklabs/spacetimedb-sdk` which
now becomes a thin wrapper
- eventually allow us to break up the `spacetimedb` package into other
packages if we want to split them up (e.g. `@spacetimedb/lib`,
`@spacetimedb/sdk`, etc.) and we can solve the build complexity that
introduces when we get to it
- eventually allow us to move `bindings-csharp` out of the crates
directory where it probably doesn't belong anyway
- organizes all TypeScript packages into the packages directory where
you'd normally expect them, with the possible exception of
`/sdks/typescript` if we wanted to leave that separate
### Cons:
- The `sdk` directory is now a bit of a ruse as to where the code
actually lives since it's just a thin wrapper. If it eventually becomes
its own independent package, we'll also have to break up spacetimedb
into `@spacetimedb/lib` and `@spacetimedb/server` so that
`@clockworklabs/spacetimedb-sdk` can depend on `@spacetimedb/lib` while
being a dependency of `spacetimedb`.
Ideally this change would have been made later, however, it became
necessary for the following **heinously disastrous chain of forcing
moves**:
1. Adding `react` support necessitated shipping react as an optional
peer dependency under `@clockworklabs/spacetimedb-sdk/react`
2. This required adding a new build target/export/bundle
3. Previously `@clockworklabs/spacetimedb-sdk` was configured to have
`noExternal` for `spacetimedb` meaning it would collect the library into
the sdk bundle. I attempted to continue this for react support but...
4. Creating a new `react` bundle which also pulled in `spacetimedb`
caused their to be nominal type conflicts between classes in the
duplicate `spacetimedb` bundles.
5. Changing `spacetimedb` to be included as `external` caused compile
errors because `@clockworklabs/spacetimedb-sdk` is configured in
`tsconfig.json` to fail on unused variables and it was now including the
source of `spacetimedb` which is not configured to error on unused
variables and has "unused" private variables which are actually used by
us secretly, but not exposed to the clients.
> SIDE NOTE: The unused variables settings cannot be turned off on a
line by line basis, so it has to be turned off entirely, but in order to
maintain the linting checks we had I used `eslint` to enforce the rule
so that we could disable it line by line. (This caused me to discover
quite a lot of things that were broken that were caught by `eslint`
being applied to the entire project. `eslint` was previously only
applied to the `quickstart-chat` and the `crates/bindings-typescript`
library)
6. Changing the build to be external, now requires `spacetimedb` to also
be published to npm as its own module which
`@clockworklabs/spacetimedb-sdk` now imports, which requires that we add
`tsup` config to `spacetimedb` to publish a built version of the
library.
7. The only way to avoid that is to move the `sdk` and `react` code from
`@clockworklabs/spacetimedb-sdk` into the existing `spacetimedb` package
to avoid the duplicate import problem on step 4 and change
`@clockworklabs/spacetimedb-sdk` back to again use `noExternal` for its
`spacetimedb` dependency.
And here we are. I chose not to move `/crates/bindings-typescript` even
though that's probably not a great place long term. It would be better
to have it in `/packages/spacetimedb` or `/npm-packages/spacetimedb` or
`/ts-packages/spacetimedb` or something, and move all our TypeScript
packages in there. But that is a different matter.
The net result however is that we have a new `spacetimedb` package which
exports the different parts of the API under:
- `spacetimedb`
- `spacetimedb/server`
- `spacetimedb/sdk`
- `spacetimedb/react`
while still not breaking the existing deploy process, nor any
users/developers who are currently using
`@clockworklabs/spacetimedb-sdk`.
I think long term should we ever decide to split `spacetimedb` up into
multiple packages or if we have additional unrelated packages, we should
publish them to the `@spacetimedb` org which I reserved for us here:
https://www.npmjs.com/org/spacetimedb
> NOTE: `spacetimedb` is a package and `@spacetimedb` is an org.
`spacetimedb/sdk` is not a separate package, it's a subpath export of
the `spacetimedb` package, whereas `@spacetimedb/sdk` would be (and
would need to be) it's own separate package. You can certainly have both
`spacetimedb/sdk` and `@spacetimedb/sdk`. We could for example host the
code for the sdk at `@spacetimedb/sdk` and just reexport it from
`spacetimedb` under the `spacetimedb/sdk` subpath.
# API and ABI breaking changes
This should not change or modify the API or ABI in any way. If it does
so accidentally it is a bug, although I carefully went through the
exports.
# Expected complexity level and risk
3 because it changes how the SDK is built a bit and rearranges a lot of
paths.
# Testing
- [x] All of the CI passes
- [x] I also ran quickstart-chat to confirm that it is not broken
- [x] I also ran test-app
# Description of Changes
Fixes#3149
- Fix for MoveAllPlayers calculation to force the circles apart
- Fix for PlayerSplit to properly calculate Scheduled_At for the
recombine timer
# API and ABI breaking changes
None
# Expected complexity level and risk
1 - Trivial
# Testing
I've loaded this up against the Unreal Blackholio comparing the
server-rust vs server-csharp.
- [x] Confirm split + recombine work correctly in the Unity client
*Note*: The demo/Blackholio/client-unity project is currently generated
with the server-rust bindings
# Description of Changes
This adds a new system table to store the jwt payloads of connected
clients. I'm planning to use this system table to expose client claims
to modules in subsequent PRs.
The new table is called `st_connection_credentials`. It is a **private**
system table which stores a mapping from `connection_id` to
`jwt_payload`. Note that a jwt payload is a json representation of the
clients claims, not a fully signed token.
The times when we need to insert and delete these rows closely mirrors
that of the existing `st_client` table, with 1.5 exceptions:
1. We weren't previously inserting to `st_client` until after the
`OnConnect` reducer ran (even though it was in the same transaction). We
want `st_connection_credentials` to be populated before calling the
reducer, so that the reducer can use it get the credentials, so I made a
change to insert to `st_client` and `st_connection_credentials` before
calling the reducer.
2. This difference has not actualized, but when clients start sending
refresh tokens, we will probably need to update the credentials stored
in this table.
This also enforces uniqueness of connection ids. A duplicate connection
id will now make the on-connect reducer fail (since it will violate
uniqueness when trying to insert to `st_connection_credentials`).
# Expected complexity level and risk
2.5
Adding a system table is a bit risky. This is almost rollback safe, with
one annoying case that is worth calling out:
If a database is created with this system table, opening it with an
older version of spacetimedb will only work if there is a snapshot of
the database. If we try to load a table without a snapshot, replaying
will fail on the first row for that table. This is because we don't
write the table schema information to the commit log when creating a
database. In practice, this is unlikely to be an issue, because new
databases asynchronously trigger a snapshot immediately after creation.
Migrating existing databases will be fine. On startup this will detect
that there is a missing system table, and add it in a way that writes it
to the commit log. Since it is in the commit log, we can open the
database with an older version and still understand the data for that
table.
# Testing
There are unit tests that cover opening a database created with an older
version (which doesn't have this table).
I manually tested opening a migrated database with an older version of
spacetimedb.
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
In order to facilitate commitlog and snapshot archival, this patch adds
channel params to `snapshot_watching_commitlog_compressor`
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
1
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
The method is no longer used.
Note that this patch must merge **after** the private companion.
# API and ABI breaking changes
Internal API change.
# Expected complexity level and risk
1
# Testing
In private.
# Description of Changes
This just constrains the types of rows from `any` to `Record<string,
any>`.
# API and ABI breaking changes
Not breaking, purely internal.
# Expected complexity level and risk
1
# Testing
- [x] Just the automated tests
# Description of Changes
PR introduces support for column-level default values via a new
`#[default(...)]` attribute.
It also validates, `default` macro is not used along with `primary_key`,
`unique` or `auto_inc`.
# API and ABI breaking changes
NA
# Expected complexity level and risk
2
# Testing
Start using macro in `module-test`.
---------
Co-authored-by: James Gilles <jameshgilles@gmail.com>
# Description of Changes
Closes
[#2686](https://github.com/clockworklabs/SpacetimeDB/issues/2686).
Add support for listening using the [PG wire
protocol](https://www.postgresql.org/docs/current/protocol.html) so `pg`
clients could be used against the database.
# API and ABI breaking changes
The output of `duration` is changed to `rfc3339`, instead of the way is
made with `sats` because is what is done in `pg`, see note below.
# Expected complexity level and risk
2
~~There is open questions that are in the [ticket
#2686](https://github.com/clockworklabs/SpacetimeDB/issues/2686). Also
the crate used here require `RustTls`, so it could be good idea to
decide if~~:
* ~~Rewrite a big chunk of code to use `OpenSSL`~~
* ~~Move to `RustTls`
https://github.com/clockworklabs/SpacetimeDB/pull/1700~~
* ~~Pay for the extra compilation cost~~.
I open another port(`5433`) to listen for `pg` connections using `ssl`.
Need to be decided if this is the way or instead try to multi-plex the
current port for both protocols.
# Testing
Only manual testing so far. Solving the above questions allow me to
implement some unit tests. Also, not yet integrated into cloud for the
same reasons.
- [x] Adding some test for the binary encoding of special and primitive
types
- [x] Smoke test using `psql` that connect to the db instance and run
some queries
- [x] Manually inspect using a UI database explorer how infer the types,
some of this tools generate special widgets when displaying `json,
duration, etc`
---------
Co-authored-by: Noa <coolreader18@gmail.com>
# Description of Changes
- Adds endpoint for for pretty printing migration plan.
- It also changes current `publish` endpoint to optionally provide
`MigrationToken` and `MigrationPolicy` to allow migration with breaking
clients.
# API and ABI breaking changes
Backward compatible change to existing API and new Api
# Expected complexity level and risk
2
# Testing
- Existing smoketest should cover changes for `publish` endpoint.
- For pretty print endpoint, smoketests can be written only after cli
changes.
---------
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
Co-authored-by: James Gilles <jameshgilles@gmail.com>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
# Description of Changes
Implements [subscribing to durable
commits](https://github.com/clockworklabs/SpacetimeDBPrivate/issues/1594).
The setting works on a per-connection level, and essentially just delays
sending transaction updates until the transaction is reported as durable
by the database.
For connectionless SQL operations, the setting works per-request. No SQL
syntax is provided by this patch to toggle the configuration.
After some deliberation, I opted to obtain the offset when a transaction
commits (as opposed to when it starts). This creates some mild
inconvenience, because we prevent the transaction from committing until
the corresponding subscription updates are enqueued.
The strategy is, however, more correct should we ever support weaker
isolation levels, and it is easier to document.
Follow-ups include:
- Provide SQL syntax (`SET synchronous_commit = ON` or something)
- C# and TypeScript SDKs
- Reference docs?
# API and ABI breaking changes
Not breaking, but adds a parameter to the subscribe and sql endpoints.
# Expected complexity level and risk
4
To the author's understanding, ordering of outbound messages is not
changed by this patch, even if there are messages that don't have a
transaction offset (such as error messages). I.e. while waiting for the
transaction offset of a message to become durable, no message enqueued
after that message will be delivered. This may not be desirable in some
cases.
The patch may contain concurrency bugs, e.g. awaiting futures that may
never resolve.
# Testing
- [x] Implemented a new test in the `module_subscription_actor` module
- [x] Added unit tests for the core logic in `ClientConnectionReceiver`
It would be desirable to also have integration-level tests, but I'm
currently unsure how to write those without being able to control if and
when the database reports an offset as durable.
---------
Signed-off-by: Kim Altintop <kim@eagain.io>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
# Description of Changes
This is a purely additive change to introduce, `TypeBuilder`s,
`ColumnBuilder`s, and a `t` factory export that has methods for creating
`TypeBuilder`s.
There are derived types from `TypeBuilder`s for each variant of
`AlgebraicType`. `TypeBuilder`s can be converted into a `ColumnBuilder`
which supports creating additional metadata specifying whether a column
is a primary key or should be unique. `ColumnBuilder`s are not allowed
within `TypeBuilder`s, but the opposite is allowed (for composite
types).
`ColumnBuilder`s can only be used at the top level and contain a
`TypeBuilder`.
`TypeBuilder` stores three pieces of information:
1. A phantom typescript type that the SATS type should be interpreted as
in TS
2. A phantom typescript type which is the type of the SATS type
3. A runtime AlgebraicType which stores the information to be reported
to host.
`ColumnBuilder` stores three pieces of information:
1. A `TypeBuilder`
2. A phantom typescript type with the column metadata
3. A runtime `ColumnMetadata` which stores the information to be
reported to the host.
Additionally, it is only possible to add metadata to `ColumnBuilder` or
`TypeBuilder` types which are compatible with that metadata. e.g. It is
a compile-time error to add `isAutoIncrementing` to a `String`
type/column.
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] I've created a set of type tests in `type_builders.test-d.ts`
- [x] I also created a set of runtime tests in `index.test.ts`
- [x] I have modified CI so that these tests run on every PR
# Description of Changes
This simplifies system table bootstrapping logic by removing
auto-incremented ids for indexes, constraints, and sequences in system
tables. By defining them all up front, we avoid the need for
`reset_system_table_schemas`, and make it easier to add new things to
system tables.
This change should not change the initial system table rows written
during bootstrapping, and the `load_1_2_*` tests in `relational_db.rs`
should verify that by loading data written by a previous version with
and without a snapshot.
I also added some sequence-related changes. The sequence schema previous
included the `allocated` field, which tracked where to start generating
new values after a restart. This made some checks awkward, since the
'schema' was changing as rows were inserted. I moved this outside of the
`schema`, so those should no longer change unless something actually
changes the shape of the tables.
As part of that, I also changed where we begin generating on startup to
be `allocated` instead of `allocated + 1`. The code that generates
values always stopped early enough to allow the next restart to use
`allocated`, so we were just being overly defensive. We also had some
issues if sequences wrapped over, or if we had sequences with negative
increments. Those were supported by the schema, but didn't really work
with the generation/allocation code. That should have better testing
now.
# Expected complexity level and risk
2. The bootstrapped system tables should match, since we have tests with
older data. The changes to sequences include some more invariant
checking, which has some risk of causing problems if previous versions
wrote values that I didn't consider.
# Testing
This has existing tests covering bootstrapping and restoring from a
snapshot.
# Description of Changes
Fixed how the multi-line regexps worked on Windows (with `\r\n` line
endings).
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
- [x] `cargo run -- 1.4.1` works properly on Linux still
- [x] Applying the same diff and running on Windows, now updates the
license files properly.
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Shubham Mishra