# Description of Changes
Despecializes direct indices for `u64` keys that have too large values
and convert them into a B-Tree index instead.
Otherwise, we are exposed to OOM aborts.
Also hardens typed index tests to cover more combinations.
This is how the bug was caught.
Some index code is also simplified.
# API and ABI breaking changes
None
# Expected complexity level and risk
2?
# Testing
Proptests were generalized to catch the bug and to then verify that it
was fixed.
# Description of Changes
Provides common traits `Index` and `RangedIndex` that all (current)
variants in `TypedIndex` adhere to.
This is then used to simplify `TypedIndex` by merging common code,
with more simplifications to come later.
The responsibility of tracking statistics is also moved into each index
type, as some can exploit their properties to provide some statistics
for free, rather than storing statistics in fields.
# API and ABI breaking changes
None
# Expected complexity level and risk
2?
# Testing
This is mostly code motion, so it is covered by existing tests.
Further test improvements will come in a follow up PR.
# Description of Changes
Closes#3673
*NOTE*: C++ part will be in another PR
# Expected complexity level and risk
2
Adding a new type touch everywhere
# Testing
- [x] Adding smoke and unit test
---------
Signed-off-by: Ryan <r.ekhoff@clockworklabs.io>
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
# Description of Changes
Closes
[#3290](https://github.com/clockworklabs/SpacetimeDB/issues/3290).
Adds a new "special" type to SATS, `UUID`, which is represented as the
product `{ __uuid__: u128 }`. Adds versions of this type to all of our
various languages' module bindings libraries and client SDKs, and
updates codegen to recognize it and output references to those named
library types. Adds methods for creating new UUIDs according to the V4
(all random) and V7 (timestamp, monotonic counter and random)
specifications.
# API and ABI breaking changes
We add a new type
# Expected complexity level and risk
2
it impacts all over the code
# Testing
- [x] Extends the Rust and Unreal SDK tests, and the associated
`module-test` modules in Rust, C# and TypeScript, with uses of UUIDs.
- [x] Extends the C# SDK regression tests with uses of UUIDs.
- [x] Extends the TypeScript test suite with tests with uses of UUIDs.
---------
Signed-off-by: Mario Montoya <mamcx@elmalabarista.com>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
# Description of Changes
When "fixing" a clippy lint, I accidentally flipped a boolean condition
in a `filter` call, breaking replay of column-type-altering
automigrations.
I'm also fixing an error message containing a typo I apparently wrote
many months ago, where I was printing the "old" layout in both places
rather than also printing the "new" layout. This is used only as a
diagnostic message, and is never programmatically inspected.
# API and ABI breaking changes
N/a
# Expected complexity level and risk
1
# Testing
- [x] Replayed the broken module manually locally.
# Description of Changes
Fixes https://github.com/clockworklabs/SpacetimeDB/issues/2824.
Defines a global pool `BsatnRowListBuilderPool` which reclaims the
buffers of a `ServerMessage<BsatnFormat>` and which is then used when
building new `ServerMessage<BsatnFormat>`s.
Notes:
1. The new pool `BsatnRowListBuilderPool` reports the same kind of
metrics to prometheus as `PagePool` does.
2. `BsatnRowListBuilder` now works in terms of `BytesMut`.
3. The trait method `fn to_bsatn_extend` is redefined to be capable of
dealing with `BytesMut` as well as `Vec<u8>`.
4. A trait `ConsumeEachBuffer` is defined from
`ServerMessage<BsatnFormat>` and down to extract buffers.
`<ServerMessage<_> as ConsumeEachBuffer>::consume_each_buffer(...)` is
then called in `messages::serialize(...)` just after bsatn-encoding the
entire message and before any compression is done. This is the place
where the pool reclaims buffers.
# Benchmarks
Benchmark numbers vs. master using `cargo bench --bench subscription --
--baseline subs` on i7-7700K, 64GB RAM:
```
footprint-scan time: [21.607 ms 21.873 ms 22.187 ms]
change: [-62.090% -61.438% -60.787%] (p = 0.00 < 0.05)
Performance has improved.
full-scan time: [22.185 ms 22.245 ms 22.324 ms]
change: [-36.884% -36.497% -36.166%] (p = 0.00 < 0.05)
Performance has improved.
```
The improvements in `footprint-scan` are mostly thanks to
https://github.com/clockworklabs/SpacetimeDB/pull/2918, but 7 ms of the
improvements here are thanks to the pool. The improvements to
`full-scan` should be only thanks to the pool.
# API and ABI breaking changes
None
# Expected complexity level and risk
2?
# Testing
- Tests for `Pool<T>` also apply to `BsatnRowListBuilderPool`.
# Description of Changes
Resolves algebraic type refs recursively in order to check the product
type of a query builder view.
This should fix the issue reported
[here](https://discord.com/channels/1037340874172014652/1448796556366057513).
However I've so far been unsuccessful in trying repro it.
Also adds further commentary to `Typespace::resolve` to make it clear
that it is not recursive.
# API and ABI breaking changes
None
# Expected complexity level and risk
0
# Testing
TODO. So far I haven't been able to repro with a smoketest
# Description of Changes
Fixes https://github.com/clockworklabs/SpacetimeDB/issues/3617.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
A proptest `empty_range_scans_dont_panic` is added.
# Description of Changes
fixes#3861.
While running prechecks for automigration, the `range` passed to
`iter_by_col_range_mut` was of type `AlgebraicValue::I128`, even though
it should have matched the column’s type.
Fix is straightforward by introducing `AlgebraicValue::from_i128`
helper.
# API and ABI breaking changes
NA
# Expected complexity level and risk
1
# Testing
- Modified smoketest to repro reported bug.
---------
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
# Description of Changes
Follow up to #3684. Moves `Error` and `Timeout` out of lib, so that we
don't have to implement `SpacetimeType` for them, and then removes the
http dependency altogether, so that `lib` can be leaner. I also got rid
of the separate `HttpValue` type, since it only really exists to mirror
the `http` crate and typescript won't make use of it.
# Expected complexity level and risk
1
# Testing
n/a - just code movement.
# Description of Changes
Updates views atomically on commit, but before downgrading to a
read-only transaction for subscription evaluation.
What this patch does:
1. Renames `ViewId` to `ViewFnPtr`
2. Renames `ViewDatabaseId` to `ViewId`
3. Removes the `module_rx` module watcher from the subscription manager
4. Refactors read sets to only track table scans (index key tracking
will be added later)
5. Drops read sets and removes rows from `st_view_sub` when dropping a
view in an auto-migrate
6. Re-evaluates and updates views (`call_views_with_tx`) from
`call_reducer_with_tx` for any view whose read set overlaps with the
reducer's write set
7. Does the same for sql dml
# API and ABI breaking changes
None
# Expected complexity level and risk
3
It's a bit of a messy diff.
# Testing
- [x] Integrate with
https://github.com/clockworklabs/SpacetimeDB/pull/3616
---------
Signed-off-by: joshua-spacetime <josh@clockworklabs.io>
Co-authored-by: Shubham Mishra <shivam828787@gmail.com>
# Description of Changes
Host implementation to invoke `call_view` method.
I also covers:
1. API `MutTxId::is_materialized`to check if existing view exisits and
updated.
2. Update in readsets logic to remove stale views.
3. sql caller implmentation.
# API and ABI breaking changes
NA
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
3
# Description of Changes
Refactored `st_view_client` and renamed it `st_view_sub` which tracks
the number of clients subscribed to a view. On disconnect, we decrement
the `num_subscribers` column in the appropriate rows. An async task will
be in charge of cleaning up views (and their read sets) whose subscriber
count has gone to zero (not in this patch).
On module init, we clear the entirety of each view table.
# API and ABI breaking changes
None. Technically this updates the schema of a system table, but the
system table was added and modified between releases.
# Expected complexity level and risk
~2
Need to make sure we cover all cases so that we don't leave dangling
data. Making these tables ephemeral in the future should simplify this.
# Testing
Will add tests once we can subscribe to views
# Description of Changes
Not many changes were required for the query compiler to be able to
resolve views. This is because the query engine can always assume a view
is materialized and therefore has a backing table. So from the
perspective of the query engine, a view is just another table with one
small caveat: The physical table in the datastore has two internal
metadata columns - `sender` and `arg_id`. These columns are not user
facing and so should be hidden from name resolution/type checking.
# API and ABI breaking changes
None
# Expected complexity level and risk
1.5
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] SQL type checking tests
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
Defines auto-migration rules for views.
> What are they?
Answer: You can always auto-migrate a view. It's just a matter of
whether you have to disconnect clients. Removal of a view or changing
the return type in a breaking way requires that we disconnect clients.
TODO: Even if a view is updated in a completely compatible way, we still
have to wipe or dirty the read set in order to force a re-computation
because the behavior could have changed.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
2.5
Not just a mechanical change.
Most of the time you can think of views as though they were tables.
However for auto-migration there is a key distinction. Views don't have
observably persistent state. While we can persist state internally, we
can always derive a view's result set from the database state. Hence
auto-migration rules are not as strict.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] Auto-migration planner tests
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
Adds type definitions for `RawViewDefV9` and `ViewDef`. Also validates
these module defs and populates the system tables:
- `st_table`
- `st_column`
- `st_view`
- `st_view_param`
- `st_view_column`
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
2
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
Integration tests to follow when views are added to
`__describe_module__`
---------
Signed-off-by: joshua-spacetime <josh@clockworklabs.io>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
# Description of Changes
<!-- Please describe your change, mention any related tickets, and so on
here. -->
This patch defines the system table schemas for views and allocates IDs
for them. It **does not** populate these tables.
# API and ABI breaking changes
<!-- If this is an API or ABI breaking change, please apply the
corresponding GitHub label. -->
None
# Expected complexity level and risk
<!--
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
This complexity rating applies not only to the complexity apparent in
the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning
ways. -->
1
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
Tests will be added in the patch that populates these tables
# Description of Changes
Host-side changes extracted from #3327
I added AUTO_INC_OVERFLOW even though we don't currently ever return it,
in order to future-proof so it's already there when we start emitting
it.
Prepublish was failing because it was expecting a wasm module
unconditionally, so now it takes ?host_type.
I tweaked JS deser to accept null/undefined when the unit type or an
option type is expected.
I switched to bsatn, because the native sats->js translator wasn't
matching what js was expecting.
I renamed the sys module: my thinking is that `spacetime:` as a scheme
will help disambiguate it, and maybe it could also be used for IMC in
the future or something? And I believe we had discussed wanting this to
be versioned, similar to wasm imports.
Trying to get a borrowed str from deserialize_js doesn't work, because
v8 strings don't store utf8.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [x] All this was done in the course of getting an actual typescript
module to successfully publish.
# Description of Changes
Necessary for pulling in rolldown.
# API and ABI breaking changes
None
# Expected complexity level and risk
1, with the caveat that this updates the Rust version and therefore
touches all the code.
# Testing
- [ ] Just the automated testing
# Description of Changes
Implements `Serialize` and `Deserialize` for tuples.
Extracted from https://github.com/clockworklabs/SpacetimeDB/pull/3276.
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
A test `roundtrip_tuples_in_different_data_formats` is added.
# Description of Changes
- `add_columns_to_table` api to handle `AutoMigrateStep::AddColumns` --
Look at doc comment for more Info.
depends on: https://github.com/clockworklabs/SpacetimeDB/pull/3261
TODO: handle `AutoMigrateStep::DisconnectAllUsers`.
# API and ABI breaking changes
N/A.
# Expected complexity level and risk
2? Changes are not in hotpath but a bug in migration can corrupt tables.
# Testing
a test to exercise the API.
---------
Signed-off-by: Mazdak Farrokhzad <twingoow@gmail.com>
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
# Description of Changes
Closes
[#2686](https://github.com/clockworklabs/SpacetimeDB/issues/2686).
Add support for listening using the [PG wire
protocol](https://www.postgresql.org/docs/current/protocol.html) so `pg`
clients could be used against the database.
# API and ABI breaking changes
The output of `duration` is changed to `rfc3339`, instead of the way is
made with `sats` because is what is done in `pg`, see note below.
# Expected complexity level and risk
2
~~There is open questions that are in the [ticket
#2686](https://github.com/clockworklabs/SpacetimeDB/issues/2686). Also
the crate used here require `RustTls`, so it could be good idea to
decide if~~:
* ~~Rewrite a big chunk of code to use `OpenSSL`~~
* ~~Move to `RustTls`
https://github.com/clockworklabs/SpacetimeDB/pull/1700~~
* ~~Pay for the extra compilation cost~~.
I open another port(`5433`) to listen for `pg` connections using `ssl`.
Need to be decided if this is the way or instead try to multi-plex the
current port for both protocols.
# Testing
Only manual testing so far. Solving the above questions allow me to
implement some unit tests. Also, not yet integrated into cloud for the
same reasons.
- [x] Adding some test for the binary encoding of special and primitive
types
- [x] Smoke test using `psql` that connect to the db instance and run
some queries
- [x] Manually inspect using a UI database explorer how infer the types,
some of this tools generate special widgets when displaying `json,
duration, etc`
---------
Co-authored-by: Noa <coolreader18@gmail.com>
# Description of Changes
Title. When validating column type changes in `Table::change_columns_to`
and comparing layouts in `RowTypeLayout::is_compatible_with`, return
structured error objects which describe the mismatch.
I made this change while debugging the issue that led to #3203 , where
`change_columns_to` returned an error during replay of an automigration
which had succeeded the first time. The original error contained enough
information to debug, but it was presented in a noisy and unhelpful way,
so I wrote this patch to get better diagnostics.
Per @Centril 's comments, these errors are for internal assertions, not
user-facing error reporting, so we value fail-fast rather than error
hygiene and choose not to use the `ErrorStream` combinator.
Also note that I sprinkled `Box`es around some large-ish error types to
quiet
https://rust-lang.github.io/rust-clippy/master/index.html#result_large_err
.
# API and ABI breaking changes
N/a
# Expected complexity level and risk
1.
# Testing
Manual testing with BitCraft. As this is purely altering the error
reporting for non-user-facing assertions, I don't believe any further
testing is necessary.
# Description of Changes
This adds auto migration support to the `schema` crate.
Additional changes are needed to fully implement this feature: see
https://github.com/clockworklabs/SpacetimeDB/issues/2912
# API and ABI breaking changes
None
# Expected complexity level and risk
1, at this level the change is fairly straightforward. The integration
will be more difficult.
# Testing
Added `schema` unit tests for success/failure cases.
---------
Co-authored-by: Shubham Mishra <shivam828787@gmail.com>
# Description of Changes
We recently merged several repos together. This PR clarifies the license
terms for several subdirectories, as well as the relationship between
the licenses.
The licenses in our subdirectories have become symbolic links to
licenses in our toplevel `licenses` directory. For any particular
subdirectory's license file in the diff, you can click `... -> View
file` and then click on the text that says "Symbolic Link" on that page.
This will take you to the license file that it links to.
I have also updated the `tools/upgrade-version` script to update the
change date in the new `licenses/BSL.txt` file.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
None. Only changes to license files.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
As requested by @gefjon
Most of this is:
- `'a` -> `'this`
- `'s` -> `'scope`
# API and ABI breaking changes
None
# Expected complexity level and risk
0
# Testing
No semantic changes.
Mazdak Farrokhzad