# Description of Changes
Fixes https://github.com/clockworklabs/SpacetimeDB/issues/2824.
Defines a global pool `BsatnRowListBuilderPool` which reclaims the
buffers of a `ServerMessage<BsatnFormat>` and which is then used when
building new `ServerMessage<BsatnFormat>`s.
Notes:
1. The new pool `BsatnRowListBuilderPool` reports the same kind of
metrics to prometheus as `PagePool` does.
2. `BsatnRowListBuilder` now works in terms of `BytesMut`.
3. The trait method `fn to_bsatn_extend` is redefined to be capable of
dealing with `BytesMut` as well as `Vec<u8>`.
4. A trait `ConsumeEachBuffer` is defined from
`ServerMessage<BsatnFormat>` and down to extract buffers.
`<ServerMessage<_> as ConsumeEachBuffer>::consume_each_buffer(...)` is
then called in `messages::serialize(...)` just after bsatn-encoding the
entire message and before any compression is done. This is the place
where the pool reclaims buffers.
# Benchmarks
Benchmark numbers vs. master using `cargo bench --bench subscription --
--baseline subs` on i7-7700K, 64GB RAM:
```
footprint-scan time: [21.607 ms 21.873 ms 22.187 ms]
change: [-62.090% -61.438% -60.787%] (p = 0.00 < 0.05)
Performance has improved.
full-scan time: [22.185 ms 22.245 ms 22.324 ms]
change: [-36.884% -36.497% -36.166%] (p = 0.00 < 0.05)
Performance has improved.
```
The improvements in `footprint-scan` are mostly thanks to
https://github.com/clockworklabs/SpacetimeDB/pull/2918, but 7 ms of the
improvements here are thanks to the pool. The improvements to
`full-scan` should be only thanks to the pool.
# API and ABI breaking changes
None
# Expected complexity level and risk
2?
# Testing
- Tests for `Pool<T>` also apply to `BsatnRowListBuilderPool`.
# Description of Changes
This PR modifies the `--delete-data` flag on `spacetime publish` and
adds the `--delete-data` flag on `spacetime dev`.
In particular instead of `--delete-data` being a boolean, it is now a an
enum:
- `always` -> corresponds to the old value of `true`
- `never` -> corresponds to the old value of `false`
- `on-conflict` -> clears the database, but only if publishing would
have required a manual migration
This flag does NOT change any behavior about prompting users to confirm
if they want to delete the data. Users will still be prompted to confirm
UNLESS they pass the separate `--yes` flag.
`spacetime dev` gets the same `--delete-data` flag. The default value of
`never` is equivalent to the existing behavior. `spacetime dev`
continues to publish with `--yes` just as before. This behavior is
unchanged.
# API and ABI breaking changes
Adds the flags specified above. This is NOT a breaking change to the
CLI. Passing `--delete-data` is the equivalent of
`--delete-data=always`.
This IS technically a breaking change to the `pre_publish` route. As far
as I'm aware this is *only* used by our CLI however.
> IMPORTANT SIDE NOTE: I would argue that `--break-clients` should
really be renamed to `--yes-break-clients` because it actually behaves
like the `--yes` force flag, but only for a subset of the user prompts.
I have not made this change because it would be a breaking change, but
if the reviewers agree, I will make this change.
# Expected complexity level and risk
2, Very small change, but if we get it wrong users could accidentally
lose data. I would ask reviewers to think about ways that users might
accidentally pass `--delete-data --yes`.
# Testing
- [ ] I have not yet tested manually.
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
This adds authorization to the relevant API endpoints, updates the CLI
commands and adds smoketests for the teams feature.
**Note**: Authorizing SQL (incl. subscriptions) is a bit more involved,
and submitted as a separate PR in the series.
Depends-on: https://github.com/clockworklabs/SpacetimeDB/pull/3519
# Description of Changes
Updates views atomically on commit, but before downgrading to a
read-only transaction for subscription evaluation.
What this patch does:
1. Renames `ViewId` to `ViewFnPtr`
2. Renames `ViewDatabaseId` to `ViewId`
3. Removes the `module_rx` module watcher from the subscription manager
4. Refactors read sets to only track table scans (index key tracking
will be added later)
5. Drops read sets and removes rows from `st_view_sub` when dropping a
view in an auto-migrate
6. Re-evaluates and updates views (`call_views_with_tx`) from
`call_reducer_with_tx` for any view whose read set overlaps with the
reducer's write set
7. Does the same for sql dml
# API and ABI breaking changes
None
# Expected complexity level and risk
3
It's a bit of a messy diff.
# Testing
- [x] Integrate with
https://github.com/clockworklabs/SpacetimeDB/pull/3616
---------
Signed-off-by: joshua-spacetime <josh@clockworklabs.io>
Co-authored-by: Shubham Mishra <shivam828787@gmail.com>
# Description of Changes
Host implementation to invoke `call_view` method.
I also covers:
1. API `MutTxId::is_materialized`to check if existing view exisits and
updated.
2. Update in readsets logic to remove stale views.
3. sql caller implmentation.
# API and ABI breaking changes
NA
How complicated do you think these changes are? Grade on a scale from 1
to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex
change.
3
# Description of Changes
This commit builds support for executing procedures in WASM modules.
This includes an HTTP endpoint,
`/v1/database/:name_or_address/procedure/:name POST`, as well as an
extension to the WS protocol. These new APIs are not wired up to the CLI
or SDKs, but I have manually tested the HTTP endpoint via `curl`. The
new WS extensions are completely untested.
Several TODOs are scattered throughout the new code, most notably for
sensibly tracking procedure execution time in the metrics.
I also expect that we will want to remove the `procedure_sleep_until`
syscall and the `ProcedureContext::sleep_until` method prior to release.
# API and ABI breaking changes
Adds new APIs and ABIs.
# Expected complexity level and risk
3? 4? Unlikely to break existing stuff, 'cause it's mostly additive, but
adds plenty of potentially-fragile new stuff. Notably is the first time
we're doing anything actually `async`hronous on a database core Tokio
worker, and we don't yet have strong evidence of how that will affect
reducer execution.
# Testing
- [x] Manually published `modules/module-test` and executed procedures
with the following `curl` invocations:
- `curl -X POST -H "Content-Type:application/json" -d '[]'
http://localhost:3000/v1/database/module-test/procedure/sleep_one_second`
- `curl -X POST -H "Content-Type:application/json" -d '[1223]'
http://localhost:3000/v1/database/module-test/procedure/return_value`
- [ ] Need to write automated tests.
---------
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
# Description of Changes
PR contains:
* CLI changes for the `pre_publish` endpoint when publishing a module.
* The regular `--yes` flag will not bypasses the *break clients* warning
prompt — an extra confirmation is now required. For CI, a hidden flag
`--break-clients` is added.
* Added smoketest.
* Some trivial naming changes in `client-api-*` crates for consistency
reasons.
* `pre_publish` route to accept similar Body size limit as `publish`
route.
# API and ABI breaking changes
an additive API change, does not break anything.
# Expected complexity level and risk
2
# Testing
- Existing smoketests passing for backward compatibility.
- New smoketest for add columns
---------
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
# Description of Changes
Closes
[#2686](https://github.com/clockworklabs/SpacetimeDB/issues/2686).
Add support for listening using the [PG wire
protocol](https://www.postgresql.org/docs/current/protocol.html) so `pg`
clients could be used against the database.
# API and ABI breaking changes
The output of `duration` is changed to `rfc3339`, instead of the way is
made with `sats` because is what is done in `pg`, see note below.
# Expected complexity level and risk
2
~~There is open questions that are in the [ticket
#2686](https://github.com/clockworklabs/SpacetimeDB/issues/2686). Also
the crate used here require `RustTls`, so it could be good idea to
decide if~~:
* ~~Rewrite a big chunk of code to use `OpenSSL`~~
* ~~Move to `RustTls`
https://github.com/clockworklabs/SpacetimeDB/pull/1700~~
* ~~Pay for the extra compilation cost~~.
I open another port(`5433`) to listen for `pg` connections using `ssl`.
Need to be decided if this is the way or instead try to multi-plex the
current port for both protocols.
# Testing
Only manual testing so far. Solving the above questions allow me to
implement some unit tests. Also, not yet integrated into cloud for the
same reasons.
- [x] Adding some test for the binary encoding of special and primitive
types
- [x] Smoke test using `psql` that connect to the db instance and run
some queries
- [x] Manually inspect using a UI database explorer how infer the types,
some of this tools generate special widgets when displaying `json,
duration, etc`
---------
Co-authored-by: Noa <coolreader18@gmail.com>
# Description of Changes
- Adds endpoint for for pretty printing migration plan.
- It also changes current `publish` endpoint to optionally provide
`MigrationToken` and `MigrationPolicy` to allow migration with breaking
clients.
# API and ABI breaking changes
Backward compatible change to existing API and new Api
# Expected complexity level and risk
2
# Testing
- Existing smoketest should cover changes for `publish` endpoint.
- For pretty print endpoint, smoketests can be written only after cli
changes.
---------
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
Co-authored-by: James Gilles <jameshgilles@gmail.com>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
# Description of Changes
Please note, much of the code changed in this PR is generated code.
This change updates the TypeScript SDK to use a new `spacetimedb`
TypeScript library which lives under the `/crates/bindings-typescript`
folder alongside `/crates/bindings-csharp`. Just like with the C#
bindings library, the types for `AlgebraicType` and `RawModuleDef` are
now code generated with a script in `/crates/codegen/examples`.
Pulling this out into a library allows us to use the same types and
serialization code on both the server and the client for TypeScript
modules.
In the process of making this change I also found and fixed several
issues with the TypeScript code generation. Namely an issue with
recursive types and an issue with the `never` type. I also removed any
use of `namespace`s since those are a TypeScript only feature, and we
want to have JavaScript + types so that we can use the generated code
with ESBuild without TSC.
I have also improved the npm/pnpm scripts to be able to generate
TypeScript code for us automatically by running `pnpm generate` for any
place that we have to generate typescript code. Namely:
- Quickstart module bindings
- AlgebraicType/ModuleDef for TypeScript module library
- Client API messages for the TypeScript API
- TestApp module bindings
# API and ABI breaking changes
IMPORTANT! This is an API breaking change for clients, as such it should
be a major version change. However, I am going to see if I can shim in
the old API as best as possible to make it compatible.
Notably, we were previously exporting APIs that end users do not need,
and I don't think it would ever occur to them to use, namely the whole
`AlgebraicValue` API and also the `AlgebraicType` API. In principle, no
one should have a need for these, although it was technically possible
for them to use it.
Indeed, we could potentially even just remove AlgebraicType completely
from the API by directly code generating the serialization code.
Listed below are all of the **BREAKING** changes to the API and their
effect:
- `AlgebraicType` is now a structural union literal type instead of a
class (nominal type), this is a consequence of generating the type with
our code gen. Users did not have a reason to use `AlgebraicType`
directly.
- The `AlgebraicValue` type has been removed entirely. This was
previously a class that was exported from the SDK, but very unlikely to
be used by users.
- The `ProductValue` type has been removed.
- The `ReducerArgsAdapter` and `ValueAdapter` types, which were used
with AlgebraicValues have been removed.
- Generated code has changed incompatibly so users will have to
regenerate code when upgrading to this version. Technically a breaking
change, but pretty easy to fix.
Listed below are the non-breaking API changes:
- I am now exporting generated product types as the default export in
addition to how I was exporting them before
- For generated sum type `T`, I am now exporting a `TVariants` namespace
which has the types of all the variants for `T`. This was previously
exposed on the namespace `T`, but was inaccessible in modules other than
the one it was defined in because I also export a type `T` in addition
to namespace `T` and in the type position `T` was being interpreted as a
type rather than a namespace. It's unclear why TypeScript resolved it as
the `T` namespace within the module in which is was defined previously.
Anyway, since the old types were apparently unobservable to users, I've
replaced them with the types in `TVariants`. (Open to other names for
this namespace).
- I fixed a bug where never types (sum types with no variants) were not
correctly generated.
# Expected complexity level and risk
3. The most complex thing about the PR are the potential impacts to the
API. I am reasonably certain, but not 100% certain that I have accounted
for everything above.
# Testing
- [x] I ran `pnpm test` which runs all the tests in the repo including
an integration test which tests the connection to SpacetimeDB. I also
fixed broken tests.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
We recently merged several repos together. This PR clarifies the license
terms for several subdirectories, as well as the relationship between
the licenses.
The licenses in our subdirectories have become symbolic links to
licenses in our toplevel `licenses` directory. For any particular
subdirectory's license file in the diff, you can click `... -> View
file` and then click on the text that says "Symbolic Link" on that page.
This will take you to the license file that it links to.
I have also updated the `tools/upgrade-version` script to update the
change date in the new `licenses/BSL.txt` file.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
None. Only changes to license files.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
# Description of Changes
Avoid building a temporary `Vec` in `execute_plan` by exposing a
list-building interface instead.
- The old `fn encode_list` is rewritten in terms of this list-building
interface.
- The `BsatnRowList` and `BsatnRowListBuilder` types are split into two
entirely separate types. The latter now tries to recognize the case
where there isn't a known static layout, but where the BSATN lengths
happen to be the same for all rows anyways. In those cases, the
allocation of `RowSizeHint::RowOffsets` is avoided in favor of just
storing the found length in bytes. This is in particular useful for
small table updates as statistically, the fewer rows, the more chance of
the lengths being all equal. In the case of a single row, the chance is
notably 100%. It is also good for the case of when we don't have
`RelValue::Row` or `Row::Ptr` but where the underlying table that
actually has a static layout.
In the future, we might want to avoid these lists in incremental as
well.
# Benchmarks
Benchmark numbers vs. master using `cargo bench --bench subscription --
--baseline subs` on i7-7700K, 64GB RAM:
```
footprint-scan time: [28.731 ms 28.924 ms 29.171 ms]
change: [-49.728% -49.006% -48.388%] (p = 0.00 < 0.05)
Performance has improved.
```
Performance goes from roughly 56.721 ms to 28.795 ms.
# API and ABI breaking changes
None
# Expected complexity level and risk
2, fairly local change to just subscriptions.
# Testing
Covered by existing tests.
* Remove `row_pk` from client API; hash rows not row_ids on client
This commit removes the `row_pk` field from `message TableRowOperation` in the client API.
This is the beginning of addressing issue #831.
This has several implications for clients:
- Because we no longer have a stable content-addressed ID for each row,
the client is responsible for hashing rows itself.
- This means that generated types must be `Hash + Eq`.
- This means that generated types must use `sats::F32` or `sats::F64`
rather than `f32` or `f64`,
as the former are `Eq + Hash` while the latter are not.
(Aside: it's stupid that Rust floats are not `Eq + Ord + Hash`,
since IEEE-754 defines a total equality and total ordering on floats.)
Exposing `sats::F32` to users is ugly, but I can't think of a better design.
This commit also makes some minor formatting changes to Rust codegen,
since it was emitting code that rustc warned about.
Still outstanding:
- [ ] Remove row_pk from the JSON API.
- [ ] Avoid computing row_pk in the subscription engine.
- [ ] Update other SDKs.
- [ ] C#
- [ ] TypeScript
- [ ] Python
* Combine similar patterns
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Signed-off-by: Phoebe Goldman <phoebe@goldman-tribe.org>
* Remove `row_pk` from JSON messages
* fmt
* SDK: ClientCache keyed on BSATN bytes, not domain types
Per Ingvar's suggestion, this commit makes the client cache
be `HashMap<Vec<u8>, T>`, where the key is a BSATN byte-buffer
containing the serialized representation of the value.
This means that generated structs and enums don't need to be `Hash + Eq`.
The key upside here is that we can revert to using `f32`/`f64`
rather than `sats::F32`/`sats::F64`.
---------
Signed-off-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Mazdak Farrokhzad