Commit Graph

3186 Commits

Author SHA1 Message Date
Zeke Foppa 2b66851560 More trace logs for TX locking (#1253)
* [bfops/logtrace]: do thing

* [bfops/logtrace]: empty

---------

Co-authored-by: Zeke Foppa <github.com/bfops>
2024-06-03 16:43:10 +00:00
Mazdak Farrokhzad f6b39c0abc IndexJoin/JoinInner: store ColId (#1166)
* IndexJoin/JoinInner: store ColId

* QueryExpr: carry full Header info for resulting query

* refactor and dedup build_query
2024-06-03 15:26:49 +00:00
Mazdak Farrokhzad 5e47b61f8a Tables only become public explicitly via #[spacetimedb(table(public))] (#1278)
* make user tables private by default and define privacy via attribute

* switch to spacetimedb(table(public)) syntax

* accept codegen snap changes

* sdk: use public in define_tables!

* bindings-macro: adjust some doc comments

* sdk-test-connect-disconnect: make Connected/Disconnected public tables

* Make Public Private again
2024-06-03 11:22:58 +00:00
Noa a8549457e9 Re-add execute_sql_mut_tx (#1313) 2024-06-03 09:49:03 +00:00
Mazdak Farrokhzad 74bcecd9d4 Restrict multi-col index scans to = (OpCmp::Eq) on all columns (#1316)
* add test test_multi_column_two_ranges, which should fail but doesn't

* restrict multi-col index scans to OpCmp::Eq
2024-06-01 13:25:06 +00:00
james gilles 638445293d Automated bot tests (#1255)
* Add script to dispatch bot tests to BitCraftBots

* Make sure results are named correctly, whoops

* Move results to a private bucket, whoops

* Pull in various fixes

* Safer for merge, don't run bot tests on every PR...

* Fix tracing patch

* Remove accidental file

* Test commit

* Test

* Test

* Fix results upload

* Address review comments
2024-05-31 17:52:22 +00:00
Shubham Mishra cf4b9aa282 metric for table size (#1319)
* table size metric

* feld blob_store_bytes in table

* address comments

* NumBlobBytes type

* table size metrics: adjust comments, visibility + harden test

---------

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
2024-05-31 17:44:12 +00:00
Noa f8beb699c7 Implement new rand api (#1283)
* Implement new rand api

* Address comments
2024-05-31 17:42:58 +00:00
Ingvar Stepanyan 24d1750b1f Bump ABI version for C# AOT (#1311)
Looks like these constants got missed during upgrade, making AOT version ABI-incompatible.

Unfortunately, there's no way to use a central constant here, so we probably should add it to some "steps to do during ABI version bump" doc.
2024-05-31 17:40:49 +00:00
Ingvar Stepanyan 9e1e66f708 Fix FilterBy regression in C# (#1309)
* Fix FilterBy regression in C#

Fixes regression accidentally introduced in #1277: if FindBy returns null, FilterBy will return an iterable with a single null item instead of an empty iterable.

* Fix snapshots
2024-05-31 17:40:26 +00:00
Ingvar Stepanyan c76c3a0a77 NFC: use record struct for auto-equality in C# (#1318)
The new `record struct` construct is the same as `struct`, but automatically derives equality and hashing by walking through the fields - exactly what we need for those 2 types.
2024-05-31 17:40:01 +00:00
joshua-spacetime 6512f35c6b feat(1231): Add a configurable row limit for queries (#1293)
Closes #1231.

Queries that are estimated to exceed this row limit are rejected.
And the same holds for subscriptions.
2024-05-30 21:35:48 +00:00
Zeke Foppa 0bc42bfe48 Bump version to 0.9.3 (#1224)
* [bfops/bump-version]: empty

* [bfops/bump-version]: version bump

* [bfops/bump-version]: update

* [bfops/bump-version]: bump C# module versions too

* [bfops/bump-version]: bump 0.9.3

---------

Co-authored-by: Zeke Foppa <github.com/bfops>
2024-05-29 17:22:51 +00:00
Mazdak Farrokhzad 2d045b6bb4 fn (walk_)sources: use internal iteration to avoid allocations (#1304) 2024-05-28 22:06:23 +00:00
Zeke Foppa 92f5a17ba7 Bump version to 0.9.2 (#95)
## Description of Changes
Just bumps the version number. Since 0.9.1 has been released, we are now
on 0.9.2+!

## API

 - [ ] This is an API breaking change to the SDK

*If the API is breaking, please state below what will break*


## Requires SpacetimeDB PRs
*List any PRs here that are required for this SDK change to work*

Co-authored-by: Zeke Foppa <github.com/bfops>
2024-05-28 15:48:11 -04:00
Mazdak Farrokhzad 4ac49e564d execution_set.check_auth(...) on initial subscription (#1274) 2024-05-28 19:44:32 +00:00
Kim Altintop 2c3fc66f21 Commitlog: panic on fsync failure (#985)
* commitlog: Panic on fsync failure

Errors returned by `fsync(2)` are particularly nefarious, as it is
mostly undefined what the state of the page cache is in this case.

Since the log is synced asynchronously and not after every write, it is
impossible to know up to which commit data can be considered durable --
except by reading the most recent segment from disk.

Therefore, the reasonable thing to do is to prevent any further use of
the log, and force users to re-load it from disk.

Note that this is only half of the solution: an application restart may
still read data from the page cache, which could be gone after a system
restart.

To fix this, we would need to employ direct I/O (i.e. `O_DIRECT`), which
however is beyond the scope of this patch as it invalidates the use of
most of `std::io`.

* commitlog: Handle duplicate commits when iterating

We cannot exclude the possibility of a false failure in I/O operations.
In particular, `EIO` errors are difficult to attribute to a particular
write, as they happen asynchronously during flush of the page cache.

Because we do not bypass the page cache, the possibility exists that a
particular commit is lost when it isn't, or that it is considered
durable when it isn't. The former could lead to duplicate commits
appearing in the log, while the latter could lead to a matching offset
number, but with different commit payload.

This patch thus ignores duplicates, and introduces a new error variant
in the event the offset matches but the checksum doesn't.

* durability: Manage the flush-and-sync task in this crate

Since syncing the commitlog may now panic, it is more obvious to handle
all async tasks here, so as to be able to handle the panic cases.

Namely, if the `FlushAndSyncTask` panics, the `PersisterTask` is
aborted. This will lead to the channel receiver being dropped, which in
turn will cause the next `append_tx` call to panic.

* commitlog: Remove async flush-and-sync

Due to panic behaviour, it is now preferable to manage periodic sync at
the use site of the commitlog crate.

Hence remove `flush_and_sync_every` method, and with it the dependency
on tokio.
2024-05-28 18:22:38 +00:00
Mazdak Farrokhzad c8225db8f5 1. Reduce visibilities, preparing to extract datastore crate (#1306)
2. More direct imports, preparing to extract datastore crate
3. Remove some dead code
2024-05-28 17:46:07 +00:00
Zeke Foppa 3b33f22c07 CLI - Drop Config lock earlier for spacetime logs (#1303)
* [bfops/lockfiles]: do thing

* [bfops/lockfiles]: empty

---------

Co-authored-by: Zeke Foppa <github.com/bfops>
2024-05-28 16:08:09 +00:00
Zeke Foppa be74474721 CLI - Fix Lockfile error message format strings (#1302)
* [bfops/lockfile-err]: do thing

* [bfops/lockfile-err]: empty

---------

Co-authored-by: Zeke Foppa <github.com/bfops>
2024-05-28 16:05:05 +00:00
Ingvar Stepanyan ac0c097f3d Implement consistent filtering rules for TypeScript (#1276)
- Limit types to those defined in the consistent filtering proposal (#1256).
- Make `filterBy` a lazy iterable for consistency and performance reasons.
- Add `findBy` for unique fields as per proposal.
2024-05-28 14:23:25 +00:00
Ingvar Stepanyan 4819292d74 Simplify update handling as per consistent filtering proposal (#93)
## Description of Changes

As a result of the consistent filtering rules proposal
(clockworklabs/SpacetimeDB#1256), all equatable types are now natively
equatable in C# as well.

This allows us to compare objects directly, without holding and
comparing AlgebraicValue in the entries map as well, which has a bit of
a domino effect and allows to optimise, simplify or even remove some
parts of the SDK.

## API

 - [x] This is an API breaking change to the SDK

New filtering rules limit types on which filtering can be done, as well
as change the return type of `FilterBy` functions to always be iterable.

## Requires SpacetimeDB PRs

https://github.com/clockworklabs/SpacetimeDB/pull/1277
2024-05-28 15:23:20 +01:00
Ingvar Stepanyan 307dfeebd3 Update C# codegen to consistent filtering rules (#1277)
* Update C# codegen to consistent filtering rules

- Limit types as per proposal.
- Add `Query` client-side SDK helper for API parity with server-side modules (on client-side it's a simple wrapper around Iter + Where).
- Change return type of `FilterBy` to always be iterable, with new `FindBy` function for unique fields.
- Simplify the way primary keys are handled - must go with https://github.com/clockworklabs/spacetimedb-csharp-sdk/pull/93 for the client SDK counterpart.

* Add using for System.Linq

* Update snapshot
2024-05-28 14:22:12 +00:00
Ingvar Stepanyan a7c75751f2 Implement consistent filtering rules for C# modules (#1282) 2024-05-28 14:08:40 +00:00
Ingvar Stepanyan f5a13b6f26 Consistent filtering in Rust client + minor fixes (#1275)
* Extend codegen tests to Rust

* Replace cursive-chat module_bindings with symlink

* Implement consistent filtering rules for Rust

* Fixup

* Regenerate tests

* Fix non-deterministic import order

* cargo fmt

* Fix chat examples

* Change symlinks to files themselves

* Revert accidental change

This needs to wait for server-side API break to be implemented as well.
2024-05-28 13:02:25 +00:00
Mazdak Farrokhzad 0e74beed16 merge_apply_inserts/deletes: do metrics work once per table, not per row (#1286)
* merge_apply_inserts: bulk update metrics

* merge_apply_deletes: bulk update metrics
2024-05-28 08:49:22 +00:00
Zeke Foppa 8096ab657d fix(1297): spacetime logs -f should not start at the very beginning (#1298)
* [bfops/logs-f]: do thing

* [bfops/logs-f]: empty

* [bfops/logs-f]: review

---------

Co-authored-by: Zeke Foppa <github.com/bfops>
2024-05-28 00:40:39 +00:00
Shubham Mishra 1f2331634d due cleanup (#1301) 2024-05-27 18:21:17 +00:00
Ingvar Stepanyan ed6a3aac38 Ignore duplicate inserts in the same subscription update (#92)
## Description of Changes

Ignore duplicate inserts in the same subscription update

## API

 - [ ] This is an API breaking change to the SDK

*If the API is breaking, please state below what will break*


## Requires SpacetimeDB PRs
*List any PRs here that are required for this SDK change to work*
2024-05-27 12:49:44 -05:00
Noa 55b7cbe486 Let ProgramStorage::external be async (#1291)
* Let ProgramStorage::external be async

* Remove core::object_db

* Remove odb_rocksdb feature

* Fix typo

* More resilient conflict avoidance
2024-05-24 21:28:54 +00:00
Shubham Mishra 8ecee762f1 Shub/st connected clients (#1288)
* trait modification

* st clients

* error handling

* fix

* make connect transactional

* handle empty reducer

* test

* fix tests
2024-05-24 18:03:31 +00:00
james gilles 7c90990118 Add table smoketest + more smoketest logging (#1272)
* Add smoketest to add table without migration, currently failing

* Add more logging to smoketests, add_table_pseudomigration passing

* Fix test failing for the wrong reason
2024-05-24 17:32:36 +00:00
Zeke Foppa 0bf5d7495d Fix workflow_dispatch for ci.yml (#1295)
* [bfops/fix-workflow-dispatch]: fix

* [bfops/fix-workflow-dispatch]: empty

* [bfops/fix-workflow-dispatch]: fix?

* [bfops/fix-workflow-dispatch]: fix?

---------

Co-authored-by: Zeke Foppa <github.com/bfops>
2024-05-24 17:09:41 +00:00
Zeke Foppa 8c3fa75907 Remove incremental-joins.md (#1296)
* [bfops/rm-include-str]: empty

* [bfops/rm-include-str]: do thing

---------

Co-authored-by: Zeke Foppa <github.com/bfops>
2024-05-24 17:03:09 +00:00
Zeke Foppa 93b9d7690d CI workflows support workflow_dispatch event (#1289)
* [bfops/ci-dispatch]: workflow update

* [bfops/ci-dispatch]: empty

* [bfops/ci-dispatch]: update

* [bfops/ci-dispatch]: update

* [bfops/ci-dispatch]: update

* [bfops/ci-dispatch]: update

---------

Co-authored-by: Zeke Foppa <github.com/bfops>
2024-05-24 02:56:23 +00:00
Kim Altintop 48a2b98d3c Disconnect dangling clients (#1132)
When replaying the commitlog, keep track of unpaired connect/disconnect
calls and call disconnect when instantiating the module.
2024-05-23 18:53:55 +00:00
Mazdak Farrokhzad 25513b37b9 Simplify btree_index module with more idiomatic Rust (#1285)
* simplify btree_index module, more idiomatic Rust

* test
2024-05-23 13:39:48 +00:00
joshua-spacetime 88a8adad70 feat(1231): Basic query cardinality estimation (#1273)
* feat(1231): Basic query cardinality estimation

This patch implements basic cardinality estimation for QueryExpr.
It utilizes table cardinalities and number of distinct values for index related operators.

* estimation tests: dedup + define constants for readability

* row_est: simplify with slice patterns

* fn ndv -> fn num_distinict_values

* simplify TypedIndex::num_keys

* is_range -> is_point (invert) + fuse arms in row_est

* estimation: fix logic for IndexJoin

---------

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
2024-05-23 08:49:07 +00:00
Zeke Foppa f813a298d5 Fix the C# module link in part 1 (#55)
* [bfops/docs]: empty

* [bfops/docs]: one more fix

---------

Co-authored-by: Zeke Foppa <github.com/bfops>
2024-05-22 22:23:32 -04:00
Zeke Foppa f10dcf2ad8 Fix the C# module link in overview (#54)
* [bfops/docs]: C# fix

* [bfops/docs]: empty

---------

Co-authored-by: Zeke Foppa <github.com/bfops>
2024-05-22 20:01:27 -04:00
Puru Vijay fb32452b71 fix: Broken docs links (#53) 2024-05-22 16:49:14 -07:00
Noa 471f4ff2ca recv-style abi (#1002)
* Bump module abi version

* recv abi module side

* recv abi host side

* Update csharp module sdk

* Fix name

* Address comments
2024-05-22 23:01:44 +00:00
Ingvar Stepanyan 552b18bc73 Consistent filtering rules for Rust bindings (#1280)
* Implement consistent filtering rules proposal

Mostly already consistent with the proposal, except:
 - Remove filtering functions for custom Hash type.
 - Allow filtering over `i128` and `u128` fields.
 - Allow filtering over `Address` fields.

* Update crates/bindings-macro/src/lib.rs

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Signed-off-by: Ingvar Stepanyan <me@rreverser.com>

---------

Signed-off-by: Ingvar Stepanyan <me@rreverser.com>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
2024-05-22 21:30:23 +00:00
Noa 6b1a3d3896 Module hotswapping (#1147) 2024-05-22 17:49:35 +00:00
Kurtis Mullins 1af72339f0 Bump version for release v0.9 (#94)
## Description of Changes

Bumped the version number to 0.9.0 as this was forgotten in the previous
release. Includes all changes to date.

## API

 - [X] This is an API breaking change to the SDK

*If the API is breaking, please state below what will break*

Not sure but its going out with the latest release. The previous package
was 5 months ago.

## Requires SpacetimeDB PRs
*List any PRs here that are required for this SDK change to work*
2024-05-22 17:49:34 +01:00
Mazdak Farrokhzad 9257208b6e 1. Privatize st_*_schema functions, only used in system_tables() now. (#1263)
2. Only call system_tables() once in bootstrap_system_tables.
3. Take `TableId` by value more
4. Rename `table_exists` -> `table_name`
5. Dedup `table_name` by using `get_schema()`.
6. Other misc deduping.
2024-05-22 15:48:39 +00:00
Mazdak Farrokhzad 670625ed11 add fn ColList::last(&self) -> ColId (#1267) 2024-05-22 12:04:32 +00:00
Kim Altintop edbca25f1a core: Reserve sequence range for system tables (#1265)
Reserves an unreasonably large number of sequence values for use by system
tables. This means that user-created tables will draw id values starting
from the reserved range + 1, as opposed to number of values taken by
system tables + 1.

Adding new system tables is thus unlikely to interfere with already-assigned
values in existing databases.
2024-05-22 06:35:59 +00:00
Kim Altintop 2de147522d core: Collapse DBIC into HostController (#1186)
Make it so `HostController` manages both the module host (wasm
machinery) and the database (`RelationalDB` / `DatabaseInstanceContext`)
of spacetime databases deployed to a server.

The `DatabaseInstanceContextController` (DBIC) is removed in the
process.

This allows to make database accesses panic-safe, in that uncaught
panics will cause all resouces to be released and the database to be
restarted on subsequent access. This is a prerequisite for #985.

It also allows to move towards storage of the module binary directly in
the database / commitlog. This patch, however, makes some contortions in
order to **not** introduce a breaking change just yet.
2024-05-21 17:30:02 +00:00
Mazdak Farrokhzad ebc921849e privatize Table::row_layout + related BTreeIndex refactoring (#1262) 2024-05-20 18:44:04 +00:00