# Description of Changes
Update the Default casing policy to `snake_case` for `RawModuleDefV10`.
Messy PR contains changes at different places, so that CI can pass:
Here are the main changes as follows:
- `bindings-macro` & `bindings` crate: `name` macro in Indexes for
canonical name and supply it to `RawModuleDefV10` via `ExplicitNames`.
- `bindings-typescript`:
- Changes has been reviewed through this PR -
https://github.com/clockworklabs/SpacetimeDB/pull/4308.
- `binding-csharp`: a single line change to pass `sourceName` of index
instead of null.
- `codegen`:
- Changes has been merged from branch -
https://github.com/clockworklabs/SpacetimeDB/pull/4337.
- Except a fix in rust codegen to use canonical name in Query buillder
instead of accessor.
- `lib/db/raw_def`: Extends `RawDefModuleV10` structure to support case
conversion.
- `schema` crate:
- `validate/v9` - Nothing itself should change or changes in v9
validation logic but the file contains a `CoreValidator` which is shared
with `validate/v10`. No test have t be updated to `validate/v9` which
ensures we aren't regressing it.
- `validate/v10`: This is the main meat, look at the new tests added in
bottom to understand what it does.
- Rest of the files are either test updates or module bindings.
## Testing:
1. Extensive unit tests have been added to verify generated `ModuleDef`
is correct.
2. I have done some e2e testing to verify rust codegen with rust and
typescript modules.
3. I would have like to do more testing for other codegens , I am
continue doing .
I have removed `sql.py` smoketest, as that seems to be already migated
in new framework and was headache to update.
## Expected complexity level and risk
4, It could have side-effect which aren't easily visible.
- - -
---------
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <bot@clockworklabs.com>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
Co-authored-by: Noa <coolreader18@gmail.com>
Co-authored-by: = <cloutiertyler@gmail.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@clockworklabs.io>
Co-authored-by: Jason Larabie <jason@clockworklabs.io>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
# Description of Changes
- [x] : Store "accessor"s names in ModuleDef (for tables, columns, and
indexes for now) and pass them to TableSchema as aliases.
- [x] Create accessor system tables for each of them:
`st_table_accessor`,`st_column_accessor`, `st_index_accessor`
- [x] Update the SQL parsing logic to also consider alias names.
Alias name stored will be same as actual names until we merge
https://github.com/clockworklabs/SpacetimeDB/pull/4263
# API and ABI breaking changes
NA
# Expected complexity level and risk
3
This patch updates SQL name resolution which isn't a very complex change
since table and column name resolution are each handled by their own
helper method.
It also adds new system tables which need to be handle correctly during
restore/replay. We do have regression tests for this that should catch
any incorrect handling or missing steps.
I would say the main risk is that we don't populate cached table and
column schemas with the alias/accessor name. Regression tests have been
added to catch this, but there are edge cases around migrations and
replay that currently aren't covered.
# Testing
Added SQL smoketests that use canonical/accessor table/column names.
They currently fail because we're currently using the accessor name for
both. They should pass once this is updated.
---------
Co-authored-by: rekhoff <r.ekhoff@clockworklabs.io>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <bot@clockworklabs.com>
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
# Description of Changes
The goal of this PR is to:
- Make it easy to switch our backing implementation of identifiers, as
seen in the commit changing to `LeanString`.
- Improve type safety of our identifiers by having every identifier
start with `RawIdentifier` and then `Identifier` is just a wrapper with
validation on construction. `TableName` and `ReducerName` are then just
wrappers around `Identifier`.
- Reduce allocation in `InstanceEnv` by using the now clone-on-write
O(1) + SSO optimized `Identifier`.
- Reduce allocations in the query engine as a consequence of improving
`RawIdentifier` and `Identifier`.
- Have `&'static str` strings be further optimized by avoiding to
allocate even for long ones. This is supported by `impl From<&'static
str> for RawIdentifier` as well as switching to `LeanString`.
The PR results in a roughly 2k TPS improvement over master for WASM +
hash indices. This should also help V8, as this is VM-agnostic.
# API and ABI breaking changes
None
# Expected complexity level and risk
3? Mostly just small changes in many places, but in a sea of small
changes, mistakes can happen.
# Testing
Covered by existing tests.
# Description of Changes
The first commit defines a type `TableName` that is used in e.g.,
`TxData` and where determined profitable and necessary to do this
change.
`TableName` is backed by
[`ecow::EcoString`](https://docs.rs/ecow/0.2.6/ecow/string/struct.EcoString.html)
which affords O(1) clones and 15 bytes of inline storage and
`mem::size_of::<EcoString>() == 16`.
The second commit does the same for `ReducerName`. This is also used in
reducer execution.
Together, these commits increase TPS by around 5-7k TPS.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Covered by existing tests.
The `Clone` impl for `ClientConnection` would create an independent
instance that could not observe module hotswapping. This would result in
methods called on a replaced `ModuleHost` to fail, because that host
exited already.
Fix by reading the `ModuleHost` from the watch channel directly, instead
of maintaining a redundant copy.
Also fix `watch_module_host` to properly mark the current module host as
seen.
# Expected complexity level and risk
2
# Testing
- [x] test suite passes
- [x] ran @bfops repro script
This patch attempts to integrate the new commitlog with the minimum
changes.
Most of the diff comes from deletions of the legacy log and the need to
adjust tests due to the requirement for a tokio runtime when a durable
database is used in tests.
The "meat" of the patch are the `RelationalDB` constructors,
`RelationalDB::commit_tx`, and the replay logic in
`locking_tx_datastore`.
While `DataKey` is gone, there is still some redundant data being passed
around, which will be addressed in the follow-up patch.
* Implement (but do not use) a fast path for BFLATN -> BSATN conversion
* fmt and clippy
* `u16` offset rather than `usize`
* Address Joshua's review
* Define methods on `RowRef` and `RelValue` which use the new serializer
* Comment in `align_to` about div-by-zero
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Signed-off-by: Phoebe Goldman <phoebe@goldman-tribe.org>
* Add benchmark comparing BFLATN -> BSATN with and without the fast path
* Add benchmark on `u64_u64_u32`, which has less interior padding than `u32_u64_u64`
* Remove `to_len` from `to_bsatn_extend`
It turns out to be slower than just eating the `realloc`s.
* Remove unused `to_bsatn_slice`
I thought I would need it, but it ended up not being useful.
* Expand comment with example; `Box<[...]>` to reduce memory footprint
* Comments from Mazdak's review
---------
Signed-off-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
- Move it and friends from sats to vm.
- MemTable now stores a Vec<PV>.
- Other related improvements.
Co-authored-by: Phoebe Goldman <phoebe@goldman-tribe.org>
* made MutTxId -> MutTx
* adopt using ass. type for tx
* fmt
* read tx in datastore
lint
test
lint
made MutTxId -> MutTx
adopt using ass. type for tx
fmt
added more methods to TxId
fix test
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
comments on Test
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
fmt
typo
* fix imports
* clippy
* read tx in datastore
lint
test
lint
added read tx in subs
fmt
fix iters
* comments
* comments
* fix schema_for_table
* fmt
* merge queue helped
* moved all_read_query anaylyzing to vm
* lint
* added StateView trait
* lint
* fix tests for read-tx-in-subs (#695)
Intention of this PR is to keep diff of main PR smaller - feat: subscription to use read type tx #685 by separating trivial unit test related changes from main logic.
naming changes begin_tx() -> begin_mut_tx() and begin_read_tx() -> begin_tx()
* rollback_tx -> release_tx
* Compiler should work with Write Tx
* execute sql with mut tx
* fix result len
* compiler to work with write tx (#716)
* Compiler should work with Write Tx
* merge queue helped
* Refactoring bootstrapping and adding full support for constraints & multi-column indexes
* Removed deleted code and add some comments
* Add comment for Constraints
* Update crates/core/src/db/datastore/locking_tx_datastore/mod.rs
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Signed-off-by: Mario Montoya <mamcx@elmalabarista.com>
* Update crates/core/src/db/datastore/locking_tx_datastore/mod.rs
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Signed-off-by: Mario Montoya <mamcx@elmalabarista.com>
* Addressing some PR comments & remove cloning
* Adding details for when fail to decode rows in replay
* Apply Kim's fix for replay bug
* Addressing some PR comments
* Merge
* Fix conflicts
* Fix conflicts
* Fix test
* Fix generation of code for csharp & typescript, now it generate FilterBy for all columns
* Fix for index generation
* Fixing C# index generation
* Merge
* Fix compilation on csharp bindings
---------
Signed-off-by: Mario Montoya <mamcx@elmalabarista.com>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Co-authored-by: John Detter <no-reply@boppygames.gg>
* Moving object defs to SATS
* move ColumnIndexAttribute to primitives, nix path attr
* revert unnecessary reformating
* ProductValue: don't take ColId
ProductValues can be more than rows and can be nested
* simplify sats::db::def wrt. Constraints + impl ser/de
* cargo fmt
---------
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Closes#463.
This metric is maintained by the datastore iterator.
It is written to the metric store when the iterator is dropped.
Only committed rows are counted.
Closes#456.
Closes#461.
Define an execution context for database operations.
Store the database id in RelationalDB.
Record metrics for rows [inserted|deleted] on commit.
* Add script to summarize benchmark results
* Mess with github actions benchmark script
* Formatting fixes
* Comment
* Bring back commit comments, upload to DO Spaces
* Remove cache
We now have servers for benchmarks outside of GH infrastructure and
downloading cache takes way more time than its worth it
* base and head refs are not needed anymore
* Run benchmarks only on master
* Test a branch build
* Fix branch becnhmarks path
* test
* Run only on master again
---------
Signed-off-by: Piotr Sarnacki <drogus@gmail.com>
Co-authored-by: Piotr Sarnacki <drogus@gmail.com>
Closes#414.
Adds a test for calling iter_by_col_eq with multiple columns.
Ensures the index is scanned if an applicable multi-column index is present.
To write this test, iter_by_col_eq was updated to take multiple columns.
* Reverts benchmarks portion of "Bench for SQL scan / where (#370)"
This partially reverts commit 05956d6.
* Get benchmarks runner to remember what it has installed, hopefully
* Fix build error
Noa