Commit Graph

125 Commits

Author SHA1 Message Date
Mario Montoya f9f38543c8 Add readmes to all implementation crates specifying that they do no offer stable interfaces (#2320) 2025-03-06 19:50:17 +00:00
Phoebe Goldman aedc601145 Rename Address to ConnectionId (#2220)
Signed-off-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: James Gilles <jameshgilles@gmail.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
2025-02-10 00:40:16 +00:00
Noa 293aebaef9 Bump to Rust 1.84 (#2001) 2025-01-28 23:11:29 +00:00
joshua-spacetime 3798f468a6 query engine integration (#2074) 2025-01-11 00:01:24 +00:00
Mario Montoya 20d397cf2b Split Iter & IterByColRange types into Tx and MutTxId versions (#2043) 2024-12-31 16:14:30 +00:00
Shubham Mishra 0a86b46152 Fix Build: Use default() to init custom hash based datastructures (#1871)
Signed-off-by: Shubham Mishra <shubham@clockworklabs.io>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
2024-10-17 14:41:38 +00:00
Noa c4e637e98b Energy metering for persistent memory usage (#766) 2024-10-15 17:58:34 +00:00
Mazdak Farrokhzad 1ac741aa6b Generic and optimized WebSocket API (#1761)
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
2024-09-30 21:57:00 +00:00
james gilles c32f297c47 Update TableSchema & system tables to resemble ABI V9 (#1697) 2024-09-25 20:03:22 +00:00
james gilles 2be42156b2 Allow converting new ModuleDef to old TableSchema (#1630) 2024-08-29 18:17:29 +00:00
Mazdak Farrokhzad 1ca9b1a933 [WASM ABI 1.0] Change ColId from u32 to u16 (#1597) 2024-08-19 17:56:28 +00:00
Mazdak Farrokhzad 6a08674ccb Allow empty ColList (#1588) 2024-08-15 16:08:51 +00:00
Mazdak Farrokhzad 1e8e18d74b Add support for I256 and U256 (#1477) 2024-08-08 18:40:35 +00:00
james gilles 45b2ceee9a Move schemas to schema crate, rename Def to RawDefV8 (#1498) 2024-07-24 17:38:30 +00:00
james gilles f81f2a7492 Move db module from spacetimedb_sats to spacetimedb_lib (#1479) 2024-07-17 20:59:44 +00:00
joshua-spacetime 39344852af refactor(1494): Remove SQL DDL per 1.0 SQL spec (#1499) 2024-07-12 18:55:28 +00:00
Mario Montoya 2ad91b6258 Implement a temporary type check validation on sql compiling (#1456)
Signed-off-by: Mario Montoya <mamcx@elmalabarista.com>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
2024-07-09 17:15:42 +00:00
Mario Montoya 114aa994bf Fix output of binary, Identity, Address for SQL output and the 'Display' of them to show a full hex value (#1087) 2024-06-10 16:49:46 +00:00
joshua-spacetime 61ee5def83 feat: System table based slow query logging (#1350) 2024-06-07 21:37:34 +00:00
Jeremie Pelletier da23401368 Jeremie/notify sql (#1198)
* Notify subscriptions from SQL code path

* Remove blocking_broadcast_event

(From another PR, but helps tests to pass)

* Update from review

* Update from rebase

* cargo fmt

* cargo fmt

* cargo fmt

* update space.rs
2024-06-03 22:51:05 +00:00
Mazdak Farrokhzad be9c958d21 Flatten ColumnOp so that we box less (#1234) 2024-06-03 17:36:44 +00:00
Mazdak Farrokhzad 89aecd15e3 Split ColumnOp into one with row indices and one with FieldName & other enabled changes (#1207)
* 1. Split ColumnOp into ColumnOp & FieldOp, former storing ColId
2. Shrink SqlAst to 80 bytes, so it can be passed in registers
3. Store end-result Header in IndexSemiJoin
4. Remove operational use of Header in ColumnOp & build_query
5. Simplify RowRef::{get, project, project_owned}

* Make parts of build_query actually infallible.

1. Make IndexSemiJoin::filter infallible.
2. Make ColumnOp::compare and friends infallible.
3. Make RowRef::{get, project, project_owned} infallible.

* Make RelOps::next itself infallible

* 1. with_select{_cmp}: ensure type safety o query exec cannot panic
2. Document RelValue::{get, read_or_take_column, project_owned}
3. Refactor optimize_select
4. Ensure in optimize_select that conditions are merged with preceding selects

* remove RelOps::{head, row_count}; head is redundant & row_count is useless

* remove Relation trait; it does not carry its weight

* make build_query infallible

* simplify IndexSemiJoin, make it slightly less branchy

* simplify try_index_join

* split IndexSemiJoin into Left & Right parts

* move get_field_pos to test code

* move test version of build_query to test code
2024-06-03 16:45:32 +00:00
Mazdak Farrokhzad f6b39c0abc IndexJoin/JoinInner: store ColId (#1166)
* IndexJoin/JoinInner: store ColId

* QueryExpr: carry full Header info for resulting query

* refactor and dedup build_query
2024-06-03 15:26:49 +00:00
Mazdak Farrokhzad 5e47b61f8a Tables only become public explicitly via #[spacetimedb(table(public))] (#1278)
* make user tables private by default and define privacy via attribute

* switch to spacetimedb(table(public)) syntax

* accept codegen snap changes

* sdk: use public in define_tables!

* bindings-macro: adjust some doc comments

* sdk-test-connect-disconnect: make Connected/Disconnected public tables

* Make Public Private again
2024-06-03 11:22:58 +00:00
Mazdak Farrokhzad 74bcecd9d4 Restrict multi-col index scans to = (OpCmp::Eq) on all columns (#1316)
* add test test_multi_column_two_ranges, which should fail but doesn't

* restrict multi-col index scans to OpCmp::Eq
2024-06-01 13:25:06 +00:00
Mazdak Farrokhzad 2d045b6bb4 fn (walk_)sources: use internal iteration to avoid allocations (#1304) 2024-05-28 22:06:23 +00:00
joshua-spacetime 88a8adad70 feat(1231): Basic query cardinality estimation (#1273)
* feat(1231): Basic query cardinality estimation

This patch implements basic cardinality estimation for QueryExpr.
It utilizes table cardinalities and number of distinct values for index related operators.

* estimation tests: dedup + define constants for readability

* row_est: simplify with slice patterns

* fn ndv -> fn num_distinict_values

* simplify TypedIndex::num_keys

* is_range -> is_point (invert) + fuse arms in row_est

* estimation: fix logic for IndexJoin

---------

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
2024-05-23 08:49:07 +00:00
Mazdak Farrokhzad 9a827f2f26 remove a few dead impls (#1214) 2024-05-13 08:41:08 +00:00
Mazdak Farrokhzad 4e444bdddd CrudExpr::{Delete, Insert, Update}: only DbTables are possible, so cleanup InMem path (#1211)
* remove dead code wrt. Table

* simplify DbProgram: Insert/Update/Delete only use DbTables

* consistency: move CrudExpr::Update logic to own function

* use TxMode::unwrap_mut more
2024-05-10 00:04:47 +00:00
Mazdak Farrokhzad 78696e5fc3 remove some dead code (#1210) 2024-05-10 00:02:42 +00:00
Mazdak Farrokhzad 6dce129494 From.{join => joins} + remove Option layer + Remove RelOps::try_fold (#1208)
* refactor From sql ast type

* remove RelOps.:try_fold (dead code)
2024-05-07 16:26:27 +00:00
Mazdak Farrokhzad e258262556 perf(JoinInner): use AVs for keys instead of PVs (#1194)
* perf(JoinInner): use AVs for keys instead of PVs

* Remove stale comment.

Co-authored-by: james gilles <jameshgilles@gmail.com>
Signed-off-by: Mazdak Farrokhzad <twingoow@gmail.com>

---------

Signed-off-by: Mazdak Farrokhzad <twingoow@gmail.com>
Co-authored-by: james gilles <jameshgilles@gmail.com>
2024-05-02 21:00:34 +00:00
Mazdak Farrokhzad 0142e14de5 Implement RelValue: Eq + Hash (#1107)
* impl Eq + Hash for RelValue

* Use Hash for RelValue in incr-eval

* naming: spell out pv, rv, and tro
2024-04-30 22:13:50 +00:00
Mazdak Farrokhzad b55121cc83 use a custom FixedBitSet + optimize Page::iter_fixed_len (#1160) 2024-04-30 21:57:28 +00:00
Mazdak Farrokhzad e144c400c5 Redefine FieldName as (TableId, ColId) (#1165)
* redefine FieldName as (TableId, ColId)

* fix & refactor tests + move MemTable/Header test helpers test code
2024-04-30 20:59:20 +00:00
Mazdak Farrokhzad e1064eef29 Remove Column.col_id field (#1136)
* Remove redundant field Column.col_id

* remove redundant fn Header::ty

* - move extract_table_field to only use (#1137)

- remove unused get_index_by_field
2024-04-22 21:23:06 +00:00
Mazdak Farrokhzad db6f1561c6 document RelValue variants (#1134) 2024-04-22 11:46:56 +00:00
Mario Montoya 7d5eb1532d Slow query log (#1052)
* Slow query log

* Addressing some PR comments
2024-04-17 13:53:55 +00:00
Mazdak Farrokhzad d6815ebf9c Shrink AV and AT to 24 & 16 bytes respectively, and also friends. (#1047) 2024-04-13 16:51:18 +00:00
Mario Montoya 29970d219a Implement 'sql' parsing for 'Identity, Address' in hex format (#786) 2024-04-11 17:31:08 +00:00
Phoebe Goldman 6d91d57f3c Detect unsatisfiable range queries; warn and short-circuit. (#1036)
* Detect unsatisfiable range queries; warn and short-circuit.

This commit fixes a panic caused by unsatisfiable range bounds on an index query,
e.g. `WHERE x < 5 AND x > 5`.
These unsatisfiable bounds made Rust's `BTreeMap` angry
(See https://doc.rust-lang.org/src/alloc/collections/btree/search.rs.html#106-124),
and panicked.
They also represent probable bugs,
as it's silly to write a query which statically will return no rows.

With this commit, we detect statically unsatisfiable bounds in two cases:
- When compiling queries, we log a message at `WARN` containing the offending query.
- When evaluating queries, we silently construct an `EmptyRelOps`
  rather than a real query iterator.

This commit also adds a test that the offending queries can be compiled and executed
without panicking, and select no rows.

* Per Joshua's review, add comments that this is a suboptimal solution

* Fix typo

---------

Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
2024-04-09 01:23:20 +00:00
Mazdak Farrokhzad 98363888ff IndexSemijoin::next: do a bit less cloning and work (#1044)
* IndexSemijoin::next: do a bit less cloning and work

* address Joshua's review
2024-04-08 23:48:41 +00:00
Mazdak Farrokhzad 344861f290 use nohasher_hash and ahash instead of siphash13 (#1040)
* use nohasher_hash and ahash instead of siphash13

* re-export types in spacetimedb_data_structures::map
2024-04-05 17:30:51 +00:00
Mario Montoya 56887f50ce Implement 'sql' parsing for simple 'Enums' (#1018) 2024-04-03 17:57:00 +00:00
Mazdak Farrokhzad ddf6048547 Split DatabaseTableUpdate in deletes/inserts vecs (#1019)
* eval_incr: add RelValue::ProjRef(&PV) to avoid cloning PVs

* 1. rename `build_source_query` -> `in_mem_to_rel_ops`
2. `SourceExpr::{MemTable -> InMemory}`
3. clariy some commentary re. SourceExpr/SourceSet and friends
4. cleanup: simplify `compile_select_eval_incr`
5. remove ProgramStore; twas dead code.

* add SourceProvider, simplifying the source set stuff

* use MemTable less

* split DatabaseTableUpdate in deletes/inserts vecs

* incr-join: avoid temp Vec<_> allocs

* store deletes/inserts separately in eval_incr results; mostly cleanup
2024-03-27 20:12:30 +00:00
Mazdak Farrokhzad b6c0e1c4d8 Add AlgebraicValue::take + move test-code in btree_index to tests (#1028)
* add AlgebraicValue::take for a neater interface

* btree_index: move test-only code to tests
2024-03-27 17:05:19 +00:00
Phoebe Goldman ba8a8d93c3 BFLATN -> BSATN fast-path for fixed-length rows (#1005)
* Implement (but do not use) a fast path for BFLATN -> BSATN conversion

* fmt and clippy

* `u16` offset rather than `usize`

* Address Joshua's review

* Define methods on `RowRef` and `RelValue` which use the new serializer

* Comment in `align_to` about div-by-zero

Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Signed-off-by: Phoebe Goldman <phoebe@goldman-tribe.org>

* Add benchmark comparing BFLATN -> BSATN with and without the fast path

* Add benchmark on `u64_u64_u32`, which has less interior padding than `u32_u64_u64`

* Remove `to_len` from `to_bsatn_extend`

It turns out to be slower than just eating the `realloc`s.

* Remove unused `to_bsatn_slice`

I thought I would need it, but it ended up not being useful.

* Expand comment with example; `Box<[...]>` to reduce memory footprint

* Comments from Mazdak's review

---------

Signed-off-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
2024-03-25 19:46:10 +00:00
Mario Montoya e270d97012 Remove useless usages of RowCount (#987) 2024-03-18 19:45:49 +00:00
Mazdak Farrokhzad 5601c18c52 perf: don't clone QueryExpr (#981)
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
2024-03-15 19:55:35 +00:00
Phoebe Goldman 96e5ef16f0 Distinguish between inner and semijoins in QueryExpr AST. (#969)
* Distinguish between inner and semijoins in `QueryExpr` AST.

This commit adds a flag `semi: bool` to `JoinExpr`, which signifies a semijoin,
as opposed to an inner join.

A new optimization pass, `QueryExpr::try_semi_join`, is defined
which can detect a certain common case of inner joins and rewrite them into semijoins.

The punchline here is that `core::vm::join_inner` used to accept a flag `semi: bool`
which it could use to avoid some expensive `Header` mutations,
but that flag was always passed as `false` because we had no way to distinguish semijoins.
With this commit, the flag is actually used,
so evaluating non-indexed semijoins should avoid allocating a new `Header`.

* Address Joshua's review

- Remove a test that was silly and backwards, and intentionally thwarted the optimizer
  in a way that will hopefully stop working soon.
- Add a test that an `IncrementalJoin`'s `virtual_plan` looks like we expect.
- Rename the `JoinExpr` argument to `core::vm::join_inner` for clarity.
- Sprinkle comments around about how we compile and optimize joins.
2024-03-15 14:09:52 +00:00