Closes#1024.
Before this change,
we would serialize messages **before** inserting into the send queue.
Because we commit the tx only after inserting into the send queue,
this meant we were holding onto the database lock unnecessarily.
After this change,
we serialize messages **after** inserting into the send queue.
This means we serialize only after committing the tx.
* Implement (but do not use) a fast path for BFLATN -> BSATN conversion
* fmt and clippy
* `u16` offset rather than `usize`
* Address Joshua's review
* Define methods on `RowRef` and `RelValue` which use the new serializer
* Comment in `align_to` about div-by-zero
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Signed-off-by: Phoebe Goldman <phoebe@goldman-tribe.org>
* Add benchmark comparing BFLATN -> BSATN with and without the fast path
* Add benchmark on `u64_u64_u32`, which has less interior padding than `u32_u64_u64`
* Remove `to_len` from `to_bsatn_extend`
It turns out to be slower than just eating the `realloc`s.
* Remove unused `to_bsatn_slice`
I thought I would need it, but it ended up not being useful.
* Expand comment with example; `Box<[...]>` to reduce memory footprint
* Comments from Mazdak's review
---------
Signed-off-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
`AlgebraicTypeLayout` and friends already include full layout information,
including properly-aligned offsets for `ProductTypeElementLayout`s.
As such, there's no need to do any alignment computation
during `serialize_value` or `write_value`.
Instead, while traversing a `ProductTypeLayout`,
we can use each element's `offset` to update the `curr_offset`.
* eval_updates: use map entry apis
* dedup logic in remove_subscription + use entry api to hash only once
* stop cloning PVs in eval_updates
* address Phoebe's comments
* add tracing for perf testing
---------
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
Closes#813.
A subscription will no longer materialize product values,
for queries with read-only row operations.
but instead it will serialize from bflatn straight to bsatn.
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
* Distinguish between inner and semijoins in `QueryExpr` AST.
This commit adds a flag `semi: bool` to `JoinExpr`, which signifies a semijoin,
as opposed to an inner join.
A new optimization pass, `QueryExpr::try_semi_join`, is defined
which can detect a certain common case of inner joins and rewrite them into semijoins.
The punchline here is that `core::vm::join_inner` used to accept a flag `semi: bool`
which it could use to avoid some expensive `Header` mutations,
but that flag was always passed as `false` because we had no way to distinguish semijoins.
With this commit, the flag is actually used,
so evaluating non-indexed semijoins should avoid allocating a new `Header`.
* Address Joshua's review
- Remove a test that was silly and backwards, and intentionally thwarted the optimizer
in a way that will hopefully stop working soon.
- Add a test that an `IncrementalJoin`'s `virtual_plan` looks like we expect.
- Rename the `JoinExpr` argument to `core::vm::join_inner` for clarity.
- Sprinkle comments around about how we compile and optimize joins.
* Log a warning when doing `iter_by_col_range` without an index
* Only warn if the table is sufficiently large for a scan to be bad
Per Tyler's review, this commit gates the warning behind `rdb_num_table_rows`,
so that the warning is only printed
if the table in question has at least `TOO_MANY_ROWS_FOR_SCAN` rows.
`TOO_MANY_ROWS_FOR_SCAN` is defined as 1000
because that's the number Tyler said in his comment.
* Gate the unindexed warning behind a feature in `core`
It turns out that the changes introduced in #734 do not result in more
reliable detection of incompatible schema updates. This is because the
datastructures involved can be converted into each other, but that
conversion is not bijective.
Fix this by manually adjusting the schema of the existing table to be
comparable to the proposed table.
Also log details about a schema mismatch to the user-retrievable database log,
in unified diff format.
* remove tracing from InstanceEnv::*_by_col_eq
* Added #[tracing::instrument(skip_all)] to call_reducer_with_tx and call_reducer
* remove tracing on some stuff in wasm_intance_env
---------
Co-authored-by: Tyler Cloutier <cloutiertyler@aol.com>
* Rewrite smoketests as python unittests
* Get all tests working and do some work on parallel unittest
* Give up on parallel unittests
* Fix CI + address comments
* Fix skip-clippy arg confusion (just use the env var)
* fix ci
* Add comments
Closes#954.
I previously avoided the evaluation of certain delta table joins as an optimization,
which relied on the fact that a tx would not include inserts and deletes for both tables,
which of course is not generally correct.
This patch includes the fully general solution.
We're seeing intermittent failures of the SDK tests in CI due to timeouts.
The timeout is meant only to fail if an expected event never happens;
the SDK tests are not interested in measuring performance at all.
This commit doubles the timeout from 30 to 60 seconds,
in the hopes that we will see fewer false failures.
* Cleaned up code and added better error handling and responses for invalid tables
* Fixed breaking test
* Implemented new logging system
* made a small change to a WS error log
As an optimization,
because we do not support strict projections in subscription queries,
selections may always assume that the op_type field is last,
when removing it.
* Incremental joins: compile once, run repeatedly.
Well, more like, compile 3 times, run repeatedly, but 3 is approximately 1.
This commit re-writes `IncrementalJoin` to be a re-usable representation
of a query plan for an incremental join,
where before it was a one-off worker for the same.
`IncrementalJoin` stores three copies of the query, compiled for `MemTable * DbTable`,
`DbTable * MemTable` and `MemTable * MemTable`.
Related to this change, `eval_incr` no longer needs an `AuthCtx`,
because we check permissions during query compilation, not execution,
and all query planning is now done ahead of time during `add_subscriber`.
As a result, many callsites, especially in tests, which used to pass an `AuthCtx`
no longer do so.
* `IncrementalJoin`: save `MemTable` headers to avoid recomputing
* Don't include `__op_type` column in incremental joins
Incremental joins never used the `__op_type` column,
as they separate deletes from inserts and eval them as separate queries.
This commit causes incremental joins to no longer include the `__op_type` column
in their `MemTable`s at all,
which simplifies the code and should remove some allocations.
* Remove `pub` on test `run_query` helper.
The module's not `pub`, so it was a pointless qualifier.
Mazdak Farrokhzad