* Add the `snapshot` crate, which implements snapshotting at a low level
- Requires making `BlobHash` be `Serialize` and `Deserialize`.
For arcane macro-ology reasons, this requires writing `BlobHash::SIZE`
instead of `Self::SIZE` (it gets embedded in a visitor struct or something).
- Requires adding two new operators to `BlobStore`.
- Adds a return value to `Page::save_content_hash`, for convenience.
- Impls `DerefMut` for `Pages`.
- **Scary change:** adds `Table::pages_mut`.
I think possibly this operator should be `unsafe`,
since write access to the `Pages` allows an undisciplined caller
to violate the `Table`'s assumptions by corrupting a `Page`.
It seems like an anti-pattern to mark a method `unsafe` on the grounds that
misusing its return value can cause UB,
but I don't see a plausible alternative
without making most methods on `Page` unsafe.
Open to feedback on this one!
* Nix `Table::pages_mut`
* Address Mazdak's feedback
* Use `thiserror` rather than `anyhow` for better error hygiene
* commitlog: Make bitflip test a proptest
The test sometimes fails. As a proptest, we'll be able to seed it with
failing inputs.
Fixes: #1167
* commitlog: Fix the bitflip test
Turns out we sometimes flipped a bit in the CRC32 itself, which makes
things go wrong in not the expected way.
* Make both TX offset counters agree
Prior to this commit, `CommittedState::next_tx_offset` got out of sync
with the commitlog/durability's notion of the tx offset,
because the former counted all committed TXes,
while the latter excluded certain empty TXes
(namely, empty TXes
which were not `__identity_connected__` or `__identity_disconnected__` reducers).
With this commit, the skipping logic is moved earlier into `CommittedState`,
so that it can maintain a counter consistent with that used by the commitlog.
* Remove duplicated increment
* Create new crate `fs-utils`; move `Lockfile` and `create_parent_dir`
The snapshot crate will need to create lockfiles.
Rather than duplicating code to do so, we choose to move our definition of `Lockfile`
into a crate that can be depended on by both `cli` and `snapshot`.
No existing crate seems like an obvious choice for this
-- a `Lockfile` is not really a data structure, so `data-structures` seems wrong --
so we add a new crate, `fs-utils`.
Currently this contains only `Lockfile` and `create_parent_dir`,
but a follow-up PR will add `DirTrie`, a Git-like on-disk object store.
* Deduplicate `map_err` closure
* Zeke's nit: simplify control flow
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Signed-off-by: Phoebe Goldman <phoebe@goldman-tribe.org>
---------
Signed-off-by: Phoebe Goldman <phoebe@goldman-tribe.org>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
* Impl `Serialize`, `Deserialize` for `Page`
Snapshotting needs to write `Page`s to files and read them back again.
To that effect, this commit implements `Serialize` and `Deserialize` for `Page`.
* Address Mazdak's review
- Fix soundness in `FixedBitSet` by moving an assert.
- Add commentary to test.
- Add commentary to `spacetimedb-lib` dependency.
* Fix commitlog `fold_transactions_from` ignoring requested offset
Prior to this commit, `fold_transactions_from` on a durability backed by a commitlog
would discard the requested offset and unconditionally yield all txes in the relevant segments.
This commit changes that behavior so that `fold_transactions_from`
skips commitlog commits (which contain many txes) less than the reqested offset,
and skips txes using `consume_record`.
* Add `Decoder::skip_record`
Lucky I asked Kim whether I was using `consume_record` and `decode_record` correctly,
because I wasn't.
This commit adds methods to `Decoder` and `Visitor` for skipping records and rows,
causing them to be extracted from the reader but not folded.
* Fix test
Add new methods to `Decoder` and `Visitor` hidden away in a test I missed.
* 1. Split ColumnOp into ColumnOp & FieldOp, former storing ColId
2. Shrink SqlAst to 80 bytes, so it can be passed in registers
3. Store end-result Header in IndexSemiJoin
4. Remove operational use of Header in ColumnOp & build_query
5. Simplify RowRef::{get, project, project_owned}
* Make parts of build_query actually infallible.
1. Make IndexSemiJoin::filter infallible.
2. Make ColumnOp::compare and friends infallible.
3. Make RowRef::{get, project, project_owned} infallible.
* Make RelOps::next itself infallible
* 1. with_select{_cmp}: ensure type safety o query exec cannot panic
2. Document RelValue::{get, read_or_take_column, project_owned}
3. Refactor optimize_select
4. Ensure in optimize_select that conditions are merged with preceding selects
* remove RelOps::{head, row_count}; head is redundant & row_count is useless
* remove Relation trait; it does not carry its weight
* make build_query infallible
* simplify IndexSemiJoin, make it slightly less branchy
* simplify try_index_join
* split IndexSemiJoin into Left & Right parts
* move get_field_pos to test code
* move test version of build_query to test code
* make user tables private by default and define privacy via attribute
* switch to spacetimedb(table(public)) syntax
* accept codegen snap changes
* sdk: use public in define_tables!
* bindings-macro: adjust some doc comments
* sdk-test-connect-disconnect: make Connected/Disconnected public tables
* Make Public Private again
* Add script to dispatch bot tests to BitCraftBots
* Make sure results are named correctly, whoops
* Move results to a private bucket, whoops
* Pull in various fixes
* Safer for merge, don't run bot tests on every PR...
* Fix tracing patch
* Remove accidental file
* Test commit
* Test
* Test
* Fix results upload
* Address review comments
Mazdak Farrokhzad