* commitlog: Make bitflip test a proptest
The test sometimes fails. As a proptest, we'll be able to seed it with
failing inputs.
Fixes: #1167
* commitlog: Fix the bitflip test
Turns out we sometimes flipped a bit in the CRC32 itself, which makes
things go wrong in not the expected way.
* Fix commitlog `fold_transactions_from` ignoring requested offset
Prior to this commit, `fold_transactions_from` on a durability backed by a commitlog
would discard the requested offset and unconditionally yield all txes in the relevant segments.
This commit changes that behavior so that `fold_transactions_from`
skips commitlog commits (which contain many txes) less than the reqested offset,
and skips txes using `consume_record`.
* Add `Decoder::skip_record`
Lucky I asked Kim whether I was using `consume_record` and `decode_record` correctly,
because I wasn't.
This commit adds methods to `Decoder` and `Visitor` for skipping records and rows,
causing them to be extracted from the reader but not folded.
* Fix test
Add new methods to `Decoder` and `Visitor` hidden away in a test I missed.
* commitlog: Panic on fsync failure
Errors returned by `fsync(2)` are particularly nefarious, as it is
mostly undefined what the state of the page cache is in this case.
Since the log is synced asynchronously and not after every write, it is
impossible to know up to which commit data can be considered durable --
except by reading the most recent segment from disk.
Therefore, the reasonable thing to do is to prevent any further use of
the log, and force users to re-load it from disk.
Note that this is only half of the solution: an application restart may
still read data from the page cache, which could be gone after a system
restart.
To fix this, we would need to employ direct I/O (i.e. `O_DIRECT`), which
however is beyond the scope of this patch as it invalidates the use of
most of `std::io`.
* commitlog: Handle duplicate commits when iterating
We cannot exclude the possibility of a false failure in I/O operations.
In particular, `EIO` errors are difficult to attribute to a particular
write, as they happen asynchronously during flush of the page cache.
Because we do not bypass the page cache, the possibility exists that a
particular commit is lost when it isn't, or that it is considered
durable when it isn't. The former could lead to duplicate commits
appearing in the log, while the latter could lead to a matching offset
number, but with different commit payload.
This patch thus ignores duplicates, and introduces a new error variant
in the event the offset matches but the checksum doesn't.
* durability: Manage the flush-and-sync task in this crate
Since syncing the commitlog may now panic, it is more obvious to handle
all async tasks here, so as to be able to handle the panic cases.
Namely, if the `FlushAndSyncTask` panics, the `PersisterTask` is
aborted. This will lead to the channel receiver being dropped, which in
turn will cause the next `append_tx` call to panic.
* commitlog: Remove async flush-and-sync
Due to panic behaviour, it is now preferable to manage periodic sync at
the use site of the commitlog crate.
Hence remove `flush_and_sync_every` method, and with it the dependency
on tokio.
The documentation promised to not collect payload values during folds
(i.e. replaying), but the code did so anyway. This patch makes it so
only values required to satisfy the `Visitor` trait are allocated when
folding.
Traversing the commitlog without also making it available for writing
would still require upfront I/O imposed by the `open` constructor.
Avoid that by introducing free-standing functions which start traversal
right away.
Fix a minor bug where completely empty transactions would still be
written to the commitlog. The bug is minor because, once we start
logging inputs, all transactions will be non-empty.
The check is done in relational DB rather than the durability crate,
because in principle empty transactions are permissible, and may be used
in the future (e.g. to confirm a certain offset).
Disk usage reporting was left unimplemented in previous patches of the
series, as its semantics are slightly different from before.
Namely, inspecting the size of the commitlog now requires to `stat(2)`
the segment files, and is thus fallible.
Also, a size reporting function is only defined for local durability
(i.e. the commitlog). The behaviour when the database is in a follower
state is left unspecified.
This patch attempts to integrate the new commitlog with the minimum
changes.
Most of the diff comes from deletions of the legacy log and the need to
adjust tests due to the requirement for a tokio runtime when a durable
database is used in tests.
The "meat" of the patch are the `RelationalDB` constructors,
`RelationalDB::commit_tx`, and the replay logic in
`locking_tx_datastore`.
While `DataKey` is gone, there is still some redundant data being passed
around, which will be addressed in the follow-up patch.
Defines traits intended to abstract over the kind of persistence a
database utilizes. The only implementation is (host-)local durability in
terms of the new commitlog crate.
The trait definitions may not be considered stable yet, but are in their
tentative form needed for further integration of the new commitlog.
Defines the canonical commitlog payload, and how to encode / decode it.
Also exposes folds alongside iterators, which allows the common case of
replaying the commitlog onto a database to be further optimized (the
`Txdata` does not have to be constructed in this case). This
optimization is, however, left for a future patch.
First in a series of patches to implement the new commitlog format.
This patch implements the base format, leaving the transaction payload
generic. Segment handling, writing and reading is implemented based on
an in-memory backend, which greatly simplifies testing.
As a notable deviation from the previous implementation, segments are
never implicitly trimmed. Instead, faulty commits are ignored if and
only if the next commit in the log sequence is valid and has the right
offset. On the write path, this entails closing the active segment when
an (I/O) error occurs, but retaining the commit in memory such that it
is written to the next segment.
Note that this patch does not define the final public API.
Shubham Mishra