The commitlog creates new segments atomically, returning EEXIST if the
segment already exists. This is to break a retry loop in case the
filesystem becomes unwritable.
This error did not contain any context about what does not exist, so
this patch adds some.
Also, an unhandled edge case has been discovered:
When opening an existing log, the commitlog will try to resume the last
segment for writing. If it finds a corrupt commit in that segment, it
won't resume, but instead create a new segment at the corrupt commit's
offset + 1.
However, if the first commit in the last segment is corrupted, the
offset will be that of the last segment -- trying to start a new segment
will thus fail with EEXIST.
Without additional recovery mechanisms, it is not obvious what to do in
this case: the segment could contain valid data after the initial
commit, so we certainly don't want to throw it away.
Instead, we now detect this case and return `InvalidData` with some
context.
# Expected complexity level and risk
1
# Testing
- [ ] A (regression) test is included
# Description of Changes
We've run into a problem on Maincloud caused by a database that was
writing a relatively small number of very large transactions. This was
accruing many commitlog segments consuming hundreds of gigabytes of
disk, but had not ever taken a snapshot, or compressed or archived any
data, as the database had not progressed past one million transactions.
With this PR, we take a snapshot every time the commitlog segment
rotates. We still also snapshot every million transactions.
One BitCraft database we looked at had 2.5 million transactions per
commitlog segment, meaning that this change will not meaningfully affect
the frequency of snapshots. The offending Maincloud database, however,
had only 50 transactions per segment!
# API and ABI breaking changes
N/a
# Expected complexity level and risk
3: Hastily made changes to finnicky code across several crates.
# Testing
I am unsure how to test these changes.
- [ ] <!-- maybe a test you want to do -->
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
# Description of Changes
Necessary for pulling in rolldown.
# API and ABI breaking changes
None
# Expected complexity level and risk
1, with the caveat that this updates the Rust version and therefore
touches all the code.
# Testing
- [ ] Just the automated testing
# Description of Changes
We recently merged several repos together. This PR clarifies the license
terms for several subdirectories, as well as the relationship between
the licenses.
The licenses in our subdirectories have become symbolic links to
licenses in our toplevel `licenses` directory. For any particular
subdirectory's license file in the diff, you can click `... -> View
file` and then click on the text that says "Symbolic Link" on that page.
This will take you to the license file that it links to.
I have also updated the `tools/upgrade-version` script to update the
change date in the new `licenses/BSL.txt` file.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
None. Only changes to license files.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Adds methods and free-standing functions to allow folds to stop at an
upper
bound, by passing a range instead of only a start offset.
# Expected complexity level and risk
1
# Testing
* commitlog: Make bitflip test a proptest
The test sometimes fails. As a proptest, we'll be able to seed it with
failing inputs.
Fixes: #1167
* commitlog: Fix the bitflip test
Turns out we sometimes flipped a bit in the CRC32 itself, which makes
things go wrong in not the expected way.
Kim Altintop