# Description of Changes
We've run into a problem on Maincloud caused by a database that was
writing a relatively small number of very large transactions. This was
accruing many commitlog segments consuming hundreds of gigabytes of
disk, but had not ever taken a snapshot, or compressed or archived any
data, as the database had not progressed past one million transactions.
With this PR, we take a snapshot every time the commitlog segment
rotates. We still also snapshot every million transactions.
One BitCraft database we looked at had 2.5 million transactions per
commitlog segment, meaning that this change will not meaningfully affect
the frequency of snapshots. The offending Maincloud database, however,
had only 50 transactions per segment!
# API and ABI breaking changes
N/a
# Expected complexity level and risk
3: Hastily made changes to finnicky code across several crates.
# Testing
I am unsure how to test these changes.
- [ ] <!-- maybe a test you want to do -->
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
# Description of Changes
Necessary for pulling in rolldown.
# API and ABI breaking changes
None
# Expected complexity level and risk
1, with the caveat that this updates the Rust version and therefore
touches all the code.
# Testing
- [ ] Just the automated testing
# Description of Changes
We recently merged several repos together. This PR clarifies the license
terms for several subdirectories, as well as the relationship between
the licenses.
The licenses in our subdirectories have become symbolic links to
licenses in our toplevel `licenses` directory. For any particular
subdirectory's license file in the diff, you can click `... -> View
file` and then click on the text that says "Symbolic Link" on that page.
This will take you to the license file that it links to.
I have also updated the `tools/upgrade-version` script to update the
change date in the new `licenses/BSL.txt` file.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
None. Only changes to license files.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Adds methods and free-standing functions to allow folds to stop at an
upper
bound, by passing a range instead of only a start offset.
# Expected complexity level and risk
1
# Testing
* commitlog: Make bitflip test a proptest
The test sometimes fails. As a proptest, we'll be able to seed it with
failing inputs.
Fixes: #1167
* commitlog: Fix the bitflip test
Turns out we sometimes flipped a bit in the CRC32 itself, which makes
things go wrong in not the expected way.
* Fix commitlog `fold_transactions_from` ignoring requested offset
Prior to this commit, `fold_transactions_from` on a durability backed by a commitlog
would discard the requested offset and unconditionally yield all txes in the relevant segments.
This commit changes that behavior so that `fold_transactions_from`
skips commitlog commits (which contain many txes) less than the reqested offset,
and skips txes using `consume_record`.
* Add `Decoder::skip_record`
Lucky I asked Kim whether I was using `consume_record` and `decode_record` correctly,
because I wasn't.
This commit adds methods to `Decoder` and `Visitor` for skipping records and rows,
causing them to be extracted from the reader but not folded.
* Fix test
Add new methods to `Decoder` and `Visitor` hidden away in a test I missed.
Phoebe Goldman