Traversing the commitlog without also making it available for writing
would still require upfront I/O imposed by the `open` constructor.
Avoid that by introducing free-standing functions which start traversal
right away.
When a server key rotation is suspected, suggest to make the new
identity the default.
This usually happens during development using an ephemeral instance. If
one follows the instructions, the `default_identity` is not set for the
existing server, which makes the CLI generate a fresh identity every
time.
See also: #333
* durability: Introduce a method to obtain the max tx offset of a history
Useful for reporting replay progress.
Include note that it is somewhat similar to `std::iter::Iterator::size_hint`.
* core: Re-instantiate replay progress reporting
The percentage is calculated as starting from the zero offset, although
that may change in the future.
- Arcing `TableSchema`, and this has benefits elsewhere too.
- Arc<[_]>ing the visitor program instructions.
The data behind the Arcs very rarely change,
which is the perfect case for an Arc.
Fix a minor bug where completely empty transactions would still be
written to the commitlog. The bug is minor because, once we start
logging inputs, all transactions will be non-empty.
The check is done in relational DB rather than the durability crate,
because in principle empty transactions are permissible, and may be used
in the future (e.g. to confirm a certain offset).
Disk usage reporting was left unimplemented in previous patches of the
series, as its semantics are slightly different from before.
Namely, inspecting the size of the commitlog now requires to `stat(2)`
the segment files, and is thus fallible.
Also, a size reporting function is only defined for local durability
(i.e. the commitlog). The behaviour when the database is in a follower
state is left unspecified.
`TxData` is the representation of a transaction after it was committed,
and is passed around for evaluation of subscription queries and sending
the result to clients.
With the new commitlog, it can be represented more compactly, such that
copies for writing to the log can be avoided.
Note that this patch stops short of refactoring `DatabaseUpdate`, which
is another representation of the same information as sent to clients.
This means that `ProductValue`s need to be cloned from `TxData`, just as
before.
This patch attempts to integrate the new commitlog with the minimum
changes.
Most of the diff comes from deletions of the legacy log and the need to
adjust tests due to the requirement for a tokio runtime when a durable
database is used in tests.
The "meat" of the patch are the `RelationalDB` constructors,
`RelationalDB::commit_tx`, and the replay logic in
`locking_tx_datastore`.
While `DataKey` is gone, there is still some redundant data being passed
around, which will be addressed in the follow-up patch.
* Swap the location of tags to go before variant data in the BFLATN encoding
* Fix a comment
* Apply suggestions from code review (@gefjon @Centril)
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Signed-off-by: james gilles <jameshgilles@gmail.com>
* Implement memcpy consolidation for sums
* Vanquish clippy
---------
Signed-off-by: james gilles <jameshgilles@gmail.com>
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Defines traits intended to abstract over the kind of persistence a
database utilizes. The only implementation is (host-)local durability in
terms of the new commitlog crate.
The trait definitions may not be considered stable yet, but are in their
tentative form needed for further integration of the new commitlog.
`spawn_module_host` was changed in #904 to ensure that the supplied
program is indeed the database's current program. There are, however,
uses which would call `spawn_module_host`, followed by either
`init_database` or `update_database` on the returned `ModuleHost`.
This would fail because the condition is not met.
While we cannot prevent misuse, this patch documents the various
lifecycle methods, and adjusts the signature of `init_module_host` to be
(somewhat) consistent with `update_module_host`. Namely, it will
optionally return the result of the `init` reducer call.
* Detect unsatisfiable range queries; warn and short-circuit.
This commit fixes a panic caused by unsatisfiable range bounds on an index query,
e.g. `WHERE x < 5 AND x > 5`.
These unsatisfiable bounds made Rust's `BTreeMap` angry
(See https://doc.rust-lang.org/src/alloc/collections/btree/search.rs.html#106-124),
and panicked.
They also represent probable bugs,
as it's silly to write a query which statically will return no rows.
With this commit, we detect statically unsatisfiable bounds in two cases:
- When compiling queries, we log a message at `WARN` containing the offending query.
- When evaluating queries, we silently construct an `EmptyRelOps`
rather than a real query iterator.
This commit also adds a test that the offending queries can be compiled and executed
without panicking, and select no rows.
* Per Joshua's review, add comments that this is a suboptimal solution
* Fix typo
---------
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
* Consistently use `--identity` `-i` for identity args to CLI
Prior to this commit, some subcommands used `--as-identity` as the long,
some of them used `-I` as the short, and there was no clear pattern.
With this commit, we just use `--identity` as the long everywhere,
and `-i` as the short.
* Rename variables per John's review
* Commit iter metrics
* test: Make full-join benchmark async
* use Vec instead of Hash
* Revert "test: Make full-join benchmark async"
This reverts commit eed5aa8a55.
* rebase on master
* lint
* refector api
* remove lifetime from ctx (#1017)
* move ctx to higher level
* fix boilterplate
* batch more
---------
Co-authored-by: joshua-spacetime <josh@clockworklabs.io>
Defines the canonical commitlog payload, and how to encode / decode it.
Also exposes folds alongside iterators, which allows the common case of
replaying the commitlog onto a database to be further optimized (the
`Txdata` does not have to be constructed in this case). This
optimization is, however, left for a future patch.
First in a series of patches to implement the new commitlog format.
This patch implements the base format, leaving the transaction payload
generic. Segment handling, writing and reading is implemented based on
an in-memory backend, which greatly simplifies testing.
As a notable deviation from the previous implementation, segments are
never implicitly trimmed. Instead, faulty commits are ignored if and
only if the next commit in the log sequence is valid and has the right
offset. On the write path, this entails closing the active segment when
an (I/O) error occurs, but retaining the commit in memory such that it
is written to the next segment.
Note that this patch does not define the final public API.
* doc: Onboarding impr, fixes, consistency, cleanup
refactor: Whats next cleanup, +unity, -bloat
Removed redundant text while there
refactor: Unity quickstart fixes, impr, prettify
refactor: Unity pt1 fixes, impr, prettify
fix(README): Rm "see test edits below" ref
* !exists
refactor(minor): General onboarding cleanup
* Shorter, prettier, consistent
fix(sdks/c#): Broken unitypackage url
feat(sdks/c#): Add OneTimeQuery api ref
* doc: Onboarding impr, fixes, consistency, cleanup
* fix: Rm redundant 'module_bindings' mention
* fix: Floating period, "arbitrary", "important":
- PR review change requests
- Additionally: hasUpdatedRecently fix and reformatting
* fix: Mentioned FilterBy, used FindBy
- Used FindBy since that was what the tutorial used, and also looking for a single Identity.
- Note: There may be a similar rust discrepancy in the Unity pt1 tutorial. It'll work with Filter, but just simply less consistent. Holding off on that since my Rust syntax knowledge !exists.
* fix(Unity-pt1): Rm copy+paste redundant comments
* Duplicate comments found both above and within funcs
* fix(unity): Rm unused using statement +merged info
* Removed `System.Runtime.CompilerServices`
* SpacetimeDB.Module seems to already include this (merged the info)
* refactor(minor): Code spacing for grouping/clarity
* feat: 'Standalone mode runs in foreground' memo
* At general quickstart for `spacetime start`
* refactor(unity-pt1): Standalone mode foreground memo
* Also, removed the "speed" loss mention of C#
* fix(syntaxErr): Fix err, keep FilterBy, handle null
- After a verbose discussion, we will eventually swap to FindBy for single-result queries, but not in this PR.
- For now, the syntax err is fixed by making the var nullable and suffixing a LINQ FirstOrDefault(). Approved by Tyler in Discord.
- We never *actually* created a player in the tutorial. This creates the player. Approved by Tyler in Discord.
* fix: Remote player `is null` check removal
Kim Altintop