Controlled shutdown of a database should drain the outstanding
transactions
queue(s) and flush them to the durability layer.
With the introduction of another queueing layer in #3868, it became
harder to
observe when or if this process is completed.
This patch thus introduces an explicit (async) shutdown method for
`RelationalDB` and below, which will wait until all submitted
transactions are
either reported durable, or an error occurs in the durability layer.
`RelationalDB` is made `!Clone`, such that shutdown can be initiated in
the
`Drop` impl. Note that this requires access to a tokio runtime, which we
thread
through via the `Persistence` services in order to allow control over
which of
the various runtimes is being used for durability-related tasks.
Also moves `RelationalDB::open` to a blocking thread when a
persistence-enabled
database is constructed by the `HostController` -- this process performs
heavy
I/O and can take a substantial amount of time, during which we don't
want to
block a worker thread.
# API and ABI breaking changes
None
# Expected complexity level and risk
3
# Testing
- [ ] some testing added
- [ ] existing tests still pass
- [ ] `impl Drop for RelationalDB` difficult to test, extra eyeballs
needed
---------
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Permissions for evaluating SQL/DML are not generally "actions", but more
a set of permissions that are checked during evaluation.
To make this work with the teams feature, this patch extends `AuthCtx`
to allow checking a set of permissions as mandated by the spec. This set
is a bit more fine-grained than "is owner", so as to avoid baking in the
concept of teams/collaborators, or assumptions about what a role might
entail. Both are likely to evolve in the future, so evaluation of
permissions / capabilities should be confined to the impl of the
`Authorization` trait.
Unlike "actions", the `AuthCtx` must be able to evaluate permission
checks quickly and without side-effects, nor can it enter an `async`
context. In that sense, it is precomputed (if you will), and stored as a
closure in the `AuthCtx` for external authorization.
A challenge posed is how to thread through the constructed `AuthCtx` for
subscriptions.
A tempting approach would have been to equip the `HostController` with
the ability to summon an `AuthCtx`. That, however, would have created a
gnarly circular dependency, because the `HostController` also controls
the controldb, which itself demands an `AuthCtx`.
Instead, the `AuthCtx` is obtained in the endpoint handler and passed to
each method call that requires one. That's less pretty, but more
effective.
---------
Signed-off-by: Kim Altintop <kim@eagain.io>
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
# Description of Changes
This exposes client credentials in reducer calls for rust.
# API and ABI breaking changes
API Changes:
The main API change is the addition of `AuthCtx` and the `sender_auth`
in `ReducerContext`. This also adds JwtClaims, which has some helpers
for getting commonly used claims.
ABI Changes:
This adds one new functions `get_jwt`. This uses
`st_connection_credentials` to look up the credentials associated with a
connection id.
This adds ABI version 10.2.
# Expected complexity level and risk
2. This adds new ABI functions
# Testing
I've done some manual testing with modified versions of the quickstart.
We should add some examples that use the new API.
# Description of Changes
We recently merged several repos together. This PR clarifies the license
terms for several subdirectories, as well as the relationship between
the licenses.
The licenses in our subdirectories have become symbolic links to
licenses in our toplevel `licenses` directory. For any particular
subdirectory's license file in the diff, you can click `... -> View
file` and then click on the text that says "Symbolic Link" on that page.
This will take you to the license file that it links to.
I have also updated the `tools/upgrade-version` script to update the
change date in the new `licenses/BSL.txt` file.
# API and ABI breaking changes
None.
# Expected complexity level and risk
1
# Testing
None. Only changes to license files.
---------
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
This patch attempts to integrate the new commitlog with the minimum
changes.
Most of the diff comes from deletions of the legacy log and the need to
adjust tests due to the requirement for a tokio runtime when a durable
database is used in tests.
The "meat" of the patch are the `RelationalDB` constructors,
`RelationalDB::commit_tx`, and the replay logic in
`locking_tx_datastore`.
While `DataKey` is gone, there is still some redundant data being passed
around, which will be addressed in the follow-up patch.
- Move it and friends from sats to vm.
- MemTable now stores a Vec<PV>.
- Other related improvements.
Co-authored-by: Phoebe Goldman <phoebe@goldman-tribe.org>
* made MutTxId -> MutTx
* adopt using ass. type for tx
* fmt
* read tx in datastore
lint
test
lint
made MutTxId -> MutTx
adopt using ass. type for tx
fmt
added more methods to TxId
fix test
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
comments on Test
Co-authored-by: Phoebe Goldman <phoebe@clockworklabs.io>
Signed-off-by: Shubham Mishra <shivam828787@gmail.com>
fmt
typo
* fix imports
* clippy
* read tx in datastore
lint
test
lint
added read tx in subs
fmt
fix iters
* comments
* comments
* fix schema_for_table
* fmt
* merge queue helped
* moved all_read_query anaylyzing to vm
* lint
* added StateView trait
* lint
* fix tests for read-tx-in-subs (#695)
Intention of this PR is to keep diff of main PR smaller - feat: subscription to use read type tx #685 by separating trivial unit test related changes from main logic.
naming changes begin_tx() -> begin_mut_tx() and begin_read_tx() -> begin_tx()
* rollback_tx -> release_tx
* Compiler should work with Write Tx
* execute sql with mut tx
* fix result len
* compiler to work with write tx (#716)
* Compiler should work with Write Tx
* merge queue helped
* Moving object defs to SATS
* move ColumnIndexAttribute to primitives, nix path attr
* revert unnecessary reformating
* ProductValue: don't take ColId
ProductValues can be more than rows and can be nested
* simplify sats::db::def wrt. Constraints + impl ser/de
* cargo fmt
---------
Co-authored-by: Mazdak Farrokhzad <twingoow@gmail.com>
Closes#456.
Closes#461.
Define an execution context for database operations.
Store the database id in RelationalDB.
Record metrics for rows [inserted|deleted] on commit.
We've had recurring issues with `println` calls sneaking in
where `log` crate macros would be more appropriate.
This commit adds a Clippy warning for uses of the global I/O macros,
i.e. `print`, `println`, `eprint`, `eprintln` and `dbg`.
The lint is disabled by a more-specific `clippy.toml` in the `cli` and `sqltest` crates,
as well as using `allow` attributes in `standalone`'s `subscommands` module.
Additionally, this commit converts a handful of prints in `core/utils` to `log::info`.
Fixes#259
(1) Updates MemTable to use RelValue instead of ProductValue
(2) Adds a DataKey member to DataRef and RelValue
(3) Subscriptions compute DataKey only when not present on row
Fixes#259
(1) Updates MemTable to use RelValue instead of ProductValue
(2) Adds a DataKey member to DataRef and RelValue
(3) Subscriptions compute DataKey only when not present on row
* Document most of SATS
* sats: simplify Serialize impls
* sats: simplify Deserialize impls
* improve sats::de docs
* document sats::bsatn
* simplify sats fmt/map notation
* value serializer: docs + opt
* docs/refactor sats::algebraic_value::{ser,de}
* sats: document serde conversions
* cargo fmt
* small fixes
* fix tests
* fix serde feature
* sats: address pheobe's review
* fix doc comment
* fix test failure
* Remove unusued import
* Fix a borrowing issue introduced by the merge of master
The definitions of these were tweaked in some way, I think,
such that references to them are no longer automatically 'static?
---------
Co-authored-by: George Kulakowski <george@clockworklabs.io>
* Initial bootstrap for private tables
* Separate Access (private, public) for kind o table (system, user)
* Validates the access to private tables
* Check auth for drop table
Kim Altintop