# Description of Changes
Makes runtime parameters explicit in query plans (prerequisite for
parameterized views).
As part of this change, `sender` is no longer baked directly into query
plans as a literal value. Instead it is represented as a parameter in
the query plan. Values are supplied at runtime via a variable
environment called `ExecutionParams`.
Note, parameterized plans are still not shared across subscriptions yet.
That will be done in a follow up.
This is mostly a mechanical change. The majority of the diff is just
threading runtime params/variables through various call sites.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
...
# Testing
Existing coverage
# Description of Changes
Required before we add formal parameters to query plans which in turn is
required to support parameterized views.
Before this change, we cached two types of query representations for a
subscription - a physical plan and what we call a pipelined executor.
This was mainly an artifact of built up technical debt, so this change
removes the physical plan and only caches the actual executor for the
subscription, in preparation for adding bind variables that are
substituted by the executor at runtime.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
...
# Testing
Existing coverage
# Description of Changes
Refactors index probes so that multi-column index keys are represented
as a single product value expression `PhysicalExpr::Product` instead of
being split across various fields and structs. As a result, this patch
also simplifies the index-scan and index-join query executor variants
now that index scans/probes share one physical shape.
This change is in preparation for adding formal parameters to query
plans. Since index probe values now have single unified representation
as a `PhysicalExpr`, a future parameterized plan can represent an index
scan or index join as follows:
```rust
IndexProbe::Point(PhysicalExpr::Product(vec![
PhysicalExpr::Param(sender_slot),
PhysicalExpr::Value(other_const),
]))
PhysicalExpr::Product(vec![
PhysicalExpr::Param(sender_slot),
PhysicalExpr::Field(lhs_join_field),
])
```
This will avoid having to duplicate parameter handling in a bunch of
different places or rules.
Note, a very nice consequence of this refactor is that it
removes/consolidates a significant amount of code as the diff shows.
# API and ABI breaking changes
N/A
# Expected complexity level and risk
...
# Testing
Existing coverage
# Description of Changes
This PR shrinks `JsWorkerRequest` so that it is (almost) as small as the
call reducer request.
To do that, a bunch of trivial changes had to be done to auth code, that
mostly revolves around `String` -> `Box<str>`.
This should help the auth code, but that is incidental.
The main goal was to improve throughput through the request tx/rx
channel for V8, which is taking quite a bit of time in flamegraphs.
I also noticed while making this change that the wrong hash map was
being used in a bunch of places, so I fixed all of those.
A follow up PR will shrink the reply side to fit within a cache line.
Yet another follow up PR will change the channel to replace flume with
`fibre::spsc`.
# API and ABI breaking changes
None
# Expected complexity level and risk
2, fairly trivial changes.
# Testing
Covered by existing tests.
# Description of Changes
The first commit defines a type `TableName` that is used in e.g.,
`TxData` and where determined profitable and necessary to do this
change.
`TableName` is backed by
[`ecow::EcoString`](https://docs.rs/ecow/0.2.6/ecow/string/struct.EcoString.html)
which affords O(1) clones and 15 bytes of inline storage and
`mem::size_of::<EcoString>() == 16`.
The second commit does the same for `ReducerName`. This is also used in
reducer execution.
Together, these commits increase TPS by around 5-7k TPS.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
Covered by existing tests.
# Description of Changes
When doing a ranged seek on a non-ranged index (none such exist yet, but
will be added in a follow up), return an (ABI) error.
Also:
- Use point scans in query execution (`IxScan(Delta)Eq`).
- Refactor table index code with macro `same_for_all_types`.
# API and ABI breaking changes
None
# Expected complexity level and risk
2?
# Testing
Testing the error handling will be possible once hash indices are added
(follow up PR).
# Description of Changes
View tables have private metadata columns that need to be dropped before
sending results to clients. Before this patch we dropped these columns
for sql queries and initial subscriptions, but we didn't drop them after
incremental update which is what this patch does.
# API and ABI breaking changes
None
# Expected complexity level and risk
1
# Testing
- [x] Smoketest
# Description of Changes
Updates views atomically on commit, but before downgrading to a
read-only transaction for subscription evaluation.
What this patch does:
1. Renames `ViewId` to `ViewFnPtr`
2. Renames `ViewDatabaseId` to `ViewId`
3. Removes the `module_rx` module watcher from the subscription manager
4. Refactors read sets to only track table scans (index key tracking
will be added later)
5. Drops read sets and removes rows from `st_view_sub` when dropping a
view in an auto-migrate
6. Re-evaluates and updates views (`call_views_with_tx`) from
`call_reducer_with_tx` for any view whose read set overlaps with the
reducer's write set
7. Does the same for sql dml
# API and ABI breaking changes
None
# Expected complexity level and risk
3
It's a bit of a messy diff.
# Testing
- [x] Integrate with
https://github.com/clockworklabs/SpacetimeDB/pull/3616
---------
Signed-off-by: joshua-spacetime <josh@clockworklabs.io>
Co-authored-by: Shubham Mishra <shivam828787@gmail.com>
# Description of Changes
This patch:
1. Materializes views on subscribe and sql calls by invoking `call_view`
on the `ModuleHost`.
2. Downgrades to a read-only transaction after view materialization but
before query execution.
3. Updates the `st_view_sub` system table on both subscribe and
unsubscribe.
4. Makes subscribe methods on the SubscriptionManager async.
# API and ABI breaking changes
None
# Expected complexity level and risk
2
# Testing
End-to-end tests to be added with atomic view updates
# Description of Changes
This patch does the following:
1. Expands views as part of query planning. Views are always assumed to
be materialized by the query planner, however a view's backing table may
have private columns such as the `sender` column. The query planner
needs to filter by this column in order to select the rows pertaining to
a particular caller.
2. Plumbs `AuthCtx` through the query optimizer. This is needed in order
to implement (1).
3. Adds a new operator for views to the query engine that drops a view's
private columns
# API and ABI breaking changes
None
# Expected complexity level and risk
2.5
# Testing
- [x] SQL http tests
- [ ] Subscription tests
- [ ] One off query tests
joshua-spacetime