mirror of
https://github.com/clockworklabs/SpacetimeDB.git
synced 2026-05-09 00:59:46 -04:00
Tyler Cloutier
9686139dbc
# Description of Changes
This PR translates all of our Python smoketests into Rust tests which
can be run from `cargo run`
## Motivation
The purpose of this fivefold:
1. All developers on the team are familiar with Rust
2. It simplifies our devops because we can drop Python as a dependency
to run the tests
3. You can now run all tests in the repo through the single `cargo test`
interface
4. Because we use the `SpacetimeDbGuard` and `cargo test`/`cargo
nextest` we can easily parallelize the smoke tests
5. The smoketests can now use machinery imported from SpacetimeDB crates
(e.g. `bsatn` etc.)
IMPORTANT NOTE!
There are several ways to implement the smoke tests in Rust (none are
great):
1. A separate xtask specifically for the smoke tests
- This doesn't solve the problem of the CLI tests which also use the
`guard` crate
- Idiosyncratic way to run the smoke tests as opposed to cargo test
- Does NOT resolve the cargo within cargo problem because we still have
to build the test modules with cargo
2. A `build.rs` script in `guard` which first builds the executables as
a compile step for compiling guard
- Deadlocks on a cargo lock file conflict (Outer cargo compiles guard →
runs build.rs, inner cargo tries to acquire the build directory lock,
outer cargo holds the directory lock, deadlock)
- If you fix the deadlock by using different target dirs, it still looks
stuck on building guard because it's actually compiling all of
spacetimedb-standalone and spacetimedb-cli.
- Still technically runs cargo inside of cargo.
3. Add `spacetimedb-cli` and `spacetimedb-standalone` as an artifact
dependency of the guard crate
- Has good and clear output but requires +nightly when running the
smoketests and CLI tests, otherwise won't do the right thing. See
https://github.com/rust-lang/cargo/issues/9096
4. Compile the executables at runtime during the tests themselves where
the first test takes a lock while the executables are building using
cargo within cargo
- Makes the tests look like they're taking a long time when they're just
waiting for the build to complete
- Requires relatively complex locking machinery across
binaries/tests/processes
5. A two step solution where the developer has to build the binaries
before calling the smoke tests
- Very error prone
None of these are good. `xtask` is not bad, but doesn't enable us to run
other integration tests in other crates (e.g. the CLI)
(3) is the correct solution and has the best user experience, but it
requires nightly and I don't want to introduce that for all of our
tests.
I have chosen to do a combination of (1) and (4). You will now run the
smoketests with `cargo smoketest`. If you run `cargo test --all` (or use
`guard`) without doing `cargo smoketest` it will fall back to (4) which
compiles the executables at runtime. Running `cargo build` is the **only
way** to ensure that the executables are not stale because of the
internal fingerprint checking. Everything else is fragile not robust.
NOTE! There is no way to avoid cargo within cargo and have the smoke
tests be run as cargo tests because the modules under test must be
compiled with cargo.
# API and ABI breaking changes
Note that this is a BREAKING CHANGE to `cargo test --all`. The
smoketests are now part of `cargo test --all` unless you specifically
exclude them.
# Expected complexity level and risk
3, this is partially AI translated. We need to carefully review to
ensure the semantics have not regressed.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [ ] <!-- maybe a test you want to do -->
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
23 lines
626 B
Markdown
23 lines
626 B
Markdown
# Python Smoketests (Legacy)
|
|
|
|
> **Note:** These Python smoketests are being replaced by Rust smoketests in `crates/smoketests/`.
|
|
> Both test suites currently run in CI to ensure consistency during the transition.
|
|
>
|
|
> For new tests, please add them to the Rust smoketests. See `crates/smoketests/DEVELOP.md` for instructions.
|
|
|
|
---
|
|
|
|
## Running the Python Smoketests
|
|
|
|
To use the smoketests, you first need to install the dependencies:
|
|
|
|
```
|
|
python -m venv smoketests/venv
|
|
smoketests/venv/bin/pip install -r smoketests/requirements.txt
|
|
```
|
|
|
|
Then, run the smoketests like so:
|
|
```
|
|
smoketests/venv/bin/python -m smoketests <args>
|
|
```
|