Tyler Cloutier
9686139dbc
# Description of Changes
This PR translates all of our Python smoketests into Rust tests which
can be run from `cargo run`
## Motivation
The purpose of this fivefold:
1. All developers on the team are familiar with Rust
2. It simplifies our devops because we can drop Python as a dependency
to run the tests
3. You can now run all tests in the repo through the single `cargo test`
interface
4. Because we use the `SpacetimeDbGuard` and `cargo test`/`cargo
nextest` we can easily parallelize the smoke tests
5. The smoketests can now use machinery imported from SpacetimeDB crates
(e.g. `bsatn` etc.)
IMPORTANT NOTE!
There are several ways to implement the smoke tests in Rust (none are
great):
1. A separate xtask specifically for the smoke tests
- This doesn't solve the problem of the CLI tests which also use the
`guard` crate
- Idiosyncratic way to run the smoke tests as opposed to cargo test
- Does NOT resolve the cargo within cargo problem because we still have
to build the test modules with cargo
2. A `build.rs` script in `guard` which first builds the executables as
a compile step for compiling guard
- Deadlocks on a cargo lock file conflict (Outer cargo compiles guard →
runs build.rs, inner cargo tries to acquire the build directory lock,
outer cargo holds the directory lock, deadlock)
- If you fix the deadlock by using different target dirs, it still looks
stuck on building guard because it's actually compiling all of
spacetimedb-standalone and spacetimedb-cli.
- Still technically runs cargo inside of cargo.
3. Add `spacetimedb-cli` and `spacetimedb-standalone` as an artifact
dependency of the guard crate
- Has good and clear output but requires +nightly when running the
smoketests and CLI tests, otherwise won't do the right thing. See
https://github.com/rust-lang/cargo/issues/9096
4. Compile the executables at runtime during the tests themselves where
the first test takes a lock while the executables are building using
cargo within cargo
- Makes the tests look like they're taking a long time when they're just
waiting for the build to complete
- Requires relatively complex locking machinery across
binaries/tests/processes
5. A two step solution where the developer has to build the binaries
before calling the smoke tests
- Very error prone
None of these are good. `xtask` is not bad, but doesn't enable us to run
other integration tests in other crates (e.g. the CLI)
(3) is the correct solution and has the best user experience, but it
requires nightly and I don't want to introduce that for all of our
tests.
I have chosen to do a combination of (1) and (4). You will now run the
smoketests with `cargo smoketest`. If you run `cargo test --all` (or use
`guard`) without doing `cargo smoketest` it will fall back to (4) which
compiles the executables at runtime. Running `cargo build` is the **only
way** to ensure that the executables are not stale because of the
internal fingerprint checking. Everything else is fragile not robust.
NOTE! There is no way to avoid cargo within cargo and have the smoke
tests be run as cargo tests because the modules under test must be
compiled with cargo.
# API and ABI breaking changes
Note that this is a BREAKING CHANGE to `cargo test --all`. The
smoketests are now part of `cargo test --all` unless you specifically
exclude them.
# Expected complexity level and risk
3, this is partially AI translated. We need to carefully review to
ensure the semantics have not regressed.
# Testing
<!-- Describe any testing you've done, and any testing you'd like your
reviewers to do,
so that you're confident that all the changes work as expected! -->
- [ ] <!-- maybe a test you want to do -->
- [ ] <!-- maybe a test you want a reviewer to do, so they can check it
off when they're satisfied. -->
---------
Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Signed-off-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
Co-authored-by: John Detter <4099508+jdetter@users.noreply.github.com>
Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
Co-authored-by: Zeke Foppa <196249+bfops@users.noreply.github.com>
SpacetimeDB's cargo ci
Overview
This document provides an overview of the cargo ci command-line tool, and documentation for each of its subcommands and options.
cargo ci
SpacetimeDB CI tasks
This tool provides several subcommands for automating CI workflows in SpacetimeDB.
It may be invoked via cargo ci <subcommand>, or simply cargo ci to run all subcommands in sequence. It is mostly designed to be run in CI environments via the github workflows, but can also be run locally
Usage:
Usage: cargo ci [OPTIONS] [COMMAND]
Options:
--skip: Skip specified subcommands when running all
When no subcommand is specified, all subcommands are run in sequence. This option allows specifying subcommands to skip when running all. For example, to skip the unreal-tests subcommand, use --skip unreal-tests.
--help: Print help (see a summary with '-h')
test
Runs tests
Runs rust tests, codegens csharp sdk and runs csharp tests. This does not include Unreal tests. This expects to run in a clean git state.
Usage:
Usage: test
Options:
--help: Print help (see a summary with '-h')
lint
Lints the codebase
Runs rustfmt, clippy, csharpier and generates rust docs to ensure there are no warnings.
Usage:
Usage: lint
Options:
--help: Print help (see a summary with '-h')
wasm-bindings
Tests Wasm bindings
Runs tests for the codegen crate and builds a test module with the wasm bindings.
Usage:
Usage: wasm-bindings
Options:
--help: Print help (see a summary with '-h')
dlls
Builds and packs C# DLLs and NuGet packages for local Unity workflows
Packs the in-repo C# NuGet packages and restores the C# SDK to populate sdks/csharp/packages/**. Then overlays Unity .meta skeleton files from sdks/csharp/unity-meta-skeleton~/** onto the restored versioned package directory, so Unity can associate stable meta files with the most recently built package.
Usage:
Usage: dlls
Options:
--help: Print help (see a summary with '-h')
smoketests
Runs smoketests
Executes the smoketests suite with some default exclusions.
Usage:
Usage: smoketests [ARGS]...
Options:
args: Additional arguments to pass to the smoketests runner. These are usually set by the CI environment, such as-- --docker--help: Print help (see a summary with '-h')
update-flow
Tests the update flow
Tests the self-update flow by building the spacetimedb-update binary for the specified target, by default the current target, and performing a self-install into a temporary directory.
Usage:
Usage: update-flow [OPTIONS]
Options:
--target: Target triple to build for, by default the current target. Used by github workflows to check the update flow on multiple platforms.--github-token-auth: Whether to enable github token authentication feature when building the update binary. By default this is disabled.--help: Print help (see a summary with '-h')
cli-docs
Usage:
Usage: cli-docs [OPTIONS]
Options:
--spacetime-path: specify a custom path to the SpacetimeDB repository root (where the main Cargo.toml is located)--help: Print help (see a summary with '-h')
self-docs
Usage:
Usage: self-docs [OPTIONS]
Options:
--check: Only check for changes, do not generate the docs--help: Print help (see a summary with '-h')
global-json-policy
Usage:
Usage: global-json-policy
Options:
--help:
help
Usage:
Usage: help [COMMAND]...
Options:
subcommand:
This document is auto-generated by running:
cargo ci self-docs