PublicArchive/SpacetimeDB
2
0
Fork 0
mirror of https://github.com/clockworklabs/SpacetimeDB.git synced 2026-05-06 07:26:43 -04:00
Code Issues Packages Projects Releases Wiki Activity

Add TESTING.md, which documents some of our testing (#2898)

Browse Source 
# Description of Changes

From the perspective of a new client SDK or module library developer,
with a focus on the SDK test suite.

# API and ABI breaking changes

N/a - it's internal docs.

# Expected complexity level and risk

0 - it's internal docs.

# Testing

N/a - it's internal docs.

---------

Signed-off-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: Mario Montoya <mamcx@elmalabarista.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>
This commit is contained in:
Phoebe Goldman
2026-04-14 18:14:12 -04:00
committed by GitHub
parent a137341f34
commit 8a4368e0e5
1 changed files with 145 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
TESTING.md
+145
View File
@@ -0,0 +1,145 @@
# A brief overview of SpacetimeDB testing
## From the perspective of a client SDK or module library developer
## By pgoldman 2025-06-25, updated 2026-04-14
SpacetimeDB has good test coverage, but it is rather haphazardly spread across several suites.
Some of the reasons for this are historical, and some have to do with our using multiple repositories.
This document is an attempt to describe the test suites which would be useful to someone
working on a new client SDK or module bindings library in a new language,
or attempting to move an existing client SDK or module bindings library from an external repository in-tree.
## The SDK tests
`crates/testing/src/sdk.rs` defines a test harness which was originally designed for testing client SDKs.
The basic flow of a test using this harness is:
- Build and freshly publish a module to construct a short-lived initially-empty database.
- Use that module to run client codegen via `spacetime generate` into a client project.
- Compile that client project with the newly-generated bindings.
- Run the client project as a subprocess, passing the database name or `Identity` in an environment variable.
- The client process connects to the database,
runs whatever tests it likes,
writes to stdout and/or stderr as it goes,
then uses its exit code to report whether the test was successful or not.
- If the subprocess's exit is non-zero, the test is treated as a failure,
and the subprocess's stdout and stderr are reported.
This framework has since been used more generally for integration testing.
In particular, we maintain equivalent Rust, C#, TypeScript, and C++ modules in the `modules/sdk-test*` family,
and run the Rust SDK client project at `sdks/rust/tests/test-client` against them through `sdks/rust/tests/test.rs`.
We similarly maintain `modules/sdk-test-connect-disconnect*` modules
which run against `sdks/rust/tests/connect_disconnect_client`.
There are also related SDK-harness-driven suites for event tables, procedures, and views,
using modules such as `modules/sdk-test-event-table`, `modules/sdk-test-procedure*`, and `modules/sdk-test-view*`.
The Unreal SDK also uses the same underlying harness through `sdks/unreal/tests/sdk_unreal_harness.rs`.
The harness is designed to support running multiple tests in parallel with the same client project,
running client codegen exactly once per test suite run.
This unfortunately still conflicts with our use of the suite to test that modules in different languages behave the same,
as each test suite invocation will only run `spacetime generate` against one module language at a time,
never all of them in the same run.
### Testing a new module library
If you are developing a new module bindings library, and wish to add it to the SDK test suite
so that the existing client test projects will run against it:
1. Create `modules/sdk-test-XX` and `modules/sdk-test-connect-disconnect-XX`, where `XX` is some mnemonic for your language.
Populate these with module code which defines all of the same tables and reducers
as `modules/sdk-test` and `modules/sdk-test-connect-disconnect` respectively.
Take care to use the same names, including casing, for tables, columns, indexes, reducers and other database objects.
2. Modify `sdks/rust/tests/test.rs` to add an additional call to `declare_tests_with_suffix!` at the bottom,
like `declare_tests_with_suffix!(xxlang, "-XX")`, if that driver is the right place for the new language.
Some capabilities now live in separate suites in that file, such as procedures and views,
so you may need to wire those up separately as well.
3. Run the tests with `cargo test -p spacetimedb-sdk --test test`.
### Testing a new client SDK
If you are developing a new client SDK, and wish to use the SDK test harness and existing modules
so that it will run against `modules/sdk-test` and `modules/sdk-test-connect-disconnect`:
1. Find somewhere sensible to define test projects `test-client` and `connect_disconnect_client` for your client SDK language.
If your client SDK is in-tree, put these within its directory, following the existing layout under `sdks/rust/tests/` or `sdks/unreal/tests/`.
2. Use `spacetime generate` manually, or via the harness, to generate those projects' `module_bindings`.
3. Populate those projects with client code
matching `sdks/rust/tests/test-client` and `sdks/rust/tests/connect_disconnect_client` respectively.
- Connect to SpacetimeDB running at `http://localhost:3000`.
- Connect to the database whose name is in the environment variable `SPACETIME_SDK_TEST_DB_NAME`.
- For `test-client`, take a test name as a command-line argument in `argv[1]`, and dispatch to the appropriate test to run.
- For `connect_disconnect_client`, there is only one test.
- The Rust code jumps through some hoops to do assertions about asynchronous events with timeouts,
using an abstraction called the `TestCounter` defined in `sdks/rust/tests/test-counter`.
This is effectively a semaphore with a timeout.
You may or may not need to replicate this behavior.
4. Create integration tests in the SDK crate which construct `spacetimedb_testing::sdk::Test` objects,
following `sdks/rust/tests/test.rs` or `sdks/unreal/tests/test.rs` as a template.
5. Define `#[test]` tests for each test case you have implemented,
which construct `spacetimedb_testing::sdk::Test` objects containing the various subcommand strings to run your client project,
then call `.run()` on them.
### Adding a new test case
If you want to add a new test case to the SDK test suite, to test some new or yet-untested functionality
of either the module libraries or client SDKs:
1. If necessary, add new tables and/or reducers to `modules/sdk-test` and friends which exercise the behavior you want to test.
2. Add a new function, `exec_foo`, to the appropriate client project,
such as `sdks/rust/tests/test-client/src/lib.rs`,
which connects to the database, subscribes to tables and invokes reducers as appropriate,
and performs assertions about the events it observes.
3. Add a branch to that client's dispatch logic,
such as the `match` in `sdks/rust/tests/test-client/src/main.rs`,
which matches the test name `foo` and dispatches to call your `exec_foo` function.
4. Add a `#[test]` test function to the relevant test driver,
such as `sdks/rust/tests/test.rs`,
which does `make_test("foo").run()`, where `"foo"` is the test name you chose in step 3.
5. Repeat steps 2 through 4 for any other client projects which ought to cover the same behavior.
6. Run the new test with the relevant `cargo test` command for that SDK.
## Schema parity tests
`crates/schema/tests/ensure_same_schema.rs` is a separate but important companion to the SDK tests.
It compares the extracted schemas of equivalent modules across languages,
and is often the first place where casing, indexes, primary keys, or other schema details drift apart.
As of writing, it covers the `benchmarks`, `module-test`, `sdk-test`, and `sdk-test-connect-disconnect` families.
If you add or update a cross-language module family,
it is worth considering whether it should also be covered here.
## The smoketests
`crates/smoketests/` defines an integration and regression test suite using a Rust harness.
These are useful primarily for testing the SpacetimeDB CLI, but can also be used to exercise publish flows,
documentation, and other end-to-end behavior.
The smoketest harness is still primarily oriented around Rust modules,
and it does not use the same client-project machinery as the SDK harness.
It could be extended to do more in that direction, but that may not be worth the effort.
As of writing, the smoketest suite includes dedicated coverage such as:
- `crates/smoketests/tests/smoketests/csharp_module.rs`, which smoke-tests C# module compilation.
- `crates/smoketests/tests/smoketests/quickstart.rs`, which replays the quickstart guide for Rust and C#.
- `crates/smoketests/DEVELOP.md`, which documents how to run and write these tests.
One practical note is that the smoketests use prebuilt `spacetimedb-cli` and `spacetimedb-standalone` binaries,
so if you modify those crates or their dependencies, you should rebuild before running the suite.
## Standalone integration test
The `spacetimedb-testing` crate has an integration test file, `crates/testing/tests/standalone_integration_test.rs`.
The tests in this file publish `modules/module-test`, `modules/module-test-cs`, `modules/module-test-ts`, and `modules/module-test-cpp`,
then invoke reducers or procedures in them and inspect their logs to verify that the behavior is expected.
These tests do not exercise the entire functionality of `module-test`,
but by virtue of publishing it do assert that it is syntactically valid and that it compiles.
To add a new module library to the Standalone integration test suite:
1. Create `modules/module-test-XX`, where `XX` is some mnemonic for your language.
2. Populate this with module code which defines all of the same tables and reducers
as the existing `module-test` family.
If you notice any discrepancies between the existing languages, those parts may be compiled but not run,
and so you are free to ignore them.
3. Modify `crates/testing/tests/standalone_integration_test.rs` to define new `#[test] #[serial]` test functions
which use your new `module-test-XX` module to do the same operations as the existing tests.
4. Run the tests with `cargo test -p spacetimedb-testing --test standalone_integration_test`.