Translate smoketests from Python to Rust #4102

cloutiertyler · 2026-01-23T03:37:05Z

Description of Changes

This PR translates all of our Python smoketests into Rust tests which can be run from cargo run

Motivation

The purpose of this fivefold:

All developers on the team are familiar with Rust
It simplifies our devops because we can drop Python as a dependency to run the tests
You can now run all tests in the repo through the single cargo test interface
Because we use the SpacetimeDbGuard and cargo test/cargo nextest we can easily parallelize the smoke tests
The smoketests can now use machinery imported from SpacetimeDB crates (e.g. bsatn etc.)

API and ABI breaking changes

None

Expected complexity level and risk

3, this is partially AI translated. We need to carefully review to ensure the semantics have not regressed.

Testing

Use OnceLock to build spacetimedb-cli and spacetimedb-standalone once per test process, then run the pre-built binary directly instead of using `cargo run`. This avoids repeated cargo overhead and ensures consistent binary reuse across parallel tests.

Create `crates/smoketests/` to translate Python smoketests to Rust: - Add `Smoketest` struct with builder pattern for test setup - Implement CLI helpers: `spacetime_cmd()`, `call()`, `sql()`, `logs()`, etc. - Translate `smoketests/tests/sql.py` → `tests/sql.rs` - Translate `smoketests/tests/call.py` → `tests/call.rs` - Reuse `ensure_binaries_built()` from guard crate (now public) Also fix Windows process cleanup in `SpacetimeDbGuard`: - Use `taskkill /F /T /PID` to kill entire process tree - Prevents orphaned `spacetimedb-standalone.exe` processes

- Translate 4 more Python smoketests to Rust: auto_inc, describe, module_nested_op, panic - Simplify the call API by removing the generic call<T: Serialize> method and renaming call_raw to call, since CLI args are strings - Remove unused serde dependency

Translate additional Python smoketests to Rust: - dml.rs: DML subscription tests - filtering.rs: Unique/non-unique index filtering tests - namespaces.rs: C# code generation namespace tests - add_remove_index.rs: Index add/remove with subscription tests - schedule_reducer.rs: Scheduled reducer tests Infrastructure improvements: - Add subscribe_background() and SubscriptionHandle for proper background subscription semantics matching Python tests - Add spacetime_local() for commands that don't need --server flag - Add timing instrumentation for debugging test performance

- Separate build and publish timing in lib.rs to identify bottlenecks - Use --bin-path to skip redundant rebuild during publish - Add DEVELOP.md explaining cargo-nextest for faster test runs Timing breakdown per test: - WASM build: ~12s (75%) - Server publish: ~2s (12%) - Server spawn: ~2s (12%) cargo-nextest runs all test binaries in parallel, reducing total runtime from ~265s to ~160s (40% faster).

Translate from Python smoketests: - detect_wasm_bindgen.rs: Tests build rejects wasm_bindgen and getrandom (2 tests) - default_module_clippy.rs: Tests default module passes clippy - delete_database.rs: Tests deleting database stops scheduled reducers - fail_initial_publish.rs: Tests failed publish doesn't corrupt control DB - modules.rs: Tests module update lifecycle and breaking changes (2 tests) Also adds spacetime_build() method to Smoketest for testing build failures. Total: 16 test files translated, 32 tests

Clear CARGO* environment variables (except CARGO_HOME) when spawning child cargo build processes. When running under `cargo test`, cargo sets env vars like CARGO_ENCODED_RUSTFLAGS that differ from a normal build, causing child cargo processes to think they need to recompile. This reduces single-test runtime from ~45s to ~18s by avoiding redundant rebuilds of spacetimedb-standalone and spacetimedb-cli.

Add test translations for: - connect_disconnect_from_cli.rs - client connection callbacks - domains.rs - database rename functionality - client_connection_errors.rs - client_connected error handling - confirmed_reads.rs - --confirmed flag for subscriptions/SQL - create_project.rs - spacetime init command Also fix subscription race condition by waiting for initial update before returning from subscribe_background_*, matching Python behavior.

Translate tests for: - views.rs: st_view_* system tables, namespace collisions, SQL views - auto_migration.rs: schema changes, add table migration

Add new_identity() method to support multi-identity tests. Translate tests for: - rls.rs: Row-level security filter tests - energy.rs: Energy balance endpoint test - permissions.rs: Private tables, lifecycle reducers, delete protection

Translate tests for: - new_user_flow.rs: Basic publish/call/SQL workflow - servers.rs: Server add/list/edit commands

jdetter · 2026-01-23T08:50:34Z

.github/workflows/ci.yml

  docker_smoketests:
-    needs: [lints]
+    # TODO: before merging re-enable this
+    # needs: [lints]


Address this before merging

@jdetter they're not actually running via cargo ci smoketests right now.. also it uses the version of this workflow from the base PR so this didn't make a difference 😅

okay wired up this command to actually run the new smoketests!

jdetter

I really like these changes so far, I'm excited to talk about this more later today 👍

jdetter · 2026-01-23T09:32:06Z

crates/smoketests/tests/add_remove_index.rs

+fn test_add_then_remove_index() {
+    let mut test = Smoketest::builder().module_code(MODULE_CODE).autopublish(false).build();
+
+    let name = format!("test-db-{}", std::process::id());


This would have been an issue if we were to parallelize the smoketests but now we're running 1 spacetime server per test right?

This is actually kind of weird now that we're appending the process ID to the name - why not just use a static name like "test-db"?

jdetter · 2026-01-23T09:39:33Z

crates/smoketests/tests/auto_inc.rs

+#[test]
+fn test_autoinc_i64() {
+    do_test_autoinc_basic("i64");
+}


Instead of just testing these 4 here I would have 1 test which iterates over the array of types we used to test before:

"u8", "u16", "u32", "u64", "u128", "i8", "i16", "i32", "i64", "i128"

jdetter · 2026-01-23T09:39:53Z

crates/smoketests/tests/auto_inc.rs

+#[test]
+fn test_autoinc_unique_i64() {
+    do_test_autoinc_unique("i64");
+}


same here, we're only test 2 of the 10 we used to test before.

jdetter · 2026-01-23T09:51:08Z

crates/smoketests/tests/describe.rs

+    let identity = test.database_identity.as_ref().unwrap();
+
+    // Describe the whole module
+    test.spacetime(&["describe", "--json", identity]).unwrap();


I know we weren't doing it before - but we're not doing any output validation here. Possibly worth a TODO.

jdetter · 2026-01-23T10:12:57Z

crates/smoketests/tests/new_user_flow.rs

@@ -0,0 +1,63 @@
+//! Tests translated from smoketests/tests/new_user_flow.py


This test originally was testing to make sure that our tutorial isn't broken. Since our onboarding has changed we should probably update this test in the future. Probably worth dropping a TODO here.

The 80/20 here might just be replacing this with testing the templates to be honest

kim · 2026-01-23T09:41:53Z

crates/smoketests/src/lib.rs

+        // First, run spacetime build to compile the WASM module (separate from publish)
+        let build_start = Instant::now();
+        let cli_path = ensure_binaries_built();
+        let build_output = Command::new(&cli_path)


This takes 8-10s on my machine for every test case.

Could we not, instead of a fresh tempdir for each test on every run, put all the module cargo projects under CARGO_TARGET_TMPDIR?

I'm investigating this, because indeed building the modules takes the vast majority of the time of the smoke tests.

Not for nothing but also rewriting the smoke tests to run in TypeScript would publish them much faster. Although probably we should have both languages so that we're smoke testing v8.

kim · 2026-01-23T11:08:50Z

crates/smoketests/tests/add_remove_index.rs

@@ -0,0 +1,89 @@
+//! Add/remove index tests translated from smoketests/tests/add_remove_index.py


I don't love that each test is linked into its own executable.
It's also possible to structure it like this:

tests/ smoketests.rs smoketests/ mod.rs add_remove_index.rs auto_inc.rs ...

Which will create a single executable and save a few seconds on link time.
The namespacing will also make it nicer to filter subsets of tests to run, which has been a frequent source of frustration with the python smoketests.

kim · 2026-01-23T11:25:13Z

crates/guard/src/lib.rs

+/// returning the path to the CLI binary.
+///
+/// This is useful for tests that need to run CLI commands directly.
+pub fn ensure_binaries_built() -> PathBuf {


Not so great about this approach is that there is no feedback as to what it is going on, all while the test runner tells the user that the tests are running for over 60s (in yellow even, when using nextest).

I'd also expect recursive cargo to create problems on certain systems, despite the env var hack.

Why not just create an xtask for running smoketests? Arguably, typing cargo smoketests isn't worse than cargo test -p spacetimedb-smoketests.

Invoking cargo from an xtask is fine, plus we can customize the output however we please.

We can fix up the feedback with cargo nextest, so at least that I can resolve.

I'm not opposed to creating an xtask per se, I just wanted it to be so that any schmo could run all of our tests without needing to know about our special special tools (i.e. cargo test runs all the tests), but I could be convinced.

In any case this PR is not a regression since I'm largely reusing existing machinery. I would request that we change that up in a separate PR.

- Add --config-path to spacetime_local() for test isolation - Fix new_identity() to not pass server arg to logout (matches Python) - Insert --server flag before -- separator in spacetime_cmd() - Update servers.rs to use spacetime_local() for local-only commands - Simplify test files by removing redundant publish_module() calls All 56 smoketests now pass.

Translate smoketests/tests/quickstart.py to Rust. This test validates that the quickstart documentation is correct by extracting code from markdown docs and running it. - Add parse_quickstart() to parse code blocks from markdown with CRLF handling - Add have_pnpm() to check for pnpm availability - Implement QuickstartTest with support for Rust, C#, and TypeScript servers - Rust test passes; C#/TypeScript skip gracefully if dependencies unavailable

…s' of github.com:clockworklabs/SpacetimeDB into tyler/translate-smoketests

Translate server restart tests from smoketests/tests/zz_docker.py to Rust. These tests verify SpacetimeDB behavior across server restarts: - Data persistence (test_restart_module) - SQL queries after restart (test_restart_sql) - Client auto-disconnection (test_restart_auto_disconnect) - Autoinc sequence integrity (test_add_remove_index_after_restart) Infrastructure changes: - Add data_dir and restart() to SpacetimeDbGuard - Add restart_server() to Smoketest - Consolidate duplicated kill/spawn logic into helpers

Simplify the Smoketest API by removing the spacetime_local method. Now spacetime() doesn't auto-add --server, and callers pass it explicitly when needed. This makes the API more consistent and explicit.

cloutiertyler added 5 commits January 22, 2026 21:25

cloutiertyler changed the title ~~Tyler/translate smoketests~~ Translate smoketests from Python to Rust Jan 23, 2026

cloutiertyler added 4 commits January 23, 2026 01:10

Add views and auto_migration smoketest translations

bea9069

Translate tests for: - views.rs: st_view_* system tables, namespace collisions, SQL views - auto_migration.rs: schema changes, add table migration

cloutiertyler force-pushed the tyler/translate-smoketests branch from bda7652 to bea9069 Compare January 23, 2026 07:06

cloutiertyler added 2 commits January 23, 2026 02:11

Add new_user_flow and servers smoketest translations

60cba8d

Translate tests for: - new_user_flow.rs: Basic publish/call/SQL workflow - servers.rs: Server add/list/edit commands

jdetter assigned cloutiertyler Jan 23, 2026

cargo fmt + don't block on lints for now

afb0a71

jdetter reviewed Jan 23, 2026

View reviewed changes

kim reviewed Jan 23, 2026

View reviewed changes

cloutiertyler and others added 12 commits January 23, 2026 11:42

[tyler/translate-smoketests]: lints

dafa004

[tyler/translate-smoketests]: more lints

8b6506b

[tyler/translate-smoketests]: more lints

447413b

[tyler/translate-smoketests]: more lints

c18ff5c

[tyler/translate-smoketests]: update ci stuff

95308f2

[tyler/translate-smoketests]: fix build

ed2735e

[tyler/translate-smoketests]: CI fixes?

fdba9e5

[tyler/translate-smoketests]: ci

09b53de

[tyler/translate-smoketests]: windows CI

017a744

[tyler/translate-smoketests]: fix windows ci

8ebf8e6

bfops and others added 7 commits January 23, 2026 11:08

[tyler/translate-smoketests]: actually fix windows ci

03bbd19

[tyler/translate-smoketests]: Merge branch 'tyler/translate-smoketest…

828cc2d

…s' of github.com:clockworklabs/SpacetimeDB into tyler/translate-smoketests

cargo fmt --all

157a814

[tyler/translate-smoketests]: fix lints

f658b20

[tyler/translate-smoketests]: lints

b8c31a9

[tyler/translate-smoketests]: lints

da951e9

cloutiertyler marked this pull request as ready for review January 23, 2026 20:19

bfops and others added 10 commits January 23, 2026 12:24

[tyler/translate-smoketests]: lints

70bae5b

[tyler/translate-smoketests]: slim lint

61cf2b8

Fix clippy warnings in guard crate

a2db6af

Remove spacetime_local in favor of explicit --server flag

ab70631

Simplify the Smoketest API by removing the spacetime_local method. Now spacetime() doesn't auto-add --server, and callers pass it explicitly when needed. This makes the API more consistent and explicit.

cargo fmt

bd146c5

Merge branch 'master' into tyler/translate-smoketests

147f273

temp fix

ba1f992

re-enable lints

3c69f75

fix

0f343aa

Merge branch 'master' into tyler/translate-smoketests

27be9ed

		@@ -0,0 +1,63 @@
		//! Tests translated from smoketests/tests/new_user_flow.py

		@@ -0,0 +1,89 @@
		//! Add/remove index tests translated from smoketests/tests/add_remove_index.py

Translate smoketests from Python to Rust #4102

Are you sure you want to change the base?

Translate smoketests from Python to Rust #4102

Uh oh!

Conversation

cloutiertyler commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of Changes

Motivation

API and ABI breaking changes

Expected complexity level and risk

Testing

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jdetter left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

cloutiertyler commented Jan 23, 2026 •

edited

Loading