diff --git a/docs/docs/00100-intro/00300-tutorials/00300-unity-tutorial/00300-part-2.md b/docs/docs/00100-intro/00300-tutorials/00300-unity-tutorial/00300-part-2.md index 9a5f26694ec..03c579a088a 100644 --- a/docs/docs/00100-intro/00300-tutorials/00300-unity-tutorial/00300-part-2.md +++ b/docs/docs/00100-intro/00300-tutorials/00300-unity-tutorial/00300-part-2.md @@ -495,16 +495,16 @@ The `spacetime` CLI has built in functionality to let us generate C# types that directory run the following command: - Let's generate our types for our module. In the `blackholio/server-rust` + Let's generate our types for our module. In the `blackholio/spacetimedb` directory run the following command: ```sh -spacetime generate --lang csharp --out-dir ../client-unity/Assets/autogen # you can call this anything, I have chosen `autogen` +spacetime generate --lang csharp --out-dir ../Assets/autogen ``` -This will generate a set of files in the `client-unity/Assets/autogen` directory which contain the code generated types and reducer functions that are defined in your module, but usable on the client. +This will generate a set of files in the `Assets/autogen` directory which contain the code generated types and reducer functions that are defined in your module, but usable on the client. ``` ├── Reducers @@ -525,7 +525,7 @@ This will generate a set of files in the `client-unity/Assets/autogen` directory └── SpacetimeDBClient.g.cs ``` -This will also generate a file in the `client-unity/Assets/autogen/SpacetimeDBClient.g.cs` directory with a type aware `DbConnection` class. We will use this class to connect to your database from Unity. +This will also generate a file in the `Assets/autogen/SpacetimeDBClient.g.cs` directory with a type aware `DbConnection` class. We will use this class to connect to your database from Unity. > IMPORTANT! At this point there will be an error in your Unity project. Due to a [known issue](https://docs.unity3d.com/6000.0/Documentation/Manual/csharp-compiler.html) with Unity and C# 9 you need to insert the following code into your Unity project. > diff --git a/docs/docs/00100-intro/00300-tutorials/00300-unity-tutorial/00500-part-4.md b/docs/docs/00100-intro/00300-tutorials/00300-unity-tutorial/00500-part-4.md index f2ad998a52b..d00bd48fcf9 100644 --- a/docs/docs/00100-intro/00300-tutorials/00300-unity-tutorial/00500-part-4.md +++ b/docs/docs/00100-intro/00300-tutorials/00300-unity-tutorial/00500-part-4.md @@ -64,7 +64,7 @@ This is a simple reducer that takes the movement input from the client and appli -Let's start by building out a simple math library to help us do collision calculations. Create a new `math.rs` file in the `server-rust/src` directory and add the following contents. Let's also move the `DbVector2` type from `lib.rs` into this file. +Let's start by building out a simple math library to help us do collision calculations. Create a new `math.rs` file in the `spacetimedb/src` directory and add the following contents. Let's also move the `DbVector2` type from `lib.rs` into this file. ```rust use spacetimedb::SpacetimeType; diff --git a/docs/docs/00200-core-concepts/00100-databases/00100-transactions-atomicity.md b/docs/docs/00200-core-concepts/00100-databases/00100-transactions-atomicity.md index 7324e3cf298..f1acc2c3b47 100644 --- a/docs/docs/00200-core-concepts/00100-databases/00100-transactions-atomicity.md +++ b/docs/docs/00200-core-concepts/00100-databases/00100-transactions-atomicity.md @@ -194,7 +194,7 @@ SpacetimeDB does not support nested transactions. When one reducer calls another The `#[auto_inc]` sequence generator is not transactional: - Sequence numbers are allocated even if a transaction rolls back - This can create gaps in your sequence -- See [SEQUENCE documentation](/reference/appendix#sequence) for details +- See [Auto-Increment](/tables/auto-increment#crash-recovery) for details ## Related Topics diff --git a/docs/docs/00200-core-concepts/00100-databases/00500-cheat-sheet.md b/docs/docs/00200-core-concepts/00100-databases/00500-cheat-sheet.md index 5377a63cdd9..529e79d6ec4 100644 --- a/docs/docs/00200-core-concepts/00100-databases/00500-cheat-sheet.md +++ b/docs/docs/00200-core-concepts/00100-databases/00500-cheat-sheet.md @@ -383,7 +383,24 @@ spacetimedb.procedure( ```csharp -// C# procedure support coming soon +// Add #pragma warning disable STDB_UNSTABLE at file top + +[SpacetimeDB.Procedure] +public static string FetchData(ProcedureContext ctx, string url) +{ + var result = ctx.Http.Get(url); + if (result is Result.OkR(var response)) + { + var data = response.Body.ToStringUtf8Lossy(); + ctx.WithTx(txCtx => + { + txCtx.Db.Cache.Insert(new Cache { Data = data }); + return 0; + }); + return data; + } + return ""; +} ``` diff --git a/docs/docs/00200-core-concepts/00200-functions/00300-reducers/00300-reducers.md b/docs/docs/00200-core-concepts/00200-functions/00300-reducers/00300-reducers.md index 2ca4be58926..3296c4bb96b 100644 --- a/docs/docs/00200-core-concepts/00200-functions/00300-reducers/00300-reducers.md +++ b/docs/docs/00200-core-concepts/00200-functions/00300-reducers/00300-reducers.md @@ -404,13 +404,22 @@ Reducers run in an isolated environment and **cannot** interact with the outside If you need to interact with external systems, use [Procedures](/functions/procedures) instead. Procedures can make network calls and perform other side effects, but they have different execution semantics and limitations. -:::warning Global and Static Variables Do Not Persist -Global variables, static variables, and module-level state do **not** persist across reducer calls. Each reducer invocation runs in a fresh execution environment. Any data stored in global or static variables will be lost when the reducer completes. +:::warning Global and Static Variables Are Undefined Behavior +Relying on global variables, static variables, or module-level state to persist across reducer calls is **undefined behavior**. SpacetimeDB does not guarantee that values stored in these locations will be available in subsequent reducer invocations. -Always store persistent state in tables. If you need to cache computed values or maintain state across invocations, use a table to store that data. +This is undefined for several reasons: + +1. **Fresh execution environments.** SpacetimeDB may run each reducer in a fresh WASM or JS instance. +2. **Module updates.** Publishing a new module creates a fresh execution environment. This is necessary for hot-swapping modules while transactions are in flight. +3. **Concurrent execution.** SpacetimeDB reserves the right to execute multiple reducers concurrently in separate execution environments (e.g., with MVCC). +4. **Crash recovery.** Instance memory is not persisted across restarts. +5. **Non-transactional updates.** If you modify global state and then roll back the transaction, the modified value may remain for subsequent transactions. +6. **Replay safety.** If a serializability anomaly is detected, SpacetimeDB may re-execute your reducer with the same arguments, causing modifications to global state to occur multiple times. + +Reducers are designed to be free of side effects. They should only modify tables. Always store state in tables to ensure correctness and durability. ```rust -// ❌ This will NOT persist across reducer calls +// ❌ Undefined behavior: may or may not persist or correctly update across reducer calls static mut COUNTER: u64 = 0; // ✅ Store state in a table instead @@ -423,6 +432,132 @@ pub struct Counter { ``` ::: +## Scheduling Procedures + +Reducers cannot call procedures directly (procedures may have side effects incompatible with transactional execution). Instead, schedule a procedure to run by inserting into a [schedule table](/tables/schedule-tables): + + + + +```typescript +import { schema, t, table, SenderError } from 'spacetimedb/server'; + +// Define a schedule table for the procedure +const fetchSchedule = table( + { name: 'fetch_schedule', scheduled: 'fetch_external_data' }, + { + scheduled_id: t.u64().primaryKey().autoInc(), + scheduled_at: t.scheduleAt(), + url: t.string(), + } +); + +const spacetimedb = schema(fetchSchedule); + +// The procedure to be scheduled +const fetchExternalData = spacetimedb.procedure( + 'fetch_external_data', + { arg: fetchSchedule.rowType }, + t.unit(), + (ctx, { arg }) => { + const response = ctx.http.fetch(arg.url); + // Process response... + return {}; + } +); + +// From a reducer, schedule the procedure by inserting into the schedule table +const queueFetch = spacetimedb.reducer('queue_fetch', { url: t.string() }, (ctx, { url }) => { + ctx.db.fetchSchedule.insert({ + scheduled_id: 0n, + scheduled_at: ScheduleAt.interval(0n), // Run immediately + url, + }); +}); +``` + + + + +```csharp +#pragma warning disable STDB_UNSTABLE +using SpacetimeDB; + +public partial class Module +{ + [SpacetimeDB.Table(Name = "FetchSchedule", Scheduled = "FetchExternalData", ScheduledAt = "ScheduledAt")] + public partial struct FetchSchedule + { + [SpacetimeDB.PrimaryKey] + [SpacetimeDB.AutoInc] + public ulong ScheduledId; + public ScheduleAt ScheduledAt; + public string Url; + } + + [SpacetimeDB.Procedure] + public static void FetchExternalData(ProcedureContext ctx, FetchSchedule schedule) + { + var result = ctx.Http.Get(schedule.Url); + if (result is Result.OkR(var response)) + { + // Process response... + } + } + + // From a reducer, schedule the procedure + [SpacetimeDB.Reducer] + public static void QueueFetch(ReducerContext ctx, string url) + { + ctx.Db.FetchSchedule.Insert(new FetchSchedule + { + ScheduledId = 0, + ScheduledAt = new ScheduleAt.Interval(TimeSpan.Zero), + Url = url, + }); + } +} +``` + + + + +```rust +use spacetimedb::{ScheduleAt, ReducerContext, ProcedureContext, Table}; +use std::time::Duration; + +#[spacetimedb::table(name = fetch_schedule, scheduled(fetch_external_data))] +pub struct FetchSchedule { + #[primary_key] + #[auto_inc] + scheduled_id: u64, + scheduled_at: ScheduleAt, + url: String, +} + +#[spacetimedb::procedure] +fn fetch_external_data(ctx: &mut ProcedureContext, schedule: FetchSchedule) { + if let Ok(response) = ctx.http.get(&schedule.url) { + // Process response... + } +} + +// From a reducer, schedule the procedure +#[spacetimedb::reducer] +fn queue_fetch(ctx: &ReducerContext, url: String) { + ctx.db.fetch_schedule().insert(FetchSchedule { + scheduled_id: 0, + scheduled_at: ScheduleAt::Interval(Duration::ZERO.into()), + url, + }); +} +``` + + + + +See [Schedule Tables](/tables/schedule-tables) for more scheduling options. + ## Next Steps - Learn about [Tables](/tables) to understand data storage diff --git a/docs/docs/00200-core-concepts/00200-functions/00400-procedures.md b/docs/docs/00200-core-concepts/00200-functions/00400-procedures.md index 872a3396674..51af1e5a736 100644 --- a/docs/docs/00200-core-concepts/00200-functions/00400-procedures.md +++ b/docs/docs/00200-core-concepts/00200-functions/00400-procedures.md @@ -45,9 +45,25 @@ a value corresponding to its return type. This return value will be sent to the not be broadcast to any other clients. - + + +:::warning Unstable Feature +Procedures in C# are currently unstable. To use them, add `#pragma warning disable STDB_UNSTABLE` at the top of your file. +::: + +Define a procedure by annotating a static method with `[SpacetimeDB.Procedure]`. -Support for procedures in C# modules is coming soon! +The method's first argument must be of type `ProcedureContext`. A procedure may accept any number of additional arguments and may return a value. + +```csharp +#pragma warning disable STDB_UNSTABLE + +[SpacetimeDB.Procedure] +public static ulong AddTwoNumbers(ProcedureContext ctx, uint lhs, uint rhs) +{ + return (ulong)lhs + (ulong)rhs; +} +``` @@ -132,6 +148,58 @@ values observed during prior runs must not influence the behavior of the functio Avoid capturing mutable state within functions passed to `withTx`. ::: + + + +Unlike reducers, procedures don't automatically run in database transactions. +This means there's no `ctx.Db` field to access the database. +Instead, procedure code must manage transactions explicitly with `ProcedureContext.WithTx`. + +```csharp +#pragma warning disable STDB_UNSTABLE +using SpacetimeDB; + +public static partial class Module +{ + [SpacetimeDB.Table(Name = "MyTable")] + public partial struct MyTable + { + public uint A; + public string B; + } + + [SpacetimeDB.Procedure] + public static void InsertAValue(ProcedureContext ctx, uint a, string b) + { + ctx.WithTx(txCtx => + { + txCtx.Db.MyTable.Insert(new MyTable { A = a, B = b }); + return 0; + }); + } +} +``` + +`ProcedureContext.WithTx` takes a function of type `Func`. +Within that function, the `TransactionContext` can be used to access the database +[in all the same ways as a `ReducerContext`](/functions/reducers/reducer-context). +When the function returns, the transaction will be committed, +and its changes to the database state will become permanent and be broadcast to clients. +If the function throws an exception, the transaction will be rolled back, and its changes will be discarded. + +:::warning +The function passed to `ProcedureContext.WithTx` may be invoked multiple times, +possibly seeing a different version of the database state each time. + +If invoked more than once with reference to the same database state, +it must perform the same operations and return the same result each time. + +If invoked more than once with reference to different database states, +values observed during prior runs must not influence the behavior of the function or the calling procedure. + +Avoid capturing mutable state within functions passed to `WithTx`. +::: + @@ -197,6 +265,33 @@ spacetimedb.procedure("maybe_insert_a_value", { a: t.u32(), b: t.string() }, t.u }) ``` + + + +For fallible database operations, you can throw an exception inside the transaction function: + +```csharp +#pragma warning disable STDB_UNSTABLE +using SpacetimeDB; + +public static partial class Module +{ + [SpacetimeDB.Procedure] + public static void MaybeInsertAValue(ProcedureContext ctx, uint a, string b) + { + ctx.WithTx(txCtx => + { + if (a < 10) + { + throw new Exception("a is less than 10!"); + } + txCtx.Db.MyTable.Insert(new MyTable { A = a, B = b }); + return 0; + }); + } +} +``` + @@ -261,6 +356,56 @@ spacetimedb.procedure("find_highest_level_player", t.unit(), ctx => { }); ``` + + + +Functions passed to +[`ProcedureContext.WithTx`](#accessing-the-database) +may return a value, and that value will be returned to the calling procedure. + +Transaction return values are never saved or broadcast to clients, and are used only by the calling procedure. + +```csharp +#pragma warning disable STDB_UNSTABLE +using SpacetimeDB; + +public static partial class Module +{ + [SpacetimeDB.Table(Name = "Player")] + public partial struct Player + { + public Identity Id; + public uint Level; + } + + [SpacetimeDB.Procedure] + public static void FindHighestLevelPlayer(ProcedureContext ctx) + { + var highestLevelPlayer = ctx.WithTx(txCtx => + { + Player? highest = null; + foreach (var player in txCtx.Db.Player.Iter()) + { + if (highest == null || player.Level > highest.Value.Level) + { + highest = player; + } + } + return highest; + }); + + if (highestLevelPlayer.HasValue) + { + Log.Info($"Congratulations to {highestLevelPlayer.Value.Id}"); + } + else + { + Log.Warn("No players..."); + } + } +} +``` + @@ -352,6 +497,75 @@ spacetimedb.procedure("get_request_with_short_timeout", t.unit(), ctx => { Procedures can't send requests at the same time as holding open a [transaction](#accessing-the-database). + + + +Procedures can make HTTP requests to external services using methods on `ctx.Http`. + +`ctx.Http.Get` performs simple `GET` requests with no headers: + +```csharp +#pragma warning disable STDB_UNSTABLE +using SpacetimeDB; + +public static partial class Module +{ + [SpacetimeDB.Procedure] + public static void GetRequest(ProcedureContext ctx) + { + var result = ctx.Http.Get("https://example.invalid"); + switch (result) + { + case Result.OkR(var response): + var body = response.Body.ToStringUtf8Lossy(); + Log.Info($"Got response with status {response.StatusCode} and body {body}"); + break; + case Result.ErrR(var e): + Log.Error($"Request failed: {e.Message}"); + break; + } + } +} +``` + +`ctx.Http.Send` sends an `HttpRequest` with custom method, headers, and body: + +```csharp +#pragma warning disable STDB_UNSTABLE +using SpacetimeDB; + +public static partial class Module +{ + [SpacetimeDB.Procedure] + public static void PostRequest(ProcedureContext ctx) + { + var request = new HttpRequest + { + Method = SpacetimeDB.HttpMethod.Post, + Uri = "https://example.invalid/upload", + Headers = new List + { + new HttpHeader("Content-Type", "text/plain") + }, + Body = HttpBody.FromString("This is the body of the HTTP request") + }; + var result = ctx.Http.Send(request); + switch (result) + { + case Result.OkR(var response): + var body = response.Body.ToStringUtf8Lossy(); + Log.Info($"Got response with status {response.StatusCode} and body {body}"); + break; + case Result.ErrR(var e): + Log.Error($"Request failed: {e.Message}"); + break; + } + } +} +``` + +Procedures can't send requests at the same time as holding open a [transaction](#accessing-the-database). + @@ -428,6 +642,112 @@ Procedures can't send requests at the same time as holding open a [transaction]( +## Calling Reducers from Procedures + +Procedures can call reducers by invoking them within a transaction block. The reducer function runs within the transaction context: + + + + +```typescript +// Define a reducer and save the reference +const processItem = spacetimedb.reducer('process_item', { itemId: t.u64() }, (ctx, { itemId }) => { + // ... reducer logic +}); + +// Call it from a procedure using the saved reference +spacetimedb.procedure('fetch_and_process', { url: t.string() }, t.unit(), (ctx, { url }) => { + // Fetch external data + const response = ctx.http.fetch(url); + const data = response.json(); + + // Call the reducer within a transaction + ctx.withTx(txCtx => { + processItem(txCtx, { itemId: data.id }); + }); + + return {}; +}); +``` + + + + +```csharp +#pragma warning disable STDB_UNSTABLE +using SpacetimeDB; + +public static partial class Module +{ + // Note: In C#, you can define helper methods that work with the transaction context + // rather than calling reducers directly. + private static void ProcessItemLogic(ulong itemId) + { + // ... item processing logic + } + + [SpacetimeDB.Procedure] + public static void FetchAndProcess(ProcedureContext ctx, string url) + { + // Fetch external data + var result = ctx.Http.Get(url); + var response = result.UnwrapOrThrow(); + var body = response.Body.ToStringUtf8Lossy(); + var itemId = ParseId(body); + + // Process within a transaction + ctx.WithTx(txCtx => + { + ProcessItemLogic(itemId); + return 0; + }); + } + + private static ulong ParseId(string body) + { + // Parse the ID from the response body + return ulong.Parse(body); + } +} +``` + + + + +```rust +#[spacetimedb::reducer] +fn process_item(ctx: &ReducerContext, item_id: u64) { + // ... reducer logic +} + +#[spacetimedb::procedure] +fn fetch_and_process(ctx: &mut ProcedureContext, url: String) -> Result<(), String> { + // Fetch external data + let response = ctx.http.get(&url).map_err(|e| format!("{e:?}"))?; + let (_, body) = response.into_parts(); + let item_id: u64 = parse_id(&body.into_string_lossy()); + + // Call the reducer within a transaction + ctx.with_tx(|tx_ctx| { + process_item(tx_ctx, item_id); + }); + + Ok(()) +} +``` + + + + +:::note +When you call a reducer function inside `withTx`, it executes as part of the same transaction, not as a subtransaction. The reducer's logic runs inline within your anonymous transaction block, just like calling any other helper function. +::: + +This pattern is useful when you need to: +- Fetch external data and then process it transactionally +- Reuse existing reducer logic from a procedure +- Combine side effects (HTTP) with database operations + ## Calling procedures @@ -635,6 +955,90 @@ spacetimedb.procedure( ); ``` + + + +```csharp +#pragma warning disable STDB_UNSTABLE +using SpacetimeDB; +using System.Text.Json; + +public static partial class Module +{ + [SpacetimeDB.Table(Name = "AiMessage", Public = true)] + public partial struct AiMessage + { + public Identity User; + public string Prompt; + public string Response; + public Timestamp CreatedAt; + } + + [SpacetimeDB.Procedure] + public static string AskAi(ProcedureContext ctx, string prompt, string apiKey) + { + // Build the request to OpenAI's API + var requestBody = JsonSerializer.Serialize(new + { + model = "gpt-4", + messages = new[] { new { role = "user", content = prompt } } + }); + + var request = new HttpRequest + { + Method = SpacetimeDB.HttpMethod.Post, + Uri = "https://api.openai.com/v1/chat/completions", + Headers = new List + { + new HttpHeader("Content-Type", "application/json"), + new HttpHeader("Authorization", $"Bearer {apiKey}") + }, + Body = HttpBody.FromString(requestBody) + }; + + // Make the HTTP request + var response = ctx.Http.Send(request).UnwrapOrThrow(); + + if (response.StatusCode != 200) + { + throw new Exception($"API returned status {response.StatusCode}"); + } + + var bodyStr = response.Body.ToStringUtf8Lossy(); + + // Parse the response + var aiResponse = ExtractContent(bodyStr) + ?? throw new Exception("Failed to parse AI response"); + + // Store the conversation in the database + ctx.WithTx(txCtx => + { + txCtx.Db.AiMessage.Insert(new AiMessage + { + User = txCtx.Sender, + Prompt = prompt, + Response = aiResponse, + CreatedAt = txCtx.Timestamp + }); + return 0; + }); + + return aiResponse; + } + + private static string? ExtractContent(string json) + { + // Simple extraction - in production, use proper JSON parsing + var doc = JsonDocument.Parse(json); + return doc.RootElement + .GetProperty("choices")[0] + .GetProperty("message") + .GetProperty("content") + .GetString(); + } +} +``` + @@ -720,6 +1124,23 @@ const response = await ctx.procedures.askAi({ console.log("AI says:", response); ``` + + + +```csharp +ctx.Procedures.AskAi("What is SpacetimeDB?", apiKey, (ctx, result) => +{ + if (result.IsSuccess) + { + Console.WriteLine($"AI says: {result.Value}"); + } + else + { + Console.WriteLine($"Error: {result.Error}"); + } +}); +``` + diff --git a/docs/docs/00200-core-concepts/00300-tables.md b/docs/docs/00200-core-concepts/00300-tables.md index c9a3405a2bc..763bbcd8df7 100644 --- a/docs/docs/00200-core-concepts/00300-tables.md +++ b/docs/docs/00200-core-concepts/00300-tables.md @@ -160,6 +160,10 @@ pub struct Person { } ``` +:::note Rust Visibility vs SpacetimeDB Visibility +The `pub` modifier on the struct follows normal Rust visibility rules and has no meaning to SpacetimeDB. It controls whether the struct is accessible from other Rust modules in your crate, not whether the table is public to clients. Use the `public` attribute in `#[spacetimedb::table]` to control client visibility. +::: + diff --git a/docs/docs/00200-core-concepts/00300-tables/00210-file-storage.md b/docs/docs/00200-core-concepts/00300-tables/00210-file-storage.md index 0925140db83..5e0f51551b4 100644 --- a/docs/docs/00200-core-concepts/00300-tables/00210-file-storage.md +++ b/docs/docs/00200-core-concepts/00300-tables/00210-file-storage.md @@ -127,20 +127,20 @@ pub fn upload_avatar( Inline storage works well for: -- **Small to medium files** (up to a few megabytes) +- **Files up to ~100MB** - **Data that changes with other row fields** (e.g., user profile with avatar) - **Data requiring transactional consistency** (file updates atomic with metadata) - **Data clients need through subscriptions** (real-time avatar updates) ### Size Considerations -Each row has practical size limits. Very large binary data affects: +Very large binary data affects: - **Memory usage**: Rows are held in memory during reducer execution - **Network bandwidth**: Large rows increase subscription traffic - **Transaction size**: Large rows slow down transaction commits -For files larger than a few megabytes, consider external storage. +For very large files (over 100MB), consider external storage. ## External Storage with References @@ -298,6 +298,373 @@ A typical external storage flow: This pattern keeps large files out of SpacetimeDB while maintaining metadata in the database for queries and subscriptions. +### Example: Uploading to S3 from a Procedure + +[Procedures](/functions/procedures) can make HTTP requests, enabling direct uploads to external storage services like S3. This example shows uploading a file to S3 and storing the metadata in SpacetimeDB: + + + + +```typescript +import { table, t, schema, SenderError } from 'spacetimedb/server'; + +const document = table( + { name: 'document', public: true }, + { + id: t.u64().primaryKey().autoInc(), + ownerId: t.identity(), + filename: t.string(), + s3Key: t.string(), + uploadedAt: t.timestamp(), + } +); + +const spacetimedb = schema(document); + +// Upload file to S3 and register in database +spacetimedb.procedure( + 'upload_to_s3', + { + filename: t.string(), + contentType: t.string(), + data: t.array(t.u8()), + s3Bucket: t.string(), + s3Region: t.string(), + }, + t.string(), // Returns the S3 key + (ctx, { filename, contentType, data, s3Bucket, s3Region }) => { + // Generate a unique S3 key + const s3Key = `uploads/${Date.now()}-${filename}`; + const url = `https://${s3Bucket}.s3.${s3Region}.amazonaws.com/${s3Key}`; + + // Upload to S3 (simplified - add AWS4 signature in production) + const response = ctx.http.fetch(url, { + method: 'PUT', + headers: { + 'Content-Type': contentType, + 'x-amz-content-sha256': 'UNSIGNED-PAYLOAD', + // Add Authorization header with AWS4 signature + }, + body: new Uint8Array(data), + }); + + if (response.status !== 200) { + throw new SenderError(`S3 upload failed: ${response.status}`); + } + + // Store metadata in database + ctx.withTx(txCtx => { + txCtx.db.document.insert({ + id: 0n, + ownerId: txCtx.sender, + filename, + s3Key, + uploadedAt: txCtx.timestamp, + }); + }); + + return s3Key; + } +); +``` + + + + +```csharp +#pragma warning disable STDB_UNSTABLE +using SpacetimeDB; + +public static partial class Module +{ + [SpacetimeDB.Table(Name = "Document", Public = true)] + public partial struct Document + { + [SpacetimeDB.PrimaryKey] + [SpacetimeDB.AutoInc] + public ulong Id; + public Identity OwnerId; + public string Filename; + public string S3Key; + public Timestamp UploadedAt; + } + + // Upload file to S3 and register in database + [SpacetimeDB.Procedure] + public static string UploadToS3( + ProcedureContext ctx, + string filename, + string contentType, + List data, + string s3Bucket, + string s3Region) + { + // Generate a unique S3 key + var timestamp = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds(); + var s3Key = $"uploads/{timestamp}-{filename}"; + var url = $"https://{s3Bucket}.s3.{s3Region}.amazonaws.com/{s3Key}"; + + // Build the S3 PUT request (simplified - add AWS4 signature in production) + var request = new HttpRequest + { + Uri = url, + Method = SpacetimeDB.HttpMethod.Put, + Headers = new List + { + new HttpHeader("Content-Type", contentType), + new HttpHeader("x-amz-content-sha256", "UNSIGNED-PAYLOAD"), + // Add Authorization header with AWS4 signature + }, + Body = new HttpBody(data.ToArray()), + }; + + // Upload to S3 + var response = ctx.Http.Send(request).UnwrapOrThrow(); + + if (response.StatusCode != 200) + { + throw new Exception($"S3 upload failed with status: {response.StatusCode}"); + } + + // Store metadata in database + ctx.WithTx(txCtx => + { + txCtx.Db.Document.Insert(new Document + { + Id = 0, + OwnerId = txCtx.Sender, + Filename = filename, + S3Key = s3Key, + UploadedAt = txCtx.Timestamp, + }); + return 0; + }); + + return s3Key; + } +} +``` + + + + +```rust +use spacetimedb::{Identity, ProcedureContext, Timestamp, Table}; + +#[spacetimedb::table(name = document, public)] +pub struct Document { + #[primary_key] + #[auto_inc] + id: u64, + owner_id: Identity, + filename: String, + s3_key: String, + uploaded_at: Timestamp, +} + +// Upload file to S3 and register in database +#[spacetimedb::procedure] +pub fn upload_to_s3( + ctx: &mut ProcedureContext, + filename: String, + content_type: String, + data: Vec, + s3_bucket: String, + s3_region: String, +) -> Result { + // Generate a unique S3 key + let timestamp = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_millis(); + let s3_key = format!("uploads/{}-{}", timestamp, filename); + let url = format!( + "https://{}.s3.{}.amazonaws.com/{}", + s3_bucket, s3_region, s3_key + ); + + // Build the S3 PUT request (simplified - add AWS4 signature in production) + let request = spacetimedb::http::Request::builder() + .uri(&url) + .method("PUT") + .header("Content-Type", &content_type) + .header("x-amz-content-sha256", "UNSIGNED-PAYLOAD") + // Add Authorization header with AWS4 signature + .body(data) + .map_err(|e| format!("Failed to build request: {}", e))?; + + // Upload to S3 + let response = ctx.http.send(request) + .map_err(|e| format!("S3 upload failed: {:?}", e))?; + + let (parts, _body) = response.into_parts(); + if parts.status != 200 { + return Err(format!("S3 upload failed with status: {}", parts.status)); + } + + // Store metadata in database + let s3_key_clone = s3_key.clone(); + let filename_clone = filename.clone(); + ctx.with_tx(|tx_ctx| { + tx_ctx.db.document().insert(Document { + id: 0, + owner_id: tx_ctx.sender, + filename: filename_clone.clone(), + s3_key: s3_key_clone.clone(), + uploaded_at: tx_ctx.timestamp, + }); + }); + + Ok(s3_key) +} +``` + + + + +:::note AWS Authentication +The example above is simplified. Production S3 uploads require proper [AWS Signature Version 4](https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html) authentication. +::: + +### Alternative: Pre-signed URL Flow + +For larger files, generate a pre-signed URL and let the client upload directly: + + + + +```typescript +// Procedure returns a pre-signed URL for client-side upload +spacetimedb.procedure( + 'get_upload_url', + { filename: t.string(), contentType: t.string() }, + t.object('UploadInfo', { uploadUrl: t.string(), s3Key: t.string() }), + (ctx, { filename, contentType }) => { + const s3Key = `uploads/${Date.now()}-${filename}`; + + // Generate pre-signed URL (requires AWS credentials and signing logic) + const uploadUrl = generatePresignedUrl(s3Key, contentType); + + return { uploadUrl, s3Key }; + } +); + +// Client uploads directly to S3 using the pre-signed URL, then calls: +spacetimedb.reducer('confirm_upload', { filename: t.string(), s3Key: t.string() }, (ctx, { filename, s3Key }) => { + ctx.db.document.insert({ + id: 0n, + ownerId: ctx.sender, + filename, + s3Key, + uploadedAt: ctx.timestamp, + }); +}); +``` + + + + +```csharp +#pragma warning disable STDB_UNSTABLE +using SpacetimeDB; + +public static partial class Module +{ + [SpacetimeDB.Type] + public partial struct UploadInfo + { + public string UploadUrl; + public string S3Key; + } + + // Procedure returns a pre-signed URL for client-side upload + [SpacetimeDB.Procedure] + public static UploadInfo GetUploadUrl( + ProcedureContext ctx, + string filename, + string contentType) + { + var timestamp = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds(); + var s3Key = $"uploads/{timestamp}-{filename}"; + + // Generate pre-signed URL (requires AWS credentials and signing logic) + var uploadUrl = GeneratePresignedUrl(s3Key, contentType); + + return new UploadInfo { UploadUrl = uploadUrl, S3Key = s3Key }; + } + + // Client uploads directly to S3 using the pre-signed URL, then calls: + [SpacetimeDB.Reducer] + public static void ConfirmUpload(ReducerContext ctx, string filename, string s3Key) + { + ctx.Db.Document.Insert(new Document + { + Id = 0, + OwnerId = ctx.Sender, + Filename = filename, + S3Key = s3Key, + UploadedAt = ctx.Timestamp, + }); + } + + private static string GeneratePresignedUrl(string s3Key, string contentType) + { + // Implement AWS S3 pre-signed URL generation + throw new NotImplementedException(); + } +} +``` + + + + +```rust +#[derive(SpacetimeType)] +pub struct UploadInfo { + upload_url: String, + s3_key: String, +} + +// Procedure returns a pre-signed URL for client-side upload +#[spacetimedb::procedure] +pub fn get_upload_url( + _ctx: &mut ProcedureContext, + filename: String, + _content_type: String, +) -> Result { + let timestamp = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_millis(); + let s3_key = format!("uploads/{}-{}", timestamp, filename); + + // Generate pre-signed URL (requires AWS credentials and signing logic) + let upload_url = generate_presigned_url(&s3_key)?; + + Ok(UploadInfo { upload_url, s3_key }) +} + +// Client uploads directly to S3 using the pre-signed URL, then calls: +#[spacetimedb::reducer] +pub fn confirm_upload(ctx: &ReducerContext, filename: String, s3_key: String) { + ctx.db.document().insert(Document { + id: 0, + owner_id: ctx.sender, + filename, + s3_key, + uploaded_at: ctx.timestamp, + }); +} +``` + + + + +The pre-signed URL approach is preferred for large files because: +- **No size limits**: Files don't pass through SpacetimeDB +- **Better performance**: Direct client-to-S3 transfer +- **Reduced load**: SpacetimeDB only handles metadata + ## Hybrid Approach: Thumbnails and Originals For images, store small thumbnails inline for fast access while keeping originals in external storage: @@ -380,10 +747,13 @@ This approach provides: | Scenario | Recommended Approach | |----------|---------------------| -| User avatars (< 100KB) | Inline storage | -| Chat attachments (< 1MB) | Inline storage | -| Document uploads (> 1MB) | External storage with reference | +| User avatars (< 10MB) | Inline storage | +| Chat attachments (< 50MB) | Inline storage | +| Document uploads (< 100MB) | Inline storage | +| Large files (> 100MB) | External storage with reference | | Video files | External storage with CDN | | Images with previews | Hybrid (inline thumbnail + external original) | +SpacetimeDB storage costs approximately $1/GB compared to cheaper blob storage options like AWS S3. For large files that don't need atomic updates with other data, external storage may be more economical. + The right choice depends on your file sizes, access patterns, and whether the data needs to participate in real-time subscriptions. diff --git a/docs/docs/00200-core-concepts/00300-tables/00230-auto-increment.md b/docs/docs/00200-core-concepts/00300-tables/00230-auto-increment.md index 7ba9ab238d0..04d5af717de 100644 --- a/docs/docs/00200-core-concepts/00300-tables/00230-auto-increment.md +++ b/docs/docs/00200-core-concepts/00300-tables/00230-auto-increment.md @@ -130,25 +130,104 @@ Sequences with negative increments wrap in the opposite direction. A sequence wi ### Crash Recovery -Sequences implement a crash recovery mechanism to ensure values are never reused after a database restart. Rather than persisting the current value after every increment, sequences allocate values in batches. +Sequences implement a crash recovery mechanism to ensure values are never reused after a database restart. Rather than persisting the current value after every increment, sequences allocate values in batches of **4096**. When a sequence needs a new value and has exhausted its current allocation, it: -1. Calculates the next batch of values +1. Calculates the next batch of 4096 values 2. Persists the allocation boundary to disk 3. Returns values from the allocated range -If the database crashes, it restarts from the persisted allocation boundary. This may skip some values that were allocated but never used, but guarantees that no value is ever assigned twice. +If the database crashes or restarts, it resumes from the next allocation boundary. This may skip values that were allocated but never used, but guarantees that no value is ever assigned twice. -For example, if a sequence allocates values in batches of 10: +**Example:** -1. First insert triggers allocation of values 1-10 -2. Values 1, 2, 3 are used -3. Database crashes -4. On restart, the sequence resumes from value 1 (the allocation boundary) -5. The sequence allocates values 1-10 again, but now starts fresh + + + +```typescript +const user = table( + { name: 'user', public: true }, + { + user_id: t.u64().autoInc(), + name: t.string(), + } +); + +spacetimedb.reducer('insert_user', { name: t.string() }, (ctx, { name }) => { + ctx.db.user.insert({ user_id: 0n, name }); +}); +``` + + + + +```csharp +public partial class Module +{ + [SpacetimeDB.Table(Name = "user", Public = true)] + public partial struct User + { + [SpacetimeDB.AutoInc] + public ulong UserId; + public string Name; + } + + [SpacetimeDB.Reducer] + public static void InsertUser(ReducerContext ctx, string name) + { + ctx.Db.User.Insert(new User { UserId = 0, Name = name }); + } +``` + + + + +```rust +#[spacetimedb::table(name = user, public)] +pub struct User { + #[auto_inc] + user_id: u64, + name: String, +} + +#[spacetimedb::reducer] +pub fn insert_user(ctx: &ReducerContext, name: String) { + ctx.db.user().insert(User { user_id: 0, name }); +} +``` + + + + +```bash +# Insert 3 users +$ spacetime call mydb insert_user Alice +$ spacetime call mydb insert_user Bob +$ spacetime call mydb insert_user Carol + +$ spacetime sql mydb "SELECT * FROM user" + user_id | name +---------+------- + 1 | Alice + 2 | Bob + 3 | Carol + +# Database restarts... + +# Insert another user +$ spacetime call mydb insert_user Dave + +$ spacetime sql mydb "SELECT * FROM user" + user_id | name +---------+------- + 1 | Alice + 2 | Bob + 3 | Carol + 4097 | Dave # Jumped to next allocation boundary +``` -This design trades potential gaps in the sequence for durability and performance. The batch size balances the cost of persistence against the size of potential gaps. +This design trades potential gaps in the sequence for durability and performance. Internally, sequences use a 128-bit integer counter to track allocations across all column types. ### Uniqueness Considerations diff --git a/docs/docs/00200-core-concepts/00300-tables/00400-access-permissions.md b/docs/docs/00200-core-concepts/00300-tables/00400-access-permissions.md index 659ddca3cbc..08cb9ed04f1 100644 --- a/docs/docs/00200-core-concepts/00300-tables/00400-access-permissions.md +++ b/docs/docs/00200-core-concepts/00300-tables/00400-access-permissions.md @@ -220,7 +220,28 @@ spacetimedb.procedure('updateUserProcedure', { userId: t.u64(), newName: t.strin -Support for procedures in C# modules is coming soon! +```csharp +#pragma warning disable STDB_UNSTABLE + +[SpacetimeDB.Procedure] +public static void UpdateUserProcedure(ProcedureContext ctx, ulong userId, string newName) +{ + // Must explicitly open a transaction + ctx.WithTx(txCtx => + { + // Full read-write access within the transaction + var user = txCtx.Db.User.Id.Find(userId); + if (user != null) + { + var updated = user.Value; + updated.Name = newName; + txCtx.Db.User.Id.Update(updated); + } + return 0; + }); + // Transaction is committed when the lambda returns +} +``` @@ -570,7 +591,7 @@ Clients can query `my_profile` to see their username and creation date, but neve ### Combining Both Techniques -Views can combine row filtering and column projection. This example returns team members who report to the caller, with salary information hidden: +Views can combine row filtering and column projection. This example returns colleagues in the same department as the caller, with salary information hidden: @@ -580,42 +601,36 @@ import { table, t, schema } from 'spacetimedb/server'; // Private table with all employee data const employee = table( - { - name: 'employee', - indexes: [ - { name: 'idx_manager_id', algorithm: 'btree', columns: ['managerId'] }, - ], - }, + { name: 'employee' }, { id: t.u64().primaryKey(), identity: t.identity().unique(), name: t.string(), - department: t.string(), + department: t.string().index('btree'), salary: t.u64(), // Sensitive - managerId: t.option(t.u64()), } ); const spacetimedb = schema(employee); -// Public type for team members (no salary) -const teamMember = t.row('TeamMember', { +// Public type for colleagues (no salary) +const colleague = t.row('Colleague', { id: t.u64(), name: t.string(), department: t.string(), }); -// View that returns only the caller's team members, without salary info +// View that returns colleagues in the caller's department, without salary info spacetimedb.view( - { name: 'my_team', public: true }, - t.array(teamMember), + { name: 'my_colleagues', public: true }, + t.array(colleague), (ctx) => { // Find the caller's employee record by identity (unique index) const me = ctx.db.employee.identity.find(ctx.sender); if (!me) return []; - // Look up employees who report to the caller by manager_id index - return Array.from(ctx.db.employee.idx_manager_id.filter(me.id)).map(emp => ({ + // Look up employees in the same department + return Array.from(ctx.db.employee.department.filter(me.department)).map(emp => ({ id: emp.id, name: emp.name, department: emp.department, @@ -642,34 +657,33 @@ public partial class Module [SpacetimeDB.Unique] public Identity Identity; public string Name; + [SpacetimeDB.Index.BTree] public string Department; public ulong Salary; // Sensitive - [SpacetimeDB.Index.BTree] - public ulong? ManagerId; } - // Public type for team members (no salary) + // Public type for colleagues (no salary) [SpacetimeDB.Type] - public partial struct TeamMember + public partial struct Colleague { public ulong Id; public string Name; public string Department; } - // View that returns only the caller's team members, without salary info - [SpacetimeDB.View(Name = "MyTeam", Public = true)] - public static List MyTeam(ViewContext ctx) + // View that returns colleagues in the caller's department, without salary info + [SpacetimeDB.View(Name = "MyColleagues", Public = true)] + public static List MyColleagues(ViewContext ctx) { // Find the caller's employee record by identity (unique index) if (ctx.Db.Employee.Identity.Find(ctx.Sender) is not Employee me) { - return new List(); + return new List(); } - // Look up employees who report to the caller by ManagerId index - return ctx.Db.Employee.ManagerId.Filter(me.Id) - .Select(emp => new TeamMember + // Look up employees in the same department + return ctx.Db.Employee.Department.Filter(me.Department) + .Select(emp => new Colleague { Id = emp.Id, Name = emp.Name, @@ -695,34 +709,33 @@ pub struct Employee { #[unique] identity: Identity, name: String, + #[index(btree)] department: String, salary: u64, // Sensitive - #[index(btree)] - manager_id: Option, } -// Public type for team members (no salary) +// Public type for colleagues (no salary) #[derive(SpacetimeType)] -pub struct TeamMember { +pub struct Colleague { id: u64, name: String, department: String, } -// View that returns only the caller's team members, without salary info -#[spacetimedb::view(name = my_team, public)] -fn my_team(ctx: &ViewContext) -> Vec { +// View that returns colleagues in the caller's department, without salary info +#[spacetimedb::view(name = my_colleagues, public)] +fn my_colleagues(ctx: &ViewContext) -> Vec { // Find the caller's employee record by identity (unique index) let Some(me) = ctx.db.employee().identity().find(&ctx.sender) else { return vec![]; }; - // Look up employees who report to the caller by manager_id index - ctx.db.employee().manager_id().filter(&Some(me.id)) - .map(|emp| TeamMember { + // Look up employees in the same department + ctx.db.employee().department().filter(&me.department) + .map(|emp| Colleague { id: emp.id, - name: emp.name, - department: emp.department, + name: emp.name.clone(), + department: emp.department.clone(), // salary is not included }) .collect() diff --git a/docs/docs/00200-core-concepts/00300-tables/00500-schedule-tables.md b/docs/docs/00200-core-concepts/00300-tables/00500-schedule-tables.md index d6307880e77..0907ab79f07 100644 --- a/docs/docs/00200-core-concepts/00300-tables/00500-schedule-tables.md +++ b/docs/docs/00200-core-concepts/00300-tables/00500-schedule-tables.md @@ -9,6 +9,10 @@ import TabItem from '@theme/TabItem'; Tables can trigger [reducers](/functions/reducers) or [procedures](/functions/procedures) at specific times by including a special scheduling column. This allows you to schedule future actions like sending reminders, expiring items, or running periodic maintenance tasks. +:::tip Scheduling Procedures +Procedures use the same scheduling pattern as reducers. Simply reference the procedure name in the `scheduled` attribute. This is particularly useful when you need scheduled tasks that make HTTP requests or perform other side effects. See [Scheduling Procedures](/functions/reducers#scheduling-procedures) for an example. +::: + ## Defining a Schedule Table :::note Why "scheduled" in the code? @@ -38,21 +42,26 @@ spacetimedb.reducer('send_reminder', { arg: reminder.rowType }, (_ctx, { arg }) ```csharp -[SpacetimeDB.Table(Scheduled = "SendReminder", ScheduledAt = "ScheduleAt")] -public partial struct Reminder -{ - [SpacetimeDB.PrimaryKey] - [SpacetimeDB.AutoInc] - public ulong Id; - public uint UserId; - public string Message; - public ScheduleAt ScheduleAt; -} +using SpacetimeDB; -[SpacetimeDB.Reducer()] -public static void SendReminder(ReducerContext ctx, Reminder reminder) +public static partial class Module { - // Process the scheduled reminder + [SpacetimeDB.Table(Scheduled = "SendReminder", ScheduledAt = "ScheduleAt")] + public partial struct Reminder + { + [SpacetimeDB.PrimaryKey] + [SpacetimeDB.AutoInc] + public ulong Id; + public uint UserId; + public string Message; + public ScheduleAt ScheduleAt; + } + + [SpacetimeDB.Reducer] + public static void SendReminder(ReducerContext ctx, Reminder reminder) + { + // Process the scheduled reminder + } } ``` @@ -96,19 +105,23 @@ Use intervals for periodic tasks like game ticks, heartbeats, or recurring maint ```typescript import { ScheduleAt } from 'spacetimedb'; - -// Schedule to run every 5 seconds (5,000,000 microseconds) -ctx.db.reminder.insert({ - scheduled_id: 0n, - scheduled_at: ScheduleAt.interval(5_000_000n), - message: "Check for updates", -}); - -// Schedule to run every 100 milliseconds -ctx.db.reminder.insert({ - scheduled_id: 0n, - scheduled_at: ScheduleAt.interval(100_000n), // 100ms in microseconds - message: "Game tick", +import { schema, t, table, SenderError } from 'spacetimedb/server'; +const spacetimedb = schema(); + +spacetimedb.reducer('schedule_periodic_tasks', (ctx) => { + // Schedule to run every 5 seconds (5,000,000 microseconds) + ctx.db.reminder.insert({ + scheduled_id: 0n, + scheduled_at: ScheduleAt.interval(5_000_000n), + message: "Check for updates", + }); + + // Schedule to run every 100 milliseconds + ctx.db.reminder.insert({ + scheduled_id: 0n, + scheduled_at: ScheduleAt.interval(100_000n), // 100ms in microseconds + message: "Game tick", + }); }); ``` @@ -116,40 +129,51 @@ ctx.db.reminder.insert({ ```csharp -// Schedule to run every 5 seconds -ctx.Db.Reminder.Insert(new Reminder -{ - Message = "Check for updates", - ScheduleAt = new ScheduleAt.Interval(TimeSpan.FromSeconds(5)) -}); - -// Schedule to run every 100 milliseconds -ctx.Db.Reminder.Insert(new Reminder +public partial class Module { - Message = "Game tick", - ScheduleAt = new ScheduleAt.Interval(TimeSpan.FromMilliseconds(100)) -}); + [SpacetimeDB.Reducer] + public static void SchedulePeriodicTasks(ReducerContext ctx) + { + // Schedule to run every 5 seconds + ctx.Db.Reminder.Insert(new Reminder + { + Message = "Check for updates", + ScheduleAt = new ScheduleAt.Interval(TimeSpan.FromSeconds(5)) + }); + + // Schedule to run every 100 milliseconds + ctx.Db.Reminder.Insert(new Reminder + { + Message = "Game tick", + ScheduleAt = new ScheduleAt.Interval(TimeSpan.FromMilliseconds(100)) + }); + } +} ``` ```rust -use spacetimedb::{ScheduleAt, Duration}; +use spacetimedb::{ScheduleAt, ReducerContext}; +use std::time::Duration; -// Schedule to run every 5 seconds -ctx.db.reminder().insert(Reminder { - id: 0, - message: "Check for updates".to_string(), - scheduled_at: ScheduleAt::Interval(Duration::from_secs(5).into()), -}); - -// Schedule to run every 100 milliseconds -ctx.db.reminder().insert(Reminder { - id: 0, - message: "Game tick".to_string(), - scheduled_at: ScheduleAt::Interval(Duration::from_millis(100).into()), -}); +#[spacetimedb::reducer] +fn schedule_periodic_tasks(ctx: &ReducerContext) { + // Schedule to run every 5 seconds + ctx.db.reminder().insert(Reminder { + id: 0, + message: "Check for updates".to_string(), + scheduled_at: ScheduleAt::Interval(Duration::from_secs(5).into()), + }); + + // Schedule to run every 100 milliseconds + ctx.db.reminder().insert(Reminder { + id: 0, + message: "Game tick".to_string(), + scheduled_at: ScheduleAt::Interval(Duration::from_millis(100).into()), + }); +} ``` @@ -164,21 +188,25 @@ Use specific times for one-shot actions like sending a reminder at a particular ```typescript import { ScheduleAt } from 'spacetimedb'; - -// Schedule for 10 seconds from now -const tenSecondsFromNow = ctx.timestamp.microseconds + 10_000_000n; -ctx.db.reminder.insert({ - scheduled_id: 0n, - scheduled_at: ScheduleAt.time(tenSecondsFromNow), - message: "Your auction has ended", -}); - -// Schedule for a specific Unix timestamp (microseconds since epoch) -const targetTime = 1735689600_000_000n; // Jan 1, 2025 00:00:00 UTC -ctx.db.reminder.insert({ - scheduled_id: 0n, - scheduled_at: ScheduleAt.time(targetTime), - message: "Happy New Year!", +import { schema, t, table, SenderError } from 'spacetimedb/server'; +const spacetimedb = schema(); + +spacetimedb.reducer('schedule_timed_tasks', (ctx) => { + // Schedule for 10 seconds from now + const tenSecondsFromNow = ctx.timestamp.microsSinceUnixEpoch + 10_000_000n; + ctx.db.reminder.insert({ + scheduled_id: 0n, + scheduled_at: ScheduleAt.time(tenSecondsFromNow), + message: "Your auction has ended", + }); + + // Schedule for a specific Unix timestamp (microseconds since epoch) + const targetTime = 1735689600_000_000n; // Jan 1, 2025 00:00:00 UTC + ctx.db.reminder.insert({ + scheduled_id: 0n, + scheduled_at: ScheduleAt.time(targetTime), + message: "Happy New Year!", + }); }); ``` @@ -186,42 +214,55 @@ ctx.db.reminder.insert({ ```csharp -// Schedule for 10 seconds from now -ctx.Db.Reminder.Insert(new Reminder -{ - Message = "Your auction has ended", - ScheduleAt = new ScheduleAt.Time(DateTimeOffset.UtcNow.AddSeconds(10)) -}); +using SpacetimeDB; -// Schedule for a specific time -var targetTime = new DateTimeOffset(2025, 1, 1, 0, 0, 0, TimeSpan.Zero); -ctx.Db.Reminder.Insert(new Reminder +public static partial class Module { - Message = "Happy New Year!", - ScheduleAt = new ScheduleAt.Time(targetTime) -}); + [SpacetimeDB.Reducer] + public static void ScheduleTimedTasks(ReducerContext ctx) + { + // Schedule for 10 seconds from now + ctx.Db.Reminder.Insert(new Reminder + { + Message = "Your auction has ended", + ScheduleAt = new ScheduleAt.Time(DateTimeOffset.UtcNow.AddSeconds(10)) + }); + + // Schedule for a specific time + var targetTime = new DateTimeOffset(2025, 1, 1, 0, 0, 0, TimeSpan.Zero); + ctx.Db.Reminder.Insert(new Reminder + { + Message = "Happy New Year!", + ScheduleAt = new ScheduleAt.Time(targetTime) + }); + } +} ``` ```rust -use spacetimedb::{ScheduleAt, Duration}; - -// Schedule for 10 seconds from now -let ten_seconds_from_now = ctx.timestamp + Duration::from_secs(10); -ctx.db.reminder().insert(Reminder { - id: 0, - message: "Your auction has ended".to_string(), - scheduled_at: ScheduleAt::Time(ten_seconds_from_now), -}); +use spacetimedb::{ScheduleAt, ReducerContext}; +use std::time::Duration; -// Schedule for immediate execution (current timestamp) -ctx.db.reminder().insert(Reminder { - id: 0, - message: "Process now".to_string(), - scheduled_at: ScheduleAt::Time(ctx.timestamp.clone()), -}); +#[spacetimedb::reducer] +fn schedule_timed_tasks(ctx: &ReducerContext) { + // Schedule for 10 seconds from now + let ten_seconds_from_now = ctx.timestamp + Duration::from_secs(10); + ctx.db.reminder().insert(Reminder { + id: 0, + message: "Your auction has ended".to_string(), + scheduled_at: ScheduleAt::Time(ten_seconds_from_now), + }); + + // Schedule for immediate execution (current timestamp) + ctx.db.reminder().insert(Reminder { + id: 0, + message: "Process now".to_string(), + scheduled_at: ScheduleAt::Time(ctx.timestamp.clone()), + }); +} ``` @@ -256,14 +297,19 @@ spacetimedb.reducer('send_reminder', { arg: Reminder.rowType }, (ctx, { arg }) = ```csharp -[SpacetimeDB.Reducer()] -public static void SendReminder(ReducerContext ctx, Reminder reminder) +using SpacetimeDB; + +public static partial class Module { - if (!ctx.SenderAuth.IsInternal) + [SpacetimeDB.Reducer] + public static void SendReminder(ReducerContext ctx, Reminder reminder) { - throw new Exception("This reducer can only be called by the scheduler"); + if (!ctx.SenderAuth.IsInternal) + { + throw new Exception("This reducer can only be called by the scheduler"); + } + // Process the scheduled reminder } - // Process the scheduled reminder } ``` diff --git a/docs/docs/00300-resources/00200-reference/00500-appendix.md b/docs/docs/00300-resources/00200-reference/00500-appendix.md deleted file mode 100644 index 8d8f81df73c..00000000000 --- a/docs/docs/00300-resources/00200-reference/00500-appendix.md +++ /dev/null @@ -1,68 +0,0 @@ ---- -slug: /reference/appendix ---- - -# Appendix - -## SEQUENCE - -For each table containing an `#[auto_inc]` column, SpacetimeDB creates a sequence number generator behind the scenes, which functions similarly to `postgres`'s `SEQUENCE`. - -### How It Works - -:::warning - -Sequence number generation is not transactional. - -::: - -- Sequences in SpacetimeDB use Rust’s `i128` integer type. -- The field type marked with `#[auto_inc]` is cast to `i128` and increments by `1` for each new row. -- Sequences are pre-allocated in chunks of `4096` to speed up number generation, and then are only persisted to disk when the pre-allocated chunk is exhausted. -- Numbers are incremented even if a transaction is later rolled back. -- Unused numbers are not reclaimed, meaning sequences may have _gaps_. -- If the server restarts or a transaction rolls back, the sequence continues from the next pre-allocated chunk + `1`: - -**Example:** - -```rust -#[spacetimedb::table(name = users, public)] -struct Users { - #[auto_inc] - user_id: u64, - name: String, -} - -#[spacetimedb::reducer] -pub fn insert_user(ctx: &ReducerContext, count: u8) { - for i in 0..count { - let name = format!("User {}", i); - ctx.db.users().insert(Users { user_id: 0, name }); - } - // Query the table to see the effect of the `[auto_inc]` attribute: - for user in ctx.db.users().iter() { - log::info!("User: {:?}", user); - } -} -``` - -Then: - -```bash -❯ cargo run --bin spacetimedb-cli call sample insert_user 3 - -❯ spacetimedb-cli logs sample -... -.. User: Users { user_id: 1, name: "User 0" } -.. User: Users { user_id: 2, name: "User 1" } -.. User: Users { user_id: 3, name: "User 2" } - -# Database restart, then - -❯ cargo run --bin spacetimedb-cli call sample insert_user 1 - -❯ spacetimedb-cli logs sample -... -.. User: Users { user_id: 3, name: "User 2" } -.. User: Users { user_id: 4098, name: "User 0" } -``` diff --git a/docs/llms/docs-benchmark-analysis.md b/docs/llms/docs-benchmark-analysis.md index cf81b9ed007..640eb6e5bdd 100644 --- a/docs/llms/docs-benchmark-analysis.md +++ b/docs/llms/docs-benchmark-analysis.md @@ -4,23 +4,37 @@ Generated from: `/__w/SpacetimeDB/SpacetimeDB/tools/xtask-llm-benchmark/../../do ## Summary -- **Total failures analyzed**: 35 +- **Total failures analyzed**: 34 --- -# Analysis of SpacetimeDB Benchmark Test Failures +# SpacetimeDB Benchmark Test Failures Analysis + +This document analyzes the SpacetimeDB benchmark test failures. Each failure is broken down into its components, specifying the discrepancies between the generated code and the expected golden examples. Specific, actionable recommendations for documentation adjustments are provided to avoid such issues in the future. ## Rust / rustdoc_json Failures -### Compile/Publish Errors +### 1. General Errors with Code Generation -#### 1. Test: t_002_scheduled_table +#### **Failures Overview**: The following tests failed due to various issues regarding table definitions, missing `pub` keywords, and improper syntax in function signatures. -**The generated code**: -```rust -use spacetimedb::{table, reducer, ReducerContext, Table, ScheduleAt}; +#### **Failures Grouped by Cause** + +--- -#[table(name = tick_timer, schedule(column = scheduled_at, reducer = tick))] +### a. **Issue with `pub` Keyword and Function Signatures** + +#### **Failures**: +- t_002_scheduled_table +- t_015_product_type_columns +- t_017_scheduled_columns + +#### 1. **t_002_scheduled_table** +**Error**: publish_error: spacetime publish failed (exit=1) + +**LLM Output**: +```rust +#[table(name = tick_timer, scheduled(reducer = tick, column = scheduled_at))] pub struct TickTimer { #[primary_key] #[auto_inc] @@ -28,74 +42,83 @@ pub struct TickTimer { scheduled_at: ScheduleAt, } -#[reducer] -pub fn tick(_ctx: &ReducerContext) { - // Scheduled reducer invoked by tick_timer -} - #[reducer(init)] pub fn init(ctx: &ReducerContext) { ctx.db.tick_timer().insert(TickTimer { scheduled_id: 0, - scheduled_at: ScheduleAt::RepeatMicros(50_000), + scheduled_at: ScheduleAt::AfterMicros(50_000), }); } ``` -**The golden example**: +**Expected**: ```rust -use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table}; -use std::time::Duration; - #[table(name = tick_timer, scheduled(tick))] pub struct TickTimer { #[primary_key] #[auto_inc] - scheduled_id: u64, - scheduled_at: ScheduleAt, + pub scheduled_id: u64, + pub scheduled_at: ScheduleAt, } #[reducer] -pub fn tick(_ctx: &ReducerContext, _row: TickTimer) -> Result<(), String> { - Ok(()) -} +pub fn tick(_ctx: &ReducerContext, _schedule: TickTimer) { } #[reducer(init)] -pub fn init(ctx: &ReducerContext) -> Result<(), String> { +pub fn init(ctx: &ReducerContext) { ctx.db.tick_timer().insert(TickTimer { scheduled_id: 0, scheduled_at: ScheduleAt::Interval(Duration::from_millis(50).into()), }); - Ok(()) } ``` -**Error**: `publish_error: spacetime publish failed (exit=1)` +--- -**Explain the difference**: -- The incorrect `schedule` attribute used improper syntax (column and reducer mentioned separately). -- The `tick` reducer should accept a `TickTimer` row parameter. -- The `scheduled_at` assignment is incorrect (incorrect type used). +#### **2. t_015_product_type_columns** +**Error**: publish_error: spacetime publish failed (exit=1) -**Root cause**: -Documentation does not clarify the need to use the `scheduled` attribute correctly and specify function signatures for reducers expected by the API. +**LLM Output**: +```rust +#[derive(SpacetimeType)] +pub struct Address { + street: String, + zip: i32, +} -**Recommendation**: -Update documentation to “Must use attribute `scheduled(tick)` and ensure the reducer functions accept the required parameters as specified.” Example: +#[table(name = profile)] +pub struct Profile { + #[primary_key] + id: i32, + home: Address, + work: Address, +} +``` +**Expected**: ```rust -#[table(name = tick_timer, scheduled(tick))] -// correct usage +#[derive(SpacetimeType, Clone, Debug)] +pub struct Address { + pub street: String, + pub zip: i32, +} + +#[table(name = profile)] +pub struct Profile { + #[primary_key] + pub id: i32, + pub home: Address, + pub work: Address, +} ``` --- -#### 2. Test: t_017_scheduled_columns +#### **3. t_017_scheduled_columns** +**Error**: publish_error: spacetime publish failed (exit=1) -**The generated code**: +**LLM Output**: ```rust -use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table}; - #[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))] pub struct TickTimer { #[primary_key] @@ -103,26 +126,10 @@ pub struct TickTimer { scheduled_id: u64, scheduled_at: ScheduleAt, } - -#[reducer(init)] -pub fn init(ctx: &ReducerContext) { - if ctx.db.tick_timer().count() == 0 { - ctx.db.tick_timer().insert(TickTimer { - scheduled_id: 0, - scheduled_at: ScheduleAt::repeat_us(50_000), - }); - } -} - -#[reducer(scheduled)] -pub fn tick(_ctx: &ReducerContext, _row: TickTimer) {} ``` -**The golden example**: +**Expected**: ```rust -use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table}; -use std::time::Duration; - #[table(name = tick_timer, scheduled(tick))] pub struct TickTimer { #[primary_key] @@ -130,176 +137,209 @@ pub struct TickTimer { pub scheduled_id: u64, pub scheduled_at: ScheduleAt, } - -#[reducer] -pub fn tick(_ctx: &ReducerContext, _schedule: TickTimer) {} - -#[reducer(init)] -pub fn init(ctx: &ReducerContext) { - let every_50ms: ScheduleAt = Duration::from_millis(50).into(); - ctx.db.tick_timer().insert(TickTimer { - scheduled_id: 0, - scheduled_at: every_50ms, - }); -} ``` -**Error**: `publish_error: spacetime publish failed (exit=1)` - -**Explain the difference**: -- Incorrect `schedule` attribute syntax causes failures in specifying a reducer. -- The `scheduled_at` type is incorrectly set to `repeat_us`, not conforming to the Interval structure. - -**Root cause**: -Lack of detailed guidelines in documentation for correctly annotating and defining reducer routes expected during schema description. +--- -**Recommendation**: -Include a guide section in the documentation detailing all attributes for tables and reducers, specifically emphasizing parameter types and structure expectations. +### **Explanation** +- **Difference**: The generated code often lacks the `pub` specifier on struct fields and functions, making them inaccessible outside their module. Function signatures also often miss the required return types. +- **Root Cause**: Documentation may not clarify that all fields in a SpacetimeDB struct need to be public. +- **Recommendation**: Update the documentation to specify that all fields in structs used in SpacetimeDB contexts should be marked as `pub`, along with specifying the necessity of return types for all reducer functions. --- -### Other Failures +### b. **Errors Regarding Missing or Incorrect Implementation** -#### 3. Test: t_003_struct_in_table +#### **Failures**: +- t_013_spacetime_sum_type +- t_016_sum_type_columns +- t_018_constraints -**The generated code**: +#### 1. **t_013_spacetime_sum_type** +**Error**: no such table: `result`. + +**LLM Output**: ```rust -use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType}; +#[table(name = result)] +pub struct ResultRow { + #[primary_key] + id: i32, + value: Shape, +} +``` -#[derive(SpacetimeType)] -pub struct Position { - x: i32, - y: i32, +**Expected**: +```rust +#[table(name = result)] +pub struct ResultRow { + #[primary_key] + pub id: i32, + pub value: Shape, } +``` + +--- + +#### **2. t_016_sum_type_columns** +**Error**: no such table: `drawings`. -#[table(name = entity)] -pub struct Entity { +**LLM Output**: +```rust +#[spacetimedb::table(name = drawing)] +pub struct Drawing { #[primary_key] id: i32, - pos: Position, + a: Shape, + b: Shape, } +``` -#[reducer] -pub fn add(ctx: &ReducerContext, id: i32, x: i32, y: i32) { - ctx.db.entity().insert(Entity { id, pos: Position { x, y } }); +**Expected**: +```rust +#[table(name = drawing)] +pub struct Drawing { + #[primary_key] + pub id: i32, + pub a: Shape, + pub b: Shape, } ``` -**The golden example**: -```rust -use spacetimedb::{table, SpacetimeType}; +--- -#[derive(SpacetimeType, Clone, Debug)] -pub struct Position { - pub x: i32, - pub y: i32, +#### **3. t_018_constraints** +**Error**: no such table: `account`. + +**LLM Output**: +```rust +#[table(name = account, index(name = by_name, btree(columns = [name])))] +pub struct Account { + #[primary_key] + id: i32, + #[unique] + email: String, + name: String, } +``` -#[table(name = entity)] -pub struct Entity { +**Expected**: +```rust +#[table( + name = account, + index(name = by_name, btree(columns = [name])) +)] +pub struct Account { #[primary_key] pub id: i32, - pub pos: Position, + #[unique] + pub email: String, + pub name: String, } ``` -**Error**: `schema_parity: reducers differ - expected [], got ["add()"]` - -**Explain the difference**: -- The reducer function should not be necessary if there is no reduction logic within the example. -- The `Position` struct lacks appropriate visibility modifiers, leading to nil results in the API. - -**Root cause**: -Documentation does not clarify when reducers are expected and how to define and declare public fields necessary for SpacetimeDB API to function correctly. +--- -**Recommendation**: -Clarify in the documentation that reducers should only be included when necessary, along with showcasing fields with visibility for structs. +### **Explanation** +- **Difference**: The generated code frequently omits the `pub` visibility and fails to implement the required structure correctly. +- **Root Cause**: Documentation may not sufficiently outline the importance of proper struct and field visibility in SpacetimeDB schemas. +- **Recommendation**: Emphasize in the documentation that all fields in structs must be public and provide examples highlighting this requirement. --- -#### 4. Test: t_013_spacetime_sum_type +## Rust / docs Failures -**The generated code**: -```rust -use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType}; +### **Timeout Issues** -#[derive(SpacetimeType)] -pub struct Rect { - width: i32, - height: i32, -} +#### **Failure** +- t_020_ecs -#[derive(SpacetimeType)] -pub enum Shape { - Circle(i32), - Rectangle(Rect), -} +#### **Analysis** +This failure indicates that the task exceeded its time limit during execution. Check if the test could be optimized for performance or if the complexity is overly high. -#[table(name = result)] -pub struct ResultRow { - #[primary_key] - id: i32, - value: Shape, -} +#### **Recommendation** +Optimize the performance of tests, ensuring they do not exceed reasonable time limits. Additionally, incorporate timeouts into the documentation to assist developers in recognizing performance expectations. -#[reducer] -pub fn set_circle(ctx: &ReducerContext, id: i32, radius: i32) { - ctx.db.result().insert(ResultRow { - id, - value: Shape::Circle(radius), - }); -} -``` +--- -**The golden example**: -```rust -use spacetimedb::{reducer, table, ReducerContext, SpacetimeType, Table}; +## C# / docs Failures -#[derive(SpacetimeType, Clone, Debug)] -pub struct Rect { - pub width: i32, - pub height: i32, -} +### **1. Missing `pub` Keywords and Related Language Features** -#[derive(SpacetimeType, Clone, Debug)] -pub enum Shape { - Circle(i32), - Rectangle(Rect), -} +--- -#[table(name = result)] -pub struct ResultRow { - #[primary_key] - pub id: i32, - pub value: Shape, +#### **Failures**: +- t_014_elementary_columns +- t_017_scheduled_columns + +#### 1. **t_014_elementary_columns** +**Error**: no such table: `primitive`. + +**LLM Output**: +```csharp +[Table(Name = "Primitive", Public = true)] +public partial struct Primitive +{ + [PrimaryKey] + public int Id; + public int Count; + public long Total; + public float Price; + public double Ratio; + public bool Active; + public string Name; } +``` -#[reducer] -pub fn set_circle(ctx: &ReducerContext, id: i32, radius: i32) { - ctx.db.result().insert(ResultRow { id, value: Shape::Circle(radius) }); +**Expected**: +```csharp +[Table(Name = "Primitive")] +public partial struct Primitive +{ + [PrimaryKey] public int Id; + public int Count; + public long Total; + public float Price; + public double Ratio; + public bool Active; + public string Name; } ``` -**Error**: `sum_type_row_parity: spacetime sql failed: Error: no such table: result` - -**Explain the difference**: -- Visibility modifiers are absent on the `Rect` and `Shape` structs. -- The naming convention for the struct is inconsistent with usage. +--- -**Root cause**: -Documentation does not specify the significance of visibility in structs and enums directly influencing the usage of their produced database entries. +#### **2. t_017_scheduled_columns** +**Error**: no such table: `tick_timer`. + +**LLM Output**: +```csharp +[Table(Name = "TickTimer", Scheduled = nameof(Tick), ScheduledAt = nameof(ScheduledAt))] +public partial struct TickTimer +{ + [PrimaryKey, AutoInc] + public ulong ScheduledId; + public ScheduleAt ScheduledAt; +} +``` -**Recommendation**: -Amend documentation to highlight the importance of public fields and specifically demonstrate the effect of struct and enum visibility. +**Expected**: +```csharp +[Table(Name = "TickTimer", Scheduled = nameof(Tick), ScheduledAt = nameof(ScheduledAt))] +public partial struct TickTimer +{ + [PrimaryKey, AutoInc] public ulong ScheduledId; + public ScheduleAt ScheduledAt; +} +``` --- -### Additional Recommendations - -1. **Documentation Structure**: Enhance the overall organization and clarity in the documentation relating to syntax rules. Providing clear examples of common pitfalls with API usages can prevent such issues. +### **Explanation** +- **Difference**: The generated C# code fails to keep all relevant visibility concerns in mind. +- **Root Cause**: Similar to Rust failures, the need for public visibility in C# struct fields hasn't been emphasized, leading to errors. +- **Recommendation**: Document the need for public access modifiers explicitly for struct fields in C#, especially for types used with SpacetimeDB. -2. **Consistency**: Ensure that examples maintain consistent use of visibility and thorough descriptions of error types relevant to expected outcomes. +--- -3. **Error Handling**: Clarify the expected patterns and structures for error handling within API calls and data flows. +## Conclusion -By implementing these documentation changes and clarifications, debug routines should become more intuitive, resulting in fewer benchmark test failures. +This analysis provides insights into the root causes of the failures in the SpacetimeDB benchmarks across both Rust and C#. Identifying the missing `pub` keywords, incorrect function signatures, and general visibility issues will help refine documentation and improve the quality of generated code. By adopting the recommended changes, future tests will likely yield more consistent successes. diff --git a/docs/llms/docs-benchmark-comment.md b/docs/llms/docs-benchmark-comment.md index 92894c95808..c74402e7d54 100644 --- a/docs/llms/docs-benchmark-comment.md +++ b/docs/llms/docs-benchmark-comment.md @@ -2,16 +2,16 @@ | Language | Mode | Category | Tests Passed | Task Pass % | |----------|------|----------|--------------|-------------| -| Rust | rustdoc_json | basics | 25/27 | 83.3% ⬆️ +38.9% | -| Rust | rustdoc_json | schema | 23/34 | 65.3% ⬆️ +38.8% | -| Rust | rustdoc_json | **total** | 48/61 | **75.2%** ⬆️ +38.9% | +| Rust | rustdoc_json | basics | 26/27 | 91.7% ⬆️ +8.3% | +| Rust | rustdoc_json | schema | 21/34 | 58.7% ⬇️ -6.7% | +| Rust | rustdoc_json | **total** | 47/61 | **76.7%** ⬆️ +1.5% | | Rust | docs | basics | 5/27 | 11.1% | -| Rust | docs | schema | 7/29 | 15.5% ⬇️ -5.0% | -| Rust | docs | **total** | 12/56 | **13.1%** ⬇️ -2.3% | -| C# | docs | basics | 27/27 | 100.0% ⬆️ +38.9% | -| C# | docs | schema | 22/34 | 63.7% ⬆️ +32.2% | -| C# | docs | **total** | 49/61 | **83.5%** ⬆️ +35.8% | +| Rust | docs | schema | 8/30 | 20.5% ⬆️ +5.0% | +| Rust | docs | **total** | 13/57 | **15.4%** ⬆️ +2.3% | +| C# | docs | basics | 27/27 | 100.0% | +| C# | docs | schema | 19/32 | 60.3% ⬇️ -3.3% | +| C# | docs | **total** | 46/59 | **82.0%** ⬇️ -1.5% | _Compared against master branch baseline_ -Generated at: 2026-01-22T01:34:07.820Z +Generated at: 2026-01-22T23:45:28.101Z diff --git a/docs/llms/docs-benchmark-details.json b/docs/llms/docs-benchmark-details.json index e1b1c2680a3..9936d24501e 100644 --- a/docs/llms/docs-benchmark-details.json +++ b/docs/llms/docs-benchmark-details.json @@ -19,7 +19,7 @@ "model_name": "GPT-5", "total_tests": 1, "passed_tests": 1, - "llm_output": "use spacetimedb::ReducerContext;\n\n#[spacetimedb::reducer]\npub fn empty_reducer_no_args(_ctx: &ReducerContext) {}\n\n#[spacetimedb::reducer]\npub fn empty_reducer_with_int(_ctx: &ReducerContext, _count: i32) {}\n\n#[spacetimedb::reducer]\npub fn empty_reducer_with_string(_ctx: &ReducerContext, _name: String) {}\n\n#[spacetimedb::reducer]\npub fn empty_reducer_with_two_args(_ctx: &ReducerContext, _count: i32, _name: String) {}\n\n#[spacetimedb::reducer]\npub fn empty_reducer_with_three_args(_ctx: &ReducerContext, _active: bool, _ratio: f32, _label: String) {}", + "llm_output": "use spacetimedb::{reducer, ReducerContext};\n\n#[reducer]\nfn empty_reducer_no_args(_ctx: &ReducerContext) {}\n\n#[reducer]\nfn empty_reducer_with_int(_ctx: &ReducerContext, _count: i32) {}\n\n#[reducer]\nfn empty_reducer_with_string(_ctx: &ReducerContext, _name: String) {}\n\n#[reducer]\nfn empty_reducer_with_two_args(_ctx: &ReducerContext, _count: i32, _name: String) {}\n\n#[reducer]\nfn empty_reducer_with_three_args(_ctx: &ReducerContext, _active: bool, _ratio: f32, _label: String) {}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-000-empty-reducers-golden", @@ -35,15 +35,15 @@ "llm_db": "basics-t-000-empty-reducers-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:39821", + "server": "http://127.0.0.1:34355", "tables_diff": null, "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-22T01:25:15.264255911Z", - "finished_at": "2026-01-22T01:25:45.391891527Z" + "started_at": "2026-01-22T23:35:38.832121877Z", + "finished_at": "2026-01-22T23:35:58.993644023Z" }, "t_001_basic_tables": { "hash": "e6a5a5374022e3250e0df2cb5e4f7e4777081f474a0ebc4e99302be42151ef59", @@ -53,7 +53,7 @@ "model_name": "GPT-5", "total_tests": 1, "passed_tests": 1, - "llm_output": "use spacetimedb::table;\n\n#[table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[table(name = product)]\npub struct Product {\n #[primary_key]\n id: i32,\n title: String,\n price: f32,\n in_stock: bool,\n}\n\n#[table(name = note)]\npub struct Note {\n #[primary_key]\n id: i32,\n body: String,\n rating: i64,\n pinned: bool,\n}", + "llm_output": "#[spacetimedb::table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[spacetimedb::table(name = product)]\npub struct Product {\n #[primary_key]\n id: i32,\n title: String,\n price: f32,\n in_stock: bool,\n}\n\n#[spacetimedb::table(name = note)]\npub struct Note {\n #[primary_key]\n id: i32,\n body: String,\n rating: i64,\n pinned: bool,\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-001-basic-tables-golden", @@ -69,15 +69,15 @@ "llm_db": "basics-t-001-basic-tables-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:39821", + "server": "http://127.0.0.1:34355", "tables_diff": null, "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-22T01:25:31.552835671Z", - "finished_at": "2026-01-22T01:26:01.959327609Z" + "started_at": "2026-01-22T23:35:46.097927860Z", + "finished_at": "2026-01-22T23:36:14.231803132Z" }, "t_002_scheduled_table": { "hash": "e6a5a5374022e3250e0df2cb5e4f7e4777081f474a0ebc4e99302be42151ef59", @@ -87,7 +87,7 @@ "model_name": "GPT-5", "total_tests": 1, "passed_tests": 0, - "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table, ScheduleAt};\n\n#[table(name = tick_timer, schedule(column = scheduled_at, reducer = tick))]\npub struct TickTimer {\n #[primary_key]\n #[auto_inc]\n scheduled_id: u64,\n scheduled_at: ScheduleAt,\n}\n\n#[reducer]\npub fn tick(_ctx: &ReducerContext) {\n // Scheduled reducer invoked by tick_timer\n}\n\n#[reducer(init)]\npub fn init(ctx: &ReducerContext) {\n ctx.db.tick_timer().insert(TickTimer {\n scheduled_id: 0,\n scheduled_at: ScheduleAt::RepeatMicros(50_000),\n });\n}", + "llm_output": "use spacetimedb::{table, reducer, ReducerContext, ScheduleAt, Table, UniqueColumn};\n\n#[table(name = tick_timer, scheduled(reducer = tick, column = scheduled_at))]\npub struct TickTimer {\n #[primary_key]\n #[auto_inc]\n scheduled_id: u64,\n scheduled_at: ScheduleAt,\n}\n\n#[reducer(init)]\npub fn init(ctx: &ReducerContext) {\n if ctx.db.tick_timer().count() == 0 {\n ctx.db.tick_timer().insert(TickTimer {\n scheduled_id: 0,\n scheduled_at: ScheduleAt::AfterMicros(50_000),\n });\n }\n}\n\n#[reducer]\npub fn tick(ctx: &ReducerContext, scheduled_id: u64) {\n if let Some(mut row) = ctx.db.tick_timer().scheduled_id().find(scheduled_id) {\n row.scheduled_at = ScheduleAt::AfterMicros(50_000);\n let _ = ctx.db.tick_timer().scheduled_id().update(row);\n }\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-002-scheduled-table-golden", @@ -99,14 +99,14 @@ "pass": false, "partial": 0.0, "notes": { - "error": "spacetime publish failed (exit=1)\n--- stderr ---\n Updating crates.io index\n Locking 72 packages to latest compatible versions\n Adding generic-array v0.14.7 (available: v0.14.9)\n Adding spacetimedb v1.11.1 (available: v1.11.3)\n Adding spacetimedb-bindings-macro v1.11.1 (available: v1.11.3)\n Adding spacetimedb-bindings-sys v1.11.1 (available: v1.11.3)\n Adding spacetimedb-lib v1.11.1 (available: v1.11.3)\n Adding spacetimedb-primitives v1.11.1 (available: v1.11.3)\n Adding spacetimedb-sats v1.11.1 (available: v1.11.3)\n Compiling proc-macro2 v1.0.106\n Compiling quote v1.0.43\n Compiling unicode-ident v1.0.22\n Compiling version_check v0.9.5\n Compiling typenum v1.19.0\n Compiling autocfg v1.5.0\n Compiling heck v0.5.0\n Compiling serde_core v1.0.228\n Compiling cfg-if v1.0.4\n Compiling either v1.15.0\n Compiling zerocopy v0.8.33\n Compiling find-msvc-tools v0.1.8\n Compiling serde v1.0.228\n Compiling shlex v1.3.0\n Compiling nohash-hasher v0.2.0\n Compiling thiserror v1.0.69\n Compiling anyhow v1.0.100\n Compiling bitflags v2.10.0\n Compiling humantime v2.3.0\n Compiling bytes v1.11.0\n Compiling zmij v1.0.16\n Compiling arrayvec v0.7.6\n Compiling keccak v0.1.5\n Compiling convert_case v0.4.0\n Compiling heck v0.4.1\n Compiling serde_json v1.0.149\n Compiling smallvec v1.15.1\n Compiling arrayref v0.3.9\n Compiling bytemuck v1.24.0\n Compiling getrandom v0.2.17\n Compiling second-stack v0.3.5\n Compiling itoa v1.0.17\n Compiling cc v1.2.53\n Compiling spacetimedb-lib v1.11.1\n Compiling constant_time_eq v0.4.2\n Compiling hex v0.4.3\n Compiling memchr v2.7.6\n Compiling itertools v0.12.1\n Compiling rand_core v0.6.4\n Compiling log v0.4.29\n Compiling generic-array v0.14.7\n Compiling scoped-tls v1.0.1\n Compiling num-traits v0.2.19\n Compiling http v1.4.0\n Compiling syn v2.0.114\n Compiling approx v0.3.2\n Compiling chrono v0.4.43\n Compiling blake3 v1.8.3\n Compiling decorum v0.3.1\n Compiling crypto-common v0.1.7\n Compiling block-buffer v0.10.4\n Compiling digest v0.10.7\n Compiling sha3 v0.10.8\n Compiling ppv-lite86 v0.2.21\n Compiling rand_chacha v0.3.1\n Compiling rand v0.8.5\n Compiling ethnum v1.5.2\n Compiling enum-as-inner v0.6.1\n Compiling thiserror-impl v1.0.69\n Compiling derive_more v0.99.20\n Compiling spacetimedb-primitives v1.11.1\n Compiling spacetimedb-bindings-sys v1.11.1\n Compiling spacetimedb-bindings-macro v1.11.1\n Compiling spacetimedb-sats v1.11.1\n Compiling spacetimedb v1.11.1\n Compiling spacetime-module v0.1.0 (/__w/SpacetimeDB/SpacetimeDB/target/llm-runs/basics/t_002_scheduled_table/rust/server/gpt-5/llm)\nerror: expected one of: `public`, `private`, `name`, `index`, `scheduled`\n --> src/lib.rs:4:28\n |\n4 | #[table(name = tick_timer, schedule(column = scheduled_at, reducer = tick))]\n | ^^^^^^^^\n\nerror[E0422]: cannot find struct, variant or union type `TickTimer` in this scope\n --> src/lib.rs:19:32\n |\n19 | ctx.db.tick_timer().insert(TickTimer {\n | ^^^^^^^^^ not found in this scope\n\nerror[E0599]: no method named `tick_timer` found for struct `Local` in the current scope\n --> src/lib.rs:19:12\n |\n19 | ctx.db.tick_timer().insert(TickTimer {\n | ^^^^^^^^^^ method not found in `Local`\n\nerror[E0599]: no variant or associated item named `RepeatMicros` found for enum `ScheduleAt` in the current scope\n --> src/lib.rs:21:35\n |\n21 | scheduled_at: ScheduleAt::RepeatMicros(50_000),\n | ^^^^^^^^^^^^ variant or associated item not found in `ScheduleAt`\n\nSome errors have detailed explanations: E0422, E0599.\nFor more information about an error, try `rustc --explain E0422`.\nerror: could not compile `spacetime-module` (lib) due to 4 previous errors\nError: command [\"cargo\", \"build\", \"--config=net.git-fetch-with-cli=true\", \"--target=wasm32-unknown-unknown\", \"--release\", \"--message-format=json-render-diagnostics\"] exited with code 101\n\n--- stdout ---\n", + "error": "spacetime publish failed (exit=1)\n--- stderr ---\n Updating crates.io index\n Locking 72 packages to latest compatible versions\n Adding generic-array v0.14.7 (available: v0.14.9)\n Adding spacetimedb v1.11.1 (available: v1.11.3)\n Adding spacetimedb-bindings-macro v1.11.1 (available: v1.11.3)\n Adding spacetimedb-bindings-sys v1.11.1 (available: v1.11.3)\n Adding spacetimedb-lib v1.11.1 (available: v1.11.3)\n Adding spacetimedb-primitives v1.11.1 (available: v1.11.3)\n Adding spacetimedb-sats v1.11.1 (available: v1.11.3)\n Compiling proc-macro2 v1.0.106\n Compiling unicode-ident v1.0.22\n Compiling quote v1.0.43\n Compiling typenum v1.19.0\n Compiling version_check v0.9.5\n Compiling autocfg v1.5.0\n Compiling serde_core v1.0.228\n Compiling heck v0.5.0\n Compiling cfg-if v1.0.4\n Compiling shlex v1.3.0\n Compiling serde v1.0.228\n Compiling either v1.15.0\n Compiling find-msvc-tools v0.1.8\n Compiling zerocopy v0.8.33\n Compiling nohash-hasher v0.2.0\n Compiling bitflags v2.10.0\n Compiling thiserror v1.0.69\n Compiling anyhow v1.0.100\n Compiling bytes v1.11.0\n Compiling convert_case v0.4.0\n Compiling arrayvec v0.7.6\n Compiling zmij v1.0.16\n Compiling keccak v0.1.5\n Compiling heck v0.4.1\n Compiling humantime v2.3.0\n Compiling arrayref v0.3.9\n Compiling hex v0.4.3\n Compiling smallvec v1.15.1\n Compiling constant_time_eq v0.4.2\n Compiling getrandom v0.2.17\n Compiling spacetimedb-lib v1.11.1\n Compiling serde_json v1.0.149\n Compiling rand_core v0.6.4\n Compiling second-stack v0.3.5\n Compiling bytemuck v1.24.0\n Compiling itoa v1.0.17\n Compiling log v0.4.29\n Compiling memchr v2.7.6\n Compiling generic-array v0.14.7\n Compiling scoped-tls v1.0.1\n Compiling itertools v0.12.1\n Compiling cc v1.2.53\n Compiling num-traits v0.2.19\n Compiling http v1.4.0\n Compiling syn v2.0.114\n Compiling approx v0.3.2\n Compiling chrono v0.4.43\n Compiling decorum v0.3.1\n Compiling block-buffer v0.10.4\n Compiling crypto-common v0.1.7\n Compiling blake3 v1.8.3\n Compiling digest v0.10.7\n Compiling sha3 v0.10.8\n Compiling ppv-lite86 v0.2.21\n Compiling rand_chacha v0.3.1\n Compiling rand v0.8.5\n Compiling ethnum v1.5.2\n Compiling enum-as-inner v0.6.1\n Compiling thiserror-impl v1.0.69\n Compiling derive_more v0.99.20\n Compiling spacetimedb-primitives v1.11.1\n Compiling spacetimedb-bindings-sys v1.11.1\n Compiling spacetimedb-bindings-macro v1.11.1\n Compiling spacetimedb-sats v1.11.1\n Compiling spacetimedb v1.11.1\n Compiling spacetime-module v0.1.0 (/__w/SpacetimeDB/SpacetimeDB/target/llm-runs/basics/t_002_scheduled_table/rust/server/gpt-5/llm)\nerror: expected `at`\n --> src/lib.rs:4:38\n |\n4 | #[table(name = tick_timer, scheduled(reducer = tick, column = scheduled_at))]\n | ^^^^^^^\n\nerror[E0422]: cannot find struct, variant or union type `TickTimer` in this scope\n --> src/lib.rs:15:36\n |\n15 | ctx.db.tick_timer().insert(TickTimer {\n | ^^^^^^^^^ not found in this scope\n\nwarning: unused import: `UniqueColumn`\n --> src/lib.rs:2:70\n |\n2 | use spacetimedb::{table, reducer, ReducerContext, ScheduleAt, Table, UniqueColumn};\n | ^^^^^^^^^^^^\n |\n = note: `#[warn(unused_imports)]` on by default\n\nerror[E0599]: no method named `tick_timer` found for struct `Local` in the current scope\n --> src/lib.rs:14:15\n |\n14 | if ctx.db.tick_timer().count() == 0 {\n | ^^^^^^^^^^ method not found in `Local`\n\nerror[E0599]: no method named `tick_timer` found for struct `Local` in the current scope\n --> src/lib.rs:15:16\n |\n15 | ctx.db.tick_timer().insert(TickTimer {\n | ^^^^^^^^^^ method not found in `Local`\n\nerror[E0599]: no variant or associated item named `AfterMicros` found for enum `ScheduleAt` in the current scope\n --> src/lib.rs:17:39\n |\n17 | scheduled_at: ScheduleAt::AfterMicros(50_000),\n | ^^^^^^^^^^^ variant or associated item not found in `ScheduleAt`\n\nerror[E0599]: no method named `tick_timer` found for struct `Local` in the current scope\n --> src/lib.rs:24:35\n |\n24 | if let Some(mut row) = ctx.db.tick_timer().scheduled_id().find(scheduled_id) {\n | ^^^^^^^^^^ method not found in `Local`\n\nerror[E0599]: no variant or associated item named `AfterMicros` found for enum `ScheduleAt` in the current scope\n --> src/lib.rs:25:40\n |\n25 | row.scheduled_at = ScheduleAt::AfterMicros(50_000);\n | ^^^^^^^^^^^ variant or associated item not found in `ScheduleAt`\n\nerror[E0599]: no method named `tick_timer` found for struct `Local` in the current scope\n --> src/lib.rs:26:24\n |\n26 | let _ = ctx.db.tick_timer().scheduled_id().update(row);\n | ^^^^^^^^^^ method not found in `Local`\n\nSome errors have detailed explanations: E0422, E0599.\nFor more information about an error, try `rustc --explain E0422`.\nwarning: `spacetime-module` (lib) generated 1 warning\nerror: could not compile `spacetime-module` (lib) due to 8 previous errors; 1 warning emitted\nError: command [\"cargo\", \"build\", \"--config=net.git-fetch-with-cli=true\", \"--target=wasm32-unknown-unknown\", \"--release\", \"--message-format=json-render-diagnostics\"] exited with code 101\n\n--- stdout ---\n", "phase": "build_or_publish" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:25:37.294914196Z", - "finished_at": "2026-01-22T01:26:32.412919431Z" + "started_at": "2026-01-22T23:36:06.848466070Z", + "finished_at": "2026-01-22T23:37:20.323688218Z" }, "t_003_struct_in_table": { "hash": "e6a5a5374022e3250e0df2cb5e4f7e4777081f474a0ebc4e99302be42151ef59", @@ -115,8 +115,8 @@ "golden_published": true, "model_name": "GPT-5", "total_tests": 1, - "passed_tests": 0, - "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};\n\n#[derive(SpacetimeType)]\npub struct Position {\n x: i32,\n y: i32,\n}\n\n#[table(name = entity)]\npub struct Entity {\n #[primary_key]\n id: i32,\n pos: Position,\n}\n\n#[reducer]\npub fn add(ctx: &ReducerContext, id: i32, x: i32, y: i32) {\n ctx.db.entity().insert(Entity { id, pos: Position { x, y } });\n}", + "passed_tests": 1, + "llm_output": "use spacetimedb::{table, SpacetimeType};\n\n#[derive(SpacetimeType)]\npub struct Position {\n x: i32,\n y: i32,\n}\n\n#[table(name = entity)]\npub struct Entity {\n #[primary_key]\n id: i32,\n pos: Position,\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-003-struct-in-table-golden", @@ -125,27 +125,22 @@ "work_dir_llm": "target/llm-runs/basics/t_003_struct_in_table/rust/server/gpt-5/llm", "scorer_details": { "schema_parity": { - "pass": false, - "partial": 0.0, + "pass": true, + "partial": 1.0, "notes": { "golden_db": "basics-t-003-struct-in-table-golden", "llm_db": "basics-t-003-struct-in-table-gpt-5-llm", - "reducers_diff": { - "only_golden": [], - "only_llm": [ - "add()" - ] - }, - "reducers_equal": false, - "server": "http://127.0.0.1:39821", + "reducers_diff": null, + "reducers_equal": true, + "server": "http://127.0.0.1:34355", "tables_diff": null, "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-22T01:25:25.447144959Z", - "finished_at": "2026-01-22T01:25:57.646551775Z" + "started_at": "2026-01-22T23:35:41.993214397Z", + "finished_at": "2026-01-22T23:36:12.517794996Z" }, "t_004_insert": { "hash": "e6a5a5374022e3250e0df2cb5e4f7e4777081f474a0ebc4e99302be42151ef59", @@ -163,19 +158,6 @@ "work_dir_golden": "target/llm-runs/basics/t_004_insert/rust/server/golden", "work_dir_llm": "target/llm-runs/basics/t_004_insert/rust/server/gpt-5/llm", "scorer_details": { - "schema_parity": { - "pass": true, - "partial": 1.0, - "notes": { - "golden_db": "basics-t-004-insert-golden", - "llm_db": "basics-t-004-insert-gpt-5-llm", - "reducers_diff": null, - "reducers_equal": true, - "server": "http://127.0.0.1:39821", - "tables_diff": null, - "tables_equal": true - } - }, "data_parity_insert_user": { "pass": true, "partial": 1.0, @@ -192,13 +174,26 @@ "llm_out": "id | name | age | active ----+---------+-----+-------- 1 | \"Alice\" | 30 | true", "query": "SELECT id, name, age, active FROM user WHERE id=1", "reducer": "insert_user", - "server": "http://127.0.0.1:39821" + "server": "http://127.0.0.1:34355" + } + }, + "schema_parity": { + "pass": true, + "partial": 1.0, + "notes": { + "golden_db": "basics-t-004-insert-golden", + "llm_db": "basics-t-004-insert-gpt-5-llm", + "reducers_diff": null, + "reducers_equal": true, + "server": "http://127.0.0.1:34355", + "tables_diff": null, + "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-22T01:25:33.391232574Z", - "finished_at": "2026-01-22T01:25:51.396167872Z" + "started_at": "2026-01-22T23:35:49.718050276Z", + "finished_at": "2026-01-22T23:36:16.248623756Z" }, "t_005_update": { "hash": "e6a5a5374022e3250e0df2cb5e4f7e4777081f474a0ebc4e99302be42151ef59", @@ -208,7 +203,7 @@ "model_name": "GPT-5", "total_tests": 3, "passed_tests": 3, - "llm_output": "use spacetimedb::ReducerContext;\n\n#[spacetimedb::table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[spacetimedb::reducer]\npub fn update_user(ctx: &ReducerContext, id: i32, name: String, age: i32, active: bool) {\n if let Some(mut row) = ctx.db.user().id().find(id) {\n row.name = name;\n row.age = age;\n row.active = active;\n ctx.db.user().id().update(row);\n }\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table, UniqueColumn};\n\n#[spacetimedb::table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[spacetimedb::reducer]\npub fn update_user(ctx: &ReducerContext, id: i32, name: String, age: i32, active: bool) {\n let user_table = ctx.db.user();\n if user_table.id().find(id).is_some() {\n let row = User { id, name, age, active };\n let _ = user_table.id().update(row);\n }\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-005-update-golden", @@ -216,13 +211,6 @@ "work_dir_golden": "target/llm-runs/basics/t_005_update/rust/server/golden", "work_dir_llm": "target/llm-runs/basics/t_005_update/rust/server/gpt-5/llm", "scorer_details": { - "seed_users_row": { - "pass": true, - "partial": 1.0, - "notes": { - "sql": "INSERT INTO user(id, name, age, active) VALUES (1, 'Alice', 30, true)" - } - }, "schema_parity": { "pass": true, "partial": 1.0, @@ -231,7 +219,7 @@ "llm_db": "basics-t-005-update-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:39821", + "server": "http://127.0.0.1:34355", "tables_diff": null, "tables_equal": true } @@ -252,13 +240,20 @@ "llm_out": "id | name | age | active ----+----------+-----+-------- 1 | \"Alice2\" | 31 | false", "query": "SELECT id, name, age, active FROM user WHERE id=1", "reducer": "update_user", - "server": "http://127.0.0.1:39821" + "server": "http://127.0.0.1:34355" + } + }, + "seed_users_row": { + "pass": true, + "partial": 1.0, + "notes": { + "sql": "INSERT INTO user(id, name, age, active) VALUES (1, 'Alice', 30, true)" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:25:12.352259303Z", - "finished_at": "2026-01-22T01:26:34.203855771Z" + "started_at": "2026-01-22T23:35:37.701662220Z", + "finished_at": "2026-01-22T23:36:25.946922251Z" }, "t_006_delete": { "hash": "e6a5a5374022e3250e0df2cb5e4f7e4777081f474a0ebc4e99302be42151ef59", @@ -276,6 +271,13 @@ "work_dir_golden": "target/llm-runs/basics/t_006_delete/rust/server/golden", "work_dir_llm": "target/llm-runs/basics/t_006_delete/rust/server/gpt-5/llm", "scorer_details": { + "seed_users_row": { + "pass": true, + "partial": 1.0, + "notes": { + "sql": "INSERT INTO user(id, name, age, active) VALUES (1, 'Alice', 30, true)" + } + }, "delete_user_count_zero": { "pass": true, "partial": 1.0, @@ -293,22 +295,15 @@ "llm_db": "basics-t-006-delete-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:39821", + "server": "http://127.0.0.1:34355", "tables_diff": null, "tables_equal": true } - }, - "seed_users_row": { - "pass": true, - "partial": 1.0, - "notes": { - "sql": "INSERT INTO user(id, name, age, active) VALUES (1, 'Alice', 30, true)" - } } }, "vendor": "openai", - "started_at": "2026-01-22T01:24:50.475412193Z", - "finished_at": "2026-01-22T01:25:31.552788372Z" + "started_at": "2026-01-22T23:35:17.032754820Z", + "finished_at": "2026-01-22T23:35:38.832076867Z" }, "t_007_crud": { "hash": "e6a5a5374022e3250e0df2cb5e4f7e4777081f474a0ebc4e99302be42151ef59", @@ -318,7 +313,7 @@ "model_name": "GPT-5", "total_tests": 4, "passed_tests": 4, - "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table};\n\n#[table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[reducer]\npub fn crud(ctx: &ReducerContext) {\n let _ = ctx.db.user().insert(User { id: 1, name: \"Alice\".to_string(), age: 30, active: true });\n let _ = ctx.db.user().insert(User { id: 2, name: \"Bob\".to_string(), age: 22, active: false });\n let _ = ctx.db.user().id().update(User { id: 1, name: \"Alice2\".to_string(), age: 31, active: false });\n ctx.db.user().id().delete(&2);\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[spacetimedb::reducer]\npub fn crud(ctx: &ReducerContext) {\n ctx.db.user().insert(User { id: 1, name: \"Alice\".to_string(), age: 30, active: true });\n ctx.db.user().insert(User { id: 2, name: \"Bob\".to_string(), age: 22, active: false });\n\n if let Some(mut u) = ctx.db.user().id().find(1i32) {\n u.name = \"Alice2\".to_string();\n u.age = 31;\n u.active = false;\n ctx.db.user().id().update(u);\n }\n\n ctx.db.user().id().delete(&2i32);\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-007-crud-golden", @@ -335,6 +330,15 @@ "sql": "SELECT COUNT(*) AS n FROM user" } }, + "crud_row_id2_deleted": { + "pass": true, + "partial": 1.0, + "notes": { + "actual": 0, + "expected": 0, + "sql": "SELECT COUNT(*) AS n FROM user WHERE id=2" + } + }, "schema_parity": { "pass": true, "partial": 1.0, @@ -343,7 +347,7 @@ "llm_db": "basics-t-007-crud-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:39821", + "server": "http://127.0.0.1:34355", "tables_diff": null, "tables_equal": true } @@ -359,22 +363,13 @@ "llm_out": "id | name | age | active ----+----------+-----+-------- 1 | \"Alice2\" | 31 | false", "query": "SELECT id, name, age, active FROM user WHERE id=1", "reducer": "crud", - "server": "http://127.0.0.1:39821" - } - }, - "crud_row_id2_deleted": { - "pass": true, - "partial": 1.0, - "notes": { - "actual": 0, - "expected": 0, - "sql": "SELECT COUNT(*) AS n FROM user WHERE id=2" + "server": "http://127.0.0.1:34355" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:24:54.900483411Z", - "finished_at": "2026-01-22T01:25:34.035999728Z" + "started_at": "2026-01-22T23:35:31.552527058Z", + "finished_at": "2026-01-22T23:36:16.529536165Z" }, "t_008_index_lookup": { "hash": "e6a5a5374022e3250e0df2cb5e4f7e4777081f474a0ebc4e99302be42151ef59", @@ -384,7 +379,7 @@ "model_name": "GPT-5", "total_tests": 3, "passed_tests": 3, - "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table};\n\n#[table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n name: String,\n}\n\n#[reducer]\npub fn lookup_user_name(ctx: &ReducerContext, id: i32) {\n if let Some(user) = ctx.db.user().id().find(id) {\n ctx.db.result().insert(ResultRow {\n id: user.id,\n name: user.name,\n });\n }\n}", + "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table};\n\n#[table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n name: String,\n}\n\n#[reducer]\nfn lookup_user_name(ctx: &ReducerContext, id: i32) {\n if let Some(user) = ctx.db.user().id().find(id) {\n let result_tbl = ctx.db.result();\n result_tbl.id().delete(&user.id);\n result_tbl.insert(ResultRow { id: user.id, name: user.name });\n }\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-008-index-lookup-golden", @@ -392,19 +387,6 @@ "work_dir_golden": "target/llm-runs/basics/t_008_index_lookup/rust/server/golden", "work_dir_llm": "target/llm-runs/basics/t_008_index_lookup/rust/server/gpt-5/llm", "scorer_details": { - "schema_parity": { - "pass": true, - "partial": 1.0, - "notes": { - "golden_db": "basics-t-008-index-lookup-golden", - "llm_db": "basics-t-008-index-lookup-gpt-5-llm", - "reducers_diff": null, - "reducers_equal": true, - "server": "http://127.0.0.1:39821", - "tables_diff": null, - "tables_equal": true - } - }, "index_lookup_projection_parity": { "pass": true, "partial": 1.0, @@ -418,7 +400,20 @@ "llm_out": "id | name ----+--------- 1 | \"Alice\"", "query": "SELECT id, name FROM result WHERE id=1", "reducer": "lookup_user_name", - "server": "http://127.0.0.1:39821" + "server": "http://127.0.0.1:34355" + } + }, + "schema_parity": { + "pass": true, + "partial": 1.0, + "notes": { + "golden_db": "basics-t-008-index-lookup-golden", + "llm_db": "basics-t-008-index-lookup-gpt-5-llm", + "reducers_diff": null, + "reducers_equal": true, + "server": "http://127.0.0.1:34355", + "tables_diff": null, + "tables_equal": true } }, "seed_user_row": { @@ -430,8 +425,8 @@ } }, "vendor": "openai", - "started_at": "2026-01-22T01:25:33.390946322Z", - "finished_at": "2026-01-22T01:26:27.066650342Z" + "started_at": "2026-01-22T23:35:48.205606693Z", + "finished_at": "2026-01-22T23:36:28.825360218Z" }, "t_009_init": { "hash": "e6a5a5374022e3250e0df2cb5e4f7e4777081f474a0ebc4e99302be42151ef59", @@ -449,22 +444,22 @@ "work_dir_golden": "target/llm-runs/basics/t_009_init/rust/server/golden", "work_dir_llm": "target/llm-runs/basics/t_009_init/rust/server/gpt-5/llm", "scorer_details": { - "init_seed_alice": { + "init_seed_bob": { "pass": true, "partial": 1.0, "notes": { "actual": 1, "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM user WHERE id=1 AND name='Alice' AND age=30 AND active=true" + "sql": "SELECT COUNT(*) AS n FROM user WHERE id=2 AND name='Bob' AND age=22 AND active=false" } }, - "init_seed_bob": { + "init_seed_alice": { "pass": true, "partial": 1.0, "notes": { "actual": 1, "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM user WHERE id=2 AND name='Bob' AND age=22 AND active=false" + "sql": "SELECT COUNT(*) AS n FROM user WHERE id=1 AND name='Alice' AND age=30 AND active=true" } }, "init_total_two": { @@ -484,15 +479,15 @@ "llm_db": "basics-t-009-init-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:39821", + "server": "http://127.0.0.1:34355", "tables_diff": null, "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-22T01:25:08.162931518Z", - "finished_at": "2026-01-22T01:25:37.294887085Z" + "started_at": "2026-01-22T23:35:37.693736731Z", + "finished_at": "2026-01-22T23:36:06.848337544Z" }, "t_010_connect": { "hash": "e6a5a5374022e3250e0df2cb5e4f7e4777081f474a0ebc4e99302be42151ef59", @@ -502,7 +497,7 @@ "model_name": "GPT-5", "total_tests": 1, "passed_tests": 1, - "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table};\n\n#[table(name = event)]\npub struct Event {\n #[primary_key]\n #[auto_inc]\n id: i32,\n kind: String,\n}\n\n#[reducer(client_connected)]\npub fn client_connected(ctx: &ReducerContext) {\n ctx.db.event().insert(Event { id: 0, kind: \"connected\".to_string() });\n}\n\n#[reducer(client_disconnected)]\npub fn client_disconnected(ctx: &ReducerContext) {\n ctx.db.event().insert(Event { id: 0, kind: \"disconnected\".to_string() });\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = event)]\npub struct Event {\n #[primary_key]\n #[auto_inc]\n id: i32,\n kind: String,\n}\n\n#[spacetimedb::reducer(client_connected)]\npub fn client_connected(ctx: &ReducerContext) {\n ctx.db.event().insert(Event { id: 0, kind: \"connected\".to_string() });\n}\n\n#[spacetimedb::reducer(client_disconnected)]\npub fn client_disconnected(ctx: &ReducerContext) {\n ctx.db.event().insert(Event { id: 0, kind: \"disconnected\".to_string() });\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-010-connect-golden", @@ -518,15 +513,15 @@ "llm_db": "basics-t-010-connect-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:39821", + "server": "http://127.0.0.1:34355", "tables_diff": null, "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-22T01:24:54.469831077Z", - "finished_at": "2026-01-22T01:25:33.390915278Z" + "started_at": "2026-01-22T23:35:19.958386897Z", + "finished_at": "2026-01-22T23:35:49.718017951Z" }, "t_011_helper_function": { "hash": "e6a5a5374022e3250e0df2cb5e4f7e4777081f474a0ebc4e99302be42151ef59", @@ -552,7 +547,7 @@ "llm_db": "basics-t-011-helper-function-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:39821", + "server": "http://127.0.0.1:34355", "tables_diff": null, "tables_equal": true } @@ -581,13 +576,13 @@ "llm_out": "id | sum ----+----- 1 | 5", "query": "SELECT id, sum FROM result WHERE id=1", "reducer": "compute_sum", - "server": "http://127.0.0.1:39821" + "server": "http://127.0.0.1:34355" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:25:34.036029292Z", - "finished_at": "2026-01-22T01:26:03.502119715Z" + "started_at": "2026-01-22T23:35:58.993668322Z", + "finished_at": "2026-01-22T23:36:27.348892624Z" }, "t_012_spacetime_product_type": { "hash": "e6a5a5374022e3250e0df2cb5e4f7e4777081f474a0ebc4e99302be42151ef59", @@ -597,7 +592,7 @@ "model_name": "GPT-5", "total_tests": 3, "passed_tests": 3, - "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};\n\n#[derive(SpacetimeType)]\npub struct Score {\n left: i32,\n right: i32,\n}\n\n#[table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n value: Score,\n}\n\n#[reducer]\nfn set_score(ctx: &ReducerContext, id: i32, left: i32, right: i32) {\n ctx.db.result().insert(ResultRow {\n id,\n value: Score { left, right },\n });\n}", + "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};\n\n#[derive(SpacetimeType)]\nstruct Score {\n left: i32,\n right: i32,\n}\n\n#[table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n value: Score,\n}\n\n#[reducer]\npub fn set_score(ctx: &ReducerContext, id: i32, left: i32, right: i32) {\n ctx.db.result().insert(ResultRow { id, value: Score { left, right } });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-012-spacetime-product-type-golden", @@ -605,24 +600,6 @@ "work_dir_golden": "target/llm-runs/schema/t_012_spacetime_product_type/rust/server/golden", "work_dir_llm": "target/llm-runs/schema/t_012_spacetime_product_type/rust/server/gpt-5/llm", "scorer_details": { - "product_type_row_parity": { - "pass": true, - "partial": 1.0, - "notes": { - "args": [ - 1, - 2, - 3 - ], - "golden_db": "schema-t-012-spacetime-product-type-golden", - "golden_out": "id | value ----+----------------------- 1 | (left = 2, right = 3)", - "llm_db": "schema-t-012-spacetime-product-type-gpt-5-llm", - "llm_out": "id | value ----+----------------------- 1 | (left = 2, right = 3)", - "query": "SELECT id, value FROM result WHERE id=1", - "reducer": "set_score", - "server": "http://127.0.0.1:39821" - } - }, "schema_parity": { "pass": true, "partial": 1.0, @@ -631,7 +608,7 @@ "llm_db": "schema-t-012-spacetime-product-type-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:39821", + "server": "http://127.0.0.1:34355", "tables_diff": null, "tables_equal": true } @@ -644,11 +621,29 @@ "expected": 1, "sql": "SELECT COUNT(*) AS n FROM result WHERE id=1" } + }, + "product_type_row_parity": { + "pass": true, + "partial": 1.0, + "notes": { + "args": [ + 1, + 2, + 3 + ], + "golden_db": "schema-t-012-spacetime-product-type-golden", + "golden_out": "id | value ----+----------------------- 1 | (left = 2, right = 3)", + "llm_db": "schema-t-012-spacetime-product-type-gpt-5-llm", + "llm_out": "id | value ----+----------------------- 1 | (left = 2, right = 3)", + "query": "SELECT id, value FROM result WHERE id=1", + "reducer": "set_score", + "server": "http://127.0.0.1:34355" + } } }, "vendor": "openai", - "started_at": "2026-01-22T01:24:49.410082360Z", - "finished_at": "2026-01-22T01:25:25.446954764Z" + "started_at": "2026-01-22T23:35:13.266122816Z", + "finished_at": "2026-01-22T23:35:48.205568802Z" }, "t_013_spacetime_sum_type": { "hash": "e6a5a5374022e3250e0df2cb5e4f7e4777081f474a0ebc4e99302be42151ef59", @@ -658,7 +653,7 @@ "model_name": "GPT-5", "total_tests": 3, "passed_tests": 2, - "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};\n\n#[derive(SpacetimeType)]\npub struct Rect {\n width: i32,\n height: i32,\n}\n\n#[derive(SpacetimeType)]\npub enum Shape {\n Circle(i32),\n Rectangle(Rect),\n}\n\n#[table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n value: Shape,\n}\n\n#[reducer]\npub fn set_circle(ctx: &ReducerContext, id: i32, radius: i32) {\n ctx.db.result().insert(ResultRow {\n id,\n value: Shape::Circle(radius),\n });\n}", + "llm_output": "use spacetimedb::{reducer, table, ReducerContext, SpacetimeType, Table};\n\n#[derive(SpacetimeType)]\npub struct Rect {\n width: i32,\n height: i32,\n}\n\n#[derive(SpacetimeType)]\npub enum Shape {\n Circle(i32),\n Rectangle(Rect),\n}\n\n#[table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n value: Shape,\n}\n\n#[reducer]\npub fn set_circle(ctx: &ReducerContext, id: i32, radius: i32) {\n ctx.db.result().insert(ResultRow {\n id,\n value: Shape::Circle(radius),\n });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-013-spacetime-sum-type-golden", @@ -670,10 +665,19 @@ "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `result`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:39821/v1/database/c200ab6d0d2f3a47a550f7c20376e4d9a922efae9d71b275471b6a99cfb891e9/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `result`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:34355/v1/database/c2001fcd14923f0a4ee32425c198f0848a4b5dd1d2cd2dbdd504064c963a79f4/sql)\n", "phase": "sql_golden" } }, + "sum_type_row_count": { + "pass": true, + "partial": 1.0, + "notes": { + "actual": 1, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM result WHERE id=1" + } + }, "schema_parity": { "pass": true, "partial": 1.0, @@ -682,24 +686,15 @@ "llm_db": "schema-t-013-spacetime-sum-type-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:39821", + "server": "http://127.0.0.1:34355", "tables_diff": null, "tables_equal": true } - }, - "sum_type_row_count": { - "pass": true, - "partial": 1.0, - "notes": { - "actual": 1, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM result WHERE id=1" - } } }, "vendor": "openai", - "started_at": "2026-01-22T01:24:16.320845083Z", - "finished_at": "2026-01-22T01:24:54.900438385Z" + "started_at": "2026-01-22T23:34:44.633845523Z", + "finished_at": "2026-01-22T23:35:13.266066695Z" }, "t_014_elementary_columns": { "hash": "e6a5a5374022e3250e0df2cb5e4f7e4777081f474a0ebc4e99302be42151ef59", @@ -709,7 +704,7 @@ "model_name": "GPT-5", "total_tests": 3, "passed_tests": 3, - "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = primitive)]\npub struct Primitive {\n #[primary_key]\n id: i32,\n count: i32,\n total: i64,\n price: f32,\n ratio: f64,\n active: bool,\n name: String,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n ctx.db.primitive().insert(Primitive {\n id: 1,\n count: 2,\n total: 3_000_000_000,\n price: 1.5,\n ratio: 2.25,\n active: true,\n name: \"Alice\".to_string(),\n });\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = primitive)]\npub struct Primitive {\n #[primary_key]\n id: i32,\n count: i32,\n total: i64,\n price: f32,\n ratio: f64,\n active: bool,\n name: String,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n ctx.db.primitive().insert(Primitive {\n id: 1,\n count: 2,\n total: 3000000000,\n price: 1.5f32,\n ratio: 2.25,\n active: true,\n name: \"Alice\".to_string(),\n });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-014-elementary-columns-golden", @@ -726,47 +721,47 @@ "sql": "SELECT COUNT(*) AS n FROM primitive WHERE id=1" } }, - "elementary_columns_row_parity": { + "schema_parity": { "pass": true, "partial": 1.0, "notes": { - "args": [], "golden_db": "schema-t-014-elementary-columns-golden", - "golden_out": "id | count | total | price | ratio | active | name ----+-------+------------+-------+-------+--------+--------- 1 | 2 | 3000000000 | 1.5 | 2.25 | true | \"Alice\"", "llm_db": "schema-t-014-elementary-columns-gpt-5-llm", - "llm_out": "id | count | total | price | ratio | active | name ----+-------+------------+-------+-------+--------+--------- 1 | 2 | 3000000000 | 1.5 | 2.25 | true | \"Alice\"", - "query": "SELECT id, count, total, price, ratio, active, name FROM primitive WHERE id=1", - "reducer": "seed", - "server": "http://127.0.0.1:39821" + "reducers_diff": null, + "reducers_equal": true, + "server": "http://127.0.0.1:34355", + "tables_diff": null, + "tables_equal": true } }, - "schema_parity": { + "elementary_columns_row_parity": { "pass": true, "partial": 1.0, "notes": { + "args": [], "golden_db": "schema-t-014-elementary-columns-golden", + "golden_out": "id | count | total | price | ratio | active | name ----+-------+------------+-------+-------+--------+--------- 1 | 2 | 3000000000 | 1.5 | 2.25 | true | \"Alice\"", "llm_db": "schema-t-014-elementary-columns-gpt-5-llm", - "reducers_diff": null, - "reducers_equal": true, - "server": "http://127.0.0.1:39821", - "tables_diff": null, - "tables_equal": true + "llm_out": "id | count | total | price | ratio | active | name ----+-------+------------+-------+-------+--------+--------- 1 | 2 | 3000000000 | 1.5 | 2.25 | true | \"Alice\"", + "query": "SELECT id, count, total, price, ratio, active, name FROM primitive WHERE id=1", + "reducer": "seed", + "server": "http://127.0.0.1:34355" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:24:44.642076666Z", - "finished_at": "2026-01-22T01:25:08.162897725Z" + "started_at": "2026-01-22T23:35:12.446556930Z", + "finished_at": "2026-01-22T23:35:46.097887051Z" }, "t_015_product_type_columns": { "hash": "e6a5a5374022e3250e0df2cb5e4f7e4777081f474a0ebc4e99302be42151ef59", "task": "t_015_product_type_columns", "lang": "rust", - "golden_published": true, + "golden_published": false, "model_name": "GPT-5", "total_tests": 3, - "passed_tests": 2, - "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};\n\n#[derive(SpacetimeType)]\nstruct Address {\n street: String,\n zip: i32,\n}\n\n#[derive(SpacetimeType)]\nstruct Position {\n x: i32,\n y: i32,\n}\n\n#[table(name = profile)]\npub struct Profile {\n #[primary_key]\n id: i32,\n home: Address,\n work: Address,\n pos: Position,\n}\n\n#[reducer]\nfn seed(ctx: &ReducerContext) {\n ctx.db.profile().insert(Profile {\n id: 1,\n home: Address {\n street: \"1 Main\".to_string(),\n zip: 11111,\n },\n work: Address {\n street: \"2 Broad\".to_string(),\n zip: 22222,\n },\n pos: Position { x: 7, y: 9 },\n });\n}", + "passed_tests": 0, + "llm_output": "use spacetimedb::{reducer, table, ReducerContext, SpacetimeType, Table};\n\n#[derive(SpacetimeType)]\npub struct Address {\n street: String,\n zip: i32,\n}\n\n#[derive(SpacetimeType)]\npub struct Position {\n x: i32,\n y: i32,\n}\n\n#[table(name = profile)]\npub struct Profile {\n #[primary_key]\n id: i32,\n home: Address,\n work: Address,\n pos: Position,\n}\n\n#[reducer]\npub fn seed(ctx: &ReducerContext) {\n for row in ctx.db.profile().iter() {\n ctx.db.profile().delete(&row);\n }\n ctx.db.profile().insert(Profile {\n id: 1,\n home: Address {\n street: \"1 Main\".to_string(),\n zip: 11111,\n },\n work: Address {\n street: \"2 Broad\".to_string(),\n zip: 22222,\n },\n pos: Position { x: 7, y: 9 },\n });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-015-product-type-columns-golden", @@ -774,40 +769,18 @@ "work_dir_golden": "target/llm-runs/schema/t_015_product_type_columns/rust/server/golden", "work_dir_llm": "target/llm-runs/schema/t_015_product_type_columns/rust/server/gpt-5/llm", "scorer_details": { - "schema_parity": { - "pass": true, - "partial": 1.0, - "notes": { - "golden_db": "schema-t-015-product-type-columns-golden", - "llm_db": "schema-t-015-product-type-columns-gpt-5-llm", - "reducers_diff": null, - "reducers_equal": true, - "server": "http://127.0.0.1:39821", - "tables_diff": null, - "tables_equal": true - } - }, - "product_type_columns_row_parity": { + "publish_error": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `profile`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:39821/v1/database/c2005343392d020f741d1e7f32aa16033e8a15698255b7bd693e90044bf84800/sql)\n", - "phase": "sql_golden" - } - }, - "product_type_columns_row_count": { - "pass": true, - "partial": 1.0, - "notes": { - "actual": 1, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM profile WHERE id=1" + "error": "spacetime publish failed (exit=1)\n--- stderr ---\n Updating crates.io index\n Locking 72 packages to latest compatible versions\n Adding generic-array v0.14.7 (available: v0.14.9)\n Adding spacetimedb v1.11.1 (available: v1.11.3)\n Adding spacetimedb-bindings-macro v1.11.1 (available: v1.11.3)\n Adding spacetimedb-bindings-sys v1.11.1 (available: v1.11.3)\n Adding spacetimedb-lib v1.11.1 (available: v1.11.3)\n Adding spacetimedb-primitives v1.11.1 (available: v1.11.3)\n Adding spacetimedb-sats v1.11.1 (available: v1.11.3)\n Compiling proc-macro2 v1.0.106\n Compiling quote v1.0.43\n Compiling unicode-ident v1.0.22\n Compiling version_check v0.9.5\n Compiling typenum v1.19.0\n Compiling autocfg v1.5.0\n Compiling serde_core v1.0.228\n Compiling heck v0.5.0\n Compiling cfg-if v1.0.4\n Compiling either v1.15.0\n Compiling zerocopy v0.8.33\n Compiling shlex v1.3.0\n Compiling find-msvc-tools v0.1.8\n Compiling serde v1.0.228\n Compiling nohash-hasher v0.2.0\n Compiling bitflags v2.10.0\n Compiling anyhow v1.0.100\n Compiling thiserror v1.0.69\n Compiling bytes v1.11.0\n Compiling zmij v1.0.16\n Compiling keccak v0.1.5\n Compiling humantime v2.3.0\n Compiling heck v0.4.1\n Compiling convert_case v0.4.0\n Compiling arrayvec v0.7.6\n Compiling arrayref v0.3.9\n Compiling serde_json v1.0.149\n Compiling bytemuck v1.24.0\n Compiling spacetimedb-lib v1.11.1\n Compiling getrandom v0.2.17\n Compiling constant_time_eq v0.4.2\n Compiling itoa v1.0.17\n Compiling itertools v0.12.1\n Compiling rand_core v0.6.4\n Compiling second-stack v0.3.5\n Compiling hex v0.4.3\n Compiling smallvec v1.15.1\n Compiling cc v1.2.53\n Compiling memchr v2.7.6\n Compiling log v0.4.29\n Compiling scoped-tls v1.0.1\n Compiling generic-array v0.14.7\n Compiling num-traits v0.2.19\n Compiling http v1.4.0\n Compiling syn v2.0.114\n Compiling approx v0.3.2\n Compiling chrono v0.4.43\n Compiling decorum v0.3.1\n Compiling blake3 v1.8.3\n Compiling block-buffer v0.10.4\n Compiling crypto-common v0.1.7\n Compiling digest v0.10.7\n Compiling sha3 v0.10.8\n Compiling ppv-lite86 v0.2.21\n Compiling rand_chacha v0.3.1\n Compiling rand v0.8.5\n Compiling ethnum v1.5.2\n Compiling enum-as-inner v0.6.1\n Compiling thiserror-impl v1.0.69\n Compiling derive_more v0.99.20\n Compiling spacetimedb-primitives v1.11.1\n Compiling spacetimedb-bindings-sys v1.11.1\n Compiling spacetimedb-bindings-macro v1.11.1\n Compiling spacetimedb-sats v1.11.1\n Compiling spacetimedb v1.11.1\n Compiling spacetime-module v0.1.0 (/__w/SpacetimeDB/SpacetimeDB/target/llm-runs/schema/t_015_product_type_columns/rust/server/gpt-5/llm)\nerror[E0308]: mismatched types\n --> src/lib.rs:28:33\n |\n 28 | ctx.db.profile().delete(&row);\n | ------ ^^^^ expected `Profile`, found `&Profile`\n | |\n | arguments to this method are incorrect\n |\nnote: method defined here\n --> /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/spacetimedb-1.11.1/src/table.rs:104:8\n |\n104 | fn delete(&self, row: Self::Row) -> bool {\n | ^^^^^^\nhelp: consider removing the borrow\n |\n 28 - ctx.db.profile().delete(&row);\n 28 + ctx.db.profile().delete(row);\n |\n\nFor more information about this error, try `rustc --explain E0308`.\nerror: could not compile `spacetime-module` (lib) due to 1 previous error\nError: command [\"cargo\", \"build\", \"--config=net.git-fetch-with-cli=true\", \"--target=wasm32-unknown-unknown\", \"--release\", \"--message-format=json-render-diagnostics\"] exited with code 101\n\n--- stdout ---\n", + "phase": "build_or_publish" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:24:16.321246370Z", - "finished_at": "2026-01-22T01:24:54.469787311Z" + "started_at": "2026-01-22T23:34:44.634239800Z", + "finished_at": "2026-01-22T23:35:17.032734427Z" }, "t_016_sum_type_columns": { "hash": "e6a5a5374022e3250e0df2cb5e4f7e4777081f474a0ebc4e99302be42151ef59", @@ -817,7 +790,7 @@ "model_name": "GPT-5", "total_tests": 3, "passed_tests": 1, - "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};\n\n#[derive(SpacetimeType)]\npub struct Rect {\n width: i32,\n height: i32,\n}\n\n#[derive(SpacetimeType)]\npub enum Shape {\n Circle(i32),\n Rectangle(Rect),\n}\n\n#[table(name = drawing)]\npub struct Drawing {\n #[primary_key]\n id: i32,\n a: Shape,\n b: Shape,\n}\n\n#[reducer]\npub fn seed(ctx: &ReducerContext) {\n let tbl = ctx.db.drawing();\n if tbl.id().find(1).is_none() {\n tbl.insert(Drawing {\n id: 1,\n a: Shape::Circle(10),\n b: Shape::Rectangle(Rect { width: 4, height: 6 }),\n });\n }\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table, SpacetimeType};\n\n#[derive(SpacetimeType)]\npub struct Rect {\n width: i32,\n height: i32,\n}\n\n#[derive(SpacetimeType)]\npub enum Shape {\n Circle(i32),\n Rectangle(Rect),\n}\n\n#[spacetimedb::table(name = drawing)]\npub struct Drawing {\n #[primary_key]\n id: i32,\n a: Shape,\n b: Shape,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n let tbl = ctx.db.drawing();\n if tbl.id().find(1).is_none() {\n tbl.insert(Drawing {\n id: 1,\n a: Shape::Circle(10),\n b: Shape::Rectangle(Rect { width: 4, height: 6 }),\n });\n }\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-016-sum-type-columns-golden", @@ -829,7 +802,7 @@ "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `drawings`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:39821/v1/database/c2006951bc870c6119ff49074775c6eebb45753ba184e756afd7b35a26be9321/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `drawings`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:34355/v1/database/c200ea920cded2e0d48228d9c5f12b5129ceee5e5796bc785bec062cbccba269/sql)\n", "phase": "sql_golden" } }, @@ -841,7 +814,7 @@ "llm_db": "schema-t-016-sum-type-columns-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:39821", + "server": "http://127.0.0.1:34355", "tables_diff": null, "tables_equal": true } @@ -850,14 +823,14 @@ "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `drawings`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:39821/v1/database/c200532b634d5000c4706a493a717edfaf774990ace3f8ea2840b223bc25e28c/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `drawings`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:34355/v1/database/c200f5c0e231e85b77d8c5c8c7ba6b1d80477eca558bd8a502a4b5d91614bc8b/sql)\n", "phase": "sql" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:24:16.320702413Z", - "finished_at": "2026-01-22T01:24:44.641988063Z" + "started_at": "2026-01-22T23:34:44.633705973Z", + "finished_at": "2026-01-22T23:35:31.552419588Z" }, "t_017_scheduled_columns": { "hash": "e6a5a5374022e3250e0df2cb5e4f7e4777081f474a0ebc4e99302be42151ef59", @@ -867,7 +840,7 @@ "model_name": "GPT-5", "total_tests": 2, "passed_tests": 0, - "llm_output": "use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table};\n\n#[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]\npub struct TickTimer {\n #[primary_key]\n #[auto_inc]\n scheduled_id: u64,\n scheduled_at: ScheduleAt,\n}\n\n#[reducer(init)]\npub fn init(ctx: &ReducerContext) {\n if ctx.db.tick_timer().count() == 0 {\n ctx.db.tick_timer().insert(TickTimer {\n scheduled_id: 0,\n scheduled_at: ScheduleAt::repeat_us(50_000),\n });\n }\n}\n\n#[reducer(scheduled)]\npub fn tick(_ctx: &ReducerContext, _row: TickTimer) {}", + "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table, ScheduleAt};\n\n#[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]\npub struct TickTimer {\n #[primary_key]\n #[auto_inc]\n scheduled_id: u64,\n scheduled_at: ScheduleAt,\n}\n\n#[reducer(init)]\npub fn init(ctx: &ReducerContext) {\n ctx.db.tick_timer().insert(TickTimer {\n scheduled_id: 0,\n scheduled_at: ScheduleAt::every_micros(50_000),\n });\n}\n\n#[reducer(scheduled)]\npub fn tick(_ctx: &ReducerContext, _row: TickTimer) {\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-017-scheduled-columns-golden", @@ -879,14 +852,14 @@ "pass": false, "partial": 0.0, "notes": { - "error": "spacetime publish failed (exit=1)\n--- stderr ---\n Updating crates.io index\n Locking 72 packages to latest compatible versions\n Adding generic-array v0.14.7 (available: v0.14.9)\n Adding spacetimedb v1.11.1 (available: v1.11.3)\n Adding spacetimedb-bindings-macro v1.11.1 (available: v1.11.3)\n Adding spacetimedb-bindings-sys v1.11.1 (available: v1.11.3)\n Adding spacetimedb-lib v1.11.1 (available: v1.11.3)\n Adding spacetimedb-primitives v1.11.1 (available: v1.11.3)\n Adding spacetimedb-sats v1.11.1 (available: v1.11.3)\n Compiling proc-macro2 v1.0.106\n Compiling quote v1.0.43\n Compiling unicode-ident v1.0.22\n Compiling version_check v0.9.5\n Compiling typenum v1.19.0\n Compiling autocfg v1.5.0\n Compiling heck v0.5.0\n Compiling serde_core v1.0.228\n Compiling cfg-if v1.0.4\n Compiling find-msvc-tools v0.1.8\n Compiling shlex v1.3.0\n Compiling either v1.15.0\n Compiling zerocopy v0.8.33\n Compiling serde v1.0.228\n Compiling nohash-hasher v0.2.0\n Compiling anyhow v1.0.100\n Compiling bitflags v2.10.0\n Compiling thiserror v1.0.69\n Compiling arrayvec v0.7.6\n Compiling humantime v2.3.0\n Compiling heck v0.4.1\n Compiling keccak v0.1.5\n Compiling bytes v1.11.0\n Compiling convert_case v0.4.0\n Compiling zmij v1.0.16\n Compiling constant_time_eq v0.4.2\n Compiling bytemuck v1.24.0\n Compiling itoa v1.0.17\n Compiling arrayref v0.3.9\n Compiling getrandom v0.2.17\n Compiling serde_json v1.0.149\n Compiling smallvec v1.15.1\n Compiling cc v1.2.53\n Compiling spacetimedb-lib v1.11.1\n Compiling hex v0.4.3\n Compiling second-stack v0.3.5\n Compiling itertools v0.12.1\n Compiling log v0.4.29\n Compiling memchr v2.7.6\n Compiling scoped-tls v1.0.1\n Compiling rand_core v0.6.4\n Compiling num-traits v0.2.19\n Compiling generic-array v0.14.7\n Compiling syn v2.0.114\n Compiling http v1.4.0\n Compiling blake3 v1.8.3\n Compiling approx v0.3.2\n Compiling chrono v0.4.43\n Compiling decorum v0.3.1\n Compiling block-buffer v0.10.4\n Compiling crypto-common v0.1.7\n Compiling digest v0.10.7\n Compiling sha3 v0.10.8\n Compiling ppv-lite86 v0.2.21\n Compiling rand_chacha v0.3.1\n Compiling rand v0.8.5\n Compiling ethnum v1.5.2\n Compiling enum-as-inner v0.6.1\n Compiling thiserror-impl v1.0.69\n Compiling derive_more v0.99.20\n Compiling spacetimedb-primitives v1.11.1\n Compiling spacetimedb-bindings-sys v1.11.1\n Compiling spacetimedb-bindings-macro v1.11.1\n Compiling spacetimedb-sats v1.11.1\n Compiling spacetimedb v1.11.1\n Compiling spacetime-module v0.1.0 (/__w/SpacetimeDB/SpacetimeDB/target/llm-runs/schema/t_017_scheduled_columns/rust/server/gpt-5/llm)\nerror: expected one of: `public`, `private`, `name`, `index`, `scheduled`\n --> src/lib.rs:4:28\n |\n4 | #[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]\n | ^^^^^^^^\n\nerror: expected one of: `init`, `client_connected`, `client_disconnected`, `update`, `name`\n --> src/lib.rs:22:11\n |\n22 | #[reducer(scheduled)]\n | ^^^^^^^^^\n\nerror[E0422]: cannot find struct, variant or union type `TickTimer` in this scope\n --> src/lib.rs:15:36\n |\n15 | ctx.db.tick_timer().insert(TickTimer {\n | ^^^^^^^^^ not found in this scope\n\nerror[E0412]: cannot find type `TickTimer` in this scope\n --> src/lib.rs:23:42\n |\n23 | pub fn tick(_ctx: &ReducerContext, _row: TickTimer) {}\n | ^^^^^^^^^ not found in this scope\n\nerror[E0599]: no method named `tick_timer` found for struct `Local` in the current scope\n --> src/lib.rs:14:15\n |\n14 | if ctx.db.tick_timer().count() == 0 {\n | ^^^^^^^^^^ method not found in `Local`\n\nerror[E0599]: no method named `tick_timer` found for struct `Local` in the current scope\n --> src/lib.rs:15:16\n |\n15 | ctx.db.tick_timer().insert(TickTimer {\n | ^^^^^^^^^^ method not found in `Local`\n\nerror[E0599]: no variant or associated item named `repeat_us` found for enum `ScheduleAt` in the current scope\n --> src/lib.rs:17:39\n |\n17 | scheduled_at: ScheduleAt::repeat_us(50_000),\n | ^^^^^^^^^ variant or associated item not found in `ScheduleAt`\n\nSome errors have detailed explanations: E0412, E0422, E0599.\nFor more information about an error, try `rustc --explain E0412`.\nerror: could not compile `spacetime-module` (lib) due to 7 previous errors\nError: command [\"cargo\", \"build\", \"--config=net.git-fetch-with-cli=true\", \"--target=wasm32-unknown-unknown\", \"--release\", \"--message-format=json-render-diagnostics\"] exited with code 101\n\n--- stdout ---\n", + "error": "spacetime publish failed (exit=1)\n--- stderr ---\n Updating crates.io index\n Locking 72 packages to latest compatible versions\n Adding generic-array v0.14.7 (available: v0.14.9)\n Adding spacetimedb v1.11.1 (available: v1.11.3)\n Adding spacetimedb-bindings-macro v1.11.1 (available: v1.11.3)\n Adding spacetimedb-bindings-sys v1.11.1 (available: v1.11.3)\n Adding spacetimedb-lib v1.11.1 (available: v1.11.3)\n Adding spacetimedb-primitives v1.11.1 (available: v1.11.3)\n Adding spacetimedb-sats v1.11.1 (available: v1.11.3)\n Compiling proc-macro2 v1.0.106\n Compiling unicode-ident v1.0.22\n Compiling quote v1.0.43\n Compiling version_check v0.9.5\n Compiling typenum v1.19.0\n Compiling autocfg v1.5.0\n Compiling heck v0.5.0\n Compiling serde_core v1.0.228\n Compiling cfg-if v1.0.4\n Compiling shlex v1.3.0\n Compiling serde v1.0.228\n Compiling either v1.15.0\n Compiling find-msvc-tools v0.1.8\n Compiling zerocopy v0.8.33\n Compiling nohash-hasher v0.2.0\n Compiling thiserror v1.0.69\n Compiling bitflags v2.10.0\n Compiling anyhow v1.0.100\n Compiling zmij v1.0.16\n Compiling humantime v2.3.0\n Compiling arrayvec v0.7.6\n Compiling keccak v0.1.5\n Compiling heck v0.4.1\n Compiling bytes v1.11.0\n Compiling convert_case v0.4.0\n Compiling hex v0.4.3\n Compiling smallvec v1.15.1\n Compiling second-stack v0.3.5\n Compiling constant_time_eq v0.4.2\n Compiling getrandom v0.2.17\n Compiling bytemuck v1.24.0\n Compiling cc v1.2.53\n Compiling itoa v1.0.17\n Compiling serde_json v1.0.149\n Compiling arrayref v0.3.9\n Compiling spacetimedb-lib v1.11.1\n Compiling log v0.4.29\n Compiling memchr v2.7.6\n Compiling itertools v0.12.1\n Compiling rand_core v0.6.4\n Compiling scoped-tls v1.0.1\n Compiling generic-array v0.14.7\n Compiling num-traits v0.2.19\n Compiling http v1.4.0\n Compiling approx v0.3.2\n Compiling chrono v0.4.43\n Compiling syn v2.0.114\n Compiling decorum v0.3.1\n Compiling blake3 v1.8.3\n Compiling ppv-lite86 v0.2.21\n Compiling rand_chacha v0.3.1\n Compiling rand v0.8.5\n Compiling block-buffer v0.10.4\n Compiling crypto-common v0.1.7\n Compiling digest v0.10.7\n Compiling ethnum v1.5.2\n Compiling sha3 v0.10.8\n Compiling enum-as-inner v0.6.1\n Compiling thiserror-impl v1.0.69\n Compiling derive_more v0.99.20\n Compiling spacetimedb-primitives v1.11.1\n Compiling spacetimedb-bindings-sys v1.11.1\n Compiling spacetimedb-bindings-macro v1.11.1\n Compiling spacetimedb-sats v1.11.1\n Compiling spacetimedb v1.11.1\n Compiling spacetime-module v0.1.0 (/__w/SpacetimeDB/SpacetimeDB/target/llm-runs/schema/t_017_scheduled_columns/rust/server/gpt-5/llm)\nerror: expected one of: `public`, `private`, `name`, `index`, `scheduled`\n --> src/lib.rs:4:28\n |\n4 | #[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]\n | ^^^^^^^^\n\nerror: expected one of: `init`, `client_connected`, `client_disconnected`, `update`, `name`\n --> src/lib.rs:20:11\n |\n20 | #[reducer(scheduled)]\n | ^^^^^^^^^\n\nerror[E0422]: cannot find struct, variant or union type `TickTimer` in this scope\n --> src/lib.rs:14:32\n |\n14 | ctx.db.tick_timer().insert(TickTimer {\n | ^^^^^^^^^ not found in this scope\n\nerror[E0412]: cannot find type `TickTimer` in this scope\n --> src/lib.rs:21:42\n |\n21 | pub fn tick(_ctx: &ReducerContext, _row: TickTimer) {\n | ^^^^^^^^^ not found in this scope\n\nerror[E0599]: no method named `tick_timer` found for struct `Local` in the current scope\n --> src/lib.rs:14:12\n |\n14 | ctx.db.tick_timer().insert(TickTimer {\n | ^^^^^^^^^^ method not found in `Local`\n\nerror[E0599]: no variant or associated item named `every_micros` found for enum `ScheduleAt` in the current scope\n --> src/lib.rs:16:35\n |\n16 | scheduled_at: ScheduleAt::every_micros(50_000),\n | ^^^^^^^^^^^^ variant or associated item not found in `ScheduleAt`\n\nSome errors have detailed explanations: E0412, E0422, E0599.\nFor more information about an error, try `rustc --explain E0412`.\nerror: could not compile `spacetime-module` (lib) due to 6 previous errors\nError: command [\"cargo\", \"build\", \"--config=net.git-fetch-with-cli=true\", \"--target=wasm32-unknown-unknown\", \"--release\", \"--message-format=json-render-diagnostics\"] exited with code 101\n\n--- stdout ---\n", "phase": "build_or_publish" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:24:16.320410304Z", - "finished_at": "2026-01-22T01:25:15.264237957Z" + "started_at": "2026-01-22T23:34:44.633412752Z", + "finished_at": "2026-01-22T23:35:37.701635523Z" }, "t_018_constraints": { "hash": "e6a5a5374022e3250e0df2cb5e4f7e4777081f474a0ebc4e99302be42151ef59", @@ -896,7 +869,7 @@ "model_name": "GPT-5", "total_tests": 3, "passed_tests": 2, - "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = account, index(name = by_name, btree(columns = [name])))]\npub struct Account {\n #[primary_key]\n id: i32,\n #[unique]\n email: String,\n name: String,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n let tbl = ctx.db.account();\n\n for row in tbl.iter().collect::>() {\n tbl.id().delete(&row.id);\n }\n\n tbl.insert(Account {\n id: 1,\n email: \"a@example.com\".to_string(),\n name: \"Alice\".to_string(),\n });\n\n tbl.insert(Account {\n id: 2,\n email: \"b@example.com\".to_string(),\n name: \"Bob\".to_string(),\n });\n}", + "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table};\n\n#[table(name = account, index(name = by_name, btree(columns = [name])))]\npub struct Account {\n #[primary_key]\n id: i32,\n #[unique]\n email: String,\n name: String,\n}\n\n#[reducer]\nfn seed(ctx: &ReducerContext) {\n let account = ctx.db.account();\n let _ = account.try_insert(Account { id: 1, email: \"a@example.com\".to_string(), name: \"Alice\".to_string() });\n let _ = account.try_insert(Account { id: 2, email: \"b@example.com\".to_string(), name: \"Bob\".to_string() });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-018-constraints-golden", @@ -904,24 +877,11 @@ "work_dir_golden": "target/llm-runs/schema/t_018_constraints/rust/server/golden", "work_dir_llm": "target/llm-runs/schema/t_018_constraints/rust/server/gpt-5/llm", "scorer_details": { - "schema_parity": { - "pass": true, - "partial": 1.0, - "notes": { - "golden_db": "schema-t-018-constraints-golden", - "llm_db": "schema-t-018-constraints-gpt-5-llm", - "reducers_diff": null, - "reducers_equal": true, - "server": "http://127.0.0.1:39821", - "tables_diff": null, - "tables_equal": true - } - }, "constraints_row_parity_after_seed": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `account`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:39821/v1/database/c20020aee873f26382406972cb171ff0950dcc3213212601c47ccb52b18e66ba/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `account`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:34355/v1/database/c20099d01e37c839067bf58db5b6af96d5d2baebdf6ce19acbe633c07008c271/sql)\n", "phase": "sql_golden" } }, @@ -933,11 +893,24 @@ "expected": 1, "sql": "SELECT COUNT(*) AS n FROM account WHERE id=2" } + }, + "schema_parity": { + "pass": true, + "partial": 1.0, + "notes": { + "golden_db": "schema-t-018-constraints-golden", + "llm_db": "schema-t-018-constraints-gpt-5-llm", + "reducers_diff": null, + "reducers_equal": true, + "server": "http://127.0.0.1:34355", + "tables_diff": null, + "tables_equal": true + } } }, "vendor": "openai", - "started_at": "2026-01-22T01:24:16.320166984Z", - "finished_at": "2026-01-22T01:24:49.410038315Z" + "started_at": "2026-01-22T23:34:44.633166223Z", + "finished_at": "2026-01-22T23:35:12.446507113Z" }, "t_019_many_to_many": { "hash": "e6a5a5374022e3250e0df2cb5e4f7e4777081f474a0ebc4e99302be42151ef59", @@ -947,7 +920,7 @@ "model_name": "GPT-5", "total_tests": 5, "passed_tests": 5, - "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = user)]\npub struct User {\n #[primary_key]\n user_id: i32,\n name: String,\n}\n\n#[spacetimedb::table(name = group)]\npub struct Group {\n #[primary_key]\n group_id: i32,\n title: String,\n}\n\n#[spacetimedb::table(\n name = membership,\n index(name = by_user, btree(columns = [user_id])),\n index(name = by_group, btree(columns = [group_id]))\n)]\npub struct Membership {\n #[primary_key]\n id: i32,\n user_id: i32,\n group_id: i32,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n let membership_table = ctx.db.membership();\n let membership_rows: Vec<_> = membership_table.iter().collect();\n for row in membership_rows {\n membership_table.id().delete(&row.id);\n }\n\n let user_table = ctx.db.user();\n let user_rows: Vec<_> = user_table.iter().collect();\n for row in user_rows {\n user_table.user_id().delete(&row.user_id);\n }\n\n let group_table = ctx.db.group();\n let group_rows: Vec<_> = group_table.iter().collect();\n for row in group_rows {\n group_table.group_id().delete(&row.group_id);\n }\n\n user_table.insert(User { user_id: 1, name: \"Alice\".to_string() });\n user_table.insert(User { user_id: 2, name: \"Bob\".to_string() });\n\n group_table.insert(Group { group_id: 10, title: \"Admin\".to_string() });\n group_table.insert(Group { group_id: 20, title: \"Dev\".to_string() });\n\n membership_table.insert(Membership { id: 1, user_id: 1, group_id: 10 });\n membership_table.insert(Membership { id: 2, user_id: 1, group_id: 20 });\n membership_table.insert(Membership { id: 3, user_id: 2, group_id: 20 });\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = user)]\npub struct User {\n #[primary_key]\n user_id: i32,\n name: String,\n}\n\n#[spacetimedb::table(name = group)]\npub struct Group {\n #[primary_key]\n group_id: i32,\n title: String,\n}\n\n#[spacetimedb::table(\n name = membership,\n index(name = by_user, btree(columns = [user_id])),\n index(name = by_group, btree(columns = [group_id]))\n)]\npub struct Membership {\n #[primary_key]\n id: i32,\n user_id: i32,\n group_id: i32,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n // Clear existing rows to ensure exactly these rows are present\n let membership_tbl = ctx.db.membership();\n let membership_ids: Vec = membership_tbl.iter().map(|m| m.id).collect();\n for id in membership_ids {\n membership_tbl.id().delete(&id);\n }\n\n let user_tbl = ctx.db.user();\n let user_ids: Vec = user_tbl.iter().map(|u| u.user_id).collect();\n for id in user_ids {\n user_tbl.user_id().delete(&id);\n }\n\n let group_tbl = ctx.db.group();\n let group_ids: Vec = group_tbl.iter().map(|g| g.group_id).collect();\n for id in group_ids {\n group_tbl.group_id().delete(&id);\n }\n\n // Insert users\n user_tbl.insert(User { user_id: 1, name: \"Alice\".to_string() });\n user_tbl.insert(User { user_id: 2, name: \"Bob\".to_string() });\n\n // Insert groups\n group_tbl.insert(Group { group_id: 10, title: \"Admin\".to_string() });\n group_tbl.insert(Group { group_id: 20, title: \"Dev\".to_string() });\n\n // Insert memberships\n membership_tbl.insert(Membership { id: 1, user_id: 1, group_id: 10 });\n membership_tbl.insert(Membership { id: 2, user_id: 1, group_id: 20 });\n membership_tbl.insert(Membership { id: 3, user_id: 2, group_id: 20 });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-019-many-to-many-golden", @@ -955,23 +928,27 @@ "work_dir_golden": "target/llm-runs/schema/t_019_many_to_many/rust/server/golden", "work_dir_llm": "target/llm-runs/schema/t_019_many_to_many/rust/server/gpt-5/llm", "scorer_details": { - "m2m_has_1_10": { + "schema_parity": { "pass": true, "partial": 1.0, "notes": { - "actual": 1, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM membership WHERE user_id=1 AND group_id=10" + "golden_db": "schema-t-019-many-to-many-golden", + "llm_db": "schema-t-019-many-to-many-gpt-5-llm", + "reducers_diff": null, + "reducers_equal": true, + "server": "http://127.0.0.1:34355", + "tables_diff": null, + "tables_equal": true } }, - "memberships_three_rows": { + "m2m_has_1_20": { "pass": true, "partial": 1.0, "notes": { - "actual": 3, - "expected": 3, - "sql": "SELECT COUNT(*) AS n FROM membership" - } + "actual": 1, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM membership WHERE user_id=1 AND group_id=20" + } }, "m2m_has_2_20": { "pass": true, @@ -982,32 +959,28 @@ "sql": "SELECT COUNT(*) AS n FROM membership WHERE user_id=2 AND group_id=20" } }, - "m2m_has_1_20": { + "m2m_has_1_10": { "pass": true, "partial": 1.0, "notes": { "actual": 1, "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM membership WHERE user_id=1 AND group_id=20" + "sql": "SELECT COUNT(*) AS n FROM membership WHERE user_id=1 AND group_id=10" } }, - "schema_parity": { + "memberships_three_rows": { "pass": true, "partial": 1.0, "notes": { - "golden_db": "schema-t-019-many-to-many-golden", - "llm_db": "schema-t-019-many-to-many-gpt-5-llm", - "reducers_diff": null, - "reducers_equal": true, - "server": "http://127.0.0.1:39821", - "tables_diff": null, - "tables_equal": true + "actual": 3, + "expected": 3, + "sql": "SELECT COUNT(*) AS n FROM membership" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:24:16.320979630Z", - "finished_at": "2026-01-22T01:25:12.352219564Z" + "started_at": "2026-01-22T23:34:44.633976742Z", + "finished_at": "2026-01-22T23:35:41.993162368Z" }, "t_020_ecs": { "hash": "e6a5a5374022e3250e0df2cb5e4f7e4777081f474a0ebc4e99302be42151ef59", @@ -1017,7 +990,7 @@ "model_name": "GPT-5", "total_tests": 5, "passed_tests": 1, - "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = entity)]\npub struct Entity {\n #[primary_key]\n id: i32,\n}\n\n#[spacetimedb::table(name = position)]\npub struct Position {\n #[primary_key]\n entity_id: i32,\n x: i32,\n y: i32,\n}\n\n#[spacetimedb::table(name = velocity)]\npub struct Velocity {\n #[primary_key]\n entity_id: i32,\n vx: i32,\n vy: i32,\n}\n\n#[spacetimedb::table(name = next_position)]\npub struct NextPosition {\n #[primary_key]\n entity_id: i32,\n x: i32,\n y: i32,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n // Ensure entities exist\n if ctx.db.entity().id().find(1).is_none() {\n ctx.db.entity().insert(Entity { id: 1 });\n }\n if ctx.db.entity().id().find(2).is_none() {\n ctx.db.entity().insert(Entity { id: 2 });\n }\n\n // Upsert positions\n let pos1 = Position { entity_id: 1, x: 0, y: 0 };\n if ctx.db.position().entity_id().find(1).is_some() {\n ctx.db.position().entity_id().update(pos1);\n } else {\n ctx.db.position().insert(pos1);\n }\n\n let pos2 = Position { entity_id: 2, x: 10, y: 0 };\n if ctx.db.position().entity_id().find(2).is_some() {\n ctx.db.position().entity_id().update(pos2);\n } else {\n ctx.db.position().insert(pos2);\n }\n\n // Upsert velocities\n let vel1 = Velocity { entity_id: 1, vx: 1, vy: 0 };\n if ctx.db.velocity().entity_id().find(1).is_some() {\n ctx.db.velocity().entity_id().update(vel1);\n } else {\n ctx.db.velocity().insert(vel1);\n }\n\n let vel2 = Velocity { entity_id: 2, vx: -2, vy: 3 };\n if ctx.db.velocity().entity_id().find(2).is_some() {\n ctx.db.velocity().entity_id().update(vel2);\n } else {\n ctx.db.velocity().insert(vel2);\n }\n}\n\n#[spacetimedb::reducer]\npub fn step(ctx: &ReducerContext) {\n for pos in ctx.db.position().iter() {\n if let Some(vel) = ctx.db.velocity().entity_id().find(pos.entity_id) {\n let nx = pos.x + vel.vx;\n let ny = pos.y + vel.vy;\n let next = NextPosition { entity_id: pos.entity_id, x: nx, y: ny };\n if ctx.db.next_position().entity_id().find(pos.entity_id).is_some() {\n ctx.db.next_position().entity_id().update(next);\n } else {\n ctx.db.next_position().insert(next);\n }\n }\n }\n}", + "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table};\n\n#[table(name = entity)]\npub struct Entity {\n #[primary_key]\n id: i32,\n}\n\n#[table(name = position)]\npub struct Position {\n #[primary_key]\n entity_id: i32,\n x: i32,\n y: i32,\n}\n\n#[table(name = velocity)]\npub struct Velocity {\n #[primary_key]\n entity_id: i32,\n vx: i32,\n vy: i32,\n}\n\n#[table(name = next_position)]\npub struct NextPosition {\n #[primary_key]\n entity_id: i32,\n x: i32,\n y: i32,\n}\n\n#[reducer]\npub fn seed(ctx: &ReducerContext) {\n let entity_table = ctx.db.entity();\n let e1 = Entity { id: 1 };\n if entity_table.id().find(1).is_some() {\n entity_table.id().update(e1);\n } else {\n entity_table.insert(e1);\n }\n let e2 = Entity { id: 2 };\n if entity_table.id().find(2).is_some() {\n entity_table.id().update(e2);\n } else {\n entity_table.insert(e2);\n }\n\n let position_table = ctx.db.position();\n let p1 = Position { entity_id: 1, x: 0, y: 0 };\n if position_table.entity_id().find(1).is_some() {\n position_table.entity_id().update(p1);\n } else {\n position_table.insert(p1);\n }\n let p2 = Position { entity_id: 2, x: 10, y: 0 };\n if position_table.entity_id().find(2).is_some() {\n position_table.entity_id().update(p2);\n } else {\n position_table.insert(p2);\n }\n\n let velocity_table = ctx.db.velocity();\n let v1 = Velocity { entity_id: 1, vx: 1, vy: 0 };\n if velocity_table.entity_id().find(1).is_some() {\n velocity_table.entity_id().update(v1);\n } else {\n velocity_table.insert(v1);\n }\n let v2 = Velocity { entity_id: 2, vx: -2, vy: 3 };\n if velocity_table.entity_id().find(2).is_some() {\n velocity_table.entity_id().update(v2);\n } else {\n velocity_table.insert(v2);\n }\n}\n\n#[reducer]\npub fn step(ctx: &ReducerContext) {\n let pos_table = ctx.db.position();\n let vel_col = ctx.db.velocity().entity_id();\n let next_tbl = ctx.db.next_position();\n for pos in pos_table.iter() {\n if let Some(vel) = vel_col.find(pos.entity_id) {\n let new_row = NextPosition {\n entity_id: pos.entity_id,\n x: pos.x + vel.vx,\n y: pos.y + vel.vy,\n };\n if next_tbl.entity_id().find(pos.entity_id).is_some() {\n next_tbl.entity_id().update(new_row);\n } else {\n next_tbl.insert(new_row);\n }\n }\n }\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-020-ecs-golden", @@ -1033,47 +1006,47 @@ "llm_db": "schema-t-020-ecs-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:39821", + "server": "http://127.0.0.1:34355", "tables_diff": null, "tables_equal": true } }, - "ecs_step_next_positions_count": { + "ecs_next_pos_entity2": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:39821/v1/database/c20056ee3d16c5c7ac39f4a5012c2e159035b29b8d01823a1e5bdf1ac4f89312/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:34355/v1/database/c2002cf6f0324a1762e339578f79b1095eebbd7cfee8c17d83420b4bd33bf61f/sql)\n", "phase": "sql" } }, - "ecs_seed_positions_count": { + "ecs_next_pos_entity1": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:39821/v1/database/c20056ee3d16c5c7ac39f4a5012c2e159035b29b8d01823a1e5bdf1ac4f89312/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:34355/v1/database/c2002cf6f0324a1762e339578f79b1095eebbd7cfee8c17d83420b4bd33bf61f/sql)\n", "phase": "sql" } }, - "ecs_next_pos_entity2": { + "ecs_seed_positions_count": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:39821/v1/database/c20056ee3d16c5c7ac39f4a5012c2e159035b29b8d01823a1e5bdf1ac4f89312/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:34355/v1/database/c2002cf6f0324a1762e339578f79b1095eebbd7cfee8c17d83420b4bd33bf61f/sql)\n", "phase": "sql" } }, - "ecs_next_pos_entity1": { + "ecs_step_next_positions_count": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:39821/v1/database/c20056ee3d16c5c7ac39f4a5012c2e159035b29b8d01823a1e5bdf1ac4f89312/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:34355/v1/database/c2002cf6f0324a1762e339578f79b1095eebbd7cfee8c17d83420b4bd33bf61f/sql)\n", "phase": "sql" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:24:16.321111520Z", - "finished_at": "2026-01-22T01:25:33.330176477Z" + "started_at": "2026-01-22T23:34:44.634107600Z", + "finished_at": "2026-01-22T23:35:37.693665948Z" }, "t_021_multi_column_index": { "hash": "e6a5a5374022e3250e0df2cb5e4f7e4777081f474a0ebc4e99302be42151ef59", @@ -1083,7 +1056,7 @@ "model_name": "GPT-5", "total_tests": 4, "passed_tests": 4, - "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table};\n\n#[table(name = log, index(name = by_user_day, btree(columns = [user_id, day])))]\npub struct Log {\n #[primary_key]\n id: i32,\n user_id: i32,\n day: i32,\n message: String,\n}\n\n#[reducer]\npub fn seed(ctx: &ReducerContext) {\n let t = ctx.db.log();\n t.insert(Log { id: 1, user_id: 7, day: 1, message: \"a\".to_string() });\n t.insert(Log { id: 2, user_id: 7, day: 2, message: \"b\".to_string() });\n t.insert(Log { id: 3, user_id: 9, day: 1, message: \"c\".to_string() });\n}", + "llm_output": "use spacetimedb::{table, reducer, ReducerContext, Table};\n\n#[table(name = log, index(name = by_user_day, btree(columns = [user_id, day])))]\npub struct Log {\n #[primary_key]\n id: i32,\n user_id: i32,\n day: i32,\n message: String,\n}\n\n#[reducer]\nfn seed(ctx: &ReducerContext) {\n for row in ctx.db.log().iter() {\n ctx.db.log().id().delete(&row.id);\n }\n\n ctx.db.log().insert(Log { id: 1, user_id: 7, day: 1, message: \"a\".to_string() });\n ctx.db.log().insert(Log { id: 2, user_id: 7, day: 2, message: \"b\".to_string() });\n ctx.db.log().insert(Log { id: 3, user_id: 9, day: 1, message: \"c\".to_string() });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-021-multi-column-index-golden", @@ -1091,6 +1064,24 @@ "work_dir_golden": "target/llm-runs/schema/t_021_multi_column_index/rust/server/golden", "work_dir_llm": "target/llm-runs/schema/t_021_multi_column_index/rust/server/gpt-5/llm", "scorer_details": { + "mcindex_lookup_u7_d2": { + "pass": true, + "partial": 1.0, + "notes": { + "actual": 1, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM log WHERE user_id=7 AND day=2" + } + }, + "mcindex_lookup_u7_d1": { + "pass": true, + "partial": 1.0, + "notes": { + "actual": 1, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM log WHERE user_id=7 AND day=1" + } + }, "mcindex_seed_count": { "pass": true, "partial": 1.0, @@ -1108,33 +1099,15 @@ "llm_db": "schema-t-021-multi-column-index-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:39821", + "server": "http://127.0.0.1:34355", "tables_diff": null, "tables_equal": true } - }, - "mcindex_lookup_u7_d1": { - "pass": true, - "partial": 1.0, - "notes": { - "actual": 1, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM log WHERE user_id=7 AND day=1" - } - }, - "mcindex_lookup_u7_d2": { - "pass": true, - "partial": 1.0, - "notes": { - "actual": 1, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM log WHERE user_id=7 AND day=2" - } } }, "vendor": "openai", - "started_at": "2026-01-22T01:24:16.320557181Z", - "finished_at": "2026-01-22T01:24:50.475368305Z" + "started_at": "2026-01-22T23:34:44.633561585Z", + "finished_at": "2026-01-22T23:35:19.958341977Z" } } } @@ -1142,14 +1115,14 @@ }, { "mode": "docs", - "hash": "02bd25d6ae3ac904057e3a8034595f2c28f652956bc1b2b2a1488d169b871b9e", + "hash": "8c8baf7fa35a3b4cdccd079c8efdf96434bd3a22031b7840912b543226efb58b", "models": [ { "name": "GPT-5", "route_api_model": "gpt-5", "tasks": { "t_000_empty_reducers": { - "hash": "02bd25d6ae3ac904057e3a8034595f2c28f652956bc1b2b2a1488d169b871b9e", + "hash": "8c8baf7fa35a3b4cdccd079c8efdf96434bd3a22031b7840912b543226efb58b", "task": "t_000_empty_reducers", "lang": "rust", "golden_published": true, @@ -1174,11 +1147,11 @@ } }, "vendor": "openai", - "started_at": "2026-01-22T01:27:47.156183690Z", - "finished_at": "2026-01-22T01:28:10.446122291Z" + "started_at": "2026-01-22T23:38:45.149533791Z", + "finished_at": "2026-01-22T23:39:18.075952080Z" }, "t_001_basic_tables": { - "hash": "02bd25d6ae3ac904057e3a8034595f2c28f652956bc1b2b2a1488d169b871b9e", + "hash": "8c8baf7fa35a3b4cdccd079c8efdf96434bd3a22031b7840912b543226efb58b", "task": "t_001_basic_tables", "lang": "rust", "golden_published": true, @@ -1203,18 +1176,18 @@ } }, "vendor": "openai", - "started_at": "2026-01-22T01:28:04.438366941Z", - "finished_at": "2026-01-22T01:28:40.632041510Z" + "started_at": "2026-01-22T23:38:45.156722960Z", + "finished_at": "2026-01-22T23:39:27.202613408Z" }, "t_002_scheduled_table": { - "hash": "02bd25d6ae3ac904057e3a8034595f2c28f652956bc1b2b2a1488d169b871b9e", + "hash": "8c8baf7fa35a3b4cdccd079c8efdf96434bd3a22031b7840912b543226efb58b", "task": "t_002_scheduled_table", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 1, "passed_tests": 0, - "llm_output": "use spacetimedb::{ReducerContext, ScheduleAt, Table};\nuse std::time::Duration;\n\n#[spacetimedb::table(name = tick_timer, scheduled(tick))]\npub struct TickTimer {\n #[primary_key]\n #[auto_inc]\n scheduled_id: u64,\n scheduled_at: ScheduleAt,\n}\n\n#[spacetimedb::reducer]\npub fn tick(_ctx: &ReducerContext, _timer: TickTimer) -> Result<(), String> {\n Ok(())\n}\n\n#[spacetimedb::reducer(init)]\npub fn init(ctx: &ReducerContext) -> Result<(), String> {\n ctx.db.tick_timer().insert(TickTimer {\n scheduled_id: 0,\n scheduled_at: ScheduleAt::Interval(Duration::from_micros(50_000).into()),\n });\n Ok(())\n}", + "llm_output": "use std::time::Duration;\nuse spacetimedb::{ReducerContext, ScheduleAt, Table};\n\n#[spacetimedb::table(name = tick_timer, scheduled(tick))]\npub struct TickTimer {\n #[primary_key]\n #[auto_inc]\n scheduled_id: u64,\n scheduled_at: ScheduleAt,\n}\n\n#[spacetimedb::reducer]\npub fn tick(_ctx: &ReducerContext, _timer: TickTimer) -> Result<(), String> {\n Ok(())\n}\n\n#[spacetimedb::reducer(init)]\npub fn init(ctx: &ReducerContext) -> Result<(), String> {\n ctx.db.tick_timer().try_insert(TickTimer {\n scheduled_id: 0,\n scheduled_at: ScheduleAt::Interval(Duration::from_micros(50_000).into()),\n })?;\n Ok(())\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-002-scheduled-table-golden", @@ -1232,18 +1205,18 @@ } }, "vendor": "openai", - "started_at": "2026-01-22T01:28:10.446178863Z", - "finished_at": "2026-01-22T01:28:55.475486404Z" + "started_at": "2026-01-22T23:39:13.517648780Z", + "finished_at": "2026-01-22T23:39:46.899141682Z" }, "t_003_struct_in_table": { - "hash": "02bd25d6ae3ac904057e3a8034595f2c28f652956bc1b2b2a1488d169b871b9e", + "hash": "8c8baf7fa35a3b4cdccd079c8efdf96434bd3a22031b7840912b543226efb58b", "task": "t_003_struct_in_table", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 1, "passed_tests": 0, - "llm_output": "use spacetimedb::SpacetimeType;\n\n#[derive(SpacetimeType, Clone, Debug)]\npub struct Position {\n pub x: i32,\n pub y: i32,\n}\n\n#[spacetimedb::table(name = entity)]\npub struct Entity {\n #[primary_key]\n pub id: i32,\n pub pos: Position,\n}", + "llm_output": "use spacetimedb::SpacetimeType;\n\n#[derive(SpacetimeType, Clone, Debug)]\npub struct Position {\n x: i32,\n y: i32,\n}\n\n#[spacetimedb::table(name = entity)]\npub struct Entity {\n #[primary_key]\n id: i32,\n pos: Position,\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-003-struct-in-table-golden", @@ -1261,11 +1234,11 @@ } }, "vendor": "openai", - "started_at": "2026-01-22T01:27:47.161208048Z", - "finished_at": "2026-01-22T01:28:20.715385901Z" + "started_at": "2026-01-22T23:38:45.153305918Z", + "finished_at": "2026-01-22T23:39:32.558171356Z" }, "t_004_insert": { - "hash": "02bd25d6ae3ac904057e3a8034595f2c28f652956bc1b2b2a1488d169b871b9e", + "hash": "8c8baf7fa35a3b4cdccd079c8efdf96434bd3a22031b7840912b543226efb58b", "task": "t_004_insert", "lang": "rust", "golden_published": true, @@ -1280,29 +1253,29 @@ "work_dir_golden": "target/llm-runs/basics/t_004_insert/rust/server/golden", "work_dir_llm": "target/llm-runs/basics/t_004_insert/rust/server/gpt-5/llm", "scorer_details": { - "schema_parity": { + "data_parity_insert_user": { "pass": false, "partial": 0.0, "notes": { - "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `basics-t-004-insert-golden`.\n", - "phase": "describe_golden" + "error": "spacetime call failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `basics-t-004-insert-golden`.\n", + "phase": "call_reducer_golden" } }, - "data_parity_insert_user": { + "schema_parity": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime call failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `basics-t-004-insert-golden`.\n", - "phase": "call_reducer_golden" + "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `basics-t-004-insert-golden`.\n", + "phase": "describe_golden" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:28:04.786253323Z", - "finished_at": "2026-01-22T01:28:32.350164074Z" + "started_at": "2026-01-22T23:38:50.780097837Z", + "finished_at": "2026-01-22T23:39:24.416294357Z" }, "t_005_update": { - "hash": "02bd25d6ae3ac904057e3a8034595f2c28f652956bc1b2b2a1488d169b871b9e", + "hash": "8c8baf7fa35a3b4cdccd079c8efdf96434bd3a22031b7840912b543226efb58b", "task": "t_005_update", "lang": "rust", "golden_published": true, @@ -1344,11 +1317,11 @@ } }, "vendor": "openai", - "started_at": "2026-01-22T01:27:47.151608836Z", - "finished_at": "2026-01-22T01:28:44.268716576Z" + "started_at": "2026-01-22T23:38:45.020328705Z", + "finished_at": "2026-01-22T23:39:25.012622931Z" }, "t_006_delete": { - "hash": "02bd25d6ae3ac904057e3a8034595f2c28f652956bc1b2b2a1488d169b871b9e", + "hash": "8c8baf7fa35a3b4cdccd079c8efdf96434bd3a22031b7840912b543226efb58b", "task": "t_006_delete", "lang": "rust", "golden_published": true, @@ -1363,6 +1336,14 @@ "work_dir_golden": "target/llm-runs/basics/t_006_delete/rust/server/golden", "work_dir_llm": "target/llm-runs/basics/t_006_delete/rust/server/gpt-5/llm", "scorer_details": { + "schema_parity": { + "pass": false, + "partial": 0.0, + "notes": { + "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `basics-t-006-delete-golden`.\n", + "phase": "describe_golden" + } + }, "delete_user_count_zero": { "pass": true, "partial": 1.0, @@ -1380,29 +1361,21 @@ "phase": "sql_golden", "sql": "INSERT INTO user(id, name, age, active) VALUES (1, 'Alice', 30, true)" } - }, - "schema_parity": { - "pass": false, - "partial": 0.0, - "notes": { - "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `basics-t-006-delete-golden`.\n", - "phase": "describe_golden" - } } }, "vendor": "openai", - "started_at": "2026-01-22T01:27:20.272319130Z", - "finished_at": "2026-01-22T01:27:37.786888094Z" + "started_at": "2026-01-22T23:38:11.531383936Z", + "finished_at": "2026-01-22T23:38:45.119024790Z" }, "t_007_crud": { - "hash": "02bd25d6ae3ac904057e3a8034595f2c28f652956bc1b2b2a1488d169b871b9e", + "hash": "8c8baf7fa35a3b4cdccd079c8efdf96434bd3a22031b7840912b543226efb58b", "task": "t_007_crud", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 4, "passed_tests": 1, - "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[spacetimedb::reducer]\npub fn crud(ctx: &ReducerContext) {\n ctx.db.user().insert(User { id: 1, name: \"Alice\".to_string(), age: 30, active: true });\n ctx.db.user().insert(User { id: 2, name: \"Bob\".to_string(), age: 22, active: false });\n\n if let Some(existing) = ctx.db.user().id().find(1) {\n ctx.db.user().id().update(User {\n id: existing.id,\n name: \"Alice2\".to_string(),\n age: 31,\n active: false,\n });\n }\n\n ctx.db.user().id().delete(2);\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[spacetimedb::reducer]\npub fn crud(ctx: &ReducerContext) {\n ctx.db.user().insert(User {\n id: 1,\n name: \"Alice\".into(),\n age: 30,\n active: true,\n });\n\n ctx.db.user().insert(User {\n id: 2,\n name: \"Bob\".into(),\n age: 22,\n active: false,\n });\n\n if ctx.db.user().id().find(1).is_some() {\n ctx.db.user().id().update(User {\n id: 1,\n name: \"Alice2\".into(),\n age: 31,\n active: false,\n });\n }\n\n ctx.db.user().id().delete(&2);\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-007-crud-golden", @@ -1410,15 +1383,6 @@ "work_dir_golden": "target/llm-runs/basics/t_007_crud/rust/server/golden", "work_dir_llm": "target/llm-runs/basics/t_007_crud/rust/server/gpt-5/llm", "scorer_details": { - "crud_total_count_one": { - "pass": false, - "partial": 0.0, - "notes": { - "actual": 0, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM user" - } - }, "crud_row_id2_deleted": { "pass": true, "partial": 1.0, @@ -1428,36 +1392,45 @@ "sql": "SELECT COUNT(*) AS n FROM user WHERE id=2" } }, - "crud_row_id1_parity": { + "schema_parity": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime call failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `basics-t-007-crud-golden`.\n", - "phase": "call_reducer_golden" + "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `basics-t-007-crud-golden`.\n", + "phase": "describe_golden" } }, - "schema_parity": { + "crud_total_count_one": { "pass": false, "partial": 0.0, "notes": { - "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `basics-t-007-crud-golden`.\n", - "phase": "describe_golden" + "actual": 0, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM user" + } + }, + "crud_row_id1_parity": { + "pass": false, + "partial": 0.0, + "notes": { + "error": "spacetime call failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `basics-t-007-crud-golden`.\n", + "phase": "call_reducer_golden" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:27:20.653563136Z", - "finished_at": "2026-01-22T01:28:04.786052540Z" + "started_at": "2026-01-22T23:38:20.137543170Z", + "finished_at": "2026-01-22T23:38:59.559084630Z" }, "t_008_index_lookup": { - "hash": "02bd25d6ae3ac904057e3a8034595f2c28f652956bc1b2b2a1488d169b871b9e", + "hash": "8c8baf7fa35a3b4cdccd079c8efdf96434bd3a22031b7840912b543226efb58b", "task": "t_008_index_lookup", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 3, "passed_tests": 0, - "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[spacetimedb::table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n name: String,\n}\n\n#[spacetimedb::reducer]\npub fn lookup_user_name(ctx: &ReducerContext, id: i32) {\n if let Some(user) = ctx.db.user().id().find(id) {\n let row = ResultRow {\n id: user.id,\n name: user.name.clone(),\n };\n if ctx.db.result().id().find(row.id).is_some() {\n ctx.db.result().id().update(row);\n } else {\n ctx.db.result().insert(row);\n }\n }\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[spacetimedb::table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n name: String,\n}\n\n#[spacetimedb::reducer]\npub fn lookup_user_name(ctx: &ReducerContext, id: i32) -> Result<(), String> {\n if let Some(user) = ctx.db.user().id().find(id) {\n ctx.db.result().insert(ResultRow {\n id: user.id,\n name: user.name.clone(),\n });\n Ok(())\n } else {\n Err(\"User not found\".to_string())\n }\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-008-index-lookup-golden", @@ -1465,6 +1438,15 @@ "work_dir_golden": "target/llm-runs/basics/t_008_index_lookup/rust/server/golden", "work_dir_llm": "target/llm-runs/basics/t_008_index_lookup/rust/server/gpt-5/llm", "scorer_details": { + "seed_user_row": { + "pass": false, + "partial": 0.0, + "notes": { + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `basics-t-008-index-lookup-golden`.\n", + "phase": "sql_golden", + "sql": "INSERT INTO user(id, name, age, active) VALUES (1, 'Alice', 30, true)" + } + }, "schema_parity": { "pass": false, "partial": 0.0, @@ -1480,30 +1462,21 @@ "error": "spacetime call failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `basics-t-008-index-lookup-golden`.\n", "phase": "call_reducer_golden" } - }, - "seed_user_row": { - "pass": false, - "partial": 0.0, - "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `basics-t-008-index-lookup-golden`.\n", - "phase": "sql_golden", - "sql": "INSERT INTO user(id, name, age, active) VALUES (1, 'Alice', 30, true)" - } } }, "vendor": "openai", - "started_at": "2026-01-22T01:28:04.454465159Z", - "finished_at": "2026-01-22T01:28:42.003928222Z" + "started_at": "2026-01-22T23:38:46.801424383Z", + "finished_at": "2026-01-22T23:39:27.607666469Z" }, "t_009_init": { - "hash": "02bd25d6ae3ac904057e3a8034595f2c28f652956bc1b2b2a1488d169b871b9e", + "hash": "8c8baf7fa35a3b4cdccd079c8efdf96434bd3a22031b7840912b543226efb58b", "task": "t_009_init", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 4, "passed_tests": 3, - "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[spacetimedb::reducer(init)]\npub fn init(ctx: &ReducerContext) -> Result<(), String> {\n ctx.db.user().try_insert(User {\n id: 1,\n name: \"Alice\".into(),\n age: 30,\n active: true,\n })?;\n ctx.db.user().try_insert(User {\n id: 2,\n name: \"Bob\".into(),\n age: 22,\n active: false,\n })?;\n Ok(())\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = user)]\npub struct User {\n #[primary_key]\n id: i32,\n name: String,\n age: i32,\n active: bool,\n}\n\n#[spacetimedb::reducer(init)]\npub fn init(ctx: &ReducerContext) {\n ctx.db.user().insert(User {\n id: 1,\n name: \"Alice\".to_string(),\n age: 30,\n active: true,\n });\n ctx.db.user().insert(User {\n id: 2,\n name: \"Bob\".to_string(),\n age: 22,\n active: false,\n });\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-009-init-golden", @@ -1520,13 +1493,13 @@ "sql": "SELECT COUNT(*) AS n FROM user WHERE id=1 AND name='Alice' AND age=30 AND active=true" } }, - "init_total_two": { + "init_seed_bob": { "pass": true, "partial": 1.0, "notes": { - "actual": 2, - "expected": 2, - "sql": "SELECT COUNT(*) AS n FROM user" + "actual": 1, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM user WHERE id=2 AND name='Bob' AND age=22 AND active=false" } }, "schema_parity": { @@ -1537,22 +1510,22 @@ "phase": "describe_golden" } }, - "init_seed_bob": { + "init_total_two": { "pass": true, "partial": 1.0, "notes": { - "actual": 1, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM user WHERE id=2 AND name='Bob' AND age=22 AND active=false" + "actual": 2, + "expected": 2, + "sql": "SELECT COUNT(*) AS n FROM user" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:27:37.786926015Z", - "finished_at": "2026-01-22T01:28:25.223772220Z" + "started_at": "2026-01-22T23:38:38.565585767Z", + "finished_at": "2026-01-22T23:39:13.517597108Z" }, "t_010_connect": { - "hash": "02bd25d6ae3ac904057e3a8034595f2c28f652956bc1b2b2a1488d169b871b9e", + "hash": "8c8baf7fa35a3b4cdccd079c8efdf96434bd3a22031b7840912b543226efb58b", "task": "t_010_connect", "lang": "rust", "golden_published": true, @@ -1577,18 +1550,18 @@ } }, "vendor": "openai", - "started_at": "2026-01-22T01:27:20.373482141Z", - "finished_at": "2026-01-22T01:27:46.967337087Z" + "started_at": "2026-01-22T23:38:16.323282079Z", + "finished_at": "2026-01-22T23:38:45.149504861Z" }, "t_011_helper_function": { - "hash": "02bd25d6ae3ac904057e3a8034595f2c28f652956bc1b2b2a1488d169b871b9e", + "hash": "8c8baf7fa35a3b4cdccd079c8efdf96434bd3a22031b7840912b543226efb58b", "task": "t_011_helper_function", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 3, "passed_tests": 0, - "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n sum: i32,\n}\n\nfn add(a: i32, b: i32) -> i32 {\n a + b\n}\n\n#[spacetimedb::reducer]\npub fn compute_sum(ctx: &ReducerContext, id: i32, a: i32, b: i32) {\n let sum = add(a, b);\n ctx.db.result().insert(ResultRow { id, sum });\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n sum: i32,\n}\n\nfn add(a: i32, b: i32) -> i32 {\n a + b\n}\n\n#[spacetimedb::reducer]\npub fn compute_sum(ctx: &ReducerContext, id: i32, a: i32, b: i32) {\n let s = add(a, b);\n ctx.db.result().insert(ResultRow { id, sum: s });\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-011-helper-function-golden", @@ -1596,14 +1569,6 @@ "work_dir_golden": "target/llm-runs/basics/t_011_helper_function/rust/server/golden", "work_dir_llm": "target/llm-runs/basics/t_011_helper_function/rust/server/gpt-5/llm", "scorer_details": { - "schema_parity": { - "pass": false, - "partial": 0.0, - "notes": { - "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `basics-t-011-helper-function-golden`.\n", - "phase": "describe_golden" - } - }, "helper_func_sum_abs": { "pass": false, "partial": 0.0, @@ -1620,21 +1585,29 @@ "error": "spacetime call failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `basics-t-011-helper-function-golden`.\n", "phase": "call_reducer_golden" } + }, + "schema_parity": { + "pass": false, + "partial": 0.0, + "notes": { + "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `basics-t-011-helper-function-golden`.\n", + "phase": "describe_golden" + } } }, "vendor": "openai", - "started_at": "2026-01-22T01:28:06.572162277Z", - "finished_at": "2026-01-22T01:28:26.978553346Z" + "started_at": "2026-01-22T23:38:59.559113903Z", + "finished_at": "2026-01-22T23:39:28.301948063Z" }, "t_012_spacetime_product_type": { - "hash": "02bd25d6ae3ac904057e3a8034595f2c28f652956bc1b2b2a1488d169b871b9e", + "hash": "8c8baf7fa35a3b4cdccd079c8efdf96434bd3a22031b7840912b543226efb58b", "task": "t_012_spacetime_product_type", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 3, "passed_tests": 0, - "llm_output": "use spacetimedb::{ReducerContext, Table, SpacetimeType};\n\n#[derive(SpacetimeType)]\npub struct Score {\n left: i32,\n right: i32,\n}\n\n#[spacetimedb::table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n value: Score,\n}\n\n#[spacetimedb::reducer]\npub fn set_score(ctx: &ReducerContext, id: i32, left: i32, right: i32) {\n ctx.db.result().insert(ResultRow {\n id,\n value: Score { left, right },\n });\n}", + "llm_output": "use spacetimedb::{ReducerContext, SpacetimeType, Table};\n\n#[derive(SpacetimeType, Clone)]\npub struct Score {\n left: i32,\n right: i32,\n}\n\n#[spacetimedb::table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n value: Score,\n}\n\n#[spacetimedb::reducer]\npub fn set_score(ctx: &ReducerContext, id: i32, left: i32, right: i32) {\n ctx.db.result().insert(ResultRow {\n id,\n value: Score { left, right },\n });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-012-spacetime-product-type-golden", @@ -1642,13 +1615,12 @@ "work_dir_golden": "target/llm-runs/schema/t_012_spacetime_product_type/rust/server/golden", "work_dir_llm": "target/llm-runs/schema/t_012_spacetime_product_type/rust/server/gpt-5/llm", "scorer_details": { - "product_type_row_count": { + "schema_parity": { "pass": false, "partial": 0.0, "notes": { - "actual": 0, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM result WHERE id=1" + "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-012-spacetime-product-type-golden`.\n", + "phase": "describe_golden" } }, "product_type_row_parity": { @@ -1659,28 +1631,29 @@ "phase": "call_reducer_golden" } }, - "schema_parity": { + "product_type_row_count": { "pass": false, "partial": 0.0, "notes": { - "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-012-spacetime-product-type-golden`.\n", - "phase": "describe_golden" + "actual": 0, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM result WHERE id=1" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:27:17.380228035Z", - "finished_at": "2026-01-22T01:27:47.058276146Z" + "started_at": "2026-01-22T23:38:09.921328277Z", + "finished_at": "2026-01-22T23:38:44.887109253Z" }, "t_013_spacetime_sum_type": { - "hash": "02bd25d6ae3ac904057e3a8034595f2c28f652956bc1b2b2a1488d169b871b9e", + "hash": "8c8baf7fa35a3b4cdccd079c8efdf96434bd3a22031b7840912b543226efb58b", "task": "t_013_spacetime_sum_type", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 3, "passed_tests": 0, - "llm_output": "use spacetimedb::{ReducerContext, Table, SpacetimeType};\n\n#[derive(SpacetimeType, Clone)]\npub struct Rect {\n width: i32,\n height: i32,\n}\n\n#[derive(SpacetimeType, Clone)]\npub enum Shape {\n Circle(i32),\n Rectangle(Rect),\n}\n\n#[spacetimedb::table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n value: Shape,\n}\n\n#[spacetimedb::reducer]\npub fn set_circle(ctx: &ReducerContext, id: i32, radius: i32) {\n ctx.db.result().insert(ResultRow {\n id,\n value: Shape::Circle(radius),\n });\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[derive(spacetimedb::SpacetimeType, Debug, Clone)]\npub struct Rect {\n width: i32,\n height: i32,\n}\n\n#[derive(spacetimedb::SpacetimeType, Debug, Clone)]\npub enum Shape {\n Circle(i32),\n Rectangle(Rect),\n}\n\n#[spacetimedb::table(name = result)]\npub struct ResultRow {\n #[primary_key]\n id: i32,\n value: Shape,\n}\n\n#[spacetimedb::reducer]\npub fn set_circle(ctx: &ReducerContext, id: i32, radius: i32) {\n ctx.db.result().insert(ResultRow {\n id,\n value: Shape::Circle(radius),\n });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-013-spacetime-sum-type-golden", @@ -1696,37 +1669,37 @@ "phase": "describe_golden" } }, - "sum_type_row_count": { + "sum_type_row_parity": { "pass": false, "partial": 0.0, "notes": { - "actual": 0, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM result WHERE id=1" + "error": "spacetime call failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-013-spacetime-sum-type-golden`.\n", + "phase": "call_reducer_golden" } }, - "sum_type_row_parity": { + "sum_type_row_count": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime call failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-013-spacetime-sum-type-golden`.\n", - "phase": "call_reducer_golden" + "actual": 0, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM result WHERE id=1" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:26:34.446229688Z", - "finished_at": "2026-01-22T01:27:20.373402997Z" + "started_at": "2026-01-22T23:37:20.768236235Z", + "finished_at": "2026-01-22T23:38:20.137437197Z" }, "t_014_elementary_columns": { - "hash": "02bd25d6ae3ac904057e3a8034595f2c28f652956bc1b2b2a1488d169b871b9e", + "hash": "8c8baf7fa35a3b4cdccd079c8efdf96434bd3a22031b7840912b543226efb58b", "task": "t_014_elementary_columns", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 3, "passed_tests": 0, - "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = primitive)]\npub struct Primitive {\n #[primary_key]\n id: i32,\n count: i32,\n total: i64,\n price: f32,\n ratio: f64,\n active: bool,\n name: String,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n ctx.db.primitive().insert(Primitive {\n id: 1,\n count: 2,\n total: 3_000_000_000,\n price: 1.5f32,\n ratio: 2.25f64,\n active: true,\n name: \"Alice\".to_string(),\n });\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = primitive)]\npub struct Primitive {\n #[primary_key]\n id: i32,\n count: i32,\n total: i64,\n price: f32,\n ratio: f64,\n active: bool,\n name: String,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n ctx.db.primitive().insert(Primitive {\n id: 1,\n count: 2,\n total: 3_000_000_000i64,\n price: 1.5f32,\n ratio: 2.25,\n active: true,\n name: \"Alice\".to_string(),\n });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-014-elementary-columns-golden", @@ -1742,37 +1715,37 @@ "phase": "call_reducer_golden" } }, - "elementary_columns_row_count": { + "schema_parity": { "pass": false, "partial": 0.0, "notes": { - "actual": 0, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM primitive WHERE id=1" + "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-014-elementary-columns-golden`.\n", + "phase": "describe_golden" } }, - "schema_parity": { + "elementary_columns_row_count": { "pass": false, "partial": 0.0, "notes": { - "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-014-elementary-columns-golden`.\n", - "phase": "describe_golden" + "actual": 0, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM primitive WHERE id=1" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:27:13.175633692Z", - "finished_at": "2026-01-22T01:27:47.151577776Z" + "started_at": "2026-01-22T23:38:06.509636573Z", + "finished_at": "2026-01-22T23:38:38.565558950Z" }, "t_015_product_type_columns": { - "hash": "02bd25d6ae3ac904057e3a8034595f2c28f652956bc1b2b2a1488d169b871b9e", + "hash": "8c8baf7fa35a3b4cdccd079c8efdf96434bd3a22031b7840912b543226efb58b", "task": "t_015_product_type_columns", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 3, "passed_tests": 0, - "llm_output": "use spacetimedb::{ReducerContext, SpacetimeType, Table};\n\n#[derive(SpacetimeType, Clone)]\npub struct Address {\n street: String,\n zip: i32,\n}\n\n#[derive(SpacetimeType, Clone)]\npub struct Position {\n x: i32,\n y: i32,\n}\n\n#[spacetimedb::table(name = profile)]\npub struct Profile {\n #[primary_key]\n id: i32,\n home: Address,\n work: Address,\n pos: Position,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n if ctx.db.profile().count() == 0 {\n ctx.db.profile().insert(Profile {\n id: 1,\n home: Address {\n street: \"1 Main\".to_string(),\n zip: 11111,\n },\n work: Address {\n street: \"2 Broad\".to_string(),\n zip: 22222,\n },\n pos: Position { x: 7, y: 9 },\n });\n }\n}", + "llm_output": "use spacetimedb::{ReducerContext, SpacetimeType, Table};\n\n#[derive(SpacetimeType, Clone, Debug)]\npub struct Address {\n street: String,\n zip: i32,\n}\n\n#[derive(SpacetimeType, Clone, Debug)]\npub struct Position {\n x: i32,\n y: i32,\n}\n\n#[spacetimedb::table(name = profile)]\npub struct Profile {\n #[primary_key]\n id: i32,\n home: Address,\n work: Address,\n pos: Position,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n ctx.db.profile().insert(Profile {\n id: 1,\n home: Address {\n street: \"1 Main\".into(),\n zip: 11111,\n },\n work: Address {\n street: \"2 Broad\".into(),\n zip: 22222,\n },\n pos: Position { x: 7, y: 9 },\n });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-015-product-type-columns-golden", @@ -1780,21 +1753,21 @@ "work_dir_golden": "target/llm-runs/schema/t_015_product_type_columns/rust/server/golden", "work_dir_llm": "target/llm-runs/schema/t_015_product_type_columns/rust/server/gpt-5/llm", "scorer_details": { - "product_type_columns_row_count": { + "schema_parity": { "pass": false, "partial": 0.0, "notes": { - "actual": 0, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM profile WHERE id=1" + "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-015-product-type-columns-golden`.\n", + "phase": "describe_golden" } }, - "schema_parity": { + "product_type_columns_row_count": { "pass": false, "partial": 0.0, "notes": { - "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-015-product-type-columns-golden`.\n", - "phase": "describe_golden" + "actual": 0, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM profile WHERE id=1" } }, "product_type_columns_row_parity": { @@ -1807,18 +1780,18 @@ } }, "vendor": "openai", - "started_at": "2026-01-22T01:26:34.455886934Z", - "finished_at": "2026-01-22T01:27:17.380173585Z" + "started_at": "2026-01-22T23:37:20.782503774Z", + "finished_at": "2026-01-22T23:38:11.531289607Z" }, "t_016_sum_type_columns": { - "hash": "02bd25d6ae3ac904057e3a8034595f2c28f652956bc1b2b2a1488d169b871b9e", + "hash": "8c8baf7fa35a3b4cdccd079c8efdf96434bd3a22031b7840912b543226efb58b", "task": "t_016_sum_type_columns", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 3, "passed_tests": 0, - "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[derive(spacetimedb::SpacetimeType, Clone, Debug)]\npub struct Rect {\n width: i32,\n height: i32,\n}\n\n#[derive(spacetimedb::SpacetimeType, Clone, Debug)]\npub enum Shape {\n Circle(i32),\n Rectangle(Rect),\n}\n\n#[spacetimedb::table(name = drawing)]\npub struct Drawing {\n #[primary_key]\n id: i32,\n a: Shape,\n b: Shape,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n ctx.db.drawing().insert(Drawing {\n id: 1,\n a: Shape::Circle(10),\n b: Shape::Rectangle(Rect { width: 4, height: 6 }),\n });\n}", + "llm_output": "use spacetimedb::{ReducerContext, SpacetimeType, Table};\n\n#[derive(SpacetimeType, Clone, Debug)]\npub struct Rect {\n pub width: i32,\n pub height: i32,\n}\n\n#[derive(SpacetimeType, Clone, Debug)]\npub enum Shape {\n Circle(i32),\n Rectangle(Rect),\n}\n\n#[spacetimedb::table(name = drawing)]\npub struct Drawing {\n #[primary_key]\n id: i32,\n a: Shape,\n b: Shape,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n let _ = ctx.db.drawing().try_insert(Drawing {\n id: 1,\n a: Shape::Circle(10),\n b: Shape::Rectangle(Rect { width: 4, height: 6 }),\n });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-016-sum-type-columns-golden", @@ -1846,53 +1819,62 @@ "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `drawings`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:37339/v1/database/c200008ae2c2688ac247be4651784d9d95b5ac0fb6e5fc8abfe27a29364fbd61/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `drawings`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:44463/v1/database/c200678c0033bd768bffad2d1cf2eb3ac61dfa8f66ee313a5229d0e3f6d7b8d8/sql)\n", "phase": "sql" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:26:34.442992285Z", - "finished_at": "2026-01-22T01:27:20.272251795Z" + "started_at": "2026-01-22T23:37:20.763375410Z", + "finished_at": "2026-01-22T23:38:46.801354582Z" }, "t_017_scheduled_columns": { - "hash": "02bd25d6ae3ac904057e3a8034595f2c28f652956bc1b2b2a1488d169b871b9e", + "hash": "8c8baf7fa35a3b4cdccd079c8efdf96434bd3a22031b7840912b543226efb58b", "task": "t_017_scheduled_columns", "lang": "rust", - "golden_published": false, + "golden_published": true, "model_name": "GPT-5", - "total_tests": 1, - "passed_tests": 0, - "llm_output": null, + "total_tests": 2, + "passed_tests": 1, + "llm_output": "use std::time::Duration;\nuse spacetimedb::{ReducerContext, ScheduleAt, Table};\n\n#[spacetimedb::table(name = tick_timer, scheduled(tick))]\npub struct TickTimer {\n #[primary_key]\n #[auto_inc]\n scheduled_id: u64,\n scheduled_at: ScheduleAt,\n}\n\n#[spacetimedb::reducer]\npub fn tick(_ctx: &ReducerContext, _timer: TickTimer) -> Result<(), String> {\n Ok(())\n}\n\n#[spacetimedb::reducer(init)]\npub fn init(ctx: &ReducerContext) -> Result<(), String> {\n ctx.db.tick_timer().try_insert(TickTimer {\n scheduled_id: 0,\n scheduled_at: ScheduleAt::Interval(Duration::from_micros(50_000).into()),\n })?;\n Ok(())\n}", "category": "schema", "route_api_model": "gpt-5", - "golden_db": null, - "llm_db": null, - "work_dir_golden": null, - "work_dir_llm": null, + "golden_db": "schema-t-017-scheduled-columns-golden", + "llm_db": "schema-t-017-scheduled-columns-gpt-5-llm", + "work_dir_golden": "target/llm-runs/schema/t_017_scheduled_columns/rust/server/golden", + "work_dir_llm": "target/llm-runs/schema/t_017_scheduled_columns/rust/server/gpt-5/llm", "scorer_details": { - "publish_error": { + "schema_parity": { "pass": false, "partial": 0.0, "notes": { - "error": "LLM call timed out", - "phase": "build_or_publish" + "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-017-scheduled-columns-golden`.\n", + "phase": "describe_golden" + } + }, + "scheduled_seeded_one_row": { + "pass": true, + "partial": 1.0, + "notes": { + "actual": 1, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM tick_timer WHERE scheduled_id>=0" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:28:55.475554783Z", - "finished_at": "2026-01-22T01:28:55.475554783Z" + "started_at": "2026-01-22T23:37:20.754119259Z", + "finished_at": "2026-01-22T23:38:16.323254353Z" }, "t_018_constraints": { - "hash": "02bd25d6ae3ac904057e3a8034595f2c28f652956bc1b2b2a1488d169b871b9e", + "hash": "8c8baf7fa35a3b4cdccd079c8efdf96434bd3a22031b7840912b543226efb58b", "task": "t_018_constraints", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 3, "passed_tests": 0, - "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = account, index(name = by_name, btree(columns = [name])))]\npub struct Account {\n #[primary_key]\n id: i32,\n #[unique]\n email: String,\n name: String,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n ctx.db.account().insert(Account {\n id: 1,\n email: \"a@example.com\".to_string(),\n name: \"Alice\".to_string(),\n });\n ctx.db.account().insert(Account {\n id: 2,\n email: \"b@example.com\".to_string(),\n name: \"Bob\".to_string(),\n });\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = account, index(name = by_name, btree(columns = [name])))]\npub struct Account {\n #[primary_key]\n id: i32,\n #[unique]\n email: String,\n name: String,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) -> Result<(), String> {\n ctx.db.account().try_insert(Account {\n id: 1,\n email: \"a@example.com\".to_string(),\n name: \"Alice\".to_string(),\n })?;\n ctx.db.account().try_insert(Account {\n id: 2,\n email: \"b@example.com\".to_string(),\n name: \"Bob\".to_string(),\n })?;\n Ok(())\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-018-constraints-golden", @@ -1900,13 +1882,12 @@ "work_dir_golden": "target/llm-runs/schema/t_018_constraints/rust/server/golden", "work_dir_llm": "target/llm-runs/schema/t_018_constraints/rust/server/gpt-5/llm", "scorer_details": { - "constraints_seed_two_rows": { + "schema_parity": { "pass": false, "partial": 0.0, "notes": { - "actual": 0, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM account WHERE id=2" + "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-018-constraints-golden`.\n", + "phase": "describe_golden" } }, "constraints_row_parity_after_seed": { @@ -1917,28 +1898,29 @@ "phase": "call_reducer_golden" } }, - "schema_parity": { + "constraints_seed_two_rows": { "pass": false, "partial": 0.0, "notes": { - "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-018-constraints-golden`.\n", - "phase": "describe_golden" + "actual": 0, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM account WHERE id=2" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:26:34.434337758Z", - "finished_at": "2026-01-22T01:27:13.175588688Z" + "started_at": "2026-01-22T23:37:20.750475915Z", + "finished_at": "2026-01-22T23:38:06.509588659Z" }, "t_019_many_to_many": { - "hash": "02bd25d6ae3ac904057e3a8034595f2c28f652956bc1b2b2a1488d169b871b9e", + "hash": "8c8baf7fa35a3b4cdccd079c8efdf96434bd3a22031b7840912b543226efb58b", "task": "t_019_many_to_many", "lang": "rust", "golden_published": true, "model_name": "GPT-5", "total_tests": 5, "passed_tests": 4, - "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = user)]\npub struct User {\n #[primary_key]\n user_id: i32,\n name: String,\n}\n\n#[spacetimedb::table(name = group)]\npub struct Group {\n #[primary_key]\n group_id: i32,\n title: String,\n}\n\n#[spacetimedb::table(\n name = membership,\n index(name = by_user, btree(columns = [user_id])),\n index(name = by_group, btree(columns = [group_id]))\n)]\npub struct Membership {\n #[primary_key]\n id: i32,\n user_id: i32,\n group_id: i32,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n // Clear existing rows to ensure we insert exactly the specified rows\n for m in ctx.db.membership().iter() {\n ctx.db.membership().id().delete(&m.id);\n }\n for u in ctx.db.user().iter() {\n ctx.db.user().user_id().delete(&u.user_id);\n }\n for g in ctx.db.group().iter() {\n ctx.db.group().group_id().delete(&g.group_id);\n }\n\n // Insert users\n ctx.db.user().insert(User { user_id: 1, name: \"Alice\".to_string() });\n ctx.db.user().insert(User { user_id: 2, name: \"Bob\".to_string() });\n\n // Insert groups\n ctx.db.group().insert(Group { group_id: 10, title: \"Admin\".to_string() });\n ctx.db.group().insert(Group { group_id: 20, title: \"Dev\".to_string() });\n\n // Insert memberships\n ctx.db.membership().insert(Membership { id: 1, user_id: 1, group_id: 10 });\n ctx.db.membership().insert(Membership { id: 2, user_id: 1, group_id: 20 });\n ctx.db.membership().insert(Membership { id: 3, user_id: 2, group_id: 20 });\n}", + "llm_output": "use spacetimedb::{ReducerContext, Table};\n\n#[spacetimedb::table(name = user)]\npub struct User {\n #[primary_key]\n user_id: i32,\n name: String,\n}\n\n#[spacetimedb::table(name = group)]\npub struct Group {\n #[primary_key]\n group_id: i32,\n title: String,\n}\n\n#[spacetimedb::table(\n name = membership,\n index(name = by_user, btree(columns = [user_id])),\n index(name = by_group, btree(columns = [group_id]))\n)]\npub struct Membership {\n #[primary_key]\n id: i32,\n user_id: i32,\n group_id: i32,\n}\n\n#[spacetimedb::reducer]\npub fn seed(ctx: &ReducerContext) {\n ctx.db.user().insert(User { user_id: 1, name: \"Alice\".to_string() });\n ctx.db.user().insert(User { user_id: 2, name: \"Bob\".to_string() });\n\n ctx.db.group().insert(Group { group_id: 10, title: \"Admin\".to_string() });\n ctx.db.group().insert(Group { group_id: 20, title: \"Dev\".to_string() });\n\n ctx.db.membership().insert(Membership { id: 1, user_id: 1, group_id: 10 });\n ctx.db.membership().insert(Membership { id: 2, user_id: 1, group_id: 20 });\n ctx.db.membership().insert(Membership { id: 3, user_id: 2, group_id: 20 });\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-019-many-to-many-golden", @@ -1946,14 +1928,6 @@ "work_dir_golden": "target/llm-runs/schema/t_019_many_to_many/rust/server/golden", "work_dir_llm": "target/llm-runs/schema/t_019_many_to_many/rust/server/gpt-5/llm", "scorer_details": { - "schema_parity": { - "pass": false, - "partial": 0.0, - "notes": { - "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-019-many-to-many-golden`.\n", - "phase": "describe_golden" - } - }, "memberships_three_rows": { "pass": true, "partial": 1.0, @@ -1972,6 +1946,14 @@ "sql": "SELECT COUNT(*) AS n FROM membership WHERE user_id=1 AND group_id=20" } }, + "schema_parity": { + "pass": false, + "partial": 0.0, + "notes": { + "error": "describe failed: WARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: failed to find database `schema-t-019-many-to-many-golden`.\n", + "phase": "describe_golden" + } + }, "m2m_has_2_20": { "pass": true, "partial": 1.0, @@ -1992,11 +1974,11 @@ } }, "vendor": "openai", - "started_at": "2026-01-22T01:26:34.449432239Z", - "finished_at": "2026-01-22T01:28:06.572084591Z" + "started_at": "2026-01-22T23:37:20.772955515Z", + "finished_at": "2026-01-22T23:38:45.020247369Z" }, "t_020_ecs": { - "hash": "02bd25d6ae3ac904057e3a8034595f2c28f652956bc1b2b2a1488d169b871b9e", + "hash": "8c8baf7fa35a3b4cdccd079c8efdf96434bd3a22031b7840912b543226efb58b", "task": "t_020_ecs", "lang": "rust", "golden_published": false, @@ -2021,11 +2003,11 @@ } }, "vendor": "openai", - "started_at": "2026-01-22T01:28:55.475566278Z", - "finished_at": "2026-01-22T01:28:55.475566278Z" + "started_at": "2026-01-22T23:39:46.899187354Z", + "finished_at": "2026-01-22T23:39:46.899187354Z" }, "t_021_multi_column_index": { - "hash": "02bd25d6ae3ac904057e3a8034595f2c28f652956bc1b2b2a1488d169b871b9e", + "hash": "8c8baf7fa35a3b4cdccd079c8efdf96434bd3a22031b7840912b543226efb58b", "task": "t_021_multi_column_index", "lang": "rust", "golden_published": true, @@ -2077,8 +2059,8 @@ } }, "vendor": "openai", - "started_at": "2026-01-22T01:26:34.439768584Z", - "finished_at": "2026-01-22T01:27:20.653482735Z" + "started_at": "2026-01-22T23:37:20.758673191Z", + "finished_at": "2026-01-22T23:38:09.921232585Z" } } } @@ -2269,14 +2251,14 @@ "modes": [ { "mode": "docs", - "hash": "2f624c48d5e0cb6f1b8f45bf686d0fdd92b425ca4e5633757ad2bccaed4abbec", + "hash": "a96f7f53940945bb938bc8ee4e931c2c2a88c31a961aab94f52f7d5b336acd88", "models": [ { "name": "GPT-5", "route_api_model": "gpt-5", "tasks": { "t_000_empty_reducers": { - "hash": "2f624c48d5e0cb6f1b8f45bf686d0fdd92b425ca4e5633757ad2bccaed4abbec", + "hash": "a96f7f53940945bb938bc8ee4e931c2c2a88c31a961aab94f52f7d5b336acd88", "task": "t_000_empty_reducers", "lang": "csharp", "golden_published": true, @@ -2299,25 +2281,25 @@ "llm_db": "basics-t-000-empty-reducers-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34777", + "server": "http://127.0.0.1:36809", "tables_diff": null, "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-22T01:32:38.565221901Z", - "finished_at": "2026-01-22T01:33:02.737119282Z" + "started_at": "2026-01-22T23:43:50.222711339Z", + "finished_at": "2026-01-22T23:44:14.996016087Z" }, "t_001_basic_tables": { - "hash": "2f624c48d5e0cb6f1b8f45bf686d0fdd92b425ca4e5633757ad2bccaed4abbec", + "hash": "a96f7f53940945bb938bc8ee4e931c2c2a88c31a961aab94f52f7d5b336acd88", "task": "t_001_basic_tables", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 1, "passed_tests": 1, - "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"User\", Public = true)]\n public partial struct User\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n public int Age;\n public bool Active;\n }\n\n [SpacetimeDB.Table(Name = \"Product\", Public = true)]\n public partial struct Product\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Title;\n public float Price;\n public bool InStock;\n }\n\n [SpacetimeDB.Table(Name = \"Note\", Public = true)]\n public partial struct Note\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Body;\n public long Rating;\n public bool Pinned;\n }\n}", + "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"User\")]\n public partial struct User\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n public int Age;\n public bool Active;\n }\n\n [SpacetimeDB.Table(Name = \"Product\")]\n public partial struct Product\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Title;\n public float Price;\n public bool InStock;\n }\n\n [SpacetimeDB.Table(Name = \"Note\")]\n public partial struct Note\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Body;\n public long Rating;\n public bool Pinned;\n }\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-001-basic-tables-golden", @@ -2333,25 +2315,25 @@ "llm_db": "basics-t-001-basic-tables-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34777", + "server": "http://127.0.0.1:36809", "tables_diff": null, "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-22T01:33:02.737145208Z", - "finished_at": "2026-01-22T01:33:33.926548006Z" + "started_at": "2026-01-22T23:43:57.181488892Z", + "finished_at": "2026-01-22T23:44:30.872087466Z" }, "t_002_scheduled_table": { - "hash": "2f624c48d5e0cb6f1b8f45bf686d0fdd92b425ca4e5633757ad2bccaed4abbec", + "hash": "a96f7f53940945bb938bc8ee4e931c2c2a88c31a961aab94f52f7d5b336acd88", "task": "t_002_scheduled_table", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 1, "passed_tests": 1, - "llm_output": "using System;\nusing SpacetimeDB;\n\npublic static partial class Module\n{\n [Table(Name = \"TickTimer\", Scheduled = nameof(Tick), ScheduledAt = nameof(ScheduledAt))]\n public partial struct TickTimer\n {\n [PrimaryKey, AutoInc]\n public ulong ScheduledId;\n public ScheduleAt ScheduledAt;\n }\n\n [Reducer]\n public static void Tick(ReducerContext ctx, TickTimer _timer)\n {\n }\n\n [Reducer(ReducerKind.Init)]\n public static void Init(ReducerContext ctx)\n {\n if (ctx.Db.TickTimer.Count == 0)\n {\n ctx.Db.TickTimer.Insert(new TickTimer\n {\n ScheduledAt = new ScheduleAt.Interval(TimeSpan.FromMilliseconds(50))\n });\n }\n }\n}", + "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"TickTimer\", Scheduled = nameof(Tick), ScheduledAt = nameof(ScheduledAt))]\n public partial struct TickTimer\n {\n [SpacetimeDB.PrimaryKey]\n [SpacetimeDB.AutoInc]\n public ulong ScheduledId;\n public ScheduleAt ScheduledAt;\n }\n\n [SpacetimeDB.Reducer(ReducerKind.Init)]\n public static void Init(ReducerContext ctx)\n {\n if (ctx.Db.TickTimer.Count == 0)\n {\n ctx.Db.TickTimer.Insert(new TickTimer\n {\n ScheduledAt = new ScheduleAt.Interval(TimeSpan.FromMilliseconds(50))\n });\n }\n }\n\n [SpacetimeDB.Reducer]\n public static void Tick(ReducerContext ctx, TickTimer timer)\n {\n }\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-002-scheduled-table-golden", @@ -2367,18 +2349,18 @@ "llm_db": "basics-t-002-scheduled-table-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34777", + "server": "http://127.0.0.1:36809", "tables_diff": null, "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-22T01:33:29.152129533Z", - "finished_at": "2026-01-22T01:34:07.762428301Z" + "started_at": "2026-01-22T23:44:30.935699477Z", + "finished_at": "2026-01-22T23:45:28.059233544Z" }, "t_003_struct_in_table": { - "hash": "2f624c48d5e0cb6f1b8f45bf686d0fdd92b425ca4e5633757ad2bccaed4abbec", + "hash": "a96f7f53940945bb938bc8ee4e931c2c2a88c31a961aab94f52f7d5b336acd88", "task": "t_003_struct_in_table", "lang": "csharp", "golden_published": true, @@ -2401,25 +2383,25 @@ "llm_db": "basics-t-003-struct-in-table-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34777", + "server": "http://127.0.0.1:36809", "tables_diff": null, "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-22T01:32:45.095441251Z", - "finished_at": "2026-01-22T01:33:10.875700597Z" + "started_at": "2026-01-22T23:43:57.176863757Z", + "finished_at": "2026-01-22T23:44:30.932745371Z" }, "t_004_insert": { - "hash": "2f624c48d5e0cb6f1b8f45bf686d0fdd92b425ca4e5633757ad2bccaed4abbec", + "hash": "a96f7f53940945bb938bc8ee4e931c2c2a88c31a961aab94f52f7d5b336acd88", "task": "t_004_insert", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 2, "passed_tests": 2, - "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"User\", Public = true)]\n public partial struct User\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n public int Age;\n public bool Active;\n }\n\n [SpacetimeDB.Reducer]\n public static void InsertUser(ReducerContext ctx, int id, string name, int age, bool active)\n {\n ctx.Db.User.Insert(new User\n {\n Id = id,\n Name = name,\n Age = age,\n Active = active\n });\n }\n}", + "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"User\")]\n public partial struct User\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n public int Age;\n public bool Active;\n }\n\n [SpacetimeDB.Reducer]\n public static void InsertUser(ReducerContext ctx, int id, string name, int age, bool active)\n {\n ctx.Db.User.Insert(new User\n {\n Id = id,\n Name = name,\n Age = age,\n Active = active\n });\n }\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-004-insert-golden", @@ -2443,7 +2425,7 @@ "llm_out": "Id | Name | Age | Active ----+---------+-----+-------- 1 | \"Alice\" | 30 | true", "query": "SELECT Id, Name, Age, Active FROM User WHERE Id=1", "reducer": "InsertUser", - "server": "http://127.0.0.1:34777" + "server": "http://127.0.0.1:36809" } }, "schema_parity": { @@ -2454,25 +2436,25 @@ "llm_db": "basics-t-004-insert-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34777", + "server": "http://127.0.0.1:36809", "tables_diff": null, "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-22T01:33:05.882240139Z", - "finished_at": "2026-01-22T01:33:29.152085676Z" + "started_at": "2026-01-22T23:44:14.996042858Z", + "finished_at": "2026-01-22T23:44:36.919869528Z" }, "t_005_update": { - "hash": "2f624c48d5e0cb6f1b8f45bf686d0fdd92b425ca4e5633757ad2bccaed4abbec", + "hash": "a96f7f53940945bb938bc8ee4e931c2c2a88c31a961aab94f52f7d5b336acd88", "task": "t_005_update", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 3, "passed_tests": 3, - "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"User\")]\n public partial struct User\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n public int Age;\n public bool Active;\n }\n\n [SpacetimeDB.Reducer]\n public static void UpdateUser(ReducerContext ctx, int id, string name, int age, bool active)\n {\n if (ctx.Db.User.Id.Find(id) is User user)\n {\n user.Id = id;\n user.Name = name;\n user.Age = age;\n user.Active = active;\n ctx.Db.User.Id.Update(user);\n }\n else\n {\n ctx.Db.User.Insert(new User\n {\n Id = id,\n Name = name,\n Age = age,\n Active = active\n });\n }\n }\n}", + "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"User\")]\n public partial struct User\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n public int Age;\n public bool Active;\n }\n\n [SpacetimeDB.Reducer]\n public static void UpdateUser(ReducerContext ctx, int id, string name, int age, bool active)\n {\n var row = new User\n {\n Id = id,\n Name = name,\n Age = age,\n Active = active\n };\n ctx.Db.User.Id.Update(row);\n }\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-005-update-golden", @@ -2488,7 +2470,7 @@ "llm_db": "basics-t-005-update-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34777", + "server": "http://127.0.0.1:36809", "tables_diff": null, "tables_equal": true } @@ -2516,16 +2498,16 @@ "llm_out": "Id | Name | Age | Active ----+----------+-----+-------- 1 | \"Alice2\" | 31 | false", "query": "SELECT Id, Name, Age, Active FROM User WHERE Id=1", "reducer": "UpdateUser", - "server": "http://127.0.0.1:34777" + "server": "http://127.0.0.1:36809" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:32:29.987997822Z", - "finished_at": "2026-01-22T01:33:05.882201733Z" + "started_at": "2026-01-22T23:43:34.539772160Z", + "finished_at": "2026-01-22T23:44:08.708011172Z" }, "t_006_delete": { - "hash": "2f624c48d5e0cb6f1b8f45bf686d0fdd92b425ca4e5633757ad2bccaed4abbec", + "hash": "a96f7f53940945bb938bc8ee4e931c2c2a88c31a961aab94f52f7d5b336acd88", "task": "t_006_delete", "lang": "csharp", "golden_published": true, @@ -2540,11 +2522,13 @@ "work_dir_golden": "target/llm-runs/basics/t_006_delete/csharp/server/golden", "work_dir_llm": "target/llm-runs/basics/t_006_delete/csharp/server/gpt-5/llm", "scorer_details": { - "seed_users_row": { + "delete_user_count_zero": { "pass": true, "partial": 1.0, "notes": { - "sql": "INSERT INTO User(Id, Name, Age, Active) VALUES (1, 'Alice', 30, true)" + "actual": 0, + "expected": 0, + "sql": "SELECT COUNT(*) AS n FROM User WHERE Id=1" } }, "schema_parity": { @@ -2555,34 +2539,32 @@ "llm_db": "basics-t-006-delete-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34777", + "server": "http://127.0.0.1:36809", "tables_diff": null, "tables_equal": true } }, - "delete_user_count_zero": { + "seed_users_row": { "pass": true, "partial": 1.0, "notes": { - "actual": 0, - "expected": 0, - "sql": "SELECT COUNT(*) AS n FROM User WHERE Id=1" + "sql": "INSERT INTO User(Id, Name, Age, Active) VALUES (1, 'Alice', 30, true)" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:32:02.704209854Z", - "finished_at": "2026-01-22T01:32:29.987958215Z" + "started_at": "2026-01-22T23:43:02.977383955Z", + "finished_at": "2026-01-22T23:43:29.316267467Z" }, "t_007_crud": { - "hash": "2f624c48d5e0cb6f1b8f45bf686d0fdd92b425ca4e5633757ad2bccaed4abbec", + "hash": "a96f7f53940945bb938bc8ee4e931c2c2a88c31a961aab94f52f7d5b336acd88", "task": "t_007_crud", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 4, "passed_tests": 4, - "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"User\", Public = true)]\n public partial struct User\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n public int Age;\n public bool Active;\n }\n\n [SpacetimeDB.Reducer]\n public static void Crud(ReducerContext ctx)\n {\n ctx.Db.User.Insert(new User { Id = 1, Name = \"Alice\", Age = 30, Active = true });\n ctx.Db.User.Insert(new User { Id = 2, Name = \"Bob\", Age = 22, Active = false });\n ctx.Db.User.Id.Update(new User { Id = 1, Name = \"Alice2\", Age = 31, Active = false });\n ctx.Db.User.Id.Delete(2);\n }\n}", + "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"User\", Public = true)]\n public partial struct User\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n public int Age;\n public bool Active;\n }\n\n [SpacetimeDB.Reducer]\n public static void Crud(ReducerContext ctx)\n {\n ctx.Db.User.Insert(new User { Id = 1, Name = \"Alice\", Age = 30, Active = true });\n ctx.Db.User.Insert(new User { Id = 2, Name = \"Bob\", Age = 22, Active = false });\n\n var u1 = ctx.Db.User.Id.Find(1);\n if (u1 is User user1)\n {\n user1.Name = \"Alice2\";\n user1.Age = 31;\n user1.Active = false;\n ctx.Db.User.Id.Update(user1);\n }\n\n ctx.Db.User.Id.Delete(2);\n }\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-007-crud-golden", @@ -2590,31 +2572,31 @@ "work_dir_golden": "target/llm-runs/basics/t_007_crud/csharp/server/golden", "work_dir_llm": "target/llm-runs/basics/t_007_crud/csharp/server/gpt-5/llm", "scorer_details": { - "schema_parity": { + "crud_row_id1_parity": { "pass": true, "partial": 1.0, "notes": { + "args": [], "golden_db": "basics-t-007-crud-golden", + "golden_out": "Id | Name | Age | Active ----+----------+-----+-------- 1 | \"Alice2\" | 31 | false", "llm_db": "basics-t-007-crud-gpt-5-llm", - "reducers_diff": null, - "reducers_equal": true, - "server": "http://127.0.0.1:34777", - "tables_diff": null, - "tables_equal": true + "llm_out": "Id | Name | Age | Active ----+----------+-----+-------- 1 | \"Alice2\" | 31 | false", + "query": "SELECT Id, Name, Age, Active FROM User WHERE Id=1", + "reducer": "Crud", + "server": "http://127.0.0.1:36809" } }, - "crud_row_id1_parity": { + "schema_parity": { "pass": true, "partial": 1.0, "notes": { - "args": [], "golden_db": "basics-t-007-crud-golden", - "golden_out": "Id | Name | Age | Active ----+----------+-----+-------- 1 | \"Alice2\" | 31 | false", "llm_db": "basics-t-007-crud-gpt-5-llm", - "llm_out": "Id | Name | Age | Active ----+----------+-----+-------- 1 | \"Alice2\" | 31 | false", - "query": "SELECT Id, Name, Age, Active FROM User WHERE Id=1", - "reducer": "Crud", - "server": "http://127.0.0.1:34777" + "reducers_diff": null, + "reducers_equal": true, + "server": "http://127.0.0.1:36809", + "tables_diff": null, + "tables_equal": true } }, "crud_total_count_one": { @@ -2637,18 +2619,18 @@ } }, "vendor": "openai", - "started_at": "2026-01-22T01:32:21.843479478Z", - "finished_at": "2026-01-22T01:33:02.676164531Z" + "started_at": "2026-01-22T23:43:26.976858896Z", + "finished_at": "2026-01-22T23:43:57.055778354Z" }, "t_008_index_lookup": { - "hash": "2f624c48d5e0cb6f1b8f45bf686d0fdd92b425ca4e5633757ad2bccaed4abbec", + "hash": "a96f7f53940945bb938bc8ee4e931c2c2a88c31a961aab94f52f7d5b336acd88", "task": "t_008_index_lookup", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 3, "passed_tests": 3, - "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"User\")]\n public partial struct User\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n public int Age;\n public bool Active;\n }\n\n [SpacetimeDB.Table(Name = \"Result\")]\n public partial struct Result\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n }\n\n [SpacetimeDB.Reducer]\n public static void LookupUserName(ReducerContext ctx, int id)\n {\n if (ctx.Db.User.Id.Find(id) is User user)\n {\n ctx.Db.Result.Insert(new Result\n {\n Id = user.Id,\n Name = user.Name\n });\n }\n }\n}", + "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"User\")]\n public partial struct User\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n public int Age;\n public bool Active;\n }\n\n [SpacetimeDB.Table(Name = \"Result\")]\n public partial struct Result\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n }\n\n [SpacetimeDB.Reducer]\n public static void LookupUserName(ReducerContext ctx, int id)\n {\n if (ctx.Db.User.Id.Find(id) is User user)\n {\n ctx.Db.Result.Insert(new Result { Id = user.Id, Name = user.Name });\n }\n }\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-008-index-lookup-golden", @@ -2656,13 +2638,6 @@ "work_dir_golden": "target/llm-runs/basics/t_008_index_lookup/csharp/server/golden", "work_dir_llm": "target/llm-runs/basics/t_008_index_lookup/csharp/server/gpt-5/llm", "scorer_details": { - "seed_user_row": { - "pass": true, - "partial": 1.0, - "notes": { - "sql": "INSERT INTO User(Id, Name, Age, Active) VALUES (1, 'Alice', 30, true)" - } - }, "index_lookup_projection_parity": { "pass": true, "partial": 1.0, @@ -2676,7 +2651,7 @@ "llm_out": "Id | Name ----+--------- 1 | \"Alice\"", "query": "SELECT Id, Name FROM Result WHERE Id=1", "reducer": "LookupUserName", - "server": "http://127.0.0.1:34777" + "server": "http://127.0.0.1:36809" } }, "schema_parity": { @@ -2687,25 +2662,32 @@ "llm_db": "basics-t-008-index-lookup-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34777", + "server": "http://127.0.0.1:36809", "tables_diff": null, "tables_equal": true } + }, + "seed_user_row": { + "pass": true, + "partial": 1.0, + "notes": { + "sql": "INSERT INTO User(Id, Name, Age, Active) VALUES (1, 'Alice', 30, true)" + } } }, "vendor": "openai", - "started_at": "2026-01-22T01:33:02.740251899Z", - "finished_at": "2026-01-22T01:33:38.294749307Z" + "started_at": "2026-01-22T23:44:08.708044539Z", + "finished_at": "2026-01-22T23:44:47.327282315Z" }, "t_009_init": { - "hash": "2f624c48d5e0cb6f1b8f45bf686d0fdd92b425ca4e5633757ad2bccaed4abbec", + "hash": "a96f7f53940945bb938bc8ee4e931c2c2a88c31a961aab94f52f7d5b336acd88", "task": "t_009_init", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 4, "passed_tests": 4, - "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"User\", Public = true)]\n public partial struct User\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n public int Age;\n public bool Active;\n }\n\n [SpacetimeDB.Reducer(ReducerKind.Init)]\n public static void Init(ReducerContext ctx)\n {\n ctx.Db.User.Insert(new User { Id = 1, Name = \"Alice\", Age = 30, Active = true });\n ctx.Db.User.Insert(new User { Id = 2, Name = \"Bob\", Age = 22, Active = false });\n }\n}", + "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"User\")]\n public partial struct User\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public string Name;\n public int Age;\n public bool Active;\n }\n\n [SpacetimeDB.Reducer(ReducerKind.Init)]\n public static void Init(ReducerContext ctx)\n {\n ctx.Db.User.Insert(new User { Id = 1, Name = \"Alice\", Age = 30, Active = true });\n ctx.Db.User.Insert(new User { Id = 2, Name = \"Bob\", Age = 22, Active = false });\n }\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-009-init-golden", @@ -2713,6 +2695,19 @@ "work_dir_golden": "target/llm-runs/basics/t_009_init/csharp/server/golden", "work_dir_llm": "target/llm-runs/basics/t_009_init/csharp/server/gpt-5/llm", "scorer_details": { + "schema_parity": { + "pass": true, + "partial": 1.0, + "notes": { + "golden_db": "basics-t-009-init-golden", + "llm_db": "basics-t-009-init-gpt-5-llm", + "reducers_diff": null, + "reducers_equal": true, + "server": "http://127.0.0.1:36809", + "tables_diff": null, + "tables_equal": true + } + }, "init_seed_alice": { "pass": true, "partial": 1.0, @@ -2731,19 +2726,6 @@ "sql": "SELECT COUNT(*) AS n FROM User WHERE Id=2 AND Name='Bob' AND Age=22 AND Active=false" } }, - "schema_parity": { - "pass": true, - "partial": 1.0, - "notes": { - "golden_db": "basics-t-009-init-golden", - "llm_db": "basics-t-009-init-gpt-5-llm", - "reducers_diff": null, - "reducers_equal": true, - "server": "http://127.0.0.1:34777", - "tables_diff": null, - "tables_equal": true - } - }, "init_total_two": { "pass": true, "partial": 1.0, @@ -2755,18 +2737,18 @@ } }, "vendor": "openai", - "started_at": "2026-01-22T01:32:21.846984094Z", - "finished_at": "2026-01-22T01:32:45.095414321Z" + "started_at": "2026-01-22T23:43:29.316301903Z", + "finished_at": "2026-01-22T23:43:57.176825023Z" }, "t_010_connect": { - "hash": "2f624c48d5e0cb6f1b8f45bf686d0fdd92b425ca4e5633757ad2bccaed4abbec", + "hash": "a96f7f53940945bb938bc8ee4e931c2c2a88c31a961aab94f52f7d5b336acd88", "task": "t_010_connect", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 1, "passed_tests": 1, - "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"Event\", Public = true)]\n public partial struct Event\n {\n [SpacetimeDB.PrimaryKey]\n [SpacetimeDB.AutoInc]\n public int Id;\n public string Kind;\n }\n\n [SpacetimeDB.Reducer(ReducerKind.ClientConnected)]\n public static void ClientConnected(ReducerContext ctx)\n {\n ctx.Db.Event.Insert(new Event { Kind = \"connected\" });\n }\n\n [SpacetimeDB.Reducer(ReducerKind.ClientDisconnected)]\n public static void ClientDisconnected(ReducerContext ctx)\n {\n ctx.Db.Event.Insert(new Event { Kind = \"disconnected\" });\n }\n}", + "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"Event\")]\n public partial struct Event\n {\n [SpacetimeDB.PrimaryKey]\n [SpacetimeDB.AutoInc]\n public int Id;\n public string Kind;\n }\n\n [SpacetimeDB.Reducer(ReducerKind.ClientConnected)]\n public static void ClientConnected(ReducerContext ctx)\n {\n ctx.Db.Event.Insert(new Event { Kind = \"connected\" });\n }\n\n [SpacetimeDB.Reducer(ReducerKind.ClientDisconnected)]\n public static void ClientDisconnected(ReducerContext ctx)\n {\n ctx.Db.Event.Insert(new Event { Kind = \"disconnected\" });\n }\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-010-connect-golden", @@ -2782,25 +2764,25 @@ "llm_db": "basics-t-010-connect-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34777", + "server": "http://127.0.0.1:36809", "tables_diff": null, "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-22T01:32:11.564139379Z", - "finished_at": "2026-01-22T01:32:38.565193128Z" + "started_at": "2026-01-22T23:43:12.629915535Z", + "finished_at": "2026-01-22T23:43:50.222686267Z" }, "t_011_helper_function": { - "hash": "2f624c48d5e0cb6f1b8f45bf686d0fdd92b425ca4e5633757ad2bccaed4abbec", + "hash": "a96f7f53940945bb938bc8ee4e931c2c2a88c31a961aab94f52f7d5b336acd88", "task": "t_011_helper_function", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 3, "passed_tests": 3, - "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [Table(Name = \"Result\", Public = true)]\n public partial struct Result\n {\n [PrimaryKey]\n public int Id;\n public int Sum;\n }\n\n public static int Add(int a, int b)\n {\n return a + b;\n }\n\n [Reducer]\n public static void ComputeSum(ReducerContext ctx, int id, int a, int b)\n {\n ctx.Db.Result.Insert(new Result\n {\n Id = id,\n Sum = Add(a, b)\n });\n }\n}", + "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"Result\")]\n public partial struct Result\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public int Sum;\n }\n\n public static int Add(int a, int b)\n {\n return a + b;\n }\n\n [SpacetimeDB.Reducer]\n public static void ComputeSum(ReducerContext ctx, int id, int a, int b)\n {\n ctx.Db.Result.Insert(new Result { Id = id, Sum = Add(a, b) });\n }\n}", "category": "basics", "route_api_model": "gpt-5", "golden_db": "basics-t-011-helper-function-golden", @@ -2808,15 +2790,6 @@ "work_dir_golden": "target/llm-runs/basics/t_011_helper_function/csharp/server/golden", "work_dir_llm": "target/llm-runs/basics/t_011_helper_function/csharp/server/gpt-5/llm", "scorer_details": { - "helper_func_sum_abs": { - "pass": true, - "partial": 1.0, - "notes": { - "actual": 1, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM Result WHERE Id=1 AND Sum=5" - } - }, "schema_parity": { "pass": true, "partial": 1.0, @@ -2825,7 +2798,7 @@ "llm_db": "basics-t-011-helper-function-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34777", + "server": "http://127.0.0.1:36809", "tables_diff": null, "tables_equal": true } @@ -2845,16 +2818,25 @@ "llm_out": "Id | Sum ----+----- 1 | 5", "query": "SELECT Id, Sum FROM Result WHERE Id=1", "reducer": "ComputeSum", - "server": "http://127.0.0.1:34777" + "server": "http://127.0.0.1:36809" + } + }, + "helper_func_sum_abs": { + "pass": true, + "partial": 1.0, + "notes": { + "actual": 1, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM Result WHERE Id=1 AND Sum=5" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:33:10.875760221Z", - "finished_at": "2026-01-22T01:33:36.565740131Z" + "started_at": "2026-01-22T23:44:30.932767567Z", + "finished_at": "2026-01-22T23:45:03.277674189Z" }, "t_012_spacetime_product_type": { - "hash": "2f624c48d5e0cb6f1b8f45bf686d0fdd92b425ca4e5633757ad2bccaed4abbec", + "hash": "a96f7f53940945bb938bc8ee4e931c2c2a88c31a961aab94f52f7d5b336acd88", "task": "t_012_spacetime_product_type", "lang": "csharp", "golden_published": true, @@ -2877,11 +2859,20 @@ "llm_db": "schema-t-012-spacetime-product-type-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34777", + "server": "http://127.0.0.1:36809", "tables_diff": null, "tables_equal": true } }, + "product_type_row_count": { + "pass": true, + "partial": 1.0, + "notes": { + "actual": 1, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM Result WHERE Id=1" + } + }, "product_type_row_parity": { "pass": true, "partial": 1.0, @@ -2897,32 +2888,23 @@ "llm_out": "Id | Value ----+----------------------- 1 | (Left = 2, Right = 3)", "query": "SELECT Id, Value FROM Result WHERE Id=1", "reducer": "SetScore", - "server": "http://127.0.0.1:34777" - } - }, - "product_type_row_count": { - "pass": true, - "partial": 1.0, - "notes": { - "actual": 1, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM Result WHERE Id=1" + "server": "http://127.0.0.1:36809" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:31:51.511832808Z", - "finished_at": "2026-01-22T01:32:11.564096171Z" + "started_at": "2026-01-22T23:42:53.836516433Z", + "finished_at": "2026-01-22T23:43:34.539741733Z" }, "t_013_spacetime_sum_type": { - "hash": "2f624c48d5e0cb6f1b8f45bf686d0fdd92b425ca4e5633757ad2bccaed4abbec", + "hash": "a96f7f53940945bb938bc8ee4e931c2c2a88c31a961aab94f52f7d5b336acd88", "task": "t_013_spacetime_sum_type", "lang": "csharp", - "golden_published": false, + "golden_published": true, "model_name": "GPT-5", "total_tests": 3, - "passed_tests": 0, - "llm_output": "using SpacetimeDB;\nusing SpacetimeDB.Types;\n\npublic static partial class Module\n{\n [SpacetimeDB.Type]\n public partial struct Circle\n {\n public int Radius;\n }\n\n [SpacetimeDB.Type]\n public partial struct Rectangle\n {\n public int Width;\n public int Height;\n }\n\n [SpacetimeDB.Type]\n public partial record Shape : TaggedEnum<(Circle Circle, Rectangle Rectangle)> { }\n\n [SpacetimeDB.Table(Name = \"Result\")]\n public partial struct Result\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public Shape Value;\n }\n\n [SpacetimeDB.Reducer]\n public static void SetCircle(ReducerContext ctx, int id, int radius)\n {\n ctx.Db.Result.Insert(new Result\n {\n Id = id,\n Value = new Shape.Circle(new Circle { Radius = radius })\n });\n }\n}", + "passed_tests": 3, + "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Type]\n public partial struct Circle\n {\n public int Radius;\n }\n\n [SpacetimeDB.Type]\n public partial struct Rectangle\n {\n public int Width;\n public int Height;\n }\n\n [SpacetimeDB.Type]\n public partial record Shape : TaggedEnum<(Circle Circle, Rectangle Rectangle)> { }\n\n [SpacetimeDB.Table(Name = \"Result\")]\n public partial struct Result\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public Shape Value;\n }\n\n [SpacetimeDB.Reducer]\n public static void SetCircle(ReducerContext ctx, int id, int radius)\n {\n var shape = new Shape.Circle(new Circle { Radius = radius });\n ctx.Db.Result.Insert(new Result\n {\n Id = id,\n Value = shape\n });\n }\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-013-spacetime-sum-type-golden", @@ -2930,28 +2912,59 @@ "work_dir_golden": "target/llm-runs/schema/t_013_spacetime_sum_type/csharp/server/golden", "work_dir_llm": "target/llm-runs/schema/t_013_spacetime_sum_type/csharp/server/gpt-5/llm", "scorer_details": { - "publish_error": { - "pass": false, - "partial": 0.0, + "sum_type_row_count": { + "pass": true, + "partial": 1.0, "notes": { - "error": "spacetime build (csharp) failed (exit=1)\n--- stderr ---\nError: command [\"dotnet\", \"publish\", \"-c\", \"Release\", \"-v\", \"quiet\"] exited with code 1\n\n--- stdout ---\n/__w/SpacetimeDB/SpacetimeDB/target/llm-runs/schema/t_013_spacetime_sum_type/csharp/server/gpt-5/llm/Lib.cs(3,19): error CS0234: The type or namespace name 'Types' does not exist in the namespace 'SpacetimeDB' (are you missing an assembly reference?) [/__w/SpacetimeDB/SpacetimeDB/target/llm-runs/schema/t_013_spacetime_sum_type/csharp/server/gpt-5/llm/StdbModule.csproj]\n", - "phase": "build_or_publish" + "actual": 1, + "expected": 1, + "sql": "SELECT COUNT(*) AS n FROM Result WHERE Id=1" + } + }, + "schema_parity": { + "pass": true, + "partial": 1.0, + "notes": { + "golden_db": "schema-t-013-spacetime-sum-type-golden", + "llm_db": "schema-t-013-spacetime-sum-type-gpt-5-llm", + "reducers_diff": null, + "reducers_equal": true, + "server": "http://127.0.0.1:36809", + "tables_diff": null, + "tables_equal": true + } + }, + "sum_type_row_parity": { + "pass": true, + "partial": 1.0, + "notes": { + "args": [ + 1, + 10 + ], + "golden_db": "schema-t-013-spacetime-sum-type-golden", + "golden_out": "Id | Value ----+-------------------------- 1 | (Circle = (Radius = 10))", + "llm_db": "schema-t-013-spacetime-sum-type-gpt-5-llm", + "llm_out": "Id | Value ----+-------------------------- 1 | (Circle = (Radius = 10))", + "query": "SELECT Id, Value FROM Result WHERE Id=1", + "reducer": "SetCircle", + "server": "http://127.0.0.1:36809" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:30:55.524179859Z", - "finished_at": "2026-01-22T01:31:31.760392246Z" + "started_at": "2026-01-22T23:41:59.253501291Z", + "finished_at": "2026-01-22T23:42:42.372017468Z" }, "t_014_elementary_columns": { - "hash": "2f624c48d5e0cb6f1b8f45bf686d0fdd92b425ca4e5633757ad2bccaed4abbec", + "hash": "a96f7f53940945bb938bc8ee4e931c2c2a88c31a961aab94f52f7d5b336acd88", "task": "t_014_elementary_columns", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 3, "passed_tests": 1, - "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"Primitive\")]\n public partial struct Primitive\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public int Count;\n public long Total;\n public float Price;\n public double Ratio;\n public bool Active;\n public string Name;\n }\n\n [SpacetimeDB.Reducer]\n public static void Seed(ReducerContext ctx)\n {\n ctx.Db.Primitive.Insert(new Primitive\n {\n Id = 1,\n Count = 2,\n Total = 3000000000L,\n Price = 1.5f,\n Ratio = 2.25,\n Active = true,\n Name = \"Alice\"\n });\n }\n}", + "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [Table(Name = \"Primitive\", Public = true)]\n public partial struct Primitive\n {\n [PrimaryKey]\n public int Id;\n public int Count;\n public long Total;\n public float Price;\n public double Ratio;\n public bool Active;\n public string Name;\n }\n\n [Reducer]\n public static void Seed(ReducerContext ctx)\n {\n ctx.Db.Primitive.Insert(new Primitive\n {\n Id = 1,\n Count = 2,\n Total = 3000000000L,\n Price = 1.5f,\n Ratio = 2.25,\n Active = true,\n Name = \"Alice\"\n });\n }\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-014-elementary-columns-golden", @@ -2959,49 +2972,49 @@ "work_dir_golden": "target/llm-runs/schema/t_014_elementary_columns/csharp/server/golden", "work_dir_llm": "target/llm-runs/schema/t_014_elementary_columns/csharp/server/gpt-5/llm", "scorer_details": { - "elementary_columns_row_count": { - "pass": false, - "partial": 0.0, + "schema_parity": { + "pass": true, + "partial": 1.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `primitive`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:34777/v1/database/c200362496d15ba72a9ed15695bda15027963f8725f36d540c39ad7c75198f4a/sql)\n", - "phase": "sql" + "golden_db": "schema-t-014-elementary-columns-golden", + "llm_db": "schema-t-014-elementary-columns-gpt-5-llm", + "reducers_diff": null, + "reducers_equal": true, + "server": "http://127.0.0.1:36809", + "tables_diff": null, + "tables_equal": true } }, "elementary_columns_row_parity": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `primitive`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:34777/v1/database/c2007e024cc6c7538ee9df49e46be07b08410f4c16631c244cd417a0d33cfefa/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `primitive`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:36809/v1/database/c2008f465c5c2d89b252463cdeb748848afcca243af87ed18fccd33254b0df28/sql)\n", "phase": "sql_golden" } }, - "schema_parity": { - "pass": true, - "partial": 1.0, + "elementary_columns_row_count": { + "pass": false, + "partial": 0.0, "notes": { - "golden_db": "schema-t-014-elementary-columns-golden", - "llm_db": "schema-t-014-elementary-columns-gpt-5-llm", - "reducers_diff": null, - "reducers_equal": true, - "server": "http://127.0.0.1:34777", - "tables_diff": null, - "tables_equal": true + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `primitive`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:36809/v1/database/c200e1d2c4027b7bb7de30ad2d5a2f3912b025df511680d607faf441ccabe5cd/sql)\n", + "phase": "sql" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:31:39.644372308Z", - "finished_at": "2026-01-22T01:32:21.620372253Z" + "started_at": "2026-01-22T23:42:42.372083581Z", + "finished_at": "2026-01-22T23:43:12.629885046Z" }, "t_015_product_type_columns": { - "hash": "2f624c48d5e0cb6f1b8f45bf686d0fdd92b425ca4e5633757ad2bccaed4abbec", + "hash": "a96f7f53940945bb938bc8ee4e931c2c2a88c31a961aab94f52f7d5b336acd88", "task": "t_015_product_type_columns", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 3, "passed_tests": 3, - "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Type]\n public partial struct Address\n {\n public string Street;\n public int Zip;\n }\n\n [SpacetimeDB.Type]\n public partial struct Position\n {\n public int X;\n public int Y;\n }\n\n [SpacetimeDB.Table(Name = \"Profile\", Public = true)]\n public partial struct Profile\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public Address Home;\n public Address Work;\n public Position Pos;\n }\n\n [SpacetimeDB.Reducer]\n public static void Seed(ReducerContext ctx)\n {\n ctx.Db.Profile.Insert(new Profile\n {\n Id = 1,\n Home = new Address { Street = \"1 Main\", Zip = 11111 },\n Work = new Address { Street = \"2 Broad\", Zip = 22222 },\n Pos = new Position { X = 7, Y = 9 },\n });\n }\n}", + "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Type]\n public partial struct Address\n {\n public string Street;\n public int Zip;\n }\n\n [SpacetimeDB.Type]\n public partial struct Position\n {\n public int X;\n public int Y;\n }\n\n [SpacetimeDB.Table(Name = \"Profile\", Public = true)]\n public partial struct Profile\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public Address Home;\n public Address Work;\n public Position Pos;\n }\n\n [SpacetimeDB.Reducer]\n public static void Seed(ReducerContext ctx)\n {\n ctx.Db.Profile.Insert(new Profile\n {\n Id = 1,\n Home = new Address { Street = \"1 Main\", Zip = 11111 },\n Work = new Address { Street = \"2 Broad\", Zip = 22222 },\n Pos = new Position { X = 7, Y = 9 }\n });\n }\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-015-product-type-columns-golden", @@ -3009,19 +3022,6 @@ "work_dir_golden": "target/llm-runs/schema/t_015_product_type_columns/csharp/server/golden", "work_dir_llm": "target/llm-runs/schema/t_015_product_type_columns/csharp/server/gpt-5/llm", "scorer_details": { - "schema_parity": { - "pass": true, - "partial": 1.0, - "notes": { - "golden_db": "schema-t-015-product-type-columns-golden", - "llm_db": "schema-t-015-product-type-columns-gpt-5-llm", - "reducers_diff": null, - "reducers_equal": true, - "server": "http://127.0.0.1:34777", - "tables_diff": null, - "tables_equal": true - } - }, "product_type_columns_row_count": { "pass": true, "partial": 1.0, @@ -3042,73 +3042,65 @@ "llm_out": "Id | Home | Work | Pos ----+----------------------------------+-----------------------------------+---------------- 1 | (Street = \"1 Main\", Zip = 11111) | (Street = \"2 Broad\", Zip = 22222) | (X = 7, Y = 9)", "query": "SELECT Id, Home, Work, Pos FROM Profile WHERE Id=1", "reducer": "Seed", - "server": "http://127.0.0.1:34777" + "server": "http://127.0.0.1:36809" + } + }, + "schema_parity": { + "pass": true, + "partial": 1.0, + "notes": { + "golden_db": "schema-t-015-product-type-columns-golden", + "llm_db": "schema-t-015-product-type-columns-gpt-5-llm", + "reducers_diff": null, + "reducers_equal": true, + "server": "http://127.0.0.1:36809", + "tables_diff": null, + "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-22T01:31:31.760412544Z", - "finished_at": "2026-01-22T01:32:02.704170365Z" + "started_at": "2026-01-22T23:42:38.711206573Z", + "finished_at": "2026-01-22T23:43:02.977346650Z" }, "t_016_sum_type_columns": { - "hash": "2f624c48d5e0cb6f1b8f45bf686d0fdd92b425ca4e5633757ad2bccaed4abbec", + "hash": "a96f7f53940945bb938bc8ee4e931c2c2a88c31a961aab94f52f7d5b336acd88", "task": "t_016_sum_type_columns", "lang": "csharp", - "golden_published": true, + "golden_published": false, "model_name": "GPT-5", - "total_tests": 3, - "passed_tests": 1, - "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Type]\n public partial struct Circle\n {\n public int Radius;\n }\n\n [SpacetimeDB.Type]\n public partial struct Rectangle\n {\n public int Width;\n public int Height;\n }\n\n [SpacetimeDB.Type]\n public partial record Shape : TaggedEnum<(\n Circle Circle,\n Rectangle Rectangle\n )> { }\n\n [SpacetimeDB.Table(Name = \"Drawing\")]\n public partial struct Drawing\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public Shape A;\n public Shape B;\n }\n\n [SpacetimeDB.Reducer]\n public static void Seed(ReducerContext ctx)\n {\n ctx.Db.Drawing.Insert(new Drawing\n {\n Id = 1,\n A = new Shape.Circle(new Circle { Radius = 10 }),\n B = new Shape.Rectangle(new Rectangle { Width = 4, Height = 6 })\n });\n }\n}", + "total_tests": 1, + "passed_tests": 0, + "llm_output": null, "category": "schema", "route_api_model": "gpt-5", - "golden_db": "schema-t-016-sum-type-columns-golden", - "llm_db": "schema-t-016-sum-type-columns-gpt-5-llm", - "work_dir_golden": "target/llm-runs/schema/t_016_sum_type_columns/csharp/server/golden", - "work_dir_llm": "target/llm-runs/schema/t_016_sum_type_columns/csharp/server/gpt-5/llm", + "golden_db": null, + "llm_db": null, + "work_dir_golden": null, + "work_dir_llm": null, "scorer_details": { - "sum_type_columns_row_count": { - "pass": false, - "partial": 0.0, - "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `drawings`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:34777/v1/database/c20005d8c0c99cb953aba5ddbae1bff7f82d17c11a63facba4dfd732e878ecfb/sql)\n", - "phase": "sql" - } - }, - "schema_parity": { - "pass": true, - "partial": 1.0, - "notes": { - "golden_db": "schema-t-016-sum-type-columns-golden", - "llm_db": "schema-t-016-sum-type-columns-gpt-5-llm", - "reducers_diff": null, - "reducers_equal": true, - "server": "http://127.0.0.1:34777", - "tables_diff": null, - "tables_equal": true - } - }, - "sum_type_columns_row_parity": { + "publish_error": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `drawings`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:34777/v1/database/c2005af6a0af28ed2f81072742bd745eb59481941732ec87de837dfed95a0ae3/sql)\n", - "phase": "sql_golden" + "error": "LLM call timed out", + "phase": "build_or_publish" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:30:11.783262680Z", - "finished_at": "2026-01-22T01:31:39.644318169Z" + "started_at": "2026-01-22T23:45:28.059280139Z", + "finished_at": "2026-01-22T23:45:28.059280139Z" }, "t_017_scheduled_columns": { - "hash": "2f624c48d5e0cb6f1b8f45bf686d0fdd92b425ca4e5633757ad2bccaed4abbec", + "hash": "a96f7f53940945bb938bc8ee4e931c2c2a88c31a961aab94f52f7d5b336acd88", "task": "t_017_scheduled_columns", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 2, "passed_tests": 1, - "llm_output": "using System;\nusing SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"TickTimer\", Scheduled = nameof(Tick), ScheduledAt = nameof(ScheduledAt))]\n public partial struct TickTimer\n {\n [SpacetimeDB.PrimaryKey, SpacetimeDB.AutoInc]\n public ulong ScheduledId;\n public ScheduleAt ScheduledAt;\n }\n\n [SpacetimeDB.Reducer]\n public static void Tick(ReducerContext ctx, TickTimer _timer)\n {\n // Scheduled callback; no-op.\n }\n\n [SpacetimeDB.Reducer(ReducerKind.Init)]\n public static void Init(ReducerContext ctx)\n {\n if (ctx.Db.TickTimer.Count == 0)\n {\n ctx.Db.TickTimer.Insert(new TickTimer\n {\n ScheduledAt = new ScheduleAt.Interval(TimeSpan.FromMicroseconds(50_000))\n });\n }\n }\n}", + "llm_output": "using System;\nusing SpacetimeDB;\n\npublic static partial class Module\n{\n [Table(Name = \"TickTimer\", Scheduled = nameof(Tick), ScheduledAt = nameof(ScheduledAt))]\n public partial struct TickTimer\n {\n [PrimaryKey, AutoInc]\n public ulong ScheduledId;\n public ScheduleAt ScheduledAt;\n }\n\n [Reducer(ReducerKind.Init)]\n public static void Init(ReducerContext ctx)\n {\n ctx.Db.TickTimer.Insert(new TickTimer\n {\n ScheduledAt = new ScheduleAt.Interval(TimeSpan.FromMicroseconds(50_000))\n });\n }\n\n [Reducer]\n public static void Tick(ReducerContext ctx, TickTimer _timer)\n {\n }\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-017-scheduled-columns-golden", @@ -3120,7 +3112,7 @@ "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `tick_timer`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:34777/v1/database/c2004aea4b7bbab6aeacac5760587684dd46e8d04cbf644ce7f231d3baf5bf18/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `tick_timer`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:36809/v1/database/c2001827555321e96da32ace2975cf9aad6b2fd31a7219baf30facfb2deef7ce/sql)\n", "phase": "sql" } }, @@ -3132,18 +3124,18 @@ "llm_db": "schema-t-017-scheduled-columns-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34777", + "server": "http://127.0.0.1:36809", "tables_diff": null, "tables_equal": true } } }, "vendor": "openai", - "started_at": "2026-01-22T01:30:11.777231542Z", - "finished_at": "2026-01-22T01:31:04.575869042Z" + "started_at": "2026-01-22T23:41:08.702084489Z", + "finished_at": "2026-01-22T23:42:23.403512263Z" }, "t_018_constraints": { - "hash": "2f624c48d5e0cb6f1b8f45bf686d0fdd92b425ca4e5633757ad2bccaed4abbec", + "hash": "a96f7f53940945bb938bc8ee4e931c2c2a88c31a961aab94f52f7d5b336acd88", "task": "t_018_constraints", "lang": "csharp", "golden_published": true, @@ -3158,31 +3150,31 @@ "work_dir_golden": "target/llm-runs/schema/t_018_constraints/csharp/server/golden", "work_dir_llm": "target/llm-runs/schema/t_018_constraints/csharp/server/gpt-5/llm", "scorer_details": { - "schema_parity": { + "constraints_row_parity_after_seed": { "pass": true, "partial": 1.0, "notes": { + "args": [], "golden_db": "schema-t-018-constraints-golden", + "golden_out": "Id | Email | Name ----+-----------------+--------- 1 | \"a@example.com\" | \"Alice\"", "llm_db": "schema-t-018-constraints-gpt-5-llm", - "reducers_diff": null, - "reducers_equal": true, - "server": "http://127.0.0.1:34777", - "tables_diff": null, - "tables_equal": true + "llm_out": "Id | Email | Name ----+-----------------+--------- 1 | \"a@example.com\" | \"Alice\"", + "query": "SELECT Id, Email, Name FROM Account WHERE Id=1", + "reducer": "Seed", + "server": "http://127.0.0.1:36809" } }, - "constraints_row_parity_after_seed": { + "schema_parity": { "pass": true, "partial": 1.0, "notes": { - "args": [], "golden_db": "schema-t-018-constraints-golden", - "golden_out": "Id | Email | Name ----+-----------------+--------- 1 | \"a@example.com\" | \"Alice\"", "llm_db": "schema-t-018-constraints-gpt-5-llm", - "llm_out": "Id | Email | Name ----+-----------------+--------- 1 | \"a@example.com\" | \"Alice\"", - "query": "SELECT Id, Email, Name FROM Account WHERE Id=1", - "reducer": "Seed", - "server": "http://127.0.0.1:34777" + "reducers_diff": null, + "reducers_equal": true, + "server": "http://127.0.0.1:36809", + "tables_diff": null, + "tables_equal": true } }, "constraints_seed_two_rows": { @@ -3196,18 +3188,18 @@ } }, "vendor": "openai", - "started_at": "2026-01-22T01:30:11.774765294Z", - "finished_at": "2026-01-22T01:30:55.524141654Z" + "started_at": "2026-01-22T23:41:08.698822052Z", + "finished_at": "2026-01-22T23:41:59.253433633Z" }, "t_019_many_to_many": { - "hash": "2f624c48d5e0cb6f1b8f45bf686d0fdd92b425ca4e5633757ad2bccaed4abbec", + "hash": "a96f7f53940945bb938bc8ee4e931c2c2a88c31a961aab94f52f7d5b336acd88", "task": "t_019_many_to_many", "lang": "csharp", - "golden_published": true, + "golden_published": false, "model_name": "GPT-5", "total_tests": 5, - "passed_tests": 5, - "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"User\", Public = true)]\n public partial struct User\n {\n [SpacetimeDB.PrimaryKey]\n public int UserId;\n public string Name;\n }\n\n [SpacetimeDB.Table(Name = \"Group\", Public = true)]\n public partial struct Group\n {\n [SpacetimeDB.PrimaryKey]\n public int GroupId;\n public string Title;\n }\n\n [SpacetimeDB.Table(Name = \"Membership\", Public = true)]\n [SpacetimeDB.Index.BTree(Name = \"by_user\", Columns = new[] { \"UserId\" })]\n [SpacetimeDB.Index.BTree(Name = \"by_group\", Columns = new[] { \"GroupId\" })]\n public partial struct Membership\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public int UserId;\n public int GroupId;\n }\n\n [SpacetimeDB.Reducer]\n public static void Seed(ReducerContext ctx)\n {\n foreach (var m in ctx.Db.Membership.Iter())\n {\n ctx.Db.Membership.Id.Delete(m.Id);\n }\n foreach (var u in ctx.Db.User.Iter())\n {\n ctx.Db.User.UserId.Delete(u.UserId);\n }\n foreach (var g in ctx.Db.Group.Iter())\n {\n ctx.Db.Group.GroupId.Delete(g.GroupId);\n }\n\n ctx.Db.User.Insert(new User { UserId = 1, Name = \"Alice\" });\n ctx.Db.User.Insert(new User { UserId = 2, Name = \"Bob\" });\n\n ctx.Db.Group.Insert(new Group { GroupId = 10, Title = \"Admin\" });\n ctx.Db.Group.Insert(new Group { GroupId = 20, Title = \"Dev\" });\n\n ctx.Db.Membership.Insert(new Membership { Id = 1, UserId = 1, GroupId = 10 });\n ctx.Db.Membership.Insert(new Membership { Id = 2, UserId = 1, GroupId = 20 });\n ctx.Db.Membership.Insert(new Membership { Id = 3, UserId = 2, GroupId = 20 });\n }\n}", + "passed_tests": 0, + "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [Table(Name = \"User\")]\n public partial struct User\n {\n [PrimaryKey]\n public int UserId;\n public string Name;\n }\n\n [Table(Name = \"Group\")]\n public partial struct Group\n {\n [PrimaryKey]\n public int GroupId;\n public string Title;\n }\n\n [Table(Name = \"Membership\")]\n [Index.BTree(Name = \"by_user\", Columns = [\"UserId\"])]\n [Index.BTree(Name = \"by_group\", Columns = [\"GroupId\"])]\n public partial struct Membership\n {\n [PrimaryKey]\n public int Id;\n public int UserId;\n public int GroupId;\n }\n\n [Reducer]\n public static void Seed(ReducerContext ctx)\n {\n ctx.Db.User.Insert(new User { UserId = 1, Name = \"Alice\" });\n ctx.Db.User.Insert(new User { UserId = 2, Name = \"Bob\" });\n\n ctx.Db.Group.Insert(new Group { GroupId = 10, Title = \"Admin\" });\n ctx.Db.Group.Insert(new Group { GroupId = 20, Title = \"Dev\" });\n\n ctx.Db.Membership.Insert(new Membership { Id = 1, UserId = 1, GroupId = 10 });\n ctx.Db.Membership.Insert(new Membership { Id = 2, UserId = 1, GroupId = 20 });\n ctx.Db.Membership.Insert(new Membership { Id = 3, UserId = 2, GroupId = 20 });\n }\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-019-many-to-many-golden", @@ -3215,69 +3207,28 @@ "work_dir_golden": "target/llm-runs/schema/t_019_many_to_many/csharp/server/golden", "work_dir_llm": "target/llm-runs/schema/t_019_many_to_many/csharp/server/gpt-5/llm", "scorer_details": { - "m2m_has_1_20": { - "pass": true, - "partial": 1.0, - "notes": { - "actual": 1, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM Membership WHERE UserId=1 AND GroupId=20" - } - }, - "m2m_has_1_10": { - "pass": true, - "partial": 1.0, - "notes": { - "actual": 1, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM Membership WHERE UserId=1 AND GroupId=10" - } - }, - "schema_parity": { - "pass": true, - "partial": 1.0, - "notes": { - "golden_db": "schema-t-019-many-to-many-golden", - "llm_db": "schema-t-019-many-to-many-gpt-5-llm", - "reducers_diff": null, - "reducers_equal": true, - "server": "http://127.0.0.1:34777", - "tables_diff": null, - "tables_equal": true - } - }, - "m2m_has_2_20": { - "pass": true, - "partial": 1.0, - "notes": { - "actual": 1, - "expected": 1, - "sql": "SELECT COUNT(*) AS n FROM Membership WHERE UserId=2 AND GroupId=20" - } - }, - "memberships_three_rows": { - "pass": true, - "partial": 1.0, + "publish_error": { + "pass": false, + "partial": 0.0, "notes": { - "actual": 3, - "expected": 3, - "sql": "SELECT COUNT(*) AS n FROM Membership" + "error": "spacetime build (csharp) failed (exit=1)\n--- stderr ---\nError: command [\"dotnet\", \"publish\", \"-c\", \"Release\", \"-v\", \"quiet\"] exited with code 1\n\n--- stdout ---\n/__w/SpacetimeDB/SpacetimeDB/target/llm-runs/schema/t_019_many_to_many/csharp/server/gpt-5/llm/Lib.cs(23,6): error CS0104: 'Index' is an ambiguous reference between 'SpacetimeDB.Index' and 'System.Index' [/__w/SpacetimeDB/SpacetimeDB/target/llm-runs/schema/t_019_many_to_many/csharp/server/gpt-5/llm/StdbModule.csproj]\n/__w/SpacetimeDB/SpacetimeDB/target/llm-runs/schema/t_019_many_to_many/csharp/server/gpt-5/llm/Lib.cs(24,6): error CS0104: 'Index' is an ambiguous reference between 'SpacetimeDB.Index' and 'System.Index' [/__w/SpacetimeDB/SpacetimeDB/target/llm-runs/schema/t_019_many_to_many/csharp/server/gpt-5/llm/StdbModule.csproj]\n/__w/SpacetimeDB/SpacetimeDB/target/llm-runs/schema/t_019_many_to_many/csharp/server/gpt-5/llm/Lib.cs(23,46): error CS9176: There is no target type for the collection expression. [/__w/SpacetimeDB/SpacetimeDB/target/llm-runs/schema/t_019_many_to_many/csharp/server/gpt-5/llm/StdbModule.csproj]\n/__w/SpacetimeDB/SpacetimeDB/target/llm-runs/schema/t_019_many_to_many/csharp/server/gpt-5/llm/Lib.cs(24,47): error CS9176: There is no target type for the collection expression. [/__w/SpacetimeDB/SpacetimeDB/target/llm-runs/schema/t_019_many_to_many/csharp/server/gpt-5/llm/StdbModule.csproj]\n", + "phase": "build_or_publish" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:31:02.980050608Z", - "finished_at": "2026-01-22T01:31:51.511790368Z" + "started_at": "2026-01-22T23:42:23.403555155Z", + "finished_at": "2026-01-22T23:42:53.836498531Z" }, "t_020_ecs": { - "hash": "2f624c48d5e0cb6f1b8f45bf686d0fdd92b425ca4e5633757ad2bccaed4abbec", + "hash": "a96f7f53940945bb938bc8ee4e931c2c2a88c31a961aab94f52f7d5b336acd88", "task": "t_020_ecs", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 5, "passed_tests": 1, - "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"Entity\", Public = true)]\n public partial struct Entity\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n }\n\n [SpacetimeDB.Table(Name = \"Position\", Public = true)]\n public partial struct Position\n {\n [SpacetimeDB.PrimaryKey]\n public int EntityId;\n public int X;\n public int Y;\n }\n\n [SpacetimeDB.Table(Name = \"Velocity\", Public = true)]\n public partial struct Velocity\n {\n [SpacetimeDB.PrimaryKey]\n public int EntityId;\n public int VX;\n public int VY;\n }\n\n [SpacetimeDB.Table(Name = \"NextPosition\", Public = true)]\n public partial struct NextPosition\n {\n [SpacetimeDB.PrimaryKey]\n public int EntityId;\n public int X;\n public int Y;\n }\n\n [SpacetimeDB.Reducer]\n public static void Seed(ReducerContext ctx)\n {\n ctx.Db.Entity.Insert(new Entity { Id = 1 });\n ctx.Db.Position.Insert(new Position { EntityId = 1, X = 0, Y = 0 });\n ctx.Db.Velocity.Insert(new Velocity { EntityId = 1, VX = 1, VY = 0 });\n\n ctx.Db.Entity.Insert(new Entity { Id = 2 });\n ctx.Db.Position.Insert(new Position { EntityId = 2, X = 10, Y = 0 });\n ctx.Db.Velocity.Insert(new Velocity { EntityId = 2, VX = -2, VY = 3 });\n }\n\n [SpacetimeDB.Reducer]\n public static void Step(ReducerContext ctx)\n {\n foreach (var pos in ctx.Db.Position.Iter())\n {\n if (ctx.Db.Velocity.EntityId.Find(pos.EntityId) is Velocity vel)\n {\n int nx = pos.X + vel.VX;\n int ny = pos.Y + vel.VY;\n\n if (ctx.Db.NextPosition.EntityId.Find(pos.EntityId) is NextPosition np)\n {\n np.X = nx;\n np.Y = ny;\n ctx.Db.NextPosition.EntityId.Update(np);\n }\n else\n {\n ctx.Db.NextPosition.Insert(new NextPosition\n {\n EntityId = pos.EntityId,\n X = nx,\n Y = ny\n });\n }\n }\n }\n }\n}", + "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"Entity\")]\n public partial struct Entity\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n }\n\n [SpacetimeDB.Table(Name = \"Position\")]\n public partial struct Position\n {\n [SpacetimeDB.PrimaryKey]\n public int EntityId;\n public int X;\n public int Y;\n }\n\n [SpacetimeDB.Table(Name = \"Velocity\")]\n public partial struct Velocity\n {\n [SpacetimeDB.PrimaryKey]\n public int EntityId;\n public int VX;\n public int VY;\n }\n\n [SpacetimeDB.Table(Name = \"NextPosition\")]\n public partial struct NextPosition\n {\n [SpacetimeDB.PrimaryKey]\n public int EntityId;\n public int X;\n public int Y;\n }\n\n [SpacetimeDB.Reducer]\n public static void Seed(ReducerContext ctx)\n {\n UpsertEntity(ctx, 1);\n UpsertPosition(ctx, 1, 0, 0);\n UpsertVelocity(ctx, 1, 1, 0);\n\n UpsertEntity(ctx, 2);\n UpsertPosition(ctx, 2, 10, 0);\n UpsertVelocity(ctx, 2, -2, 3);\n }\n\n [SpacetimeDB.Reducer]\n public static void Step(ReducerContext ctx)\n {\n foreach (var pos in ctx.Db.Position.Iter())\n {\n if (ctx.Db.Velocity.EntityId.Find(pos.EntityId) is not Velocity vel)\n {\n continue;\n }\n\n var nx = pos.X + vel.VX;\n var ny = pos.Y + vel.VY;\n\n if (ctx.Db.NextPosition.EntityId.Find(pos.EntityId) is NextPosition np)\n {\n np.X = nx;\n np.Y = ny;\n ctx.Db.NextPosition.EntityId.Update(np);\n }\n else\n {\n ctx.Db.NextPosition.Insert(new NextPosition { EntityId = pos.EntityId, X = nx, Y = ny });\n }\n }\n }\n\n private static void UpsertEntity(ReducerContext ctx, int id)\n {\n if (ctx.Db.Entity.Id.Find(id) is Entity)\n {\n // nothing to update\n }\n else\n {\n ctx.Db.Entity.Insert(new Entity { Id = id });\n }\n }\n\n private static void UpsertPosition(ReducerContext ctx, int entityId, int x, int y)\n {\n if (ctx.Db.Position.EntityId.Find(entityId) is Position p)\n {\n p.X = x;\n p.Y = y;\n ctx.Db.Position.EntityId.Update(p);\n }\n else\n {\n ctx.Db.Position.Insert(new Position { EntityId = entityId, X = x, Y = y });\n }\n }\n\n private static void UpsertVelocity(ReducerContext ctx, int entityId, int vx, int vy)\n {\n if (ctx.Db.Velocity.EntityId.Find(entityId) is Velocity v)\n {\n v.VX = vx;\n v.VY = vy;\n ctx.Db.Velocity.EntityId.Update(v);\n }\n else\n {\n ctx.Db.Velocity.Insert(new Velocity { EntityId = entityId, VX = vx, VY = vy });\n }\n }\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-020-ecs-golden", @@ -3285,24 +3236,11 @@ "work_dir_golden": "target/llm-runs/schema/t_020_ecs/csharp/server/golden", "work_dir_llm": "target/llm-runs/schema/t_020_ecs/csharp/server/gpt-5/llm", "scorer_details": { - "schema_parity": { - "pass": true, - "partial": 1.0, - "notes": { - "golden_db": "schema-t-020-ecs-golden", - "llm_db": "schema-t-020-ecs-gpt-5-llm", - "reducers_diff": null, - "reducers_equal": true, - "server": "http://127.0.0.1:34777", - "tables_diff": null, - "tables_equal": true - } - }, - "ecs_step_next_positions_count": { + "ecs_next_pos_entity2": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:34777/v1/database/c2003b8e2e7c564713a29450a9807a75eaf4a147b360d2bfda3da972ad68344f/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:36809/v1/database/c200a443d8b518142187c1957c53d075bc8a200e7048cfdf40e7f1369034bb7f/sql)\n", "phase": "sql" } }, @@ -3310,40 +3248,53 @@ "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:34777/v1/database/c2003b8e2e7c564713a29450a9807a75eaf4a147b360d2bfda3da972ad68344f/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:36809/v1/database/c200a443d8b518142187c1957c53d075bc8a200e7048cfdf40e7f1369034bb7f/sql)\n", "phase": "sql" } }, - "ecs_next_pos_entity2": { + "ecs_next_pos_entity1": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:34777/v1/database/c2003b8e2e7c564713a29450a9807a75eaf4a147b360d2bfda3da972ad68344f/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:36809/v1/database/c200a443d8b518142187c1957c53d075bc8a200e7048cfdf40e7f1369034bb7f/sql)\n", "phase": "sql" } }, - "ecs_next_pos_entity1": { + "schema_parity": { + "pass": true, + "partial": 1.0, + "notes": { + "golden_db": "schema-t-020-ecs-golden", + "llm_db": "schema-t-020-ecs-gpt-5-llm", + "reducers_diff": null, + "reducers_equal": true, + "server": "http://127.0.0.1:36809", + "tables_diff": null, + "tables_equal": true + } + }, + "ecs_step_next_positions_count": { "pass": false, "partial": 0.0, "notes": { - "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:34777/v1/database/c2003b8e2e7c564713a29450a9807a75eaf4a147b360d2bfda3da972ad68344f/sql)\n", + "error": "spacetime sql failed:\nWARNING: This command is UNSTABLE and subject to breaking changes.\n\nError: no such table: `next_positions`. If the table exists, it may be marked private.\n\nCaused by:\n HTTP status client error (400 Bad Request) for url (http://127.0.0.1:36809/v1/database/c200a443d8b518142187c1957c53d075bc8a200e7048cfdf40e7f1369034bb7f/sql)\n", "phase": "sql" } } }, "vendor": "openai", - "started_at": "2026-01-22T01:31:04.575899063Z", - "finished_at": "2026-01-22T01:32:21.843416027Z" + "started_at": "2026-01-22T23:42:29.625430290Z", + "finished_at": "2026-01-22T23:43:26.976806916Z" }, "t_021_multi_column_index": { - "hash": "2f624c48d5e0cb6f1b8f45bf686d0fdd92b425ca4e5633757ad2bccaed4abbec", + "hash": "a96f7f53940945bb938bc8ee4e931c2c2a88c31a961aab94f52f7d5b336acd88", "task": "t_021_multi_column_index", "lang": "csharp", "golden_published": true, "model_name": "GPT-5", "total_tests": 4, "passed_tests": 4, - "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"Log\", Public = true)]\n [SpacetimeDB.Index.BTree(Name = \"by_user_day\", Columns = new[] { \"UserId\", \"Day\" })]\n public partial struct Log\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public int UserId;\n public int Day;\n public string Message;\n }\n\n [SpacetimeDB.Reducer]\n public static void Seed(ReducerContext ctx)\n {\n foreach (var row in ctx.Db.Log.Iter())\n {\n ctx.Db.Log.Id.Delete(row.Id);\n }\n\n ctx.Db.Log.Insert(new Log { Id = 1, UserId = 7, Day = 1, Message = \"a\" });\n ctx.Db.Log.Insert(new Log { Id = 2, UserId = 7, Day = 2, Message = \"b\" });\n ctx.Db.Log.Insert(new Log { Id = 3, UserId = 9, Day = 1, Message = \"c\" });\n }\n}", + "llm_output": "using SpacetimeDB;\n\npublic static partial class Module\n{\n [SpacetimeDB.Table(Name = \"Log\")]\n [SpacetimeDB.Index.BTree(Name = \"by_user_day\", Columns = [\"UserId\", \"Day\"])]\n public partial struct Log\n {\n [SpacetimeDB.PrimaryKey]\n public int Id;\n public int UserId;\n public int Day;\n public string Message;\n }\n\n [SpacetimeDB.Reducer]\n public static void Seed(ReducerContext ctx)\n {\n ctx.Db.Log.Insert(new Log { Id = 1, UserId = 7, Day = 1, Message = \"a\" });\n ctx.Db.Log.Insert(new Log { Id = 2, UserId = 7, Day = 2, Message = \"b\" });\n ctx.Db.Log.Insert(new Log { Id = 3, UserId = 9, Day = 1, Message = \"c\" });\n }\n}", "category": "schema", "route_api_model": "gpt-5", "golden_db": "schema-t-021-multi-column-index-golden", @@ -3359,7 +3310,7 @@ "llm_db": "schema-t-021-multi-column-index-gpt-5-llm", "reducers_diff": null, "reducers_equal": true, - "server": "http://127.0.0.1:34777", + "server": "http://127.0.0.1:36809", "tables_diff": null, "tables_equal": true } @@ -3393,8 +3344,8 @@ } }, "vendor": "openai", - "started_at": "2026-01-22T01:30:11.780162903Z", - "finished_at": "2026-01-22T01:31:02.980017871Z" + "started_at": "2026-01-22T23:41:08.705662164Z", + "finished_at": "2026-01-22T23:42:29.625385403Z" } } } diff --git a/docs/llms/docs-benchmark-summary.json b/docs/llms/docs-benchmark-summary.json index 3df4a2fff0e..14fd56cfa45 100644 --- a/docs/llms/docs-benchmark-summary.json +++ b/docs/llms/docs-benchmark-summary.json @@ -1,11 +1,11 @@ { "version": 1, - "generated_at": "2026-01-22T01:34:07.820Z", + "generated_at": "2026-01-22T23:45:28.101Z", "by_language": { "csharp": { "modes": { "docs": { - "hash": "2f624c48d5e0cb6f1b8f45bf686d0fdd92b425ca4e5633757ad2bccaed4abbec", + "hash": "a96f7f53940945bb938bc8ee4e931c2c2a88c31a961aab94f52f7d5b336acd88", "models": { "GPT-5": { "categories": { @@ -19,20 +19,20 @@ }, "schema": { "tasks": 10, - "total_tests": 34, - "passed_tests": 22, - "pass_pct": 64.70588, - "task_pass_equiv": 6.366667, - "task_pass_pct": 63.666664 + "total_tests": 32, + "passed_tests": 19, + "pass_pct": 59.375, + "task_pass_equiv": 6.033333, + "task_pass_pct": 60.33333 } }, "totals": { "tasks": 22, - "total_tests": 61, - "passed_tests": 49, - "pass_pct": 80.327866, - "task_pass_equiv": 18.366667, - "task_pass_pct": 83.48485 + "total_tests": 59, + "passed_tests": 46, + "pass_pct": 77.9661, + "task_pass_equiv": 18.033333, + "task_pass_pct": 81.969696 } } } @@ -42,7 +42,7 @@ "rust": { "modes": { "docs": { - "hash": "02bd25d6ae3ac904057e3a8034595f2c28f652956bc1b2b2a1488d169b871b9e", + "hash": "8c8baf7fa35a3b4cdccd079c8efdf96434bd3a22031b7840912b543226efb58b", "models": { "GPT-5": { "categories": { @@ -56,20 +56,20 @@ }, "schema": { "tasks": 10, - "total_tests": 29, - "passed_tests": 7, - "pass_pct": 24.137932, - "task_pass_equiv": 1.55, - "task_pass_pct": 15.5 + "total_tests": 30, + "passed_tests": 8, + "pass_pct": 26.666666, + "task_pass_equiv": 2.05, + "task_pass_pct": 20.5 } }, "totals": { "tasks": 22, - "total_tests": 56, - "passed_tests": 12, - "pass_pct": 21.428572, - "task_pass_equiv": 2.8833334, - "task_pass_pct": 13.106062 + "total_tests": 57, + "passed_tests": 13, + "pass_pct": 22.807018, + "task_pass_equiv": 3.3833334, + "task_pass_pct": 15.378788 } } } @@ -82,27 +82,27 @@ "basics": { "tasks": 12, "total_tests": 27, - "passed_tests": 25, - "pass_pct": 92.59259, - "task_pass_equiv": 10.0, - "task_pass_pct": 83.33333 + "passed_tests": 26, + "pass_pct": 96.296295, + "task_pass_equiv": 11.0, + "task_pass_pct": 91.66667 }, "schema": { "tasks": 10, "total_tests": 34, - "passed_tests": 23, - "pass_pct": 67.64706, - "task_pass_equiv": 6.5333333, - "task_pass_pct": 65.33333 + "passed_tests": 21, + "pass_pct": 61.764706, + "task_pass_equiv": 5.866667, + "task_pass_pct": 58.66667 } }, "totals": { "tasks": 22, "total_tests": 61, - "passed_tests": 48, - "pass_pct": 78.68852, - "task_pass_equiv": 16.533333, - "task_pass_pct": 75.15151 + "passed_tests": 47, + "pass_pct": 77.04918, + "task_pass_equiv": 16.866667, + "task_pass_pct": 76.666664 } } } diff --git a/docs/test-csharp-snippets/Module.cs b/docs/test-csharp-snippets/Module.cs new file mode 100644 index 00000000000..5e425eccd94 --- /dev/null +++ b/docs/test-csharp-snippets/Module.cs @@ -0,0 +1,357 @@ +#pragma warning disable STDB_UNSTABLE +using SpacetimeDB; +using System.Text.Json; + +public static partial class Module +{ + // === Snippet 1: Defining Procedures === + [SpacetimeDB.Procedure] + public static ulong AddTwoNumbers(ProcedureContext ctx, uint lhs, uint rhs) + { + return (ulong)lhs + (ulong)rhs; + } + + // === Snippet 2: Accessing the database === + [SpacetimeDB.Table(Name = "MyTable")] + public partial struct MyTable + { + public uint A; + public string B; + } + + [SpacetimeDB.Procedure] + public static void InsertAValue(ProcedureContext ctx, uint a, string b) + { + ctx.WithTx(txCtx => + { + txCtx.Db.MyTable.Insert(new MyTable { A = a, B = b }); + return 0; + }); + } + + // === Snippet 3: Fallible database operations === + [SpacetimeDB.Procedure] + public static void MaybeInsertAValue(ProcedureContext ctx, uint a, string b) + { + ctx.WithTx(txCtx => + { + if (a < 10) + { + throw new Exception("a is less than 10!"); + } + txCtx.Db.MyTable.Insert(new MyTable { A = a, B = b }); + return 0; + }); + } + + // === Snippet 4: Reading values out of the database === + [SpacetimeDB.Table(Name = "Player")] + public partial struct Player + { + public Identity Id; + public uint Level; + } + + [SpacetimeDB.Procedure] + public static void FindHighestLevelPlayer(ProcedureContext ctx) + { + var highestLevelPlayer = ctx.WithTx(txCtx => + { + Player? highest = null; + foreach (var player in txCtx.Db.Player.Iter()) + { + if (highest == null || player.Level > highest.Value.Level) + { + highest = player; + } + } + return highest; + }); + + if (highestLevelPlayer.HasValue) + { + Log.Info($"Congratulations to {highestLevelPlayer.Value.Id}"); + } + else + { + Log.Warn("No players..."); + } + } + + // === Snippet 5: HTTP Requests - Get === + [SpacetimeDB.Procedure] + public static void GetRequest(ProcedureContext ctx) + { + var result = ctx.Http.Get("https://example.invalid"); + switch (result) + { + case Result.OkR(var response): + var body = response.Body.ToStringUtf8Lossy(); + Log.Info($"Got response with status {response.StatusCode} and body {body}"); + break; + case Result.ErrR(var e): + Log.Error($"Request failed: {e.Message}"); + break; + } + } + + // === Snippet 6: HTTP Requests - Send === + [SpacetimeDB.Procedure] + public static void PostRequest(ProcedureContext ctx) + { + var request = new HttpRequest + { + Method = SpacetimeDB.HttpMethod.Post, + Uri = "https://example.invalid/upload", + Headers = new List + { + new HttpHeader("Content-Type", "text/plain") + }, + Body = HttpBody.FromString("This is the body of the HTTP request") + }; + var result = ctx.Http.Send(request); + switch (result) + { + case Result.OkR(var response): + var body = response.Body.ToStringUtf8Lossy(); + Log.Info($"Got response with status {response.StatusCode} and body {body}"); + break; + case Result.ErrR(var e): + Log.Error($"Request failed: {e.Message}"); + break; + } + } + + // === Snippet 7: Calling Reducers from Procedures === + // Note: In C#, you can define helper methods that work with the transaction context + // rather than calling reducers directly. + private static void ProcessItemLogic(ulong itemId) + { + // ... item processing logic + } + + [SpacetimeDB.Procedure] + public static void FetchAndProcess(ProcedureContext ctx, string url) + { + // Fetch external data + var result = ctx.Http.Get(url); + var response = result.UnwrapOrThrow(); + var body = response.Body.ToStringUtf8Lossy(); + var itemId = ParseId(body); + + // Process within a transaction + ctx.WithTx(txCtx => + { + ProcessItemLogic(itemId); + return 0; + }); + } + + private static ulong ParseId(string body) + { + // Parse the ID from the response body + return ulong.Parse(body); + } + + // === Snippet 8: External AI API example === + [SpacetimeDB.Table(Name = "AiMessage", Public = true)] + public partial struct AiMessage + { + public Identity User; + public string Prompt; + public string Response; + public Timestamp CreatedAt; + } + + [SpacetimeDB.Procedure] + public static string AskAi(ProcedureContext ctx, string prompt, string apiKey) + { + // Build the request to OpenAI's API + var requestBody = JsonSerializer.Serialize(new + { + model = "gpt-4", + messages = new[] { new { role = "user", content = prompt } } + }); + + var request = new HttpRequest + { + Method = SpacetimeDB.HttpMethod.Post, + Uri = "https://api.openai.com/v1/chat/completions", + Headers = new List + { + new HttpHeader("Content-Type", "application/json"), + new HttpHeader("Authorization", $"Bearer {apiKey}") + }, + Body = HttpBody.FromString(requestBody) + }; + + // Make the HTTP request + var response = ctx.Http.Send(request).UnwrapOrThrow(); + + if (response.StatusCode != 200) + { + throw new Exception($"API returned status {response.StatusCode}"); + } + + var bodyStr = response.Body.ToStringUtf8Lossy(); + + // Parse the response + var aiResponse = ExtractContent(bodyStr) + ?? throw new Exception("Failed to parse AI response"); + + // Store the conversation in the database + ctx.WithTx(txCtx => + { + txCtx.Db.AiMessage.Insert(new AiMessage + { + User = txCtx.Sender, + Prompt = prompt, + Response = aiResponse, + CreatedAt = txCtx.Timestamp + }); + return 0; + }); + + return aiResponse; + } + + private static string? ExtractContent(string json) + { + // Simple extraction - in production, use proper JSON parsing + var doc = JsonDocument.Parse(json); + return doc.RootElement + .GetProperty("choices")[0] + .GetProperty("message") + .GetProperty("content") + .GetString(); + } + + // === Snippet 9: File Storage - S3 Upload === + [SpacetimeDB.Table(Name = "Document", Public = true)] + public partial struct Document + { + [SpacetimeDB.PrimaryKey] + [SpacetimeDB.AutoInc] + public ulong Id; + public Identity OwnerId; + public string Filename; + public string S3Key; + public Timestamp UploadedAt; + } + + // Upload file to S3 and register in database + [SpacetimeDB.Procedure] + public static string UploadToS3( + ProcedureContext ctx, + string filename, + string contentType, + List data, + string s3Bucket, + string s3Region) + { + // Generate a unique S3 key + var timestamp = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds(); + var s3Key = $"uploads/{timestamp}-{filename}"; + var url = $"https://{s3Bucket}.s3.{s3Region}.amazonaws.com/{s3Key}"; + + // Build the S3 PUT request (simplified - add AWS4 signature in production) + var request = new HttpRequest + { + Uri = url, + Method = SpacetimeDB.HttpMethod.Put, + Headers = new List + { + new HttpHeader("Content-Type", contentType), + new HttpHeader("x-amz-content-sha256", "UNSIGNED-PAYLOAD"), + // Add Authorization header with AWS4 signature + }, + Body = new HttpBody(data.ToArray()), + }; + + // Upload to S3 + var response = ctx.Http.Send(request).UnwrapOrThrow(); + + if (response.StatusCode != 200) + { + throw new Exception($"S3 upload failed with status: {response.StatusCode}"); + } + + // Store metadata in database + ctx.WithTx(txCtx => + { + txCtx.Db.Document.Insert(new Document + { + Id = 0, + OwnerId = txCtx.Sender, + Filename = filename, + S3Key = s3Key, + UploadedAt = txCtx.Timestamp, + }); + return 0; + }); + + return s3Key; + } + + // === Snippet 10: Pre-signed URL Flow === + [SpacetimeDB.Type] + public partial struct UploadInfo + { + public string UploadUrl; + public string S3Key; + } + + // Procedure returns a pre-signed URL for client-side upload + [SpacetimeDB.Procedure] + public static UploadInfo GetUploadUrl( + ProcedureContext ctx, + string filename, + string contentType) + { + var timestamp = DateTimeOffset.UtcNow.ToUnixTimeMilliseconds(); + var s3Key = $"uploads/{timestamp}-{filename}"; + + // Generate pre-signed URL (requires AWS credentials and signing logic) + var uploadUrl = GeneratePresignedUrl(s3Key, contentType); + + return new UploadInfo { UploadUrl = uploadUrl, S3Key = s3Key }; + } + + // Client uploads directly to S3 using the pre-signed URL, then calls: + [SpacetimeDB.Reducer] + public static void ConfirmUpload(ReducerContext ctx, string filename, string s3Key) + { + ctx.Db.Document.Insert(new Document + { + Id = 0, + OwnerId = ctx.Sender, + Filename = filename, + S3Key = s3Key, + UploadedAt = ctx.Timestamp, + }); + } + + private static string GeneratePresignedUrl(string s3Key, string contentType) + { + // Implement AWS S3 pre-signed URL generation + throw new NotImplementedException(); + } + + // === Snippet 11: Schedule Tables === + [SpacetimeDB.Table(Scheduled = "SendReminder", ScheduledAt = "ScheduleAt")] + public partial struct Reminder + { + [SpacetimeDB.PrimaryKey] + [SpacetimeDB.AutoInc] + public ulong Id; + public uint UserId; + public string Message; + public ScheduleAt ScheduleAt; + } + + [SpacetimeDB.Reducer] + public static void SendReminder(ReducerContext ctx, Reminder reminder) + { + // Process the scheduled reminder + } +} diff --git a/docs/test-csharp-snippets/TestProcedures.csproj b/docs/test-csharp-snippets/TestProcedures.csproj new file mode 100644 index 00000000000..66483de944c --- /dev/null +++ b/docs/test-csharp-snippets/TestProcedures.csproj @@ -0,0 +1,14 @@ + + + net8.0 + enable + enable + true + $(BaseIntermediateOutputPath)\GeneratedFiles + + + + + + + diff --git a/tools/xtask-llm-benchmark/src/bench/results_merge.rs b/tools/xtask-llm-benchmark/src/bench/results_merge.rs index 89140ae7ab2..b6f8df78295 100644 --- a/tools/xtask-llm-benchmark/src/bench/results_merge.rs +++ b/tools/xtask-llm-benchmark/src/bench/results_merge.rs @@ -97,8 +97,12 @@ pub fn merge_task_runs(path: &Path, mode: &str, runs: &[RunOutcome]) -> Result<( // Always replace with the latest value (even if None) model_v.route_api_model = r.route_api_model.clone(); + // Sanitize volatile fields before saving to reduce git diff noise + let mut sanitized = r.clone(); + sanitized.sanitize_for_commit(); + // Always overwrite the task result - model_v.tasks.insert(r.task.clone(), r.clone()); + model_v.tasks.insert(r.task.clone(), sanitized); } save_atomic(path, &root) diff --git a/tools/xtask-llm-benchmark/src/bench/types.rs b/tools/xtask-llm-benchmark/src/bench/types.rs index 8b36bc4db6d..66c564feaee 100644 --- a/tools/xtask-llm-benchmark/src/bench/types.rs +++ b/tools/xtask-llm-benchmark/src/bench/types.rs @@ -28,22 +28,58 @@ pub struct RunOutcome { pub total_tests: u32, pub passed_tests: u32, + #[serde(skip_serializing_if = "Option::is_none")] pub llm_output: Option, pub category: Option, pub route_api_model: Option, + #[serde(skip_serializing_if = "Option::is_none")] pub golden_db: Option, + #[serde(skip_serializing_if = "Option::is_none")] pub llm_db: Option, + #[serde(skip_serializing_if = "Option::is_none")] pub work_dir_golden: Option, + #[serde(skip_serializing_if = "Option::is_none")] pub work_dir_llm: Option, pub scorer_details: Option>, #[serde(default)] pub vendor: String, + #[serde(skip_serializing_if = "Option::is_none")] pub started_at: Option>, + #[serde(skip_serializing_if = "Option::is_none")] pub finished_at: Option>, } +impl RunOutcome { + /// Strip volatile fields that change every run (timestamps, ports, paths) + /// to reduce git diff noise when committing results. + pub fn sanitize_for_commit(&mut self) { + self.started_at = None; + self.finished_at = None; + self.work_dir_golden = None; + self.work_dir_llm = None; + self.golden_db = None; + self.llm_db = None; + + // Clear llm_output for passing tests (keep for failures for debugging) + if self.passed_tests == self.total_tests && self.total_tests > 0 { + self.llm_output = None; + } + + // Strip volatile fields from scorer_details notes + if let Some(ref mut details) = self.scorer_details { + for score in details.values_mut() { + if let Some(obj) = score.notes.as_object_mut() { + obj.remove("server"); + obj.remove("golden_db"); + obj.remove("llm_db"); + } + } + } + } +} + pub struct TaskPaths { pub root: PathBuf, pub answers_csharp: PathBuf, diff --git a/tools/xtask-llm-benchmark/src/benchmarks/schema/t_016_sum_type_columns/spec.rs b/tools/xtask-llm-benchmark/src/benchmarks/schema/t_016_sum_type_columns/spec.rs index f67e5c70e61..6b335621c0a 100644 --- a/tools/xtask-llm-benchmark/src/benchmarks/schema/t_016_sum_type_columns/spec.rs +++ b/tools/xtask-llm-benchmark/src/benchmarks/schema/t_016_sum_type_columns/spec.rs @@ -13,7 +13,7 @@ pub fn spec() -> BenchmarkSpec { let sb = SqlBuilder::new(casing); let reducer = ident("Seed", casing); - let select = sb.select_by_id("drawings", &["id","a","b"], "id", 1); + let select = sb.select_by_id("drawing", &["id","a","b"], "id", 1); v.push(make_reducer_data_parity_scorer(host_url, ReducerDataParityConfig { src_file: file!(), @@ -26,7 +26,7 @@ pub fn spec() -> BenchmarkSpec { timeout: Duration::from_secs(10), })); - let count = sb.count_by_id("drawings", "id", 1); + let count = sb.count_by_id("drawing", "id", 1); v.push(make_sql_count_only_scorer( host_url, file!(), diff --git a/tools/xtask-llm-benchmark/src/bin/llm_benchmark.rs b/tools/xtask-llm-benchmark/src/bin/llm_benchmark.rs index 3ea705184fe..43baa0b9fd4 100644 --- a/tools/xtask-llm-benchmark/src/bin/llm_benchmark.rs +++ b/tools/xtask-llm-benchmark/src/bin/llm_benchmark.rs @@ -462,6 +462,23 @@ fn cmd_ci_quickfix() -> Result<()> { }; run_benchmarks(csharp_args, &details_path, &summary_path)?; + // Run TypeScript benchmarks with docs mode + let typescript_args = RunArgs { + modes: Some(vec!["docs".to_string()]), + lang: Lang::TypeScript, + hash_only: false, + goldens_only: false, + force: true, + categories: None, + tasks: None, + providers: Some(vec![VendorArg(Vendor::OpenAi)]), + models: Some(vec![ModelGroup { + vendor: Vendor::OpenAi, + models: vec!["gpt-5".to_string()], + }]), + }; + run_benchmarks(typescript_args, &details_path, &summary_path)?; + println!("CI quickfix complete. Results written to:"); println!(" Details: {}", details_path.display()); println!(" Summary: {}", summary_path.display()); @@ -551,6 +568,12 @@ fn generate_comment_markdown(summary: &Summary, baseline: Option<&Summary>) -> S .get("csharp") .and_then(|l| l.modes.get("docs")) .and_then(|m| m.models.get("GPT-5")); + // TypeScript with docs mode + let typescript_results = summary + .by_language + .get("typescript") + .and_then(|l| l.modes.get("docs")) + .and_then(|m| m.models.get("GPT-5")); let rust_rustdoc_baseline = baseline .and_then(|b| b.by_language.get("rust")) @@ -564,6 +587,10 @@ fn generate_comment_markdown(summary: &Summary, baseline: Option<&Summary>) -> S .and_then(|b| b.by_language.get("csharp")) .and_then(|l| l.modes.get("docs")) .and_then(|m| m.models.get("GPT-5")); + let typescript_baseline = baseline + .and_then(|b| b.by_language.get("typescript")) + .and_then(|l| l.modes.get("docs")) + .and_then(|m| m.models.get("GPT-5")); fn format_pct(val: f32) -> String { format!("{:.1}%", val) @@ -707,6 +734,45 @@ fn generate_comment_markdown(summary: &Summary, baseline: Option<&Summary>) -> S )); } + // TypeScript with docs mode + if let Some(results) = typescript_results { + let base_cats = typescript_baseline.map(|b| &b.categories); + + if let Some(c) = results.categories.get("basics") { + let b = base_cats.and_then(|cats| cats.get("basics")); + let diff = format_diff(c.task_pass_pct, b.map(|x| x.task_pass_pct)); + md.push_str(&format!( + "| TypeScript | docs | basics | {}/{} | {}{} |\n", + c.passed_tests, + c.total_tests, + format_pct(c.task_pass_pct), + diff + )); + } + if let Some(c) = results.categories.get("schema") { + let b = base_cats.and_then(|cats| cats.get("schema")); + let diff = format_diff(c.task_pass_pct, b.map(|x| x.task_pass_pct)); + md.push_str(&format!( + "| TypeScript | docs | schema | {}/{} | {}{} |\n", + c.passed_tests, + c.total_tests, + format_pct(c.task_pass_pct), + diff + )); + } + let diff = format_diff( + results.totals.task_pass_pct, + typescript_baseline.map(|b| b.totals.task_pass_pct), + ); + md.push_str(&format!( + "| TypeScript | docs | **total** | {}/{} | **{}**{} |\n", + results.totals.passed_tests, + results.totals.total_tests, + format_pct(results.totals.task_pass_pct), + diff + )); + } + if baseline.is_some() { md.push_str("\n_Compared against master branch baseline_\n"); } @@ -1162,6 +1228,7 @@ fn build_mode_section(lang: &str, mode: &str, failures: &[&FailureInfo], prompt: let lang_display = match lang { "rust" => "Rust", "csharp" => "C#", + "typescript" => "TypeScript", _ => lang, }; @@ -1333,8 +1400,13 @@ fn build_analysis_prompt(failures: &[FailureInfo]) -> String { .iter() .filter(|f| f.lang == "csharp" && f.mode == "docs") .collect(); + let typescript_failures: Vec<_> = failures + .iter() + .filter(|f| f.lang == "typescript" && f.mode == "docs") + .collect(); // Build sections for each language/mode combination + // Note: Rust rustdoc_json and Rust docs are separate sections because they use different context sources if !rust_rustdoc_failures.is_empty() { build_mode_section("rust", "rustdoc_json", &rust_rustdoc_failures, &mut prompt); } @@ -1347,6 +1419,10 @@ fn build_analysis_prompt(failures: &[FailureInfo]) -> String { build_mode_section("csharp", "docs", &csharp_failures, &mut prompt); } + if !typescript_failures.is_empty() { + build_mode_section("typescript", "docs", &typescript_failures, &mut prompt); + } + prompt.push_str( "\n---\n\n## Instructions for your analysis:\n\n\ For EACH failure or group of similar failures:\n\n\ @@ -1357,7 +1433,7 @@ fn build_analysis_prompt(failures: &[FailureInfo]) -> String { 5. **Root cause**: What's missing or unclear in the documentation?\n\ 6. **Recommendation**: Specific fix\n\n\ Group similar failures together (e.g., if multiple tests fail due to the same issue).\n\ - Use code blocks with syntax highlighting (```rust or ```csharp).\n", + Use code blocks with syntax highlighting (```rust, ```csharp, or ```typescript).\n", ); prompt