Keynote Spacetime simulation comparisons

templates/keynote-2/.dockerignore

@@ -0,0 +1,6 @@
+node_modules
+**/node_modules
+.git
+dist
+build
+.tmp

templates/keynote-2/.env.example

@@ -0,0 +1,63 @@
+# ===== Runtime toggles =====
+USE_DOCKER=1          # 1 = run docker compose up for pg/crdb; 0 = skip
+#SKIP_PG=1             # 1 = don't init Postgres in prep
+#SKIP_CRDB=1           # 1 = don't init Cockroach in prep
+#SKIP_SQLITE=1         # 1 = don't init SQLite in prep
+#SKIP_SUPABASE=1       # 1 = don't init Supabase in prep
+SKIP_CONVEX=1          # 1 = don't init Convex in prep
+USE_SPACETIME_METRICS_ENDPOINT=0
+
+# ===== PostgreSQL =====
+POSTGRES_USER=postgres
+POSTGRES_PASSWORD=postgres
+POSTGRES_DB=postgres
+PG_URL=postgres://postgres:postgres@127.0.0.1:5432/postgres
+
+# ===== CockroachDB =====
+CRDB_URL=postgresql://root@127.0.0.1:26257/defaultdb?sslmode=disable
+# Example managed CockroachDB URL (fill in your own values):
+#CRDB_URL=postgresql://username:password@your-cluster.region.cockroachlabs.cloud:26257/defaultdb?sslmode=verify-full
+
+# ===== SQLite =====
+SQLITE_FILE=./.data/accounts.sqlite
+SQLITE_MODE=default
+
+# ===== SpacetimeDB =====
+STDB_URL=ws://127.0.0.1:3000
+STDB_MODULE=test-1
+STDB_MODULE_PATH=./modules/test-1/server
+STDB_METRICS_URL=http://127.0.0.1:3000/v1/metrics
+
+# ===== Supabase  =====
+# Replace with your actual Supabase project ref
+SUPABASE_URL=http://127.0.0.1:54321
+SUPABASE_ANON_KEY=your_supabase_anon_key_here
+SUPABASE_DB_URL=postgresql://postgres:postgres@127.0.0.1:54322/postgres
+
+# ===== Convex =====
+#CONVEX_URL=https://your-deployment.convex.cloud
+CONVEX_URL=http://127.0.0.1:3210
+CONVEX_SITE_URL=http://127.0.0.1:3210
+CLEAR_CONVEX_ON_PREP=0
+CONVEX_USE_SHARDED_COUNTER=1
+
+# ===== Bun =====
+BUN_URL=http://127.0.0.1:4001
+BUN_PG_URL=postgres://postgres:postgres@pg_bun:5432/bun_bench
+
+# ===== RPC Servers =====
+PG_RPC_URL=http://127.0.0.1:4101
+CRDB_RPC_URL=http://127.0.0.1:4102
+SQLITE_RPC_URL=http://127.0.0.1:4103
+SUPABASE_RPC_URL=http://127.0.0.1:4106
+
+# ===== PlanetScale (optional) =====
+#PLANETSCALE_PG_URL=postgresql://user:password@your-planetscale-host:5432/database
+#PLANETSCALE_RPC_URL=http://127.0.0.1:4104
+
+# ===== Seeding knobs  =====
+SEED_ACCOUNTS=100000
+SEED_INITIAL_BALANCE=10000000
+
+VERIFY=0
+ENABLE_RPC_SERVERS=0

templates/keynote-2/.gitignore

@@ -0,0 +1,39 @@
+# Environment files
+.env
+.env.*
+!.env.example
+
+# Data files
+.data/
+
+# Auto-generated files
+convex-app/convex/_generated/
+modules/test-1/module_bindings/
+
+# SpacetimeDB
+.spacetime/
+
+# Logs
+logs
+*.log
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+pnpm-debug.log*
+lerna-debug.log*
+
+node_modules
+dist
+dist-ssr
+*.local
+
+# Editor directories and files
+.vscode/
+.idea
+.DS_Store
+*.suo
+*.ntvs*
+*.njsproj
+*.sln
+*.sw?
+/runs

templates/keynote-2/.prettierrc

@@ -0,0 +1,5 @@
+{
+  "singleQuote": true,
+  "printWidth": 80,
+  "tabWidth": 2
+}

templates/keynote-2/DEVELOP.md

@@ -0,0 +1,221 @@
+## Overview
+
+This app runs repeatable load tests against multiple connectors (Bun+Postgres, CockroachDB, SQLite, Supabase, Convex, SpacetimeDB, etc.).
+
+Each run:
+
+- Loads a scenario from `src/tests/<test-name>/`
+- Runs it against one or more connectors
+- Writes a JSON report into `./runs/` with TPS and latency stats
+
+---
+
+## Prerequisites
+
+- **Node.js** ≥ 20.x
+- **pnpm** installed globally
+- **Docker** for local Postgres / Cockroach / Supabase
+- Local/Cloud Convex
+
+From a fresh clone:
+
+```bash
+pnpm install
+````
+
+---
+
+## Configuration (`.env`)
+
+Copy `.env.example` to `.env` and adjust.
+
+**Seeding / verification:**
+
+* `SEED_ACCOUNTS` – number of accounts to seed (if unset, code defaults to `100_000`)
+* `SEED_INITIAL_BALANCE` – starting balance per account
+* `VERIFY` – enable extra verification passes when non-zero
+* `ENABLE_RPC_SERVERS` – flag used by scripts that start the RPC benchmark servers
+
+**Runtime toggles:**
+
+* `USE_DOCKER` – `1` = run Docker Compose for Postgres / CockroachDB; `0` = skip
+* `SKIP_PG` – `1` = don't init Postgres in prep
+* `SKIP_CRDB` – `1` = don't init CockroachDB in prep
+* `SKIP_SQLITE` – `1` = don't init SQLite in prep
+* `SKIP_SUPABASE` – `1` = don't init Supabase in prep
+* `SKIP_CONVEX` – `1` = don't init Convex in prep
+* `USE_SPACETIME_METRICS_ENDPOINT` – `1` = read committed transfer counts from the SpacetimeDB metrics endpoint; otherwise only local counters are used
+
+**PostgreSQL / CockroachDB:**
+
+* `PG_URL` – Postgres connection string
+* `CRDB_URL` – CockroachDB connection string
+
+**SQLite:**
+
+* `SQLITE_FILE` – path to the SQLite file
+* `SQLITE_MODE` – tuning preset for the SQLite connector
+
+**SpacetimeDB:**
+
+* `STDB_URL` – WebSocket URL for SpacetimeDB
+* `STDB_MODULE` – module name to load (e.g. `test-1`)
+* `STDB_MODULE_PATH` – filesystem path to the module source (for local dev)
+* `STDB_METRICS_URL` – HTTP URL for the SpacetimeDB metrics endpoint
+
+**Supabase:**
+
+* `SUPABASE_URL` – Supabase project URL
+* `SUPABASE_ANON_KEY` – Supabase anon/public key
+* `SUPABASE_DB_URL` – Postgres connection string for the Supabase database
+
+**Convex:**
+
+* `CONVEX_URL` – Convex deployment URL
+* `CONVEX_SITE_URL` – Convex site URL
+* `CLEAR_CONVEX_ON_PREP` – Convex prep flag (clears data when enabled)
+* `CONVEX_USE_SHARDED_COUNTER` – flag for using the sharded-counter implementation
+
+**Bun / RPC helpers:**
+
+* `BUN_URL` – Bun HTTP benchmark server URL
+* `BUN_PG_URL` – Postgres connection string for the Bun benchmark service
+
+**RPC benchmark servers:**
+
+* `PG_RPC_URL` – HTTP URL for the Postgres RPC server
+* `CRDB_RPC_URL` – HTTP URL for the CockroachDB RPC server
+* `SQLITE_RPC_URL` – HTTP URL for the SQLite RPC server
+
+---
+
+## Setup
+
+### Generate bindings (first time after clone)
+
+**SpacetimeDB module bindings:**
+```bash
+cd modules/test-1/server
+spacetimedb generate --lang typescript --out-dir ../module_bindings
+cd ../../..
+```
+
+**Convex generated files:**
+```bash
+cd convex-app
+npx convex dev
+# Wait for it to generate files, then Ctrl+C
+cd ..
+```
+
+### Start services
+
+1. Start SpacetimeDB (`spacetimedb start`)
+2. Start Convex (inside convex-app run `npx convex dev`)
+3. Init Supabase (run `supabase init`) inside project root.
+4. `npm run prep` to seed the databases.
+5. `npm run bench` to run the test against all connectors. 
+
+## Commands & Examples
+
+### 1. Run a test
+
+```bash
+npm run bench [test-name] [--seconds N] [--concurrency N] [--alpha A] [--connectors list]
+```
+
+Examples:
+
+```bash
+# Default test (test-1), default args (note: only 1 test right now, and it's embedded)
+npm run bench
+
+# Explicit test name
+npm run bench test-1
+
+# Short run, 100 concurrent workers
+npm run bench test-1 --seconds 10 --concurrency 100
+
+# Heavier skew on hot accounts
+npm run bench test-1 --alpha 2.0
+
+# Only run selected connectors
+npm run bench test-1 --connectors spacetimedb,sqlite
+```
+
+---
+
+## CLI Arguments
+
+From `src/cli.ts`:
+
+* **`test-name`** (positional)
+
+    * Name of the test folder under `src/tests/`
+    * Default: `test-1`
+
+* **`--seconds N`**
+
+    * Duration of the benchmark in seconds
+    * Default: `1`
+
+* **`--concurrency N`**
+
+    * Number of workers / in-flight operations
+    * Default: `10`
+
+* **`--alpha A`**
+
+    * Zipf α parameter for account selection (hot vs cold distribution)
+    * Default: `0.5`
+
+* **`--connectors list`**
+
+    * Optional, comma-separated list of connector `system` names
+    * Example:
+
+      ```bash
+      --connectors spacetimedb,sqlite,postgres
+      ```
+    * If omitted, all connectors for that test are run
+    * The valid names come from `tc.system` in the test modules and the keys in `CONNECTORS`
+
+* **`--contention-tests startAlpha endAlpha step concurrency`**
+
+    * Runs a sweep over Zipf α values for a single connector
+    * Uses `startAlpha`, `endAlpha`, and `step` to choose the α values
+    * Uses the provided `concurrency` for all runs
+
+* **`--concurrency-tests startConc endConc step alpha`**
+
+    * Runs a sweep over concurrency levels for a single connector
+    * Uses `startConc`, `endConc`, and `step` to choose the concurrency values
+    * Uses the provided `alpha` for all runs
+
+---
+
+### Running in Docker
+
+You can also run the benchmark via Docker instead of Node directly:
+
+```bash
+docker compose run --rm bench \
+  --seconds 5 \
+  --concurrency 50 \
+  --alpha 1 \
+  --connectors convex
+```
+
+If using Docker, make sure to set `USE_DOCKER=1` in `.env`, verify docker-compose env variables, verify you've run supabase init, and run `npm prep` before running bench.
+
+## Output
+
+Every run writes a JSON file into `./runs/`:
+
+* Directory: `./runs/`
+* Filename: `<test-name>-<timestamp>.json`
+
+    * Example: `test-1-2025-11-17T16-45-12-345Z.json`
+
+
+Point your visualizations / CSV exports at `./runs/` and you’re good.

templates/keynote-2/Dockerfile.bench

@@ -0,0 +1,17 @@
+# Dockerfile.bench
+FROM node:20.18.1-bookworm-slim
+
+WORKDIR /app
+RUN npm install -g pnpm@9.15.3
+
+# Install build tools for better-sqlite3
+RUN apt-get update && apt-get install -y python3 make g++ && rm -rf /var/lib/apt/lists/*
+
+COPY package.json pnpm-lock.yaml* tsconfig.json* ./
+RUN pnpm install --frozen-lockfile || pnpm install
+
+COPY src ./src
+COPY modules ./modules
+
+# Correct: use bench script with passed args
+ENTRYPOINT ["pnpm", "tsx", "src/cli.ts"]

templates/keynote-2/Dockerfile.bun

@@ -0,0 +1,14 @@
+FROM oven/bun:latest
+
+WORKDIR /app
+
+COPY package.json bun.lockb* tsconfig.json ./
+
+RUN bun install --ignore-scripts
+
+# Copy source
+COPY bun ./bun
+
+ENV PORT=4001
+
+CMD ["bun", "bun/bun-server.ts"]

templates/keynote-2/Dockerfile.rpc

@@ -0,0 +1,14 @@
+# Dockerfile.rpc
+FROM node:20.18.1-bookworm-slim
+
+WORKDIR /app
+RUN apt-get update && apt-get install -y python3 make g++ && rm -rf /var/lib/apt/lists/*
+
+RUN npm install -g pnpm@9.15.3
+
+COPY package.json pnpm-lock.yaml* tsconfig.json* ./
+RUN pnpm install --frozen-lockfile || pnpm install
+
+COPY src ./src
+
+CMD ["pnpm", "tsx", "src/rpc-servers/postgres-rpc-server.ts"]