feat: multi-SDK expansion with auto-discovery registry #49
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…stry
Introduce a registry pattern that auto-discovers integrations from
src/integrations/*/index.ts at startup, eliminating the need to edit
shared files (constants.ts, config.ts, run-with-core.ts) when adding
new SDK integrations. This enables parallel agent-team development
for the multi-SDK expansion.
Key changes:
- Add src/lib/registry.ts with IntegrationRegistry (auto-discovery via
fs.readdirSync + dynamic import)
- Add src/lib/language-detection.ts for multi-language project detection
- Migrate 5 existing JS integrations to src/integrations/{name}/
- Replace Integration enum with string type alias
- Expand Bash tool permissions for Python, Ruby, PHP, Go, .NET, Elixir,
and Kotlin package managers
- Add language, stability, and priority fields to FrameworkConfig
Add support for SvelteKit, Node.js/Express, Python/Django, Ruby/Rails, Go/Gin, PHP, PHP/Laravel, Kotlin/Spring Boot, .NET/ASP.NET Core, and Elixir/Phoenix. Each integration includes a FrameworkConfig + skill file and was implemented in parallel by an agent team with zero merge conflicts. Non-JS integrations use a custom run() that bypasses the JS-centric runAgentInstaller and calls initializeAgent/runAgent directly. Also fixes port-detection crash for unknown integrations by adding optional chaining with sensible defaults.
Add graders, scenarios, and infrastructure for SvelteKit, Node.js, Python, Ruby, Go, PHP, PHP-Laravel, Kotlin, .NET, and Elixir evals. - Language-aware fixture manager (pip, bundle, go mod, composer, mix, dotnet restore, gradle) - Generic checkCommand() in BuildGrader for non-JS validation - Agent executor uses string-keyed skill names, writes .env for non-JS frameworks - 10 new grader files with framework-specific checks - 34 total scenarios (24 existing + 10 new)
Minimal project scaffolds matching WorkOS AuthKit quickstart patterns: - SvelteKit (svelte.config.js, app.html, routes) - Node.js/Express (server.js, CJS matching quickstart) - Python/Flask (server.py, requirements.txt) - Ruby/Sinatra (server.rb, Gemfile) - Go/Gin (main.go, go.mod) - PHP (composer.json, public/index.php) - PHP-Laravel (artisan, routes/web.php, Blade template) - Kotlin/Spring Boot (build.gradle.kts, Application.kt) - .NET/ASP.NET Core (Example.csproj, Program.cs) - Elixir/Phoenix (mix.exs, router.ex, controller) Each fixture is the "before" state — a working app before WorkOS is added. The eval system runs the installer agent against these and grades the result.
Build artifacts shouldn't be in the fixture — they're regenerated by `pnpm build` during eval runs. Added .gitignore to prevent.
- Node/Python/Ruby: sealed session (loadSealedSession) is now a bonus check, not required. Agent reliably adds SDK + auth endpoints (steps 1-2 of quickstart) but doesn't always complete session handling (step 3). Required checks: SDK installed + authorization URL + authenticateWithCode. - SvelteKit: fix package name from @workos-inc/authkit-sveltekit to regex /authkit-sveltekit/ (actual package is @workos/authkit-sveltekit). Loosen hooks.server.ts and callback route patterns to match workos OR authkit references. - PHP-Laravel: broaden auth check from Controllers/**/*.php to **/*.php since agent may put WorkOS code in routes or service providers.
Skills now detect the user's framework and adapt instead of forcing a specific one: - Python: Django → Flask → FastAPI → vanilla (install Flask) - Ruby: Rails → Sinatra → vanilla (install Sinatra) - Node: Express → Fastify → Hono → Koa → vanilla (install Express) Each skill has framework-specific integration paths with the right patterns (e.g., Django uses views.py + urls.py, Flask uses route decorators, Rails uses controllers + routes.rb, Sinatra uses inline routes).
yargs .version() with no args reads from cwd's package.json, which fails when the CLI is invoked via npx (different working directory). Use getVersion() from settings which reads from the bundled config.
Add Auth0 migration scenarios for 8 backend SDKs (node, python, ruby, go, php, php-laravel, kotlin, elixir). Each fixture has a working Auth0 integration that the agent must replace with WorkOS AuthKit, testing code comprehension and migration rather than greenfield generation. - Relax errorHandling quality rubric so SDK-default delegation scores 3-4 - Add /api/health route to each fixture as a preservation check (bonus) - Refactor backend graders to required/bonus pattern - 42 total scenarios (was 34)
Kotlin: pin SDK version (4.18.1), add JVM target guidance, correct API
method names (userManagement not sso, builder pattern)
Go: clarify provider is the string "authkit" (not a constant), add
specific method names (GetAuthorizationURL, AuthenticateWithCode)
Elixir: add prerequisite check for {AppName}Web module with :controller
macro — agent was creating it from scratch without guidance
…ures Kotlin: Include gradlew + gradle/wrapper/ in both kotlin fixtures so the grader's `./gradlew build` check doesn't depend on the agent creating a wrapper. Previously caused flaky failures. Fixture cleanup: Add maxRetries to rm() and swallow cleanup errors. The ENOTEMPTY error on tanstack-start temp dirs was turning passing scenarios into false failures — cleanup is now best-effort.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Integrationenum with auto-discovery registry — each SDK is a self-contained directory undersrc/integrations/that's discovered at startup viafs.readdirSync+ dynamicimport()Why
The installer only supported 5 JS AuthKit SDKs. WorkOS has 10+ additional SDKs across Python, Ruby, PHP, Go, Kotlin, .NET, and Elixir that developers had to integrate manually. The registry pattern was needed to eliminate shared-file merge conflicts and enable parallel agent-team development.
Notes
stability: 'experimental'— graduate tostableafter full eval suite coverageworkos.com/docs/authkit/vanilla/{language}run()functions that bypassrunAgentInstaller(which has JS-centric assumptions) and callinitializeAgent/runAgentdirectly — a follow-up can makerunAgentInstallerlanguage-aware.NETeval requiresdotnetSDK installed to rundocs/directory is not committed (local ideation artifacts per CLAUDE.md)