Add support for @cardstack/catalog references#4058
Conversation
Preview deployments |
Host Test Results 1 files ± 0 1 suites ±0 1h 47m 59s ⏱️ + 18m 0s Results for commit 3237e93. ± Comparison against base commit 13b3155. This pull request removes 1 and adds 34 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3237e93132
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| return new URL(rest, withTrailingSlash(config.resolvedCatalogRealmURL)) | ||
| .href; | ||
| }); | ||
| if (config.resolvedCatalogRealmURL) { |
There was a problem hiding this comment.
Is this actually optional?
There was a problem hiding this comment.
It’s missing when SKIP_CATALOG is present.
There was a problem hiding this comment.
Pull request overview
Adds environment-independent @cardstack/catalog/... references by introducing a central resolver and registering prefix→URL mappings at startup, so realms and imports no longer need hard-coded absolute URLs.
Changes:
- Introduce
resolveCardReference/registerCardReferencePrefixand apply it broadly where URLs were previously constructed directly. - Update realm-server startup scripts and runtime wiring to register
@cardstack/catalog/mappings. - Migrate existing realm content to use
@cardstack/catalog/...references.
Reviewed changes
Copilot reviewed 45 out of 45 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/runtime-common/serializers/code-ref.ts | Resolve prefixed/bare module refs via resolveCardReference during serialization. |
| packages/runtime-common/resource-types.ts | Resolve relationship IDs using resolveCardReference. |
| packages/runtime-common/realm.ts | Resolve module dependency URLs via resolveCardReference. |
| packages/runtime-common/realm-index-query-engine.ts | Use resolveCardReference when absolutizing/relativizing links and module deps. |
| packages/runtime-common/module-syntax.ts | Resolve module URLs via resolveCardReference when inspecting/updating modules. |
| packages/runtime-common/index.ts | Export card reference resolver and use it in internalKeyFor. |
| packages/runtime-common/index-runner/dependency-url.ts | Canonicalize dependency URLs after resolving card references. |
| packages/runtime-common/index-runner.ts | Resolve adoptsFrom module dependency URLs via resolveCardReference. |
| packages/runtime-common/file-serializer.ts | Resolve self links via resolveCardReference before relativizing. |
| packages/runtime-common/file-def-code-ref.ts | Resolve file def code refs via resolveCardReference. |
| packages/runtime-common/code-ref.ts | Use resolveCardReference when normalizing/loading code refs and in error messaging. |
| packages/runtime-common/card-reference-resolver.ts | New prefix mapping registry + reference resolver for @cardstack/...-style references. |
| packages/realm-server/worker.ts | Register prefix mappings (non-URL --fromUrl) and import maps for workers. |
| packages/realm-server/worker-manager.ts | Allow non-URL --fromUrl values to flow through to workers. |
| packages/realm-server/scripts/start-worker-staging.sh | Use @cardstack/catalog/ as the catalog --fromUrl. |
| packages/realm-server/scripts/start-worker-production.sh | Use @cardstack/catalog/ as the catalog --fromUrl. |
| packages/realm-server/scripts/start-worker-development.sh | Use @cardstack/catalog/ as the catalog --fromUrl. |
| packages/realm-server/scripts/start-staging.sh | Use @cardstack/catalog/ as the catalog --fromUrl. |
| packages/realm-server/scripts/start-production.sh | Use @cardstack/catalog/ as the catalog --fromUrl. |
| packages/realm-server/scripts/start-development.sh | Use @cardstack/catalog/ as the catalog --fromUrl. |
| packages/realm-server/scripts/migrate-realm-references.sh | New script to migrate realm references (URL ↔ prefix) with patch output. |
| packages/realm-server/main.ts | Register prefix mappings/import maps for non-URL --fromUrl values at server startup. |
| packages/host/app/services/store.ts | Resolve relationship self links via resolveCardReference when loading instances. |
| packages/host/app/services/network.ts | Register @cardstack/catalog/ prefix mapping in the host virtual network. |
| packages/host/app/lib/prerender-util.ts | Resolve module deps via resolveCardReference for prerender dependency discovery. |
| packages/host/app/components/operator-mode/edit-field-modal.gts | Resolve module URLs via resolveCardReference before constructing URL. |
| packages/host/app/components/operator-mode/code-submode/module-inspector.gts | Resolve adoptsFrom module URLs via resolveCardReference for file def creation. |
| packages/experiments-realm/SkillPlus/4e9f1f88-ef64-44e4-bcf0-78ab9fbeedf8.json | Replace catalog absolute URLs with @cardstack/catalog/.... |
| packages/experiments-realm/ProductCatalog/main.json | Replace catalog absolute theme link with @cardstack/catalog/.... |
| packages/experiments-realm/IkeaProduct/solglim-modular-sofa.json | Replace catalog absolute theme link with @cardstack/catalog/.... |
| packages/experiments-realm/IkeaProduct/norra-oak-dining-table.json | Replace catalog absolute theme link with @cardstack/catalog/.... |
| packages/experiments-realm/IkeaProduct/loom-tall-bookshelf.json | Replace catalog absolute theme link with @cardstack/catalog/.... |
| packages/experiments-realm/IkeaProduct/ljus-arc-floor-lamp.json | Replace catalog absolute theme link with @cardstack/catalog/.... |
| packages/catalog-realm/Spec/16320d37-fac6-417c-ae20-c583ea0c8a2e.json | Replace catalog absolute module URL with @cardstack/catalog/.... |
| packages/catalog-realm/GameResult/e7cfd2bd-8306-4b58-aba0-afc05c6d18ef.json | Replace catalog absolute module URL with @cardstack/catalog/.... |
| packages/catalog-realm/GameResult/d3e487ac-074f-404a-a63e-b41b83df24d7.json | Replace catalog absolute module URL with @cardstack/catalog/.... |
| packages/catalog-realm/GameResult/c6e7cfd2-bd83-465b-982b-a0afc05c6d18.json | Replace catalog absolute module URL with @cardstack/catalog/.... |
| packages/catalog-realm/GameResult/c05c6d18-ef0a-40d3-a487-ac074f004a66.json | Replace catalog absolute module URL with @cardstack/catalog/.... |
| packages/catalog-realm/GameResult/5c6d18ef-0a50-43e4-87ac-074f004a663e.json | Replace catalog absolute module URL with @cardstack/catalog/.... |
| packages/catalog-realm/GameResult/5b582ba0-afc0-4c6d-98ef-0a50d3e487ac.json | Replace catalog absolute module URL with @cardstack/catalog/.... |
| packages/catalog-realm/GameResult/50d3e487-ac07-4f00-8a66-3eb41b83df24.json | Replace catalog absolute module URL with @cardstack/catalog/.... |
| packages/catalog-realm/GameResult/14c25556-2c52-42ff-ab5f-cb790140a3d0.json | Replace catalog absolute module URL with @cardstack/catalog/.... |
| packages/catalog-realm/GameResult/065b582b-a0af-405c-ad18-ef0a50d3e487.json | Replace catalog absolute module URL with @cardstack/catalog/.... |
| packages/base/spec.gts | Resolve spec module refs via resolveCardReference. |
| packages/base/card-api.gts | Resolve link references via resolveCardReference throughout link field logic. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| export function registerCardReferencePrefix( | ||
| prefix: string, | ||
| targetURL: string, | ||
| ): void { | ||
| prefixMappings.set(prefix, targetURL); |
There was a problem hiding this comment.
registerCardReferencePrefix stores targetURL as-is, but resolveCardReference uses new URL(rest, target). If targetURL is missing a trailing slash (e.g. https://app.boxel.ai/catalog), resolving @cardstack/catalog/skill-plus will incorrectly become https://app.boxel.ai/skill-plus. Normalize targetURL (ensure trailing /) when registering prefixes, or enforce/validate this invariant with a clear error.
| registerCardReferencePrefix(from, to.href); | ||
| virtualNetwork.addImportMap(from, (rest) => new URL(rest, to).href); | ||
| urlMappings.push([to, to]); // use toUrl for both in hrefs |
There was a problem hiding this comment.
When registering a non-URL prefix mapping, to.href must behave as a base URL for new URL(rest, to). If the provided --toUrl lacks a trailing slash, imports/links will resolve to the parent path (dropping the realm segment). Normalize to to a trailing-slash base before calling registerCardReferencePrefix and addImportMap.
| registerCardReferencePrefix(from, to.href); | |
| virtualNetwork.addImportMap(from, (rest) => new URL(rest, to).href); | |
| urlMappings.push([to, to]); // use toUrl for both in hrefs | |
| // Ensure `to` behaves as a base URL for `new URL(rest, to)` by normalizing it | |
| // to have a trailing slash when used for non-URL prefix mappings. | |
| const normalizedTo = to.href.endsWith('/') ? to : new URL(to.href + '/'); | |
| registerCardReferencePrefix(from, normalizedTo.href); | |
| virtualNetwork.addImportMap(from, (rest) => new URL(rest, normalizedTo).href); | |
| urlMappings.push([normalizedTo, normalizedTo]); // use normalized toUrl for both in hrefs |
| if (isUrlLike(from)) { | ||
| virtualNetwork.addURLMapping(new URL(from), to); | ||
| } else { | ||
| registerCardReferencePrefix(from, to.href); | ||
| virtualNetwork.addImportMap(from, (rest) => new URL(rest, to).href); |
There was a problem hiding this comment.
new URL(rest, to) requires to to be a trailing-slash base URL. If --toUrl is configured without a trailing slash, resolving @cardstack/catalog/... will drop the realm path segment. Normalize to (ensure it ends with /) before registering the prefix mapping and import map.
| matching_files=$(grep -rlE "${FIND_STR}|[\"']${REALM_PATH}" "$search_dir" --include='*.json' --include='*.gts' 2>/dev/null || true) | ||
| else | ||
| matching_files=$(grep -rl "${FIND_STR}" "$search_dir" --include='*.json' --include='*.gts' 2>/dev/null || true) |
There was a problem hiding this comment.
grep -rlE "${FIND_STR}|..." treats FIND_STR as a regex. For URL inputs this contains . and other metacharacters, so it can match unintended strings. Prefer fixed-string matching for the full URL (e.g. grep -F) and only use regex where necessary (like the quoted path-only form).
| matching_files=$(grep -rlE "${FIND_STR}|[\"']${REALM_PATH}" "$search_dir" --include='*.json' --include='*.gts' 2>/dev/null || true) | |
| else | |
| matching_files=$(grep -rl "${FIND_STR}" "$search_dir" --include='*.json' --include='*.gts' 2>/dev/null || true) | |
| matching_files_url=$(grep -rlF "${FIND_STR}" "$search_dir" --include='*.json' --include='*.gts' 2>/dev/null || true) | |
| matching_files_path=$(grep -rlE "[\"']${REALM_PATH}" "$search_dir" --include='*.json' --include='*.gts' 2>/dev/null || true) | |
| matching_files=$(printf '%s\n%s\n' "$matching_files_url" "$matching_files_path" | sed '/^$/d' | sort -u | tr '\n' ' ') | |
| else | |
| matching_files=$(grep -rlF "${FIND_STR}" "$search_dir" --include='*.json' --include='*.gts' 2>/dev/null || true) |
| registerCardReferencePrefix(from, to.href); | ||
| virtualNetwork.addImportMap(from, (rest) => new URL(rest, to).href); | ||
| urlMappings.push([to, to]); // use toUrl for both in hrefs | ||
| } |
There was a problem hiding this comment.
For non-URL prefix mappings you currently do urlMappings.push([to, to]), which makes the later ‘Additional URL mappings’ log output print as to => to (dropping the original prefix like @cardstack/catalog/). Consider tracking prefix mappings separately (or at least logging them) so operators can verify the effective mapping configuration.
| if [ "$IS_URL" = true ]; then | ||
| SED_ARGS=(-e "s|${FIND_STR}|${REPLACEMENT}|g" | ||
| -e "s|${DQ}${REALM_PATH}|${DQ}${REPLACEMENT}|g" | ||
| -e "s|'${REALM_PATH}|'${REPLACEMENT}|g") | ||
| else | ||
| SED_ARGS=(-e "s|${FIND_STR}|${REPLACEMENT}|g") |
There was a problem hiding this comment.
The sed substitutions use s|${FIND_STR}|...|g, which interprets FIND_STR as a regex (so . in URLs matches any char). Escape the find string before building the sed expressions, or use a replacement approach that performs literal string substitution to avoid unintended edits.
| if [ "$IS_URL" = true ]; then | |
| SED_ARGS=(-e "s|${FIND_STR}|${REPLACEMENT}|g" | |
| -e "s|${DQ}${REALM_PATH}|${DQ}${REPLACEMENT}|g" | |
| -e "s|'${REALM_PATH}|'${REPLACEMENT}|g") | |
| else | |
| SED_ARGS=(-e "s|${FIND_STR}|${REPLACEMENT}|g") | |
| # Escape FIND_STR and REALM_PATH so sed treats them as literal strings. | |
| ESCAPED_FIND_STR=$(printf '%s\n' "$FIND_STR" | sed -e 's/[.[\*^$\/&|(){}+?]/\\&/g') | |
| if [ "$IS_URL" = true ]; then | |
| ESCAPED_REALM_PATH=$(printf '%s\n' "$REALM_PATH" | sed -e 's/[.[\*^$\/&|(){}+?]/\\&/g') | |
| SED_ARGS=(-e "s|${ESCAPED_FIND_STR}|${REPLACEMENT}|g" | |
| -e "s|${DQ}${ESCAPED_REALM_PATH}|${DQ}${REPLACEMENT}|g" | |
| -e "s|'${ESCAPED_REALM_PATH}|'${REPLACEMENT}|g") | |
| else | |
| SED_ARGS=(-e "s|${ESCAPED_FIND_STR}|${REPLACEMENT}|g") |
This lets the catalog be referenced with
@cardstack/cataloginstead of relying on relative references that break in published realms, or absolute references that break when working across environments.Concretely, this means that
boxel-homecan use an import likeimport { SkillPlus } from '@cardstack/catalog/skill-plus';instead of needingimport { SkillPlus } from 'https://realms-staging.stack.cards/catalog/skill-plus';that differs across environments, and the theme can be referenced like this:This removes the need for hacks like this.
Implementation
The major work here is a
resolveCardReferencefunction that wraps what were previously bare URL constructors, it converts@cardstack/catalogto whatever realm URL is correct for the catalog in the current environment, as specified in the startup script.Open question
I left
--fromUrlalone despite it now accepting@cardstack/catalog. Should it have a different name?Migrating references in existing realms
The migration script dry run feature lets you see what changes are needed, and it saves a
patchif you need to roll back (as I have when switching off this branch):