feat(model-registry): add model registry with profiles, cost estimation, and visibility logging#2399
feat(model-registry): add model registry with profiles, cost estimation, and visibility logging#2399conversun wants to merge 5 commits intocode-yeongyu:devfrom
Conversation
|
All contributors have signed the CLA. Thank you! ✅ |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
7 issues found across 29 files
Confidence score: 2/5
- Several high-confidence, user-impacting regressions are flagged in model selection flow, including missing
provider/prefixes insrc/shared/model-registry/profiles.ts(Opencode compatibility) and first-run profile overrides being overwritten insrc/agents/builtin-agents/hephaestus-agent.ts. - Agent behavior may be incorrectly disabled or misrouted:
src/tools/delegate-task/categories.tscan returnnullbefore profile overrides are considered, andsrc/tools/delegate-task/category-resolver.tsmay fail to detect custom/unstable models due to stricter matching and case sensitivity. - The score is 2 because there are multiple severity 7–8 issues with strong confidence that can change runtime agent/model resolution, not just tests or logging; this creates meaningful regression risk before merge.
- Pay close attention to
src/shared/model-registry/profiles.ts,src/tools/delegate-task/categories.ts,src/agents/builtin-agents/hephaestus-agent.ts,src/tools/delegate-task/category-resolver.ts, andsrc/plugin-handlers/agent-config-handler.ts- these paths contain the highest-risk model/profile forwarding and resolution bugs.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/shared/model-registry/profiles.ts">
<violation number="1" location="src/shared/model-registry/profiles.ts:11">
P1: Custom agent: **Opencode Compatibility**
Profile models must include the `provider/` prefix to comply with the Opencode SDK's requirement for model strings.</violation>
</file>
<file name="src/shared/model-resolution-pipeline.ts">
<violation number="1" location="src/shared/model-resolution-pipeline.ts:74">
P2: Inconsistent agent context logging: `agentPrefix` is only applied to uiSelectedModel and userModel resolution logs but omitted from category-default, provider-fallback, and system-default logs, making it impossible to trace which agent triggered fallback/default resolutions</violation>
</file>
<file name="src/tools/delegate-task/categories.ts">
<violation number="1" location="src/tools/delegate-task/categories.ts:27">
P1: Categories are prematurely disabled due to checking hardcoded `requiresModel` availability before evaluating profile overrides. If the default required model is unavailable, the function returns null immediately, ignoring any valid alternative models a profile might provide.</violation>
</file>
<file name="src/shared/model-registry/cost-estimation.test.ts">
<violation number="1" location="src/shared/model-registry/cost-estimation.test.ts:112">
P2: Test for stable sort order does not actually verify element ordering - it only checks labels which would pass regardless of order</violation>
</file>
<file name="src/agents/builtin-agents/hephaestus-agent.ts">
<violation number="1" location="src/agents/builtin-agents/hephaestus-agent.ts:76">
P1: Model profile override is ignored during first run. The code unconditionally overwrites the model resolution with the fallback model on first run, even when a profile (e.g., 'economy') was explicitly configured. The condition should also check for `!modelProfile` to preserve profile-based selections.</violation>
</file>
<file name="src/tools/delegate-task/category-resolver.ts">
<violation number="1" location="src/tools/delegate-task/category-resolver.ts:217">
P1: Unstable agent detection regression: switching from flexible substring matching to strict registry lookup breaks detection for unlisted/custom unstable models and introduces case sensitivity issues. Models like "gemini-1.5-pro-preview" or case variants like "GEMINI-3.1-PRO" will no longer be flagged as unstable.</violation>
</file>
<file name="src/plugin-handlers/agent-config-handler.ts">
<violation number="1" location="src/plugin-handlers/agent-config-handler.ts:162">
P1: Missing `modelProfile` forwarding to sisyphus-junior and prometheus agents. The `modelProfile` config is extracted and passed to `createBuiltinAgents`, but not forwarded to `createSisyphusJuniorAgentWithOverrides` or `buildPrometheusAgentConfig`. This causes these agents to bypass model profile overrides (premium/balanced/economy), breaking the feature's consistency.</violation>
</file>
Since this is your first cubic review, here's how it works:
- cubic automatically reviews your code and comments on bugs and improvements
- Teach cubic by replying to its comments. cubic learns from your replies and gets better over time
- Add one-off context when rerunning by tagging
@cubic-dev-aiwith guidance or docs links (includingllms.txt) - Ask questions if you need clarification on any suggestion
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
| export const PROFILE_PRESETS: Record<ProfileName, Record<string, ProfileOverride>> = { | ||
| premium: {}, | ||
| balanced: {}, | ||
| economy: { |
There was a problem hiding this comment.
P1: Custom agent: Opencode Compatibility
Profile models must include the provider/ prefix to comply with the Opencode SDK's requirement for model strings.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/shared/model-registry/profiles.ts, line 11:
<comment>Profile models must include the `provider/` prefix to comply with the Opencode SDK's requirement for model strings.</comment>
<file context>
@@ -0,0 +1,31 @@
+export const PROFILE_PRESETS: Record<ProfileName, Record<string, ProfileOverride>> = {
+ premium: {},
+ balanced: {},
+ economy: {
+ sisyphus: { model: "claude-sonnet-4-6" },
+ oracle: { model: "gemini-3-flash" },
</file context>
| if (isFirstRunNoCache && !hephaestusOverride?.model) { | ||
| hephaestusResolution = getFirstFallbackModel(hephaestusRequirement) | ||
| } | ||
| if (isFirstRunNoCache && !hephaestusOverride?.model) { |
There was a problem hiding this comment.
P1: Model profile override is ignored during first run. The code unconditionally overwrites the model resolution with the fallback model on first run, even when a profile (e.g., 'economy') was explicitly configured. The condition should also check for !modelProfile to preserve profile-based selections.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/agents/builtin-agents/hephaestus-agent.ts, line 76:
<comment>Model profile override is ignored during first run. The code unconditionally overwrites the model resolution with the fallback model on first run, even when a profile (e.g., 'economy') was explicitly configured. The condition should also check for `!modelProfile` to preserve profile-based selections.</comment>
<file context>
@@ -1,90 +1,121 @@
- if (isFirstRunNoCache && !hephaestusOverride?.model) {
- hephaestusResolution = getFirstFallbackModel(hephaestusRequirement)
- }
+ if (isFirstRunNoCache && !hephaestusOverride?.model) {
+ hephaestusResolution = getFirstFallbackModel(hephaestusRequirement);
+ }
</file context>
| if (isFirstRunNoCache && !hephaestusOverride?.model) { | |
| if (isFirstRunNoCache && !hephaestusOverride?.model && !modelProfile) { |
| sisyphus: builtinAgents.sisyphus, | ||
| }; | ||
|
|
||
| agentConfig["sisyphus-junior"] = createSisyphusJuniorAgentWithOverrides( |
There was a problem hiding this comment.
P1: Missing modelProfile forwarding to sisyphus-junior and prometheus agents. The modelProfile config is extracted and passed to createBuiltinAgents, but not forwarded to createSisyphusJuniorAgentWithOverrides or buildPrometheusAgentConfig. This causes these agents to bypass model profile overrides (premium/balanced/economy), breaking the feature's consistency.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/plugin-handlers/agent-config-handler.ts, line 162:
<comment>Missing `modelProfile` forwarding to sisyphus-junior and prometheus agents. The `modelProfile` config is extracted and passed to `createBuiltinAgents`, but not forwarded to `createSisyphusJuniorAgentWithOverrides` or `buildPrometheusAgentConfig`. This causes these agents to bypass model profile overrides (premium/balanced/economy), breaking the feature's consistency.</comment>
<file context>
@@ -1,235 +1,264 @@
+ sisyphus: builtinAgents.sisyphus,
+ };
+
+ agentConfig["sisyphus-junior"] = createSisyphusJuniorAgentWithOverrides(
+ params.pluginConfig.agents?.["sisyphus-junior"],
+ undefined,
</file context>
| const models = ["claude-haiku-4-5", "gpt-5-nano"] | ||
| const sorted = compareModelCosts(models) | ||
| expect(sorted.length).toBe(2) | ||
| expect(sorted[0].label).toBe("cheap") |
There was a problem hiding this comment.
P2: Test for stable sort order does not actually verify element ordering - it only checks labels which would pass regardless of order
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/shared/model-registry/cost-estimation.test.ts, line 112:
<comment>Test for stable sort order does not actually verify element ordering - it only checks labels which would pass regardless of order</comment>
<file context>
@@ -0,0 +1,161 @@
+ const models = ["claude-haiku-4-5", "gpt-5-nano"]
+ const sorted = compareModelCosts(models)
+ expect(sorted.length).toBe(2)
+ expect(sorted[0].label).toBe("cheap")
+ expect(sorted[1].label).toBe("cheap")
+ })
</file context>
…, and registry-based unstable detection
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a009d456e7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (!resolvedUserModel && profileName && agentName) { | ||
| const profileOverride = getProfileOverride(profileName, agentName); | ||
| if (profileOverride) { | ||
| resolvedUserModel = profileOverride.model; | ||
| resolvedVariant = profileOverride.variant; |
There was a problem hiding this comment.
This assigns profile presets directly into userModel, but resolveModelPipeline() treats userModel as a hard override and returns before any fallback-chain availability checks. In environments where the selected profile model is not connected (for example model_profile: "economy" with only OpenAI connected), agents like oracle are configured with an unavailable model and fail at runtime instead of falling back to a valid provider model.
Useful? React with 👍 / 👎.
| [actualModelID, configModelID].some((m) => | ||
| m ? MODEL_REGISTRY[m]?.isUnstable === true : false, |
There was a problem hiding this comment.
Unstable detection now relies on exact MODEL_REGISTRY key matches, but this lookup uses raw parsed model IDs and does not normalize provider aliases like gemini-3.1-pro-preview/gemini-3-flash-preview. Those variants are common in provider-transformed IDs, so unstable Gemini executions can be misclassified as stable, which disables unstable-agent handling paths that previously triggered from family-based matching.
Useful? React with 👍 / 👎.
conversun
force-pushed
the
feat/model-registry
branch
from
a009d45 to
aa701f6
Compare
|
I have read the CLA Document and I hereby sign the CLA |
… gating, and robust unstable detection
1 participant
Summary
Adds a Model Registry + Profile Overlay system that sits on top of existing model resolution, providing:
premium/balanced/economy) that override per-agent model selection via configMotivation
Configuring agent models today requires manually cross-referencing multiple documents, editing each agent individually, and updating configs whenever models change. This feature enables zero-config smart defaults via profiles. Set
model_profile: "economy"in config and all agents get appropriate model overrides automatically.Changes
New:
src/shared/model-registry/types.tsregistry.tsprofiles.tsgetProfileOverride()cost-estimation.tsestimateModelCost(),compareModelCosts()index.tsNew:
src/config/schema/model-profile.tsmodel_profileconfig fieldModified (Integration Points)
src/agents/builtin-agents/model-resolution.ts- Profile override injection + variant passthroughsrc/agents/builtin-agents/general-agents.ts- modelProfile param forwardingsrc/agents/builtin-agents/sisyphus-agent.ts- modelProfile param forwardingsrc/agents/builtin-agents/hephaestus-agent.ts- modelProfile param forwardingsrc/agents/builtin-agents/atlas-agent.ts- modelProfile param forwardingsrc/agents/builtin-agents.ts- modelProfile routingsrc/plugin-handlers/agent-config-handler.ts- Extract model_profile from configsrc/tools/delegate-task/category-resolver.ts- Profile override + registry-based unstable detectionsrc/shared/model-resolution-pipeline.ts- Visibility logging with agent contextsrc/config/schema/oh-my-opencode-config.ts- model_profile field additionTests (48 new tests)
registry.test.ts,profiles.test.ts,cost-estimation.test.tsprofile-injection.test.ts,integration.test.tscategory-profile.test.ts,category-resolver.test.tsWhat is NOT Changed (by design)
AGENT_MODEL_REQUIREMENTS/CATEGORY_MODEL_REQUIREMENTSdata - untouchedFallbackEntrytype structure - untouchedresolveModelPipeline()/resolveModelForDelegateTask()- not mergedmodel-selection.ts,subagent-resolver.ts,hook.ts) - untouchedResolution Priority (unchanged)
Usage
Test Results
Summary by cubic
Adds a model registry with profile-based overrides for agents and delegate tasks, plus cost estimation and consistent model-selection logging. Category selection now respects profiles, and unstable models are flagged via the registry.
New Features
premium,balanced(default),economy; per-agent overrides with variant passthrough, applied across builtin agents and the category/delegate-task resolver.[model-resolution]logs include selected model, provenance, reason, and agent name.estimateModelCost()andcompareModelCosts()for pre-selection estimates.Migration
"model_profile": "economy" | "balanced" | "premium"to.opencode/oh-my-opencode.jsonc(defaults tobalanced).Written for commit ef41209. Summary will update on new commits.