Repository navigation
fix: protect verified Bedrock Claude prices from regeneration #36
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,137 @@ | ||
| { | ||
| "description": [ | ||
| "Prices verified by hand against the vendor. rake models:update re-applies every entry here after it merges the provider listings with models.dev (RubyLLM::Models.apply_overrides), so a regeneration cannot replace them.", | ||
| "Each entry sets the model's pricing.text_tokens to { standard: ... } only and replaces metadata.cost whole, so no long-context tier is kept for these ids in either place.", | ||
| "rake models:update also keeps every models.json entry with metadata source 'manual' that no provider listing or models.dev returns (RubyLLM::Models.keep_manual_models). Such an entry stays until it is deleted from models.json by hand.", | ||
| "No long-context tier exists for these models. Anthropic's pricing page (platform.claude.com/docs/en/about-claude/pricing, read 2026-10-14) says: 'Claude 4.6 and later models (except Claude Haiku 5.5) ... include the full 1M token context window at standard pricing.' The AWS Price List has long_ctx SKUs only for Claude Haiku 5.5 and Claude Sonnet 4.", | ||
| "Edit models.json and this file together. spec/ruby_llm/models_overrides_spec.rb fails when they disagree." | ||
| ], | ||
| "overrides": { | ||
| "bedrock:us.anthropic.claude-fable-5-1": { | ||
| "source": "AWS Price List API, AmazonBedrockFoundationModels offer, us-west-2, publication 2026-10-12: on-demand regional ('Standard' / 'Regional') SKUs", | ||
| "cost": { | ||
| "input": 11, | ||
| "output": 55, | ||
| "cache_read": 0.275, | ||
| "cache_write": 13.75 | ||
| } | ||
| }, | ||
| "bedrock:us.anthropic.claude-fable-5": { | ||
| "source": "AWS Price List API, AmazonBedrockFoundationModels offer, us-west-2, publication 2026-10-12: on-demand regional ('Standard' / 'Regional') SKUs", | ||
| "cost": { | ||
| "input": 11, | ||
| "output": 55, | ||
| "cache_read": 1.1, | ||
| "cache_write": 13.75 | ||
| } | ||
| }, | ||
| "bedrock:us.anthropic.claude-sonnet-5-5": { | ||
| "source": "AWS Price List API, AmazonBedrockFoundationModels offer, us-west-2, publication 2026-10-12: on-demand regional ('Standard' / 'Regional') SKUs", | ||
| "cost": { | ||
| "input": 2.2, | ||
| "output": 11, | ||
| "cache_read": 0.11, | ||
| "cache_write": 2.75 | ||
| } | ||
| }, | ||
| "bedrock:us.anthropic.claude-sonnet-5": { | ||
| "source": "AWS Price List API, AmazonBedrockFoundationModels offer, us-west-2, publication 2026-10-12: on-demand regional ('Standard' / 'Regional') SKUs", | ||
| "cost": { | ||
| "input": 2.2, | ||
| "output": 11, | ||
| "cache_read": 0.22, | ||
| "cache_write": 2.75 | ||
| } | ||
| }, | ||
| "bedrock:us.anthropic.claude-opus-5-5": { | ||
| "source": "AWS Price List API, AmazonBedrockFoundationModels offer, us-west-2, publication 2026-10-12: on-demand regional ('Standard' / 'Regional') SKUs", | ||
| "cost": { | ||
| "input": 4.4, | ||
| "output": 22, | ||
| "cache_read": 0.22, | ||
| "cache_write": 5.5 | ||
| } | ||
| }, | ||
| "bedrock:us.anthropic.claude-opus-5": { | ||
| "source": "AWS Price List API, AmazonBedrockFoundationModels offer, us-west-2, publication 2026-10-12: on-demand regional ('Standard' / 'Regional') SKUs", | ||
| "cost": { | ||
| "input": 5.5, | ||
| "output": 27.5, | ||
| "cache_read": 0.55, | ||
| "cache_write": 6.875 | ||
| } | ||
| }, | ||
| "bedrock:us.anthropic.claude-opus-4-8": { | ||
| "source": "AWS Price List API, AmazonBedrockFoundationModels offer, us-west-2, publication 2026-10-12: on-demand regional ('Standard' / 'Regional') SKUs", | ||
| "cost": { | ||
| "input": 5.5, | ||
| "output": 27.5, | ||
| "cache_read": 0.55, | ||
| "cache_write": 6.875 | ||
| } | ||
| }, | ||
| "bedrock:us.anthropic.claude-opus-4-7": { | ||
| "source": "AWS Price List API, AmazonBedrockFoundationModels offer, us-west-2, publication 2026-10-12: on-demand regional ('Standard' / 'Regional') SKUs", | ||
| "cost": { | ||
| "input": 5.5, | ||
| "output": 27.5, | ||
| "cache_read": 0.55, | ||
| "cache_write": 6.875 | ||
| } | ||
| }, | ||
| "bedrock:us.anthropic.claude-opus-4-6-v1": { | ||
| "source": "AWS Price List API, AmazonBedrockFoundationModels offer, us-west-2, publication 2026-10-12: on-demand regional ('Standard' / 'Regional') SKUs", | ||
| "cost": { | ||
| "input": 5.5, | ||
| "output": 27.5, | ||
| "cache_read": 0.55, | ||
| "cache_write": 6.875 | ||
| } | ||
| }, | ||
| "bedrock:us.anthropic.claude-opus-4-5-20251101-v1:0": { | ||
| "source": "AWS Price List API, AmazonBedrockFoundationModels offer, us-west-2, publication 2026-10-12: on-demand regional ('Standard' / 'Regional') SKUs", | ||
| "cost": { | ||
| "input": 5.5, | ||
| "output": 27.5, | ||
| "cache_read": 0.55, | ||
| "cache_write": 6.875 | ||
| } | ||
| }, | ||
| "bedrock:us.anthropic.claude-sonnet-4-6": { | ||
| "source": "AWS Price List API, AmazonBedrockFoundationModels offer, us-west-2, publication 2026-10-12: on-demand regional ('Standard' / 'Regional') SKUs", | ||
| "cost": { | ||
| "input": 3.3, | ||
| "output": 16.5, | ||
| "cache_read": 0.33, | ||
| "cache_write": 4.125 | ||
| } | ||
| }, | ||
| "bedrock:us.anthropic.claude-sonnet-4-5-20250929-v1:0": { | ||
| "source": "AWS Price List API, AmazonBedrockFoundationModels offer, us-west-2, publication 2026-10-12: on-demand regional ('Standard' / 'Regional') SKUs", | ||
| "cost": { | ||
| "input": 3.3, | ||
| "output": 16.5, | ||
| "cache_read": 0.33, | ||
| "cache_write": 4.125 | ||
| } | ||
| }, | ||
| "bedrock:us.anthropic.claude-haiku-4-5-20251001-v1:0": { | ||
| "source": "Global rate on purpose: the AWS bill for 2026-10-05 to 2026-10-12 shows us. Haiku 4.5 at 94% of its regional rate (reason unknown), and the global rate (AWS Price List API, AmazonBedrockFoundationModels offer, us-west-2, publication 2026-10-12: on-demand 'Standard, Global' SKUs) is closer to the bill.", | ||
| "cost": { | ||
| "input": 1, | ||
| "output": 5, | ||
| "cache_read": 0.1, | ||
| "cache_write": 1.25 | ||
| } | ||
| }, | ||
| "bedrock:global.anthropic.claude-fable-5-1": { | ||
| "source": "AWS Price List API, AmazonBedrockFoundationModels offer, us-west-2, publication 2026-10-12: on-demand 'Standard, Global' SKUs", | ||
| "cost": { | ||
| "input": 10, | ||
| "output": 50, | ||
| "cache_read": 0.25, | ||
| "cache_write": 12.5 | ||
| } | ||
| } | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,136 @@ | ||
| # frozen_string_literal: true | ||
|
|
||
| require 'spec_helper' | ||
|
|
||
| RSpec.describe RubyLLM::Models do | ||
| include_context 'with configured RubyLLM' | ||
|
|
||
| overrides = described_class.read_overrides | ||
|
|
||
| def bedrock_entry(id) | ||
| described_class.read_from_json.find { |m| m.provider == 'bedrock' && m.id == id } | ||
| end | ||
|
|
||
| def standard_pricing(model) | ||
| model.pricing.text_tokens.standard.to_h.slice( | ||
| :input_per_million, :output_per_million, | ||
| :cache_read_input_per_million, :cache_write_input_per_million | ||
| ) | ||
| end | ||
|
|
||
| def expected_standard(cost) | ||
| { | ||
| input_per_million: cost[:input], | ||
| output_per_million: cost[:output], | ||
| cache_read_input_per_million: cost[:cache_read], | ||
| cache_write_input_per_million: cost[:cache_write] | ||
| } | ||
| end | ||
|
|
||
| it 'has overrides to check' do | ||
| expect(overrides).not_to be_empty | ||
| end | ||
|
|
||
| # models.json must already hold each override, so the JSON file and a regenerated registry | ||
| # agree. A hand edit to one file without the other fails here. | ||
| overrides.each do |key, override| | ||
| provider, id = key.split(':', 2) | ||
|
|
||
| it "keeps #{key} in models.json at the override price" do | ||
| model = described_class.read_from_json.find { |m| m.provider == provider && m.id == id } | ||
|
Author
There was a problem hiding this comment. nit: each of the ~21 generated examples here and in |
||
|
|
||
| expect(model).not_to be_nil | ||
| expect(model.metadata[:cost]).to eq(override[:cost]) | ||
| expect(standard_pricing(model)).to eq(expected_standard(override[:cost])) | ||
| end | ||
| end | ||
|
|
||
| # Anthropic's pricing page (platform.claude.com/docs/en/about-claude/pricing, read | ||
| # 2026-10-14), "Long context pricing": "Claude 4.6 and later models (except Claude Haiku 5.5) | ||
| # ... include the full 1M token context window at standard pricing." The AWS Price List | ||
| # (AmazonBedrockFoundationModels, us-west-2, publication 2026-10-12) has long_ctx SKUs only | ||
| # for Claude Haiku 5.5 and Claude Sonnet 4. A tier on any of these ids would overcharge every | ||
| # request over 200k tokens. | ||
| %w[ | ||
| us.anthropic.claude-opus-4-8 us.anthropic.claude-opus-5 us.anthropic.claude-opus-5-5 | ||
| us.anthropic.claude-sonnet-5 us.anthropic.claude-sonnet-5-5 | ||
| us.anthropic.claude-fable-5 us.anthropic.claude-fable-5-1 | ||
| ].each do |id| | ||
| it "prices #{id} with no long-context tier" do | ||
| model = bedrock_entry(id) | ||
|
|
||
| expect(model.context_window).to eq(1_000_000) | ||
| expect(model.metadata[:cost].keys).to contain_exactly(:input, :output, :cache_read, :cache_write) | ||
| expect(model.pricing.to_h[:text_tokens].keys).to eq([:standard]) | ||
| end | ||
| end | ||
|
|
||
| describe '.apply_overrides' do | ||
| let(:cost) { { input: 4.4, output: 22, cache_read: 0.22, cache_write: 5.5 } } | ||
| let(:tiered_model) do | ||
| RubyLLM::Model::Info.new( | ||
| id: 'us.anthropic.claude-opus-5-5', name: 'Claude Opus 5.5 (US)', provider: 'bedrock', | ||
| pricing: { | ||
| text_tokens: { | ||
| standard: { input_per_million: 4, output_per_million: 20 }, | ||
| context_over_200k: { input_per_million: 8, output_per_million: 30 } | ||
| } | ||
| }, | ||
| metadata: { | ||
| cost: { input: 4, output: 20, context_over_200k: { input: 8, output: 30 }, | ||
| tiers: [{ input: 8, output: 30, tier: { type: 'context', size: 200_000 } }] } | ||
| } | ||
| ) | ||
| end | ||
|
|
||
| # Model::Pricing already drops any text tier but standard and batch, so the tier that | ||
| # matters is the one in metadata.cost; this pins both places. | ||
| it 'drops a long-context tier from both pricing and metadata.cost' do | ||
| model = described_class.apply_overrides([tiered_model], | ||
| { 'bedrock:us.anthropic.claude-opus-5-5' => { cost: cost } }) | ||
| .first | ||
|
|
||
| expect(model.pricing.to_h[:text_tokens]).to eq(standard: expected_standard(cost)) | ||
| expect(model.metadata[:cost]).to eq(cost) | ||
| end | ||
| end | ||
|
|
||
| describe 'refresh' do | ||
| let(:overridden_id) { 'us.anthropic.claude-opus-5-5' } | ||
| let(:models_dev_model) do | ||
| RubyLLM::Model::Info.new( | ||
| id: overridden_id, name: 'Claude Opus 5.5 (US)', provider: 'bedrock', | ||
| pricing: { text_tokens: { standard: { input_per_million: 4, output_per_million: 20 } } }, | ||
| metadata: { | ||
| source: 'models.dev', | ||
| cost: { input: 4, output: 20, tiers: [{ input: 8, output: 30, tier: { type: 'context', size: 200_000 } }] } | ||
| } | ||
| ) | ||
| end | ||
| let(:manual_model) do | ||
| RubyLLM::Model::Info.new(id: 'us.anthropic.claude-hand-added', name: 'Hand added', provider: 'bedrock', | ||
| metadata: { source: 'manual', cost: { input: 1, output: 2 } }) | ||
| end | ||
|
|
||
| before do | ||
| allow(described_class).to receive(:read_existing_models).and_return([manual_model]) | ||
| allow(described_class).to receive_messages( | ||
| fetch_provider_models: { models: [], fetched_providers: ['bedrock'], configured_names: [], failed: [] }, | ||
| fetch_models_dev_models: { models: [models_dev_model], fetched: true } | ||
| ) | ||
| end | ||
|
|
||
| it 'applies the override over the models.dev price, tiers included' do | ||
| model = described_class.fetch_merged_models.find { |m| m.id == overridden_id } | ||
|
|
||
| expect(model.metadata[:cost]).to eq(overrides["bedrock:#{overridden_id}"][:cost]) | ||
| expect(standard_pricing(model)).to eq(expected_standard(overrides["bedrock:#{overridden_id}"][:cost])) | ||
| end | ||
|
|
||
| it 'keeps a manual model that no source returns' do | ||
| ids = described_class.fetch_merged_models.map(&:id) | ||
|
|
||
| expect(ids).to include('us.anthropic.claude-hand-added') | ||
| end | ||
| end | ||
| end | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
keep_manual_modelschanges how every provider's registry is regenerated, but the PR description doesn't mention it. The description only covers the Bedrock override table.What it does: any existing model with
metadata.source == 'manual'now stays in the registry after a refresh, even when its provider was fetched successfully and no longer lists it. So a retired hand-added model stays inmodels.jsonuntil someone deletes it by hand. That is a different policy from the provider-authoritative merge, and it isn't needed for DUX-13714.Fix: move it to its own PR, or at least describe it under "What this does" so reviewers know it's an intentional change to all providers.