Repository navigation
fix: correct Bedrock us. regional premium & cache-read rates - #34
Conversation
|
🤖 Supernova Code Review — View trace — via automation "Code Review Opus 5.5 Medium" (run), installed by Sam Boland (sam.boland@appfolio.com) |
There was a problem hiding this comment.
Reviewed the models.json changes and the new and updated specs. I checked every repriced us. entry by hand: input/output/cache-write are each 1.1x the global price, and cache-read uses the right multiplier (0.05x for Sonnet 5.5 and Opus 5.5, 0.025x for Fable 5.1, 0.1x for the rest). Each one matches its spec table, and metadata.cost agrees with pricing.text_tokens.standard everywhere.
One finding (inline): the Sonnet 5.5 cache-read fix only covers the Bedrock bare, global., eu. and us. entries. The anthropic and vertexai entries are still at 0.1x (0.20), even though the PR says that bug isn't region-specific. The same goes for the au./jp. Bedrock entries, which also lack the 10% regional premium. spec/ruby_llm/models_sonnet_5_5_spec.rb lines 27–39 now assert those known-wrong values.
— via automation "Code Review Opus 5.5 Medium" (run), installed by Sam Boland (sam.boland@appfolio.com)
| standard = { input: 2, output: 10, cache_read: 0.20, cache_write: 2.5 } | ||
| eu = { input: 2.2, output: 11, cache_read: 0.22, cache_write: 2.75 } | ||
| global = { input: 2, output: 10, cache_read: 0.1, cache_write: 2.5 } | ||
| regional = { input: 2.2, output: 11, cache_read: 0.11, cache_write: 2.75 } | ||
|
|
||
| { | ||
| %w[anthropic claude-sonnet-5-5] => standard, | ||
| %w[vertexai claude-sonnet-5-5] => standard, | ||
| %w[bedrock anthropic.claude-sonnet-5-5] => standard, | ||
| %w[bedrock global.anthropic.claude-sonnet-5-5] => standard, | ||
| %w[bedrock us.anthropic.claude-sonnet-5-5] => standard, | ||
| %w[bedrock eu.anthropic.claude-sonnet-5-5] => eu, | ||
| %w[bedrock anthropic.claude-sonnet-5-5] => global, | ||
| %w[bedrock global.anthropic.claude-sonnet-5-5] => global, | ||
| %w[bedrock us.anthropic.claude-sonnet-5-5] => regional, | ||
| %w[bedrock eu.anthropic.claude-sonnet-5-5] => regional, | ||
| %w[bedrock au.anthropic.claude-sonnet-5-5] => standard, | ||
| %w[bedrock jp.anthropic.claude-sonnet-5-5] => standard |
There was a problem hiding this comment.
This spec locks in Sonnet 5.5 cache-read prices that the PR's own source says are wrong. The anthropic, vertexai, au. and jp. entries stay at cache_read: 0.20, which is the 0.1x multiplier, but the footnote quoted on lines 12–13 says Sonnet 5.5 cache hits are 0.05x on every endpoint. The PR description also says the bug "wasn't region-specific." So anyone using the first-party Anthropic or Vertex provider gets cache reads costed at 2x the real price. au./jp. are regional endpoints too, so they should be at 2.2 / 11 / 0.11 / 2.75, the same as us./eu.. "Not called by Supernova" is a reason about one consumer, but this registry ships to every RubyLLM user. Fix: set cache_read to 0.1 on the anthropic/vertexai entries, both metadata.cost and pricing.text_tokens.standard. Then either move au./jp. to the regional rate or leave them out of this spec, so it doesn't assert values we know are wrong.
Anthropic's pricing page (read 14 Oct 2026) says regional and multi-region Bedrock endpoints carry a 10% premium over global endpoints, and that Sonnet 5.5 and Opus 5.5 cache hits are 0.05x input. The registry priced us. ids at the global rate and Sonnet 5.5 cache reads at 0.1x. The AWS bill for 5-12 Oct 2026 shows us. Sonnet 5 at 2.20/11/0.22/2.75 and us. Sonnet 5.5 at 2.20/11/0.11/2.75, which matches the regional rule. Set 12 us. Claude entries to the regional price in both pricing fields, and set the eu., global. and bare bedrock Sonnet 5.5 cache reads to 0.05x input. Haiku 4.5, Opus 4.1 and Sonnet 4 are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Sam Boland <sam.boland@appfolio.com>
4c2ad34 to
0d10799
Compare
|
Review of 4c2ad34. The data and specs are right. One change is needed before this can merge, because every test job fails for a reason outside this PR.
Verified and fine:
I will correct the PR description myself after your push (it says 13 repriced |
I've queued the Faraday fix as a separate commit and am working on it now. It makes three changes:
I'll run only the gemspec spec and the four pricing specs. I'm leaving the |
faraday-net_http 3.5.0 was released on 2026-10-11. No Faraday from 2.0.1 to 2.14.4 accepts it: they all require faraday-net_http >= 2.0, < 3.5. Faraday 2.0.0 declares no faraday-net_http dependency at all, so Bundler keeps faraday-net_http 3.5.0 and falls back to Faraday 2.0.0. That pair fails at connection setup with ":net_http is not registered on Faraday::Adapter", which breaks every CI test job. Exclude Faraday 2.0.0 in the gemspec so Bundler picks a Faraday 2.x that pins a compatible faraday-net_http. This is the same change as upstream crmne/ruby_llm commit ef2bebd ("Exclude Faraday 2.0.0, which has no default adapter"). The gemspec spec now checks that the requirement accepts Faraday 1.10.3 and 2.14.4 and rejects 2.0.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pushed Changed 2 files (+9 −2): |
samuelboland
left a comment
There was a problem hiding this comment.
Verified: parsed models.json diff shows exactly 15 bedrock entries / 102 fields changed, all matching the regional price table in both copies; Faraday != 2.0.0 matches upstream ef2bebd; all CI test jobs green.
What this does
Two commits.
1. Price
us.Claude ids at the regional rate and fix Sonnet 5.5 cache readsSupernova prices every LLM call from this gem's
lib/ruby_llm/models.json, and calls Claude on Bedrock only throughus.inference profiles. Anthropic's pricing page (read 14 Oct 2026) says regional and multi-region endpoints for Claude 4.5 and later carry a 10% premium over global endpoints. The registry priced theus.ids at the global rate (theeu.ids already had the premium), so Supernova's Claude costs ran 10% low. The same page prices Sonnet 5.5 and Opus 5.5 cache hits at 0.05x input; the registry had Sonnet 5.5 at 0.1x.Changes to
lib/ruby_llm/models.json, in bothpricing.text_tokens.standardandmetadata.cost, written as exact decimal literals:us.ids set to global price x 1.1 (input / output / cache read / cache write per million):eu.0.22 to 0.11,global.and bareanthropic.claude-sonnet-5-50.2 to 0.1.Opus 5.5 and Fable 5.1 already had the right cache multipliers (0.05x and 0.025x); only their
us.rows gained the 10%.Left unchanged on purpose:
au.,jp., and theanthropicandvertexaiproviders. Supernova does not call them.The AWS bill for 5 to 12 Oct 2026 (Cost Explorer,
USW2-MP:USW2_*usage types) shows Sonnet 5 onus.at 2.20 / 11 / 0.22 / 2.75 and Sonnet 5.5 at 2.20 / 11 / 0.11 / 2.75, matching these prices.Specs:
models_sonnet_5_5_spec.rb,models_opus_5_5_spec.rbandmodels_fable_5_1_spec.rbupdated; newmodels_bedrock_regional_pricing_spec.rbchecks both price copies for the 12us.ids and that Haiku 4.5, Opus 4.1 and Sonnet 4 keep today's values.How "nothing else changed" was checked: a script parsed
models.jsonbefore and after and compared every entry field by field. Both sides have 1,238 entries in the same order; exactly 15 bedrock entries and 102 fields changed (12 entries x 8 fields, 3 entries x 2 fields), and every new value matches the list above.2. Exclude Faraday 2.0.0, which has no default adapter
Every CI test job failed with
:net_http is not registered on Faraday::Adapter, onmainas well as this branch. faraday-net_http 3.5.0 was released on 2026-10-11. Faraday 2.0.1 to 2.14.4 all requirefaraday-net_http >= 2.0, < 3.5, but Faraday 2.0.0 declares no faraday-net_http dependency, so with no lockfile Bundler installed Faraday 2.0.0 with faraday-net_http 3.5.0. The gemspec now requires['>= 1.10.0', '!= 2.0.0'], the same change as upstream crmne/ruby_llmef2bebdb.spec/ruby_llm_gemspec_spec.rbchecks the requirement accepts 1.10.3 and 2.14.4 and rejects 2.0.0.A companion PR in
appfolio/supernova(RubyLLM2 W1-07b) pins Supernova to the commit this PR merges. No other Ruby code changed.Testing
models_json_validation_spec.rbandruby_llm_gemspec_spec.rbpass locally.bundle exec rspecsuite passes locally on a fresh resolve, and all 21 CI test jobs pass.Type of change
models.jsonwas edited by hand on purpose: this is a correction to its price data.🤖 Generated with Claude Code