Inferock Bench: the company that bills you also decides what counts as a failure
That line is the entire product thesis, and it's uncomfortable because it's just true. When an LLM call dies mid-stream, when a reply comes back empty with tokens billed against it, when a retry silently doubles a charge β the only detailed record of what happened is held by the party charging you for it. Inferock Bench is a local proxy that keeps its own. It took the top slot on Product Hunt on August 15 with 241 upvotes. Source at https://github.com/inferock/inferock-bench
Setup is two lines of config. It sits between your app and OpenAI, Anthropic, Gemini or pinned OpenRouter calls, and you point your existing SDK at it by changing apiKey and baseURL. Nothing else in your code changes. Run it with npx inferock-bench and it starts capturing every call as an independent per-call record with token usage, failures and retries. License is FSL-1.1-Apache-2.0, which converts to plain Apache-2.0 after two years.
What it's specifically hunting for is worth listing, because each item is a real failure mode people have hit and been unable to prove. Answers truncated mid-stream that still got billed in full. Empty replies with token counts attached. Token counts that don't match the visible output. Retries that doubled a charge without anyone noticing. Cache discounts that never showed up on the invoice.
Every one of those gets worse in agent workloads and it isn't close. A chat app makes one call a user can eyeball. An agent makes thousands per task, most of them unread by any human, many of them retried automatically by a framework that considers retrying its job. The exact conditions where silent billing errors compound are the exact conditions agents run in, and the volume that makes an independent ledger worth having is the volume that makes checking by hand impossible.
The natural pairing is Anthropic's own session token economics writeup from yesterday, where a vendor documented in detail how its harness bills you. That was genuinely useful and also self-reported. Inferock is the other half β an independent check on the same numbers. Both should exist, and the fact that we're only now getting the second one, years into paying per token, says something about how little scrutiny the invoice has gotten.
← Back to all articles
Setup is two lines of config. It sits between your app and OpenAI, Anthropic, Gemini or pinned OpenRouter calls, and you point your existing SDK at it by changing apiKey and baseURL. Nothing else in your code changes. Run it with npx inferock-bench and it starts capturing every call as an independent per-call record with token usage, failures and retries. License is FSL-1.1-Apache-2.0, which converts to plain Apache-2.0 after two years.
What it's specifically hunting for is worth listing, because each item is a real failure mode people have hit and been unable to prove. Answers truncated mid-stream that still got billed in full. Empty replies with token counts attached. Token counts that don't match the visible output. Retries that doubled a charge without anyone noticing. Cache discounts that never showed up on the invoice.
Every one of those gets worse in agent workloads and it isn't close. A chat app makes one call a user can eyeball. An agent makes thousands per task, most of them unread by any human, many of them retried automatically by a framework that considers retrying its job. The exact conditions where silent billing errors compound are the exact conditions agents run in, and the volume that makes an independent ledger worth having is the volume that makes checking by hand impossible.
The natural pairing is Anthropic's own session token economics writeup from yesterday, where a vendor documented in detail how its harness bills you. That was genuinely useful and also self-reported. Inferock is the other half β an independent check on the same numbers. Both should exist, and the fact that we're only now getting the second one, years into paying per token, says something about how little scrutiny the invoice has gotten.
Comments