We audited 110 AI usage tools. Here is where the numbers go wrong.

A recent audit of 110 open-source AI usage tracking tools has uncovered significant inaccuracies in how AI consumption is measured and billed. The study identified 45 verified bugs across five recurring categories, including stale pricing tables, incorrect cache multipliers, and flawed stream aggregation that leads to double-counting. Researchers found that some tools incorrectly treat missing data as zero, effectively providing unauthorized discounts, while others suffer from window boundary errors. The audit resulted in 23 upstream fixes and highlighted a lack of transparency among commercial vendors, who currently lack standardized dispute processes for billing discrepancies. To address these issues, the researchers released an open-source conformance suite, AgentMeasure, which allows users to independently verify their AI usage data. The findings emphasize the need for better accountability in usage-based billing models as AI adoption shifts from flat subscriptions to token-based pricing.