Our Recall Was 0.087 and the Model Was Innocent: How Domain-Scoped Replay Doubled It

The developer behind CauterRule, an open-source tool for automating agent failure recovery, recently addressed a persistent issue where recall metrics remained stubbornly low at 0.087. Initially, the team suspected the matcher's calibration, but further investigation revealed that the issue was the evaluation methodology. By using a global reference pool, domain-specific rules were being unfairly penalized against irrelevant failure cases. By implementing 'domain-scoped replay'—filtering the reference pool to match the candidate's specific domain—the team successfully doubled the recall scores without changing the underlying models or prompts. This shift highlights the importance of evaluation-set composition, demonstrating that uniform poor performance across models often points to an evaluation bug rather than a capability ceiling. The update is available in CauterRule v0.3.0, which includes refined metrics and improved diagnostic capabilities for developers working on agentic workflows.