I Sell Memory APIs. I'm Also Building the Benchmark. Here's How I'm Trying Not to Rig It.

Woochan, a developer at Wontopos, has introduced a new, transparent benchmark designed to evaluate the memory capabilities of LLMs. Addressing widespread industry skepticism regarding vendor-published performance metrics, the project aims to establish a fair, reproducible standard. The benchmark utilizes a massive corpus of 1.9 million tokens, testing models across 14 axes, including stale fact detection, contradiction handling, and multilingual recall. To prevent bias, the project is open-source under the Apache 2.0 license, and the author has implemented strict rules, such as mandatory publication of per-question records and independent verification. The benchmark also introduces a nuanced approach to measuring latency, separating network overhead from actual compute performance. By prioritizing transparency and community-driven validation, the author seeks to provide a reliable way for developers to compare memory APIs without the influence of potentially rigged marketing claims.