"OpenAI-compatible" is a spectrum, not a boolean — here's an 11-check conformance suite

Many API endpoints claim to be "OpenAI-compatible," but implementation details often vary, leading to subtle bugs in production applications. Developer and gateway maintainer Seven7763 argues that compatibility is a spectrum rather than a binary state. To address this, the author provides an 11-check Python conformance suite designed to test critical surfaces, including streaming deltas, tool calling, JSON mode, logprobs, and error-body structures. The article emphasizes that developers should not assume uniform behavior across different providers. Instead, they should integrate these automated checks into CI pipelines to detect regressions early. By treating "partial" compatibility as a design input rather than a failure, teams can build more resilient systems. The author also recommends pinning model IDs as dependencies and monitoring for drift to ensure that production traffic remains stable when relying on third-party LLM endpoints.