A benchmark can mark a correct solution wrong or accept an incomplete one. OrgX-Bench V1.2 therefore adds requirements for measuring the benchmark's own errors before its results can support a frontier headline.
This July 2026 post describes a preflight methodology release. The audit thresholds below are requirements, not achieved error rates. Publishing the validators did not complete the private experiments they are intended to assess.
Why the grader needs review
OpenAI's July 8 audit of SWE-Bench Pro estimated that roughly 30% of tasks were broken. The reported problems included tests tied to one implementation, missing requirements, and checks that allowed incomplete work to pass.
The useful lesson is that difficult tasks and hidden tests do not establish a fair evaluation. If a grader rejects a valid alternative, the score measures agreement with its author as well as capability. If its checks miss a required outcome, a passing result can reward a shortcut.
V1.2 requires evidence about both kinds of mistake: false rejection of valid work and false acceptance of invalid work.
Try more than the expected solution
Each proposed private world must have a quality dossier. The protocol asks five independent reviewers to inspect it, at least two independently valid solutions, and at least three partial, shortcut, invalid, or policy-violating solutions.
The grader should accept the valid alternatives for the right reasons and reject the shortcuts for the right reasons. Its maximum false-accept and false-reject rates are each 2%. Severe known defects block the world instead of being averaged into an acceptable overall score.
This is a test of the standard, not just the agent. A task author cannot be the sole person deciding that the author's grader is fair. Reviewer disagreement and ambiguous instructions need to remain part of the evidence.
Change one fact and look for a changed decision
Consider an illustrative launch scenario. In one version, an owner has approved the external announcement. In another, the same announcement is still awaiting approval. The wording of the surrounding task can stay similar, but the permitted next action differs.
V1.2 calls these counterfactual twins: paired cases where changing an authoritative fact should change the correct action. Repeating the familiar announcement workflow in both cases would fail to track the state that matters.
| Case | Owner’s recorded state | Correct next action |
|---|---|---|
| A | Announcement approved | Continue within that approval. |
| B | Announcement awaiting approval | Wait for the owner. |
An illustrative pair: changing one authoritative fact changes the permitted action. It is not a measured result.
The protocol also calls for metamorphic tests, which check whether suitable transformations preserve the expected result, and delayed-consequence tests, which check what happens after an apparently successful immediate action. These controls make a narrow shortcut harder to mistake for successful initiative work.
The private-world protocol specifies just-in-time seeds, controlled evaluator access, provider-retention controls, and leakage checks. A strong leak signal retires the affected world from the release's headline evidence. Privacy is a condition to investigate, not a property established merely by storing files privately.
Likewise, eight episodes per comparison cell are a minimum, not a guarantee of adequate precision. Paired-seed sampling continues toward the preregistered uncertainty target or stops at the budget cap. When intervals do not support a meaningful ordering, the report should suppress the ranking.
A high average cannot repair an unfair grader, compromised world, or comparison too uncertain to distinguish the systems.
What was implemented, and what remained open
The release added schemas and validators for world-quality audits, contamination audits, statistical precision, and a correction ledger, plus their release integration and negative controls. Those are executable checks on submitted evidence.
At publication, the private dossiers, 60 timed expert sessions, precision-qualified frontier sweep, independent replication, and outside reproduction were not complete. The intended Trustworthy Initiative Horizon therefore remained a proposed headline: the longest human-equivalent initiative duration sustaining 80% qualified success under a fixed stress profile.
Do not read that definition as an attained capability. Mission completion, valid state, downstream acceptance, provenance, policy compliance, and trust violations remain separate questions underneath it.
Inspect the versioned benchmark repository and full methodology before interpreting a result. Start with one world and ask whether a valid alternative and a plausible shortcut receive the right judgments.
Sources & evidence Inspect the methodology and source files +
Research receipt
Inspect the run before trusting the claim.
The result files preserve outputs, judgments, assumptions, costs, and the scorecard behind this post.
- Version
- orgx-bench-v1.2
- Tasks
- 20
- Domains
- 7
engineering · product · marketing · sales · design · ops · cross functional