The May 30, 2026 benchmark repaired two evaluation problems: an outdated judge panel and incomplete cost estimates. The new run produced 30 artifacts, made 90 judge calls, and flagged 23 artifacts for human review.
Those results describe a public catalog run. They do not show that 30 real initiatives completed without human judgment or that OrgX delivers the reported benchmark ratios on customer work.
What changed between runs
The generator remained gpt-5-nano. The recorded judges changed to gpt-5-nano, gpt-5-mini, and gpt-5.1, with low, medium, and high reasoning. These were the identifiers used in May; they are not a statement about the models available today.
The cost table also gained missing entries for two judges. Previously, unknown prices returned null and could contribute zero to the aggregate. The corrected accounting makes the estimate more complete; it is still a calculation from recorded usage and configured prices, not an invoice reconciliation.
There were other experimental changes. The new bundle used two repeats per task instead of three, and its generator output cap was 3,000 tokens rather than 5,000. The change in review flags therefore cannot be attributed solely to the new judge panel.
What the bundle records
| Item | May 30 result |
|---|---|
| Catalog | 15 tasks, seven domains |
| Repeats | Two per task |
| Generated artifacts | 30 |
| Judge calls | 90 |
| Judge failures | One |
| Human review recommended | 23 of 30, or 76.7% |
| Estimated total model cost | 52.9898 cents |
These counts describe one public catalog run; a generated artifact is not an accepted initiative.
The metadata contains model settings, usage, and the review policy. The summary reports 1.7663 cents per artifact for generation plus judging. Judging accounts for most of the total.
These figures are useful for understanding the experiment's cost boundary. They exclude the broader cost of developing the benchmark, human adjudication, and operating a production workflow.
Read the review flag before the completion rate
The summary records a 100% completion metric. Alongside it, the same bundle recommends human review for most artifacts. Those values answer different questions: whether the run produced its outputs and whether the assessment warranted further judgment.
For example, a generated design specification may be complete enough to score while judges disagree about whether an engineer could implement it. Treating output production as accepted delivery would erase the disagreement that the review flag preserves.
Open the artifact in examples.json, find its criterion scores in judgments.json, and inspect the corresponding baseline in tasks.json. The recorded quality difference is against that baseline, which may be an expert estimate. It is not a blinded, timed comparison with a representative workforce.
How autonomy should be counted
The methodology amendment distinguishes clean autonomous completion from merely producing an artifact. The required work must reach an approved state through a permitted non-human path. Work requiring human approval stays visible and can be scored, but cannot count as autonomous completion under that definition.
OrgX's autonomy policy separates Manual, Gated, and Autopilot modes. It retains a human boundary for consequential actions, and an agent cannot raise its own autonomy or approve its own work. That policy is a product mechanism. Establishing that a particular benchmark run cleared it requires the artifact's actual approval-path evidence.
A fluent completion message or a high judge score cannot supply a missing approval path. Nor should correctly waiting for a person be treated as a trust failure.
The panel's thresholds still needed human calibration. Baselines had limited samples and mixed collection methods. Public catalog tasks were development evidence, not private-holdout frontier evidence. The reported speed and quality ratios should be read within those limits.
Compare the April judged run with this bundle's raw files, including the changed repeat count and output cap. Then ask whether the evaluation and approval evidence support the particular claim you want to make. A repaired cost estimate is progress in measurement, not proof that the underlying work is ready to ship.
Sources & evidence Inspect the methodology and source files +
Research receipt
Inspect the run before trusting the claim.
The result files preserve outputs, judgments, assumptions, costs, and the scorecard behind this post.
- Version
- 2026-q1
- Tasks
- 15
- Domains
- 7
engineering · product · marketing · sales · design · ops · cross functional