OrgX-Bench asks whether coordinating agents improves initiative-shaped work compared with one capable agent, after counting quality, elapsed time, cost, and failures. A longer workflow is only better if its result justifies the extra coordination.
This page combines the original March 2026 methodology with the July V1.2 amendment. The distinction matters: public catalog runs demonstrate the measurement mechanism. They are not sufficient evidence for a frontier capability headline.
Compare the same work
The original design compares three modes:
| Mode | Role in the comparison |
|---|---|
| OrgX-orchestrated | Decompose the request, perform specialist work, and synthesize the result |
| Single agent | Give a capable model the task without that orchestration |
| Human reference | Supply a baseline with an explicit collection method and provenance |
These are comparison roles in the protocol, not evidence that any mode has won.
The task, evidence, constraints, and acceptance criteria need to be comparable. A human estimate is not a timed human run. A stronger generation model, a larger token budget, or extra source material can explain a score difference without establishing that orchestration caused it.
The public repository contains the methodology and bundle contract. Use the version attached to a run; catalog sizes, judge panels, and rollout requirements changed after the original twelve-task suite.
Follow one initiative to an outcome
Consider an illustrative launch-planning task. The agent must prepare the brief and supporting work, but an owner decision controls whether an external announcement may be sent.
A polished brief is only one part of success. The evaluator also needs to check the final work state, the evidence behind the output, and whether the system respected the approval boundary. Correctly waiting for a person should stay visible. Sending without the required approval cannot become a success because the copy was good.
V1.2 proposes private, generated organizational scenarios with evaluator-held truth for those checks. Its qualified-success condition combines mission completion, valid final state, downstream acceptance, provenance, policy compliance, and no critical trust violation. A schema-valid result or completed agent turn satisfies none of those conditions by itself.
Inspect several measurements together
A useful report makes the tradeoffs visible:
- Quality: criterion scores and the actual artifact, including disagreements.
- Time: first artifact, decision-ready work, and final completion, with the clock boundary stated.
- Cost: generation, judging, and coordination usage, with estimation assumptions.
- Reliability: repeated outcomes, errors, retries, and work that remained unresolved.
- Human involvement: which approvals or corrections were required.
The original flow multiplier divides the baseline time by the OrgX time. If the numerator is an expert estimate, the ratio inherits that limitation. It cannot establish a customer productivity gain.
Model judges can help inspect qualitative work at scale, but they need calibration. Separate judge calls can still share biases. Pairwise comparisons should conceal system identity and randomize order; the underlying artifacts and criterion judgments should remain available for review. Research on LLM judges documents why position, verbosity, and other biases matter.
Check the benchmark as well as the agent
Repeated runs expose instability that an average can conceal. The public protocol excludes sufficiently unstable tasks from its flow headline and reports the coverage that remains. A result should show losses and uncertainty, not quietly discard inconvenient tasks.
The July V1.2 amendment adds a separate quality gate for the worlds themselves: independent reviewers, multiple valid solutions, plausible invalid shortcuts, measured false acceptance and rejection, and tests that change one authoritative fact. The V1.2 explanation describes these requirements.
Private files are not automatically uncontaminated. The protocol also requires a contamination audit, a rule for retiring affected worlds, sufficient statistical precision, and a public correction process. These are publication conditions, not evidence that the conditions have already been met.
The July amendment reported incomplete private world dossiers, timed-expert coverage, replication, and outside reproduction. This article announces no new headline result and does not turn a public development run into private-holdout evidence.
Choose a bundle in the benchmark hub. Start with a task and its generated artifact. Read the rubric and individual judgments, then compare the summary with the raw scorecard and usage metadata. Check what the human baseline actually represents.
For a published bundle, the public repository documents validation and scorecard recomputation:
node runner/validate-bundle.mjs results/<week>
node runner/recompute-scorecard.mjs results/<week>Those checks establish bundle consistency, not that the work is useful or the benchmark is fair. Keep the result, the evidence supporting it, and the unanswered questions together. The comparison is valuable when a reviewer can identify both where orchestration helped and where it added work without improving the outcome.
Sources & evidence Inspect the methodology and source files +
Research receipt
Inspect the run before trusting the claim.
The result files preserve outputs, judgments, assumptions, costs, and the scorecard behind this post.
- Version
- 2026-q1
- Tasks
- 12
- Domains
- 7
engineering · product · marketing · sales · design · ops · cross functional