Build a small set of coding tasks that frontier models fail for a real reason.
What we want from you
Three tasks from one task family you choose. Each task meets the acceptance bar below, and each comes from a different real codebase. We run strong models on every task you send, and the tasks that count are the ones those models fail.
Why three, and why different codebases: one hard task can be luck. Three hard tasks in three codebases show that you understand why the family is hard, and that you can find that difficulty again in code you have not seen before. That is the skill we are looking for. Three good tasks are worth more to us than more tasks of lower quality, so retire a weak task rather than keep it to reach the number.
Some details depend on your family, so we agree them with you after you send your family brief (Step 1). Do not run Sol xhigh yourself.
How it works
- Pick a family and write a one-page brief. We agree it with you.
- Build one task and send it. We run strong models on it and review it against the bar. You fix what we find.
- Build the rest. Pilot each one against a strong model as you go. Send each task when it is ready; do not wait to send them all at once.
- We run the panel on every task and send you rewards, full trajectories and our review.
- One revision round. Harden, repair or retire each task, then send the final set with a short report.
- Walkthrough. 30 minutes on your brief, your tasks and what you changed.
Step 1: Pick a task family
A task family is a kind of work, in a kind of codebase, that frontier agents get wrong for a reason that holds up. Pick one you know well, with real codebases you can build and run. Each family below has a public benchmark you can start from. Read its tasks and model trajectories, and find where models fail. Use it as a baseline to critique, not a source to copy.
| Family | What a real task looks like | Public benchmark to start from |
|---|---|---|
| Mobile | A feature or fix in a published cross-platform, iOS or Android app | Android Bench |
| Game development | A feature or fix in a game project that builds and runs | GameDevBench |
| 3D | Work in a real 3D toolchain or on real 3D data, checked numerically | BlenderGym |
| Your own | Any other kind of real software work you know well | Find one at https://benchmarklist.com/ |
The family brief
One page that answers these questions for your family. It sets what the acceptance bar means for your tasks, so we agree it before you build.
| Question | What we want to know |
|---|---|
| Codebases | Which real projects you will use (one per task), why they are realistic, and that you have the right to use them |
| Work | What a professional actually does in these codebases |
| Failure mechanism | Why strong agents fail at this work, from benchmark trajectories you have read |
| Verifier | How you will judge the result automatically, without a human looking at it |
| Environment | How each task builds offline and reproducibly |
| Attack surface | How an agent could fake success in this family, and how you will stop it |
| Out of scope | What you will not build |
The acceptance bar
A task counts only if it meets all ten criteria. Your brief says what each one means for your family.
| # | Criterion | What it means |
|---|---|---|
| 1 | Hard | Sol passes at most 2 of 8 trials. See "Difficulty" below. |
| 2 | Fair | Every failure is a real mistake in the agent's work. No check is stricter than the instruction: your reference plus any change the instruction allows still passes. |
| 3 | Cannot be gamed | No shortcut, test detection or tampering earns reward, including the attacks in your brief. |
| 4 | Isolated verifier | Grading runs in its own environment, from only the agent's changes to the editable paths. |
| 5 | Natural instruction | Reads like a real request. Does not list every rule the verifier checks, or describe how grading works. |
| 6 | Correct reference and controls | The reference gets full reward. The unmodified project and plausible shortcut solutions get zero. |
| 7 | Real and original | A real codebase or real data from your brief, and work a professional would actually do. Not a toy, not synthetic data, and not copied from a published benchmark. You have the right to use every codebase and asset, and you list each one with its license. |
| 8 | Reproducible | Self-contained, no network, dependencies and base images pinned by digest, builds on amd64 and arm64 where the stack allows. |
| 9 | Complete metadata | Family, platform, language, base repository and commit as structured fields in task.toml. |
| 10 | Distinct | Each task uses a different codebase and tests a different behavior. Together, your tasks show the range of the family, not three versions of the easiest kind. |
Difficulty
Difficulty is the criterion most tasks fail, so here is exactly how we measure it.
- Final test: we run Sol xhigh 8 times on each task. The task is hard if Sol passes at most 2 of the 8.
- Your own check: you may not run Sol, so before you send a task, run the strongest model you can access 4 times. Aim for at most 1 pass in 4. Report the model, the settings and each result.
- The failure has to be the right kind. A failure counts only if the agent made a real mistake in the work. Failures from a broken environment, a missing dependency, an unclear instruction or a timeout do not count, and they make the task invalid (criterion 2).
- Weaker models failing is not evidence. If only weaker models fail, the task is not hard.
- Know why it is hard. For each task, state the failure mechanism and point to the trajectory steps that show it. A task that is hard for a reason you cannot explain is luck.
How we judge the take-home
- Valid: all three tasks meet criteria 2 to 10. A task that does not is retired and does not count.
- Hard: at least one task meets criterion 1. Two or three is a strong result.
- Judgment: your predicted Sol pass counts and your report. A forecast that matches what Sol does, with a clear reason for each task, counts in your favor. A wrong forecast with good reasoning is better than a right one by luck.
- We probe every task for fairness and for ways to score without solving it. Assume we will try.
Building a task
Design for difficulty first
Most tasks fail the bar on difficulty, not packaging. So test difficulty before you polish:
- Pick a behavior where the codebase, not the instruction, holds the hard part.
- Write the instruction and a rough verifier.
- Run the strongest model you can access 2 to 4 times. If it passes easily, redesign. Weaker models failing is not evidence.
- Only then write the controls, the reference solution and the documentation.
If the instruction states every rule the verifier checks, the task is a transcription exercise, not a hard task.
The package (Harbor format)
| Artifact | Contents |
|---|---|
instruction.md |
A short request in the voice of a user, PM or teammate: the context, what should change, and what "done" looks like. State only what a real requester would state. Do not prescribe the implementation, list every check, or mention tests, resets or the grading command. |
environment/ |
The pinned codebase, agent-visible inputs, pinned dependencies and base image, and a reproducible offline setup. |
solution/ |
A reference patch, with evidence that it builds and gets full reward. Keep it out of anything the agent can see. |
tests/ |
A headless, behavioral verifier with a result per check. It grades in its own environment: it applies only the agent's changes to the editable paths onto a clean copy of the base. The agent must not be able to change anything the verifier relies on. Include missing-feature and shortcut controls. |
task.toml |
Identity, family, platform, language, stack, base repository and commit, resources, network policy, timeouts, reward definition, and your predicted Sol pass count out of 8. |
Use a binary reward with a diagnostic result for each check.
Prove it is valid
For each task, show that:
- the reference gets full reward from a clean state;
- the unmodified project and every control get zero;
- a different but valid solution also passes;
- each attack in your brief gets zero, and so do version-control hooks or config, build-tool init scripts, and files outside the editable paths;
- your pilot runs, with the model you used and its results.
After the panel: revise and report
We send per-trial rewards, full trajectories and our review for every task. Harden, repair or retire each one. Do not keep a weak task only to reach three. Then send the final set and a one-page report.md:
- what you changed after the panel, and why;
- for each task: the failure mechanism, the trajectory steps that show it, and your final predicted Sol pass count out of 8;
- which tasks you retired, and why;
- how you used AI tools to generate, implement, critique and test.
Final evaluation
We run 8 Sol xhigh trials per final task from a clean state. A task counts when it meets all ten criteria and Sol passes at most 2 of 8. We rerun infrastructure failures, and we audit model failures for ambiguity, missing dependencies, unfair grading and shortcuts. We also compare your predicted pass counts with the results.
Guidance
Fair, real and hard, together. Fair: the agent has everything it needs, and every correct solution passes. Real: a developer would actually do this work. Hard: a strong model fails for a meaningful reason. Missing inputs, ambiguous instructions or broken environments make a task look hard without making it valid.
Hidden is not unfair. You may leave a rule out of the instruction if a competent developer would find it in the code, the docs or by running the software. If they could not, it is an ambiguity and fails criterion 2.
Read trajectories, not only rewards. Find where the model misread the requirement, missed a dependency, picked the wrong approach or did not check its work. Tell capability failures apart from unfair tasks and infrastructure failures.
Using AI tools. You may use AI for any part of the work. Document how you used it, and check the results yourself.