Diffuse Labs: Engineering Take-home - Notes & Reports

Build a small set of coding tasks that frontier models fail for a real reason.

What we want from you

Three tasks from one task family you choose. Each task meets the acceptance bar below, and each comes from a different real codebase. We run strong models on every task you send, and the tasks that count are the ones those models fail.

Why three, and why different codebases: one hard task can be luck. Three hard tasks in three codebases show that you understand why the family is hard, and that you can find that difficulty again in code you have not seen before. That is the skill we are looking for. Three good tasks are worth more to us than more tasks of lower quality, so retire a weak task rather than keep it to reach the number.

Some details depend on your family, so we agree them with you after you send your family brief (Step 1). Do not run Sol xhigh yourself.

How it works

  1. Pick a family and write a one-page brief. We agree it with you.
  2. Build one task and send it. We run strong models on it and review it against the bar. You fix what we find.
  3. Build the rest. Pilot each one against a strong model as you go. Send each task when it is ready; do not wait to send them all at once.
  4. We run the panel on every task and send you rewards, full trajectories and our review.
  5. One revision round. Harden, repair or retire each task, then send the final set with a short report.
  6. Walkthrough. 30 minutes on your brief, your tasks and what you changed.

Step 1: Pick a task family

A task family is a kind of work, in a kind of codebase, that frontier agents get wrong for a reason that holds up. Pick one you know well, with real codebases you can build and run. Each family below has a public benchmark you can start from. Read its tasks and model trajectories, and find where models fail. Use it as a baseline to critique, not a source to copy.

Family What a real task looks like Public benchmark to start from
Mobile A feature or fix in a published cross-platform, iOS or Android app Android Bench
Game development A feature or fix in a game project that builds and runs GameDevBench
3D Work in a real 3D toolchain or on real 3D data, checked numerically BlenderGym
Your own Any other kind of real software work you know well Find one at https://benchmarklist.com/

The family brief

One page that answers these questions for your family. It sets what the acceptance bar means for your tasks, so we agree it before you build.

Question What we want to know
Codebases Which real projects you will use (one per task), why they are realistic, and that you have the right to use them
Work What a professional actually does in these codebases
Failure mechanism Why strong agents fail at this work, from benchmark trajectories you have read
Verifier How you will judge the result automatically, without a human looking at it
Environment How each task builds offline and reproducibly
Attack surface How an agent could fake success in this family, and how you will stop it
Out of scope What you will not build

The acceptance bar

A task counts only if it meets all ten criteria. Your brief says what each one means for your family.

# Criterion What it means
1 Hard Sol passes at most 2 of 8 trials. See "Difficulty" below.
2 Fair Every failure is a real mistake in the agent's work. No check is stricter than the instruction: your reference plus any change the instruction allows still passes.
3 Cannot be gamed No shortcut, test detection or tampering earns reward, including the attacks in your brief.
4 Isolated verifier Grading runs in its own environment, from only the agent's changes to the editable paths.
5 Natural instruction Reads like a real request. Does not list every rule the verifier checks, or describe how grading works.
6 Correct reference and controls The reference gets full reward. The unmodified project and plausible shortcut solutions get zero.
7 Real and original A real codebase or real data from your brief, and work a professional would actually do. Not a toy, not synthetic data, and not copied from a published benchmark. You have the right to use every codebase and asset, and you list each one with its license.
8 Reproducible Self-contained, no network, dependencies and base images pinned by digest, builds on amd64 and arm64 where the stack allows.
9 Complete metadata Family, platform, language, base repository and commit as structured fields in task.toml.
10 Distinct Each task uses a different codebase and tests a different behavior. Together, your tasks show the range of the family, not three versions of the easiest kind.

Difficulty

Difficulty is the criterion most tasks fail, so here is exactly how we measure it.

How we judge the take-home

Building a task

Design for difficulty first

Most tasks fail the bar on difficulty, not packaging. So test difficulty before you polish:

  1. Pick a behavior where the codebase, not the instruction, holds the hard part.
  2. Write the instruction and a rough verifier.
  3. Run the strongest model you can access 2 to 4 times. If it passes easily, redesign. Weaker models failing is not evidence.
  4. Only then write the controls, the reference solution and the documentation.

If the instruction states every rule the verifier checks, the task is a transcription exercise, not a hard task.

The package (Harbor format)

Artifact Contents
instruction.md A short request in the voice of a user, PM or teammate: the context, what should change, and what "done" looks like. State only what a real requester would state. Do not prescribe the implementation, list every check, or mention tests, resets or the grading command.
environment/ The pinned codebase, agent-visible inputs, pinned dependencies and base image, and a reproducible offline setup.
solution/ A reference patch, with evidence that it builds and gets full reward. Keep it out of anything the agent can see.
tests/ A headless, behavioral verifier with a result per check. It grades in its own environment: it applies only the agent's changes to the editable paths onto a clean copy of the base. The agent must not be able to change anything the verifier relies on. Include missing-feature and shortcut controls.
task.toml Identity, family, platform, language, stack, base repository and commit, resources, network policy, timeouts, reward definition, and your predicted Sol pass count out of 8.

Use a binary reward with a diagnostic result for each check.

Prove it is valid

For each task, show that:

After the panel: revise and report

We send per-trial rewards, full trajectories and our review for every task. Harden, repair or retire each one. Do not keep a weak task only to reach three. Then send the final set and a one-page report.md:

Final evaluation

We run 8 Sol xhigh trials per final task from a clean state. A task counts when it meets all ten criteria and Sol passes at most 2 of 8. We rerun infrastructure failures, and we audit model failures for ambiguity, missing dependencies, unfair grading and shortcuts. We also compare your predicted pass counts with the results.

Guidance

Fair, real and hard, together. Fair: the agent has everything it needs, and every correct solution passes. Real: a developer would actually do this work. Hard: a strong model fails for a meaningful reason. Missing inputs, ambiguous instructions or broken environments make a task look hard without making it valid.

Hidden is not unfair. You may leave a rule out of the instruction if a competent developer would find it in the code, the docs or by running the software. If they could not, it is an ambiguity and fails criterion 2.

Read trajectories, not only rewards. Find where the model misread the requirement, missed a dependency, picked the wrong approach or did not check its work. Tell capability failures apart from unfair tasks and infrastructure failures.

Using AI tools. You may use AI for any part of the work. Document how you used it, and check the results yourself.