Instrument 13 / task framing

Does your AI task have an error signal?

Eight checks separate work an AI system can execute against a target from work that still requires a human to invent the target.

Inspect one task

Can the system tell when it is wrong?

A strong model cannot optimize against a target that nobody has defined. Check each statement that is already true for the workflow.

T1The desired end state is explicitA human can describe success before seeing the model output. E2Trusted evidence is availableThe workflow can inspect a source, dataset, environment, or observation that is not generated by the model itself. J3Correctness has an independent judgeA test, schema, measurement, or reviewer can reject a plausible but wrong result. F4Feedback arrives in time to correctThe signal is available before the next costly or irreversible action. C5Failure and edge cases are representedThe evaluation includes the cases that would make the workflow unsafe, expensive, or misleading. I6The workflow can test an ideaIt can run a safe experiment or tool action instead of only producing another explanation. B7Actions and resources are boundedTools, permissions, data, time, attempts, and spend have explicit limits. P8A human owns premise changesThe system may propose a new goal or assumption, but a named person is accountable for accepting it.

Decision model

No target: human jump.
The workflow is being asked to choose the premise, not execute against one.

No judge or cadence: weak signal.
The system cannot correct itself reliably.

Target + signal + bounds: execution-ready.
Start small and verify the trace before increasing autonomy.

Task-framing guide

A model can optimize only after the target exists.

This diagnostic applies one practical lesson from Tom Zahavy’s position paper: current systems are strongest where a target, test, proof, or metric already exists. The checklist does not claim that open-ended discovery is impossible; it shows where a workflow still depends on human premise selection.

Read the ResearchAudio analysis: Einstein Had No Error Signal →

What is an error signal in an AI system?

It is observable feedback that measures the gap between an output and a defined target. A useful signal tells the workflow whether to continue, correct, retry, or stop.

What makes a task suitable for an AI agent?

A suitable task has an explicit end state, trusted evidence, an independent judge, timely feedback, representative failure cases, bounded actions, and clear ownership of assumptions.

Can an LLM choose the goal for an open-ended task?

It can propose goals, but a proposal is not proof that the goal is correct. Keep a human accountable when the success criteria or premises must be invented.

Is this AI task-fit score a benchmark?

No. It is a framing aid, not a measure of model intelligence or a guarantee of production performance. Verify the design with real trials and inspectable evidence.

Primary source

Read the argument behind the diagnostic.

This worksheet is an applied interpretation of the task-framing problem, not a claim made verbatim by the paper.