AI agents now beat humans at desktop tasks. On long jobs the best finishes 31%.
Two kinds of benchmark, two very different answers, and almost every AI pitch you receive quotes the flattering one. Here is what each actually measures, and which one describes your firm.
Published 6 min read Sources checked 24 September 2026
Filed under AI agents, OSWorld benchmark, Agent reliability, Enterprise AI
The short answer
On OSWorld-Verified (369 bounded desktop tasks) the best model scores 86.0%, above the 72.36% human baseline. On OSWorld 2.0, the same team’s 108 long-horizon tasks, 69.6% of them over an hour long, the best agent completes 31.4%. On TheAgentCompany, a simulated firm, the best checked system resolves 42.9%. Same technology, different job shape. Scope decides which number you get.
On this page · 8 sections
You will be shown the 86%. You will not be shown the 31%. Both are real, both are current, and the difference between them is not a difference in the models.
What is OSWorld actually measuring?
369 tasks on a real desktop, each one execution-verified: open this, change that, save it there. Bounded, single-sitting, checkable by a script.
A model scoring 86.0% on that is genuinely impressive, and it is above what the original paper measured humans at. An agentic framework (several models wired together) reaches 90.2%. If your work looks like those tasks, buy the thing.
What do the long-horizon benchmarks measure?
The same team built OSWorld 2.0 in June 2026 to measure the other kind of work: 108 tasks across 31 self-hosted websites, 69.6% of them over an hour long, averaging more than 250 agent steps. The best agent completes 31.4% of them.
TheAgentCompany, from Carnegie Mellon, is a simulated company: software, HR and finance work that runs long and crosses applications. Its paper found the most competitive agent completing 30% of tasks and concluded that “more difficult long-horizon tasks are still beyond the reach of current systems”. Its leaderboard’s best checked result is now 42.9%. Better, and still well under half.
A broker case is fifteen to twenty rounds of messages across five or more people. That is not a desktop task. That is a long-horizon task, and long-horizon is the column where the best number is 31%.
Why does the number collapse?
Because errors compound and nobody is watching.
A bounded task fails visibly: the file did not save, you can see it. A long-horizon task fails at step four and produces a confident, complete-looking result at step eleven that is built on the step-four mistake. Nothing errors. Something is simply wrong.
| What is being compared | Bounded task | Long-horizon task |
|---|---|---|
| Example | Reformat this spreadsheet | Package and submit this case |
| Steps | Under ten, one sitting | Dozens, over days |
| Failure looks like | An error you can see | A plausible wrong answer |
| Who notices | You, immediately | The lender, in a week |
| Best benchmark score | 86.0% | 31.4% |
So how do you get the 86% number in a real firm?
By refusing to buy the long-horizon version. You cut the long task into bounded ones and put a named person at each join.
- Find the joins. Where does the work change hands, systems or days? Those are your checkpoints, and they already exist informally.
- Scope each agent to one leg. Not “handles the case”. “Assembles the pack for lender X and stops.”
- Make the handover explicit. A person confirms, or it does not proceed. This is the whole difference between the two columns above.
- Log every action. When something is wrong in a week, you need to see which leg produced it.
- Measure per leg. An agent that is right 95% of the time on one bounded leg is worth buying. The same agent trusted end-to-end is not.
What should you ask a vendor?
One question: on what benchmark, and is the task shape like mine?
If the answer is a percentage with no benchmark named, it is marketing. If the benchmark is bounded and your work is long-horizon, the number is real and irrelevant. Neither is dishonest exactly, but only one of them is about you.
What this does not tell you
How an agent will do on your work. These benchmarks score tasks with automatic checkers, and TheAgentCompany is a simulated firm built by researchers; none is your case file. They measure whether a task finished, and not what it cost or how bad the failures were.
OSWorld 2.0 does publish a partial score (the best agent reaches 68.3% of its checkpoints) which says most failures are not total: the work gets most of the way and then goes wrong, which is exactly why a person belongs at each join.
The leaderboards also move. They are re-read every 30 days for that reason, and the date beside each is the date it was true.
Corrected 24 September 2026. This note gave the best OSWorld score as 86.1%, by Qwen3.8-Max. Read in a browser today, the leaderboard shows no such entry: its best model scores 86.0% and an agentic framework 90.2%, and we cannot reproduce our own figure. It also gave TheAgentCompany’s 30% as current; that is the paper’s December 2024 result, and the benchmark’s leaderboard now shows 42.9%. The long-horizon figure here is now OSWorld 2.0’s, from the same team as the first. The entry is on the register.
Questions people actually ask
How accurate are AI agents in 2026?
It depends entirely on task shape. On OSWorld-Verified, 369 bounded desktop tasks, the best model scored 86.0% when read on 24 September 2026, above the 72.36% human baseline. On OSWorld 2.0, 108 long-horizon tasks from the same team, the best agent completed 31.4%, and on TheAgentCompany, a simulated firm, the best checked system resolved 42.9%.
What is OSWorld?
A benchmark of 369 execution-verified desktop tasks used to measure computer-use agents. Because the tasks are bounded and checkable, scores on it are much higher than on benchmarks that test long, multi-application work, including OSWorld 2.0, the same team’s long-horizon version, launched in June 2026.
Why do AI agents fail at long tasks?
Errors compound without being visible. A mistake at step four produces a confident, complete-looking output at step eleven. Nothing throws an error, so nobody notices until a downstream party does.
Sources
- 01 OSWorld-Verified leaderboard, read in a browser on 24 September 2026 (the table is loaded by script; os-world.github.io now redirects here). “Foundation E2E GUI” view, first: claude-fable-5 (Anthropic), 1 August 2026, general model, 86.0%; second, claude-opus-5, 83.4%. “All” view, first: Intelligence-Indeed Agent, an agentic framework, 25 July 2026, 90.2%. No entry reads 86.1%. Human baseline from the original paper: “humans can accomplish over 72.36% of the tasks”. osworld-v1.xlang.ai(opens in a new tab)
Weight: public leaderboard, primary Re-checked: every 30 days Last read: 24 September 2026
- 02 OSWorld 2.0, “Benchmarking computer-use agents on long-horizon real-world tasks”, launched 26 June 2026: 108 long-horizon tasks, 31 self-hosted websites, 69.6% of tasks over an hour, more than 250 agent steps on average. Leaderboard read 24 September 2026: best binary completion 31.4% (Claude Opus 5, maximum effort, version of 8 August 2026), partial score 68.3%. At launch the best was 20.6%. osworld-v2.xlang.ai(opens in a new tab)
Weight: public benchmark and leaderboard, primary Re-checked: every 30 days Last read: 24 September 2026
- 03 TheAgentCompany leaderboard, read 24 September 2026: best checked result 42.9% resolved (TTE-MatrixAgent with DeepSeek-V3.2, 10 November 2025); “Checked” means the benchmark’s team reproduced it. the-agent-company.com(opens in a new tab)
Weight: public leaderboard, primary Re-checked: every 30 days Last read: 24 September 2026
- 04 TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks, Carnegie Mellon, arXiv:2412.14161 (at publication, “the most competitive agent can complete 30% of tasks autonomously”). arxiv.org(opens in a new tab)
Weight: peer-reviewable preprint, primary Re-checked: never: it cannot go out of date Last read: 24 September 2026
Checked 24 September 2026. Next scheduled check 24 October 2026. Numbers that move (leaderboards, live indices) are re-checked every 30 days; annual datasets and rules in force every six months; dated research once a year. If something here has gone stale before we got to it, tell us and we will correct it and say what changed.
Cite this note
Noxia, “AI agents now beat humans at desktop tasks. On long jobs the best finishes 31%”, Field notes, 8 September 2026; sources checked 24 September 2026. https://www.noxia.co.uk/field-notes/agents-beat-humans-and-still-finish-30-percent
Want your long process cut into legs an agent can actually finish?
That is the work: find the joins, scope one leg, put a person on the handover. We quote it as a fixed build, and you can measure it per leg.