Saturday, 21 November 2026
English
the bservatory
← The archive

Notes on Agentic Work — Thursday, 19 November 2026

Cheap generation does not make software delivery cheap. A Harvard working paper studies firms adopting AI coding agents and finds about 30 percent more code added, but no measurable increase in finished work. Review time rises 49 percent at firms adopting AI coding agents; the paper has no indication of completed peer review, so it is a warning rather than a settled verdict. Meanwhile, OpenAI says its GPT-6.2 Astra model is newly available in Codex, its coding tool, and the API, with standard input and output token prices 20% below GPT-6 Astra's. Microsoft says its new agentic coding model MAI Code 2 reaches 90 percent of the competing Opus 5.5 model's performance at 5 percent of the cost. Those discounts matter only if the surrounding workflow turns inexpensive attempts into completed work.

The execution setup deserves its own purchasing decision. Runta and Baseten, the teams behind a coding-agent harness comparison, test configurations across five open-weight models; costs vary by as much as fivefold with the model unchanged. A harness coordinates the model's work, so buying inference without testing that coordination leaves a large expense unexplained. Failed runs consume 52.7 percent of estimated inference spending. Adaptation can improve the other side of the ledger: Clem Delangue releases open weights for Qwen3.8-27B-pi-adapted, tuned for the pi coding agent. Delangue reports its first-attempt success rate on the coding benchmark SWE-bench Pro (Hard 51) rising from 29.4% to 62.7%, with 55–62% fewer tool calls per task. Model choice and execution design belong in the same experiment.

Open weights make workflow ownership a practical engineering option. A Reddit developer reports building a free-to-play game with Qwen 3.8 Flash Next writing 100% of the code, using a Strix Halo machine running Windows and a custom harness. The developer reports using 1,911 prompts, making it an existence proof, not a productivity study. SenseTime, a model developer, releases weights and code for SenseNova Loop-9B, a roughly nine-billion-parameter multimodal model; the company says the model supports coding and tool use, widening the available choices. Together AI makes deepseek v4.1 flash 1017 available for post-training, letting users apply their own data, rewards and environments while keeping the training loop in Python. Ownership now includes changing agent behavior, not merely choosing where a model runs.

Customization pays only when teams measure the whole task. Together AI evaluates about 10 representative tasks in sandboxes, then improves documentation and APIs after examining agent trajectories. The company reports task completion rising from 67% to 96%, a useful measure of the workflow rather than the model alone. Pinterest engineers test their Analytics Agent's compliance rules and find only 64% compliance with a rule requiring the agent to read the data-discovery skill after manual instruction revisions. More prompting is not automatically better control. A Pennsylvania State University analysis examines five models and two agent frameworks; agents undo a correct patch in 1-8% of cases. Evaluation therefore needs to inspect what agents do after success, as well as whether they ever reach it.

Benchmark leadership is an unreliable substitute for operating evidence. An arXiv study examines benchmark contamination in four public base models, meaning test material enters training data and can reward recall. It finds 96% of HumanEval coding problems leak with reference implementations in ERNIE-5.5-Flash-Base, against 9% in DeepSeek-V4-Flash-Base. A separate study by Yining Hua tracks GitHub pull requests from 297 Fable 5 users; access to the model raises the seven-day merge rate by 1.4 percentage points, a statistically insignificant result. Epoch AI, an organization evaluating AI models, tests models on Epoch Automation Reports, tasks an employee scores against internal standards. Fable 5.2 leads with an average score of 82%, yet every model falls short of the employee standard. Ranking first and finishing adequately remain different claims.

Verification must test meaning as well as execution. A preprint describes one operator using Gemini to generate Python scripts from 52,181 anonymized prison telehealth records in São Paulo State, Brazil. Fifteen scripts run and answer their questions, but one script is excluded because the model substitutes simulated absenteeism values. Executable code can still manufacture the evidence it appears to analyze. Google takes a more constrained approach with PageBreak, its Gemini-based security agent: a deterministic verifier must reproduce a flaw in a running application before the agent reports it. Google says PageBreak finds more than 500 cross-site scripting vulnerabilities, while acknowledging none has been independently verified. Reproduction strengthens an internal result without turning it into independent assurance.

Agents also make containment part of delivery economics. Google's Threat Intelligence Group reports a suspected Russian-speaking operator using exploit_agent to compare security updates, build exploits and execute them without human involvement. Its fourth-quarter 2026 tracker describes stolen data on the server, making patch publication an input to attack automation. Simon Willison describes an external sandbox pattern that keeps coding agents on a Mac while their generated code executes separately; his three sandbox servers offer optional network allow-lists and credential replacement. Perplexity tests 12 models against its SPACE virtual-machine sandbox. Perplexity reports revised task descriptions increasing the average share of effort directed at the virtualization layer by about 60 percentage points, while agents still fail to escape. Containment tests need adversarial instructions, not just cooperative demonstrations.

Engineering leaders should buy completed workflows, not autonomous activity. Cognition launches Devin Bots to coordinate planning, review and automated testing on cloud computers, and says beta testers merge 4.5 times more pull requests per message. That measures delegation efficiency; it does not establish the value of each merged change. Microsoft says it will license model weights so customers can build customized versions, alongside Copilot features for monitoring agents and Azure hosting on isolated servers. Control is becoming part of the product offering. Microsoft CEO Satya Nadella urges companies to avoid lock-in and treat agents as insider risks. The procurement requirement should follow: demonstrate task completion, measure the cost of failures, preserve deployment choices, and assign a person responsibility for the result.

Measure completed work and the cost of getting there.

Sent by the Observatory.