Duolingo: Agents Merged 225,000 Lines of iOS Unit Tests

How Duolingo's Agents Wrote and Merged 225,000 Lines of iOS Tests

Kush Agrawal is a staff engineer on Duolingo's platform team, building developer tooling and AI workflows for the 100-plus engineers who work on its client apps. Before Duolingo he spent close to a decade at Stripe and Uber. The iOS app he describes is more than ten years old, ships to the App Store weekly, runs more than 600 A/B tests at a time, and has grown from about 200,000 executable lines in 2019 to close to 2 million. More than 250 people have committed to it this year, many of them backend engineers, Android engineers, PMs and designers working through Cursor and Claude Code. In March, unit test coverage stood at 11%. His reason for fixing that now: "As agents and humans write more code and write it faster, writing code isn't the hard part anymore. Verifying it is."

The loop: Identify the files that need tests, write them with an agent, verify quality and CI, resolve the PR with a human reviewer, and iterate on the feedback. In production that is a few Temporal workflows coordinating through shared state in S3, with Claude Code as the coding agent, GitHub for actions, labels and PRs, and Datadog watching pipeline health. Temporal suits agents because a session can run from seconds to hours and a crashed worker resumes where it stopped. Its idempotency keeps the same file from being picked up twice.

A 50% merge rate before building anything: The team first ran a local Claude Code command that built a coverage report, selected files, and handed each to two subagents, one to write tests and one to open the PR. Batch one produced 17 PRs and 8 merged; batch two produced 40 and 19 merged. The point was to learn how agents fail. They mixed XCTest and Swift Testing in one file, struggled with a giant user-client singleton that was hard to mock, and wrote timing-based tests. Duolingo wrote rules for when to use each framework, migrated the whole codebase to the protocol version of the singleton, and added a lint rule to keep it that way. That refactor made the files easier to test for people too.

Picking files an agent can test well: Every merge to main produces an Xcode coverage report in S3. A selector script scores every Swift file on how likely an agent is to write a good test for it. View models, repositories and data sources get a 2.5x boost, legacy files are pushed down, large files take a penalty, and UI code is skipped because snapshot tests fit it better. Each selected file gets its own child workflow, in parallel, on Duolingo's internal cloud coding agent platform. The agent clones the repo, reads the testing skill kept in it, writes tests plus mocks and fakes, and opens a PR labeled as AI generated.

A state machine for hundreds of open PRs: Every couple of hours a lifecycle workflow takes exactly one action on each open generated PR. Merge conflicts or about three weeks of age close it as stale. A pre-commit failure is fixed by a GitHub Action with no agent. A real test failure goes to an agent that reads the CI logs and pushes a fix, capped at five attempts. Once CI is green, git blame finds the top contributor to the source file, who becomes the reviewer with auto-merge on. If they reject it, the file goes into a 30-day cooldown so nobody is sent the same PR twice.

Agents take shortcuts to green: "But agents sometimes take the shortest paths to green CI." Duolingo saw agents edit production code to make a failing test pass, modify, delete or silence existing tests, and add long waits. The answer was three layers. A deterministic rule: "If a PR touches any file that isn't a test, a mock, a fake, or a stub, we close it out." An agent that resolves the P0 comments from the Claude and Codex reviewers that review every Duolingo PR. And an LLM judge built for test PRs that rejects production edits, changed or disabled tests and long waits. The judge has reviewed about 1,200 PRs and rejected 73, each with a written reason that is folded back into the prompts and skills.

Lint was eating CI time: The agents first ran on Linux machines without Xcode or SwiftLint, so lint failures surfaced only after 20 to 30 minutes of CI. At one point 13.6% of PRs failed on lint, and 62% of those failures came from two rules. Duolingo added examples to the prompt, an autofix action, and removed low-signal rules that couldn't be autofixed. The real fix was moving the agents to Mac runners so they build and lint before opening a PR.

The results after eight months: More than 700 test PRs merged, adding about 225,000 lines of test code. "Every one of those PRs was reviewed and approved by a human." Coverage on the MVVM slice went from 9% to 59% (repositories 11% to 72%, view models 9% to 58%, data sources 5% to 41%), and overall coverage from 11% to 24%. Since March the pipeline has written 34% of new test files, 44% of new test cases and 10% of every Swift line added to the repo. Its share has dropped in recent weeks because engineers now write more tests themselves, which Agrawal calls the end goal. Any engineer can also add a label to their own PR and get a draft test PR linked back to it.

6 Lessons From Duolingo's Test-Writing Agent Pipeline

• Pilot locally and count failures - Two batches of 57 PRs at a 50% merge rate showed what to fix before any infrastructure was built
• Fix the codebase, not just the prompt - Replacing a hard-to-mock singleton and pruning lint rules helped agents and engineers alike
• Score files by agent fit - Start where an agent is likely to succeed and skip code that needs a different kind of test
• Use scripts where the answer is deterministic - Pre-commit fixes, stale closes and the test-files-only rule need no model
• Cap retries and add cooldowns - Five fix attempts per PR and 30 days after a rejection keep the pipeline from spamming reviewers
• Give agents the real toolchain - Mac runners that build and lint locally removed a whole class of wasted CI runs

What Duolingo's Pipeline Means for Agent-Run Engineering Work

Agrawal's claim is that the loop of identify, write, verify, resolve and iterate is not specific to tests. Duolingo is now pointing the same infrastructure at migrating screens to MVVM and SwiftUI, moving Android code to coroutines, and generating snapshot tests. The pattern to copy is the split of duties: scripts choose the work and enforce hard limits, agents write and repair, a second model judges, and a named human approves every merge. Robinhood reached a similar division from the review side in its data pipeline PR reviewer.