Teamday
← Back

Lauren Tan: 11,000 Agent PRs a Month, 0.5% Reverted

AI Engineer
agentsautomationproductivityenterprisetutorial

How Lauren Tan's Grok Bot Team Ships 11,000 Agent PRs a Month

Lauren Tan, known online as poteto, works on Grok Bot at SpaceXSI. She previously worked on the React Compiler and is the author of pstack, an open-source set of agent skills. In this AI Engineer talk she reports that she personally shipped 3,000 pull requests to production in a month, many of them while she slept, and that it took her six months of work on her agents' environment to get there. Her thesis is that the engineer's job is now to keep quality steady or better while output multiplies, and that the way to do it is to invest in the environment until the agents can be trusted.

The inverse pyramid: Tan ranks where an engineer's time should go. The widest layer is the codebase itself: API design, architecture and refactoring. Next comes static analysis: lint rules, type systems, compilers and tests. Then verification, both runtime and formal. Prose that needs judgment (rules, skills, agents.md) is the thinnest layer. Her rule for moving work down the pyramid: "So whenever you're correcting your agent, think about how you can make that a deterministic check instead of adding another line to your rules or your agents.md."

The codebase is the biggest prompt: Agents copy what they find. Workarounds, deprecated code and tech debt look like the established pattern to the next agent, so they spread like weeds in a garden. In 2024 she reviewed about three pull requests a day from colleagues. Now the number is closer to 300, so the patterns in the repository matter more than any reviewer.

Ban what needs nuance: The team's internal client framework for Grok Bot encodes what it learned building desktop apps. React's useEffect is banned in product code and allowed only inside framework APIs, because agents misuse it. Two top-level directories, one for the Electron renderer and one for the main process, cannot import from each other; a dependency graph analyzer and lint rules enforce it. That removed a class of long-task jank the team had seen in Cursor, where agents pulled main-thread code into the render thread.

Comments are an escape hatch: In Cursor, agents reached for setTimeout and requestAnimationFrame to paper over a clash between VS Code's imperative style and React's declarative style, then wrote comments explaining why the quick fix was justified. The timers spread and performance suffered. Grok Bot bans 99% of code comments. Comments go stale, people don't read them, and they can't be enforced. Lint rules replace them, with error messages that tell the agent to fix the root cause.

Review the evidence, not the diff: Code review has shrunk to looking at verification artifacts: videos, screenshots, diagrams and formal verification. Agents run the app themselves through the Chrome DevTools Protocol, guided by a "feature map" (a sitemap of every feature from the user's point of view) and custom CLIs that cut token use. The team model-checks major user flows such as cross-device notifications with TLA+, uses Lean to prove properties of its own virtualization library, and has started property-based UI testing with Bombadil. Tan says most of her token usage now goes to verification. She calls formal methods promising but "certainly not a silver bullet."

The numbers: Last month the Grok Bot team shipped 11,000 pull requests to production and cut a new stable client release about once a day. In October about 227 people contributed at least one pull request, many of them drive-by contributors with little context. Reverts held steady at about half a percent of PRs, and startup, typing, opening chats and switching chats all got faster than at launch. After each release a Cursor project fuzzes the whole build by using the app like a real user, opens a fix PR for each bug it finds and auto-merges it. Tan's summary: "most product code that we land no longer really requires deep human review."

6 Rules From Lauren Tan for Trusting Agent-Written Code

  • Turn every correction into a check - A lint rule or an API change is reproducible; another line in agents.md is not
  • Clean up old patterns, not just new ones - Agents copy whatever the codebase already does, including the anti-patterns
  • Ban features that need nuance - useEffect and cross-process imports are forbidden outright instead of being reviewed case by case
  • Don't let agents explain themselves in comments - Comments become the place where a shortcut gets justified and then copied
  • Spend tokens on verification - Runtime checks, TLA+ models, Lean proofs and property-based tests are affordable now that agents write and maintain them
  • Fix the inner loop before automating it - "if your inner coding loop produces slop, then automating that loop just produces more slop."

What Grok Bot's Revert Rate Says About Agent-Run Engineering

Tan's result is a strong data point for the claim that volume and quality are not a trade-off: 11,000 PRs a month, a 0.5% revert rate, and agents merging most of their own work. The cost was six months of unglamorous engineering on constraints and verification, done before any "software factory" automation. Duolingo reached the same conclusion from a different direction when it fixed its codebase so agents could write tests. Tan and her colleague Roman Ugarte describe how the same approach extends beyond engineering in their Y Combinator conversation on AI coworkers.