Robinhood: 60% of Data PRs Merge With No Human Review

How Robinhood Reviews Data Pipeline PRs at Machine Speed

Chetas Joshi is an engineer on Robinhood's data platform team, which builds the ingestion and processing frameworks behind the company's data lake and owns review of every change to its production data pipelines: schema changes, pipeline code, configuration and access grants. AI coding assistants made those pull requests cheap to open, but the number of people who know what is safe to merge didn't grow. His one-sentence problem statement: "Pull requests to our production data pipelines now arrive at machine speed, but reviewing them safely still runs at human speed."

Three duties, three owners: Robinhood's answer is a hybrid reviewer. "Scripts prove what can be proven. An agent judges what needs context and people own the calls only they should make." Layer one is deterministic CI checks against live systems: data contracts against the schema registry, required owner and PII tags in the catalog, a lineage blast-radius check over every registered downstream job, and a performance test that runs the change in staging at production data volume to check cluster sizing and freshness SLAs. A failed check turns CI red with no model involved. Layer two is an agent whose rules tell it what layer one already covers, so it spends its judgment, and the AI budget, on what scripts can't see.

The PR that was green everywhere: Joshi walks one PR through the system. It adds a column and widens another from integer to big integer, about the safest schema change there is. Every layer-one check passed. The agent, which can only read, search and list repository files, walked pipeline definitions and access grants and found a job missing from the lineage tool that copied the widened column into a table where it was still an integer. The first oversized value would have failed that write. The agent scored the risk high and escalated with the full chain in its review comment; a human confirmed the job was live, the downstream table was widened first, and the PR merged with no incident. Lineage misses jobs because each query engine has to publish its own, and ad hoc queries and dashboards never show up in a repo search. So the agent is not allowed to claim an all clear: "None found is not the same as none exist."

Changes the agent can never approve: Most changes the agent can clear on its own when checks pass and it is confident. Three categories always go to a person: grants touching sensitive data, migrations that change what data means rather than how it moves, and any verdict the agent itself isn't confident in.

Learning from corrections, by hand: Engineers reply to the agent's comments agreeing or disagreeing and say why. Once a week the on-call engineer reads that feedback and edits the review rules manually; nothing retrains itself. One correction: a column tagged PII upstream is encrypted before it lands in the lake, so a non-sensitive label there was fine. One miss: privacy re-tagged columns as PII, moving them out of an open copy of the data, and a downstream reader of the open copy broke. That became a new rule to search the code for readers of any column about to be re-tagged.

The numbers, as of August 31, 2026: About 60% of data pipeline PRs merge without human review, manual review effort is down about 75%, roughly 132 engineering hours a month are saved, and incidents tied to these PRs are down about 30%. The miss rate is about 2%, mostly from gaps in the rules. Joshi also names what they got wrong: the two layers run as separate systems that can't see each other's results, and the agent can't call live checks. The team is exploring a single entry point that classifies the whole PR and gives the agent layer-one checks as tools, but it isn't built.

5 Lessons From Robinhood's Agent PR Reviewer

• Prove facts with scripts first - Anything a source of truth can answer shouldn't cost a model call; it's cleaner and cheaper
• Give the agent the gaps, not the whole job - Its rules list what CI already checks so it hunts for unregistered consumers, query anti-patterns and risky grants
• Flag, never clear, the scary changes - Sensitive grants, intent-changing migrations and low-confidence verdicts always reach a person
• Make the agent honest about its blind spots - It reports what it searched, never that no consumers exist
• Count false escalations like misses - A gate that escalates everything moves the work instead of removing it

What Robinhood's Results Mean for AI Code Review in the Enterprise

Robinhood's design is a template for any team drowning in agent-written PRs: let deterministic checks do the proving, let the agent read the code a human would otherwise trawl, and keep accountability for the protected calls with people. As Joshi puts it, "A gate that escalates everything is not actually saving anyone any time." The measurable win is less human review on the routine 60%, with specific, reasoned comments that make the remaining reviews faster. See also how Harvey's Spectre platform handled the PR surge on the other side.