Shiv Indap

About Shiv Indap

Shiv Indap leads voice AI at Upstart, the AI lending company. Upstart handles thousands of calls a day, and its voice agents talk to applicants and borrowers in a regulated industry where a wrong question can become a compliance problem, not just a bad customer experience.

Indap's team launched a voice agent that automates verification calls for the roughly 10% of applicants who aren't decisioned instantly. The agent confirms the caller's identity and asks three questions about income, use of funds and outstanding obligations, while following rules such as never asking the amount when funds are for medical purposes.

Notable Positions

On Validating the Judge

Upstart scores every call with an LLM judge for compliance, then has humans review the judge's verdicts. At launch, judge and humans agreed only 33% of the time. Writing down tribal knowledge, resolving disagreements between human reviewers and improving the judge raised agreement to 92%.

On Metrics

Upstart started with many metrics and cut them down to accuracy, clarity, completeness, policy alignment, correct escalation and CSAT. Indap advises teams not to get attached to their first metrics.

On Gated Changes

Even with automated fixes, every change ships behind a feature flag. Engineers sign off on judges, and the compliance team approves any change to compliance behavior.

Key Quotes

• "if things go wrong, it's not just a customer experience issue. It could be a potential regulatory issue, which is worse." (on voice AI in lending)
• "don't be too attached to your metrics." (on choosing eval metrics)

Related Reading

• Upstart and Coval: Voice AI Evals in Regulated Lending - Full breakdown of the talk
• Agent Evaluation - How organizations test agents in production