Teamday
← Back

Bumble's AI Photo Feedback: 78% Wanted It, Few Used It

AI Engineer
businessenterpriseproductivityinference

Why A/B Testing AI Features Is Different: Lessons from Bumble

Supriya Subudhi is a product manager at Bumble, the dating app with over 20 million monthly active users. Her AI Engineer talk takes apart one shipped feature, AI photo feedback, which gives users suggestions on how to improve the pictures in their dating profile. Her argument: for generative AI features, "a positive result from an A-B test is actually just where the journey begins." She gives five reasons, each from a problem Bumble had to solve.

Users ask for it, then ignore it: Research was almost unanimous that users didn't know whether their profile was good enough. When shown examples of AI guidance, almost 78% said they would love the feedback. In the A/B test, only 10% to 15% of people who had the feature used it. Her conclusion: "openness to AI doesn't actually predict whether people are going to use the feature." Adoption rose once Bumble stopped treating it as a standalone feature and offered it inside existing journeys: asking for feedback while users added photos, and pointing to it when users weren't getting matches.

The average wins, the tail fails: Top-line results were good. More people added photos and made more profile changes. Reading the details showed the AI telling people to smile more in photos where they were already smiling. Bumble added guardrails, such as never suggesting changes to things people can't control, like skin color or height. It also used evals to read real outputs and judged success by segment: whether women and men, and users in different languages, responded the same way.

Green dashboard, red bill: Bumble began with the best model, about 10x the cost of a simpler approach, plus the cost of running evals. "With generative AI, the more successful your product, the more people that use it, and the more often they use it, the more your bill increases." Bumble's matching algorithm has high upfront compute and training costs but little extra cost per user. Generative AI costs grow with every use.

The win belonged to the model: Switching to a cheaper model cut costs and also cut impact, because accuracy dropped. That showed "the results that we saw from our A-B test was due to the model that we were testing, not the concept of giving feedback itself." Bumble then used its eval setup to test models, datasets and prompts quickly. It ended up on the cheapest option that still delivered the result.

No global rollout: Bumble shipped the feature in the US and then had to stop, because AI regulation differs by country. Subudhi argues that PMs, product engineers and forward-deployed engineers all need to think about compliance from the start. Teams have to choose between building within every market's constraints up front, launching the best version in one market only, or paying to maintain different versions for different countries.

5 Questions Bumble Asks Before Shipping an AI Feature

  • Are people using it, or just saying they will? - Survey enthusiasm (78%) and real usage (10-15%) were far apart. Put the feature where users already are.
  • Do you read real outputs or only averages? - Positive averages hid tips that were plainly wrong. Read outputs and break them down by gender, language and country.
  • Does it still pay off after all costs? - Include inference and eval costs, and decide whether the feature is a short-term ROI play or a long-term brand bet that justifies a bigger budget.
  • Is the win on the best model one you can afford? - Rerun the test on the model you can afford to run at scale. The first result may only reflect the model.
  • Are you clear legally on where you can ship? - Plan market-by-market compliance before building, not after the US launch.

What Bumble's Experience Means for AI Product Teams

Bumble's photo feedback shows that an AI feature has no fixed cost or quality the way normal software does. Cost grows with usage, quality depends on the model, and the results in the long tail can be worse than the average. A green A/B test only confirms the starting point. To keep a feature that adoption, cost and compliance can sustain, teams need agent evaluation on real outputs, and a cost model rerun on the model they will actually ship. For a related cost lesson, see why a cheaper model can cost more per task.