Back
EP 006
25 Min
Rohit Tripathi joined a two-person team supporting six brand-new AI products at Zapier, inheriting a backlog of 700 tickets. When the product itself is unpredictable, telling a real bug from a hallucination is the whole job.
Supporting AI products is harder than supporting normal software, because the thing you are debugging can be wrong in ways no one designed. Rohit Tripathi has spent more than ten years in technical support, including a pilot team at Zapier that supported its entire AI line while it was still being built. On this episode of the Fini Podcast, he explained how he triages AI issues, how he gets engineering to act, and what he would automate first.
Meet Rohit Tripathi
Rohit is an application support engineer at Last Yard, and over the past decade he has worked in technical support at HP, Microsoft, Automate.io, Notion, and Zapier. At Zapier he was one of two people supporting six new AI products at once (chatbots, tables, interfaces, and more), debugging live systems, logging bugs in Jira, and working directly with product managers to prioritize fixes by real customer impact.
Clearing a 700-ticket backlog from both ends
The pilot inherited about 700 pending tickets. With two people, Rohit ran a simple play: one person worked from the newest ticket backward, the other from the oldest forward, and they met in the middle. The logic was to protect the experience of new customers without abandoning the people who had been waiting for months. It worked, taking the backlog down to roughly 100 to 150 in three to four months.
Bug, hallucination, or user error?
When you are not sure whether an issue is a product bug, a hallucination, or a user mistake, Rohit relies on two habits: watch for patterns and reproduce the issue. If several users report the same experience, it is probably a real bug, so he recreates it to see how far it goes before deciding to escalate. He is blunt that hallucination is a genuine problem, because AI will confidently make up answers, so the person receiving an AI suggestion has to double-check it before passing it to a customer.
Getting engineering to actually prioritize
Logging a bug and hoping is not a strategy. Rohit escalates with impact: which customers are affected, whether they are paying, and how badly the bug blocks them from using the product. A white-space error is not a refund-blocking failure, and he sets the noise level accordingly. His larger point is that the feedback loop matters more than anything. When communication between support and engineering is flawless and detailed, engineers who never talk to customers can still serve them well.
What to automate first, and what to automate last
First, capture knowledge before it disappears. In fast teams, a tier-one engineer hits a problem, posts in Slack, and three teams solve it, then the answer is lost in the thread and someone re-solves it a week later. Rohit wants a tool that watches those discussions, files the answer, and surfaces it on the next matching ticket. That is the knowledge architecture problem most teams ignore. Last to automate: anything touching payment or personal data, and any multi-turn conversation that drags on. If a customer comes back more than about five times unsatisfied, a human should step in, which is exactly where good guardrails earn their keep.
What support leaders should take from this
Reproduce before you escalate. Recreate the issue on your end to confirm it is a bug, not a feature request or user error.
Triage by pattern. Multiple customers reporting the same thing is your strongest signal of a real bug.
Escalate with impact, not volume. Name who is affected, whether they pay, and how badly they are blocked. Set the urgency to match.
Capture tribal knowledge. Get answers out of Slack threads and into a repo the next agent can find, or you will solve the same issue forever.
Cap the back-and-forth. If the AI cannot resolve a conversation in a few turns, hand it to a human before a hallucination does damage.
First fix the customer, then fix the issue. Acknowledge the frustration before you debug. That is still what separates a person from a robotic tone.
Listen to the full episode
Rohit shares more on supporting AI in production, the support-to-engineering loop, and advice for tier-one agents who want to level up, in the full episode of the Fini Podcast. You can connect with him on LinkedIn.
Resolution that stays grounded instead of guessing is what Fini is built for. Book a demo to see it on your own tickets.
How do you tell an AI bug from a hallucination or a user error?
Watch for patterns and reproduce the issue. If several customers report the same behavior, it is likely a real bug, so recreate it before escalating. Because AI can confidently produce wrong answers, treat unverified AI output as a possible hallucination and double-check it against a reliable source.
How do you get engineering to prioritize support escalations?
Escalate with impact rather than raw volume. Specify which customers are affected, whether they are paying, and how badly the issue blocks them from using the product, then set the urgency to match. A tight, detailed feedback loop helps engineers who never talk to customers fix the right things.
What should support teams automate first?
Rohit would first capture knowledge that gets lost in Slack threads, filing it where the next agent can find it so the same issue is not solved repeatedly. Most of his automations are simple and aimed at reducing day-to-day noise rather than building complex workflows.
When should a human take over from AI?
For anything involving payment or personal data, and whenever a conversation runs long. If a customer comes back more than about five times still unsatisfied, a human should review the conversation, since hallucinations make unmonitored long exchanges risky.
