Who Trains the AI? Human in the Loop and the Future of Support: A Conversation with Stephanie Watson of Prisma by Mediaocean

Watch the Live Recording

Introduction

Stephanie Watson, Chief Operating Officer at Prisma by Mediaocean, explains how her team turned off almost all support channels, went back to web forms, and rebuilt around self-service and AI-ready content without hurting CSAT. Self-service now handles about 70% of support volume, and a small team trains the AI behind it. She also covers how central enablement keeps employee-built AI projects from overlapping and why she thinks tickets are dead. Andrew Bothwell, Chief Revenue Officer at Kustomer, hosts this episode of the CX Now Podcast.

How do you rebuild customer service around AI and self-service?

Stephanie Watson's team rebuilt service by deliberately stepping backward first. About three years ago, they started automating so they could handle more work without adding headcount. Two years ago, they turned off almost all service channels and went back to web forms, which she says took the team "about 10 steps backwards" and was painful for customers too. Meanwhile, they trained their search, reworked content for generative AI, and trained both employees and customers on the new model. CSAT stayed strong, and about 70% of support volume is now self-service. Stephanie says it took more than a year to know it was working.

What does human in the loop mean in AI customer support?

For Stephanie Watson, human in the loop means people step in when an issue needs judgment, and keep training the AI that handles the rest. She says the best thing a team can do is train its self-service model to recognize when an issue should move to a person. That work never stops, because products change and new questions keep arriving. Her support team used to be a large group split into pods by skill set. Now a small group focuses mainly on training the AI and writing documentation to fill gaps. She says that makes the job more exciting. Learn more about how to hand a conversation off to an AI Agent in Kustomer.

Should employees build their own AI tools or should a central team do it?

Both, according to Stephanie Watson: let employees build, and back them with central enablement. Without it, she says, three people can build the same thing in silos, and busy teams like implementation know what AI could speed up but lack the time or the tool knowledge. In her model, an enablement team picks the tool, works with legal on compliance, and helps find budget. She likes host Andrew Bothwell's approach at Kustomer, where employees log what they plan to build before they start, though her company hasn't done that. She sees work that could drop from three months to three days.

How do you measure the quality of an AI agent in customer service?

Stephanie Watson says she does not have a complete answer yet and calls it her team's next hurdle. She finds a bot's mistakes forgivable when customers know it is a bot and get faster service, but she still wants a human checking quality. Her test for her team: if customers couldn't find an answer, can you find it, and if not, does it exist? She also says teams should watch AI go wrong, because it can take a task too far. She wants an enablement function to help, since quality now spans multiple tools and a data layer. Curious how Kustomer approaches AI quality? Learn more about AI QA.

Are support tickets going away?

Stephanie Watson thinks so: "I think tickets are dead." She says everything that happens in a ticket is now handled through agentic work, so the work moves to the data layer and to direct work with customers, where teams can pull data and get insights while keeping it protected. She also expects the feedback loop to product teams to run without a human reading every conversation, with people stepping in for QA. Her timeline is loose: she mentions two to five years, then credits host Andrew Bothwell's estimate of about a year. Andrew's term for the human role is reviewer mode, where a person approves what the AI recommends.

Meet the Guest

Stephanie Watson is Chief Operating Officer at Prisma by Mediaocean. Her team spent about two years rebuilding service around self-service and AI, and it is now in another reorganization that includes an AI enablement function. She describes herself as a vendor loyalist who prefers longer-term commitments. Her rule: if she invests in a vendor, she treats it as a partnership.

Full Transcript

This interview has been lightly edited for clarity.