From voice AI pilot to production: the operating model behind trusted AI
Yoni sat down with Nate Christiansen of Henry Schein One to unpack how his team moved voice AI from a $40,000 pilot to 1,500+ daily customer interactions, and what they learned along the way.
“The hardest part of this actually wasn’t even training the AI.”
That was one of the comments that stayed with me after my conversation with Nate Christiansen on Building with AI: Promises and Heartbreaks.
Nate is Director of Ascend Operations at Henry Schein One, where he leads strategy and AI operations. The company serves more than 70,000 dental practices worldwide. His team has moved AI beyond a demo, building a voice agent that now handles more than 1,500 customer support interactions every day.
This sounds like a story about voice technology. It is actually a story about everything that needs to exist around the model before AI can be trusted in production.
The model mattered, of course. But so did the knowledge, evaluation, financial model, escalation paths, ownership, and the people operating it every day.
Listen to it now: YouTube, Apple Podcasts, Spotify
Start with a real operational constraint
The project did not begin with an executive saying, “We need an AI agent.”
It began with a customer support problem.
A major healthcare outage caused support volume and wait times to shoot up. Henry Schein One has a large and complex customer base, and its support teams need to understand many products and thousands of features. You cannot hire and train 100 people overnight when demand suddenly spikes.
Nate’s team also found that roughly half of its calls were “how-to” questions. These were important questions, but many did not require the deepest technical expertise.
The obvious answer might have been to push customers toward chat. Henry Schein One deliberately avoided that. Many dental practices had been trained for decades to pick up the phone when they needed help. Forcing them into a new channel would make the company’s operational problem the customer’s problem.
So the team looked for a voice AI system that could:
Meet customers in the channel they already preferred
Answer common questions at any hour
Support multiple languages
Escalate difficult cases to a human
Improve the customer experience, not merely lower its cost
That is a much better starting point than “Where can we use AI?” It begins with a workflow, a constraint, and a measurable outcome.
The demo is not the product
A voice agent can sound impressive for five minutes. That does not mean it will work when a frustrated customer calls about a problem affecting their dental practice.
Nate’s team evaluated vendors with real support representatives and customers. They tested whether the agent could understand the product, retrieve current information, respond naturally, recognize emotion, and de-escalate difficult conversations.
They also tried to break it.
They yelled at it. They swore at it. They pushed it through the kinds of edge cases that do not show up in a polished vendor demo.
One of the most important criteria was how quickly the system could absorb corrections. Henry Schein One releases new features constantly. A good answer from last month can become a wrong answer today. The team needed to update the agent’s knowledge and see the change reflected quickly and reliably.
The evaluation continued after launch. They A/B tested introductions, reviewed hundreds of calls, and learned that seemingly small changes could make performance better or worse.
This is the same reason we built a structured approach to testing Solid’s chat. A single successful interaction proves very little. Production trust comes from repeated evaluation across expected questions, difficult questions, edge cases, and changes over time.
Make the pilot answer financial questions
When Nate took the idea to the CFO, he could not build a serious business case around tokens and theoretical model behavior.
As he joked in the episode, tokens sounded a little like “magic beans.”
So the team asked for a $40,000 pilot. They started with roughly 100 calls per day and used the pilot to answer the questions that actually mattered:
How much did each interaction cost?
How many calls could the agent resolve?
When did it need to escalate?
How quickly were issues resolved?
What happened to customer satisfaction?
How did performance change as the system improved?
This is an important distinction. The pilot was not just a smaller version of the final launch. It was an instrumented learning system.
Over a couple of months, the team improved the agent daily and built its ROI model from observed behavior. According to Nate, the AI could resolve appropriate calls in about one-fifth of the time of a live agent. It also expanded support to 24/7 and multiple languages.
Most importantly, Nate said customer satisfaction reached its highest level in the company’s history.
The business case became credible because it was based on production-like evidence, not a spreadsheet full of optimistic assumptions.
Treat the AI like a new employee
Looking back, Nate said the biggest misconception was that the team could plug in an AI tool, turn it on, and let it do the work.
“It’s more like onboarding an employee, a brand new employee.”
A new employee needs clear responsibilities, access to the right knowledge, examples of good work, rules for difficult situations, and guidance on when to ask for help. An AI agent needs the same things, except it cannot fill in organizational gaps by casually asking five coworkers what an internal term really means.
Nate made another observation that will sound familiar to anyone building AI for analytics:
“Sometimes we blame it as a hallucination when in reality we just weren’t clear on our training.”
For a support agent, the required context includes product documentation, troubleshooting steps, release information, customer language, emotional-response guidance, and escalation rules.
For an AI analyst, the context is different, but the problem is the same. It includes metric definitions, joins, filters, business terminology, validated queries, permissions, and the logic hidden across the data warehouse and BI environment.
The model can generate language or SQL. The context tells it what those words and numbers mean inside your business.
We have written before about using AI to create and maintain the context required by other AI systems. Nate’s experience reinforces the point: this context is not setup work you finish once. It is part of the production system and must evolve with the business.
Your frontline team becomes the AI team
The technical implementation was only one part of the work. The other part was helping employees understand what would change.
Support representatives naturally had questions about job security and the future of their roles. The successful response was not to ignore those concerns or make vague promises. It was to involve the team in the implementation.
The human support agents became essential to the system. They handled the more complex cases. They reviewed interactions the AI could not resolve. They identified gaps, corrected the knowledge, and trained the agent to perform better.
Their work moved up the stack, from repeatedly answering simple questions to becoming expert operators of the support system.
With the benefit of hindsight, Nate said he would spend even more time with the people affected by the project. They understand the exceptions, the customer language, and the practical reality of the workflow. They are also the people required to make the AI improve after launch.
There is no production AI without someone owning that last mile.
Once the operating model works, it can travel
After the support project showed results, Henry Schein One began applying the same approach to inbound sales.
The AI can learn the company’s product portfolio, answer initial questions, qualify a lead, and schedule a meeting with the right account executive. This allows the existing inbound sales development team to spend more time on outbound work.
At the time of our conversation, the pilot had already contributed to its first closed-won opportunity. The rollout remained cautious. A bad support interaction creates frustration, but a bad sales interaction can also lose revenue.
What is interesting here is not simply that Henry Schein One found another agent use case. It is that the company had built reusable organizational capabilities: vendor evaluation, knowledge ingestion, testing, monitoring, human handoffs, ROI measurement, and clear ownership.
Once those capabilities exist, the next use case is no longer a completely new experiment.
The lesson for AI analytics
A voice support agent and an AI analyst look very different to the end user. Underneath, they share the same production challenge.
Both need accurate, current business context.
Both need evals that test more than a happy path.
Both need observability, escalation, and a human owner.
Both can lose user trust with one confident, incorrect answer.
And both become more valuable when the people closest to the work help train and govern them.
This is the work of semantic engineering: turning scattered organizational knowledge into governed context that an AI can use and humans can maintain.
The visible model is only a small part of the system. The operating model around it is what turns an AI capability into a trusted product.
That is the real journey from pilot to production.
I thank Nate for sharing the practical details, including the parts that did not work immediately. You can listen to the full episode here: YouTube, Apple Podcasts, Spotify