For much of the past two years, the analytics industry has been focused on proving what AI can do.

Can it generate SQL? Can it summarize data? Can it explain trends? Can it answer business questions without requiring users to understand the underlying data model?

For the most part, those questions have been answered. Modern AI systems are increasingly capable of performing analytical tasks that would have seemed ambitious only a few years ago. Business users can ask questions in natural language, analysts can accelerate workflows, and organizations are beginning to rethink how people interact with data.

As AI capabilities mature, however, a different question is emerging inside enterprise data teams. It’s no longer enough to ask whether AI can answer a question. Data leaders are beginning to ask whether AI should answer it in the first place.

That distinction may sound philosophical, but it has practical implications for everything from analytics strategy and governance to user adoption and cost management.

The Risk of Treating Every Question the Same

One of the most common assumptions surrounding AI analytics is that natural language interfaces will eventually become the primary way people consume information. If users can simply ask questions and receive answers, why maintain dashboards, reports, and other traditional analytics assets?

The answer lies in the nature of the question being asked.

Not all business questions require exploration. In fact, many of the most important questions inside an organization are remarkably consistent. Executives review the same KPIs every week. Operations teams monitor the same performance metrics every day. Finance organizations distribute the same reports every month. These workflows exist because the business benefits from consistency. Everyone is looking at the same numbers, interpreted through the same lens, using the same definitions.

In these situations, AI may be fully capable of generating the answer. The more important question is whether generating the answer dynamically creates more value than delivering it through a standardized experience.

The answer is often no.

Where AI Creates the Most Value

The strongest use cases for AI analytics tend to involve uncertainty.

A business leader wants to understand why churn increased. A sales manager notices a decline in pipeline performance and wants to identify contributing factors. An operations team needs to investigate a sudden shift in customer behavior. These questions rarely have predefined paths. Each answer leads to another question, and the analytical process becomes iterative.

This is where AI excels.

Rather than forcing users to navigate dashboards, locate datasets, or wait for analysts to produce custom reports, conversational analytics enables a more natural process of discovery. Users can investigate, refine, and explore ideas in real time.

The value isn’t simply that AI produces an answer. The value is that it accelerates the process of finding the right answer.

That distinction is important because it highlights where AI provides unique capabilities rather than simply replicating existing ones.

The Economics Matter Too

As organizations move from experimentation to production, another factor is becoming increasingly important: efficiency.

Historically, dashboards were designed once and consumed repeatedly. A single report could support hundreds or thousands of users with minimal incremental cost. AI introduces a different model. Answers are often generated dynamically, which means organizations need to think more carefully about how analytical workloads are distributed.

If hundreds of users repeatedly ask the same question, it may be technically possible to generate a new response every time. That doesn’t necessarily make it the most effective approach.

This is why many organizations are beginning to think about analytics experiences in a more nuanced way. Some information is best delivered through standardized dashboards. Some is best generated through AI-powered exploration. Increasingly, some may be delivered through agents that proactively analyze data and distribute insights before users even ask for them.

The goal is not to maximize AI usage. The goal is to maximize business value.

The Future of Analytics Is Intentional

The most successful analytics strategies over the next several years will not be defined by how aggressively organizations adopt AI. They will be defined by how intentionally they deploy it.

Dashboards will continue to play an important role because many business processes depend on consistency and governance. Conversational analytics will become increasingly valuable because exploration and discovery are central to decision-making. AI agents will likely emerge as a third layer, automating recurring analytical workflows and surfacing insights proactively.

Rather than replacing one another, these experiences will coexist.

This is why the conversation around AI analytics is beginning to mature. The debate is no longer about whether AI can answer business questions. For most organizations, that capability is already becoming a reality.

The more strategic question is where AI creates the most value, where traditional approaches remain effective, and how the two can work together to support better decisions.

Because ultimately, the future of analytics won’t be determined by the sophistication of the technology alone.

It will be determined by an organization’s ability to apply the right technology to the right question.

Few topics in analytics generate more debate right now than the future of business intelligence.

Depending on who you ask, dashboards are either on the verge of extinction or as essential as they’ve ever been. Every week seems to bring a new prediction about conversational analytics replacing BI tools, AI agents replacing analysts, or natural language interfaces becoming the primary way people interact with data.

The reality is likely much less dramatic.

Most organizations aren’t choosing between dashboards and AI. They’re trying to determine where each creates the most value.

That’s a far more important question.

Why the “Dashboards Are Dead” Narrative Falls Short

The argument against dashboards is easy to understand. If users can simply ask questions in natural language and receive immediate answers, why would they need to navigate reports, filters, and visualizations?

For exploratory analysis, that’s a compelling vision. Many business users don’t know how to write SQL, build dashboards, or formulate complex analytical queries. Conversational analytics lowers those barriers and makes data more accessible to a broader audience.

But the assumption that every business question should be answered through a chat interface ignores an important reality: not all questions are exploratory.

Organizations have countless metrics that need to be monitored consistently. Executive teams need standardized scorecards. Regulatory reporting requires repeatability. Operational leaders need visibility into performance indicators that are reviewed daily, weekly, or monthly.

In these scenarios, dashboards aren’t a limitation. They’re an efficient delivery mechanism for information that needs to be consumed repeatedly and interpreted consistently.

Not All Questions Are Created Equal

One of the most useful ways to think about the future of analytics is to categorize questions by the type of answer they require.

Some questions are standardized.

What was revenue last quarter?

How are we performing against plan?

What is our claims processing volume this month?

These questions typically have agreed-upon definitions, established metrics, and recurring audiences. Dashboards are often the best solution because they provide consistency, governance, and a shared understanding of performance.

Other questions are exploratory.

Why did revenue decline in one region but not another?

What factors are driving customer churn?

Which operational changes contributed to an increase in claim volume?

These questions require investigation. The user may not know the next question until they see the first answer. This is where conversational analytics and AI-powered exploration can be transformative.

The mistake many organizations make is assuming both categories should be handled the same way.

The Emerging Middle Layer

The discussion often focuses on two endpoints: dashboards and chat. Increasingly, however, organizations are discovering a third category that sits between them.

AI agents.

Unlike dashboards, agents don’t wait for someone to open a report. Unlike conversational analytics, they don’t require users to initiate every interaction. Instead, they proactively perform recurring analysis and deliver insights through the channels people already use.

A sales leader might receive a weekly pipeline analysis. A customer success manager might get a summary of at-risk accounts. An executive team might receive an automated business review before a leadership meeting.

These workflows have traditionally required significant analyst effort. AI agents are making it possible to automate much of that work while still providing context and explanation.

As a result, the future analytics stack is becoming more nuanced than many people expected.

A Better Framework for the Future of BI

Rather than asking whether AI will replace dashboards, organizations should be asking which experience is best suited for a particular type of question.

Dashboards remain highly effective for monitoring business performance, tracking KPIs, and creating alignment around core metrics.

Conversational analytics excels when users need to investigate, explore, and ask follow-up questions.

AI agents are emerging as a powerful mechanism for recurring analysis, automated reporting, and proactive insight delivery.

Each serves a different purpose. The goal isn’t to choose one over the others. It’s to understand where each fits within the broader analytics strategy.

The Real Opportunity

The most successful organizations won’t be the ones that replace dashboards with AI. They’ll be the ones that create a more intentional relationship between users and data.

That means recognizing that some questions deserve standardized answers. Others deserve exploration. And increasingly, some deserve automation.

The future of BI isn’t a battle between dashboards and AI.

It’s a much more practical exercise: determining which questions deserve each.

Organizations that get that balance right won’t just improve analytics adoption. They’ll create a more scalable, efficient, and trusted decision-making environment—one that combines the strengths of traditional BI with the flexibility of modern AI.

If you read my last two blog posts, Human Analyst vs. AI Analyst and Human Plus AI Analyst: Defying Gravity, you know I’m a believer in AI tools augmenting analytics processes, rather than it “replacing” anyone. Those demos, especially with Cursor & Google Antigravity, were really cool, and it’s only become much more powerful since I wrote those articles 7 months ago.

I think I cracked something even better for BI developers, analysts, and even non-technical business people who want to be able to develop reports. Agentic BI Development.

I pointed Cursor at a semantic model built in Solid, gave it a scaffold, and asked it to build a Power BI report. A few minutes later I had a complete Power BI project: semantic model, relationships, DAX measures, report pages, the whole thing. All of which opened cleanly in Power BI Desktop.

I never opened Power BI to build any of it. That still feels a little revolutionary to say out loud.

PBIR Changed Everything

For years, a Power BI report was a sealed .pbix file. It was a binary blob you could only edit in Power BI. You couldn’t diff it, you couldn’t script it, and you definitely couldn’t hand it to an AI agent and say “here, go build” or “change this visual to show x.”

Microsoft changed that with the Power BI Enhanced Report Format (PBIR) and the Power BI Project (PBIP) structure.

Now a report is a folder of human/agent-readable files:

• TMDL for the semantic model (tables, columns, relationships, DAX measures)

• PBIR JSON for the report itself (pages, visuals, and the fields each visual binds to)

Individual files mean git, deterministic edits, and perhaps most importantly: an agent can author the whole project without a single click in the Power BI UI.

How it Actually Works

First, a quick bit about the problem: pointing any AI at a data warehouse is the same one I hit in my analyst bake-off. On a clean, well-labeled dataset, AI tools look brilliant, but the second you connect them to a real database with cryptic table names and tribal-knowledge business logic, they start guessing. Guessing is how you get a beautiful dashboard that’s quietly, confidently wrong.

Solid solves that by being the governed semantic layer in the middle that remains consistent as data drifts. Those models can be consumed by agents to take actions in various systems, grounded in SQL generation that has been benchmarked and validated automatically. The model already knows how “net margin” is defined, which tables are authoritative, and how everything joins.

The appetite for agents using data is growing, and Solid helps make this reliable and maintainable. But, not everybody is thinking about agents. People still ask “What about my dashboards?” I wanted to answer that question.

So the flow looks like this:

• Start from the model. A Solid semantic model built for a business domain is the baseline, containing tables, columns, metrics, and relationships. Building a model in Solid takes minutes, not weeks.

• Ground with the Solid MCP. Cursor talks to Solid’s MCP server directly and invokes its text2sql tool to generate grounded SQL from the semantic model. It returns precise joins and filters directly from the source, ensuring the agent never hallucinates business rules. It can also look up business terms and asset metadata on the fly if it doesn’t understand something in the model fully.

• Translate SQL → DAX. That grounded SQL becomes the source of truth for the DAX measures the agent writes into TMDL. A metric like a SUM or an AVG maps to a clean DAX measure; the model’s relationships become TMDL relationships one-for-one.

• Build the report. The agent scaffolds the PBIR. It builds pages, cards, charts, tables, and binds each visual to the measures and columns it just generated. A simple Python script assembles it all into a Desktop-ready .pbip.

• Open in Power BI. Drop the .pbip into Power BI Desktop against live Snowflake, and it renders. From there you can keep editing in Cursor, or push to the Power BI cloud when you’re ready.

The important part: at no point did I build the semantic model by hand in Power BI, and I never manually dropped a single visual onto the canvas. The model I’d already certified in Solid became the model in Power BI. That’s the whole pitch.

“Skill Issue”

The first time I tried this, it took a lot of iteration. Power BI’s project format has real landmines for an AI agent to go at it blind. Examples include a schema path that’s subtly wrong, a TMDL comment style that breaks the parser, a theme declared without its resource package. If you hit any one of these, the project won’t open. The difference between “magic” and “why won’t this load” came down to writing skill files that encode all of those rules up front along with a couple simple Python scripts the agent can reuse for generation and validation.

Now the agent knows the exact folder layout, the required files, the schema URLs, the traps to avoid, and the SQL-to-DAX translation rules before it writes a single line. It grounds every number through the Solid MCP, stays inside the model’s bounds, and hands me a project that opens on the first try. It turned a fiddly, expert-only workflow into something I’d comfortably hand to an analyst who’s never seen TMDL.

Let’s Build

Talk is cheap, so let me show you exactly what I’m about to run. I’m using ACMEBank, a demo bank environment we built in Solid to show off banking use cases, and I’m going to build a complete report from scratch, backed by whichever semantic model actually fits the use case.

That last part matters. I’m not hand-picking tables. My prompt literally tells Cursor to go find the right model, and Solid’s MCP does the routing to the right certified semantic model in its system.

Here’s the prompt, verbatim:

Build a new Power BI report dashboard + relevant raw table pages from scratch with the correct Solid model with the right visuals on the dashboard page to best model the below use case:

Customer Profitability & Risk Segmentation
-Show net margin trends for customers in the top profitability tier
-Which customers have overdue KYC reviews relative to their last review date?
-How does profitability vary by customer risk rating?
-Which household groups show declining net margin over consecutive periods?

Solid routes the use case to the right model, grounds every metric (net margin, estimated annual value, days-since-outreach) in real SQL, the agent translates that into DAX and PBIR, and I get two .pbip projects I can open side by side in Desktop.

No modeling TMDL, writing DAX expressions, or building report tiles. Here’s the final result:

A final report dashboard (though underlying data is missing in one of the visuals) + table tabs.

Cursor modeled the TMDL beautifully with the right DAX expressions and joins, all based on Solid’s model and the skill file we created.

Reality Check

I promised that I’d always give the honest take when evaluating tools, so here’s where the shine comes off a little.

• Not every metric always maps 1:1. Simple aggregates translate to DAX beautifully. But some of these ACMEBank questions lean on window and share-of-total math e.g. “declining net margin over consecutive periods,” “no outreach in the last 90 days,” etc., and those don’t always have a tidy scalar DAX equivalent. The skill file flags these rather than emitting something broken so you can review or edit these.

• Sometimes Cursor hits a snag. This is to be expected with pretty much all AI tools, and agents are getting better at self-correcting. I had to restart the demo video recording a couple of times because Cursor hit a random Python error. It self-corrected without my input, but it made for a messier demo. This really is less of an issue as it is just some honesty about how agents make mistakes, but fortunately self-correct (most of the time).

• Desktop is still the final judge. The validation I run catches structural problems, but Power BI Desktop opening the file is the real test. I still reopen and eyeball it every time before I’d ever push it. BUT, the sheer number of reports you could generate more quickly frees you up to do more high value work.

Cursor hitting a snag with one of the scripts it built, and self-correcting.

Neither of these are dealbreakers. They’re the kind of rough edges you’d expect from a proof of concept, and honestly, I haven’t found a limitation yet that made me want to stop.

Why I think this matters

In my analyst comparison, the takeaway was that the job wasn’t going away, it was evolving. I was still the one asking the right questions and directing the work; the AI just did the building. This is the same story, just with BI development instead.

The reason that “chat with your data” solutions never fully delivered on visual reports is that unless you specifically prescribe the exact visuals you want, you’ll always get something different, even if the underlying data is governed.

A governed semantic model plus a files-based report format gives you something durable: a real, refreshable, shareable Power BI report that traces every number back to a certified definition. You get the speed of vibe-building with the trust of a governed layer underneath it. That combination is what I couldn’t stop thinking about.

What would you build if the report just… built itself? Let us know in the comments!

Leave a comment

A few weeks ago, I met Eldar Bar-Or, who is using AI in one of the most impressive ways I’ve encountered.

Eldar is not a software engineer, and he does not run a technology startup. He owns and operates 603 Auto Salvage, an automotive recycling business.

The company buys vehicles that have been totaled, have major mechanical problems, or are too old to sell through traditional channels. Depending on the vehicle, 603 Auto Salvage might dismantle it and sell the parts, resell it through another channel, or recycle it for metal.

Eldar has been doing this for more than a decade. Over that time, he built a substantial operation with several storage lots, hundreds of vehicles, major investments in SEO and online advertising, and a team handling incoming inquiries.

The basic workflow was straightforward:

• A seller found the business through Google, Facebook, the company’s website, or another channel.

• An employee collected information about the vehicle.

• The employee evaluated it and made an offer.

• The team followed up, negotiated, and tried to close the deal.

• If the seller accepted, the company coordinated the next steps.

Then, last December, Eldar started experimenting with OpenClaw.

He had no coding experience. Within a few months, he had rebuilt much of the company’s digital operation around AI agents.

Subscribe now

From answering questions to closing transactions

Calling what Eldar built a collection of chatbots would undersell it.

Prospective sellers can contact 603 Auto Salvage through its website, Facebook, third-party platforms, and the phone. In many cases, an AI agent handles most or all of the interaction.

The agents collect vehicle information, ask follow-up questions, evaluate the opportunity, make an offer, answer objections, and continue the conversation until the transaction is closed. The company even has an AI voice agent that can answer incoming calls and provide an offer during the conversation.

The automation extends beyond sales. The system also helps manage online advertising, generate SEO content, follow up with leads, update internal systems, and track results across channels.

The system currently uses Claude Opus 4.8, and Eldar’s AI bill is approximately $10,000 per month.

That sounds expensive when described as a software subscription. It looks different when compared with the cost of running a sales, marketing, engineering, and call-center function that operates around the clock.

New inquiries do not wait for someone to start a shift. Conversations continue at night, on weekends, and while Eldar is asleep.

The agents changed the jobs

603 Auto Salvage still hires employees. The difference is what those employees are being hired to do.

Some oversee the agents’ work, review unusual outcomes, and identify where the system needs to improve. Others step into sales conversations that are currently too complex for the agents to handle reliably.

That “currently” matters.

The boundary between what the agents can manage and what requires a person keeps moving. As Eldar finds new edge cases, improves the instructions, and changes the workflow, more tasks move from the human queue into the automated one.

People are still needed to:

• Supervise agent performance

• Handle complicated transactions

• Operate machinery in the vehicle lots

• Move vehicles and remove parts

• Inspect physical inventory

• Confirm that listed parts are actually available

This is not a business without people. It is a business where people spend less time collecting the same information and chasing the same leads, and more time managing exceptions, supervising automation, and performing physical work.

A photograph can answer questions too

One of the most interesting challenges was figuring out how to evaluate a vehicle without forcing the seller through a long questionnaire.

Traditionally, an employee might ask whether there is visible body damage, whether important components are missing, whether the vehicle has been partially dismantled, or whether a specific valuable part is still present.

Every extra question creates work for the seller. Some people do not know the answer. Others lose patience and abandon the process.

Eldar realized that photographs could provide some of this information more efficiently.

Instead of relying only on manually entered details, the system can request photos and use them as part of the evaluation. The agent can inspect the images, identify relevant details, and determine which questions may already have been answered visually.

This saves time for the seller and may improve the quality of the offer. But it also creates new process questions.

Which details can be inferred reliably from a photo? Which still require confirmation? What happens when an image is blurry or incomplete? How should the agent distinguish between something it can see and something it is merely guessing?

Eldar had to define which visual signals matter, how they affect the evaluation, and when the agent should request another image or escalate the case.

One seller does not always mean one vehicle

Another complication appeared when the system encountered sellers offering several vehicles at once.

An individual might submit one car. An auto shop, towing company, or another business might want to sell three, four, or five vehicles in one transaction.

The agent must collect a separate set of details for each vehicle while maintaining the context of the overall deal. It needs to understand which photo, title, condition report, and offer belongs to which car.

It also needs to avoid mixing information between vehicles. A missing engine in one car cannot affect the valuation of another. A photo sent halfway through the conversation needs to be connected to the correct record. Negotiations may happen at both the individual-vehicle level and the package level.

Eldar had to teach the system to:

• Create a distinct record for every vehicle

• Associate each answer and image with the right record

• Track missing information separately

• Generate individual valuations

• Understand whether the seller is negotiating one offer or the full package

• Preserve the relationship between each vehicle and the combined transaction

This looks like a narrow edge case, but it exposes a broader challenge.

Real business conversations are not clean forms. People change subjects, provide information out of order, correct themselves, send several photos at once, and assume the other person understands what they mean.

The agent needs more than language comprehension. It needs a structured model of the transaction.

The model was not the hard part

It is tempting to tell this story as evidence that powerful models can automate almost anything.

Give someone access to OpenClaw and Claude, and the business runs itself.

That is not what happened.

Eldar spent months redesigning processes, testing conversations, identifying bugs, connecting systems, changing call scripts, and figuring out where the agents made poor decisions.

More importantly, he had to explain how his business works.

He had to define:

• What makes a vehicle worth buying

• Which information is required before making an offer

• Which details can be inferred from a photo

• How vehicle condition should affect the price

• When the agent should negotiate or stop

• Which cases need human review

• How multiple vehicles should be represented in one deal

• How inventory affects the value of another vehicle or part

• Which system contains the authoritative information

After more than a decade in the industry, many of these decisions were intuitive to Eldar. They were embedded in employee training, scripts, spreadsheets, and experience.

The agents could not rely on intuition. The logic had to be made explicit.

Claude could generate software, and OpenClaw could give agents access to tools. Neither could independently determine how to value a damaged vehicle, structure a multi-car transaction, or decide whether a photograph contained enough evidence to skip a question.

Eldar had to supply the meaning behind the workflow.

In other words, he had to build a semantic model of his business.

Autonomy required a single source of truth

As Eldar added agents, he ran into another major problem. The company’s information lived across several disconnected systems.

One contained incoming inquiries. Another contained vehicle or parts inventory. Other tools supported calls, advertising, offers, or operations.

Humans are surprisingly good at working around this fragmentation. An experienced employee might know that one field is outdated, another system has the real number, and a particular status means something different from what it appears to mean.

Agents do not automatically know any of that.

Eldar rearchitected the operation around a single database. He also built a custom web application for managing the business and monitoring the agents.

This was not just a software cleanup project. It was a prerequisite for autonomy.

An agent cannot make a reliable offer if it does not know which information is authoritative. It cannot sell a part if the inventory record is disconnected from what is actually in the lot. It cannot coordinate Facebook, website, and phone conversations if each channel has a different understanding of the customer.

We see the same problem in enterprise analytics. A company wants an AI analyst, but the AI encounters conflicting definitions, undocumented business logic, and multiple systems claiming to contain the truth.

Before the agents could operate the business, the business had to become legible to them.

Autonomous does not mean unobserved

Eldar did not automate the process and walk away.

His custom application tracks agent activity, business performance, inventory, conversations, logs, call scripts, and errors. He effectively built his own version of Gong so he could review calls and understand why conversations succeeded or failed.

That monitoring system may be the most important thing he built.

We have previously written about how GenAI is consistently inconsistent and how we approach testing Solid’s Chat. When software is probabilistic, logs and performance metrics are not secondary engineering tools. They are part of the operating system.

Eldar needs to know where an agent misunderstood a customer, why it arrived at a particular offer, which scripts produce better results, which cases get escalated, and whether a model or process change improved performance.

A failed conversation is not only a lost lead. It is also evaluation data.

He can inspect the failure, identify the missing rule or context, and improve the system. He is effectively building evals from the real edge cases of his business.

Zero coding experience does not mean zero expertise

The most surprising detail is that Eldar had never written code before.

But focusing on his lack of software experience misses what he did know.

He knew what a good opportunity looked like. He understood how a vehicle’s condition should affect an offer. He knew which questions sellers ask, where deals get stuck, and which mistakes create losses.

The AI supplied much of the software engineering. Eldar supplied the business context.

Without his knowledge, the agents might have produced impressive demos. They could have answered a few messages or generated generic SEO articles. They would not have been able to run a real operation.

This is semantic engineering in the wild.

It is the work of converting implicit organizational knowledge into entities, relationships, instructions, constraints, and evaluation criteria that machines can use.

In this case, those entities include sellers, vehicles, parts, photographs, offers, conversations, and multi-vehicle transactions. The relationships matter just as much as the individual data points.

A photo must belong to the correct vehicle. A vehicle must belong to the correct transaction. An offer must reflect the right condition and inventory context. A human escalation must preserve everything the agent has already learned.

The principle is the same whether you are building an AI analyst, a data agent, or an automotive recycling operation.

The AI needs more than access to the company’s tools. It needs to understand what the information means, how it relates, and how the business uses it to make decisions.

Business logic is becoming the bottleneck

For years, building custom business software required a long translation process.

An operator described the business to a product manager. The product manager wrote requirements. Designers created interfaces. Engineers implemented the logic. Months later, the operator received software that hopefully resembled what they had requested.

That translation layer is becoming dramatically smaller.

Eldar could explain a process to the AI, test what it built, identify what was wrong, and continue iterating. He created integrations, dashboards, agents, monitoring systems, and internal applications without becoming a traditional software engineer.

The code was no longer the primary bottleneck.

The bottleneck was his ability to describe the business clearly enough.

That is the part enterprises should pay attention to. Most companies will have access to increasingly capable models and agent frameworks. Access to the technology alone will not create a trustworthy operation.

The advantage will belong to companies that can make their business logic legible to machines.

Eldar turned more than a decade of experience at 603 Auto Salvage into data structures, instructions, tools, and feedback loops that AI agents could use.

The result is not a business without people. It is a business where people supervise a growing system of automated work, handle the cases that remain too difficult, and improve the agents based on what happens in production.

Eldar taught the AI how his business works.

Now the system keeps working while he sleeps.

“The hardest part of this actually wasn’t even training the AI.”

That was one of the comments that stayed with me after my conversation with Nate Christiansen on Building with AI: Promises and Heartbreaks.

Nate is Director of Ascend Operations at Henry Schein One, where he leads strategy and AI operations. The company serves more than 70,000 dental practices worldwide. His team has moved AI beyond a demo, building a voice agent that now handles more than 1,500 customer support interactions every day.

This sounds like a story about voice technology. It is actually a story about everything that needs to exist around the model before AI can be trusted in production.

The model mattered, of course. But so did the knowledge, evaluation, financial model, escalation paths, ownership, and the people operating it every day.

Listen to it now: YouTube, Apple Podcasts, Spotify

Subscribe now

Start with a real operational constraint

The project did not begin with an executive saying, “We need an AI agent.”

It began with a customer support problem.

A major healthcare outage caused support volume and wait times to shoot up. Henry Schein One has a large and complex customer base, and its support teams need to understand many products and thousands of features. You cannot hire and train 100 people overnight when demand suddenly spikes.

Nate’s team also found that roughly half of its calls were “how-to” questions. These were important questions, but many did not require the deepest technical expertise.

The obvious answer might have been to push customers toward chat. Henry Schein One deliberately avoided that. Many dental practices had been trained for decades to pick up the phone when they needed help. Forcing them into a new channel would make the company’s operational problem the customer’s problem.

So the team looked for a voice AI system that could:

• Meet customers in the channel they already preferred

• Answer common questions at any hour

• Support multiple languages

• Escalate difficult cases to a human

• Improve the customer experience, not merely lower its cost

That is a much better starting point than “Where can we use AI?” It begins with a workflow, a constraint, and a measurable outcome.

The demo is not the product

A voice agent can sound impressive for five minutes. That does not mean it will work when a frustrated customer calls about a problem affecting their dental practice.

Nate’s team evaluated vendors with real support representatives and customers. They tested whether the agent could understand the product, retrieve current information, respond naturally, recognize emotion, and de-escalate difficult conversations.

They also tried to break it.

They yelled at it. They swore at it. They pushed it through the kinds of edge cases that do not show up in a polished vendor demo.

One of the most important criteria was how quickly the system could absorb corrections. Henry Schein One releases new features constantly. A good answer from last month can become a wrong answer today. The team needed to update the agent’s knowledge and see the change reflected quickly and reliably.

The evaluation continued after launch. They A/B tested introductions, reviewed hundreds of calls, and learned that seemingly small changes could make performance better or worse.

This is the same reason we built a structured approach to testing Solid’s chat. A single successful interaction proves very little. Production trust comes from repeated evaluation across expected questions, difficult questions, edge cases, and changes over time.

Make the pilot answer financial questions

When Nate took the idea to the CFO, he could not build a serious business case around tokens and theoretical model behavior.

As he joked in the episode, tokens sounded a little like “magic beans.”

So the team asked for a $40,000 pilot. They started with roughly 100 calls per day and used the pilot to answer the questions that actually mattered:

• How much did each interaction cost?

• How many calls could the agent resolve?

• When did it need to escalate?

• How quickly were issues resolved?

• What happened to customer satisfaction?

• How did performance change as the system improved?

This is an important distinction. The pilot was not just a smaller version of the final launch. It was an instrumented learning system.

Over a couple of months, the team improved the agent daily and built its ROI model from observed behavior. According to Nate, the AI could resolve appropriate calls in about one-fifth of the time of a live agent. It also expanded support to 24/7 and multiple languages.

Most importantly, Nate said customer satisfaction reached its highest level in the company’s history.

The business case became credible because it was based on production-like evidence, not a spreadsheet full of optimistic assumptions.

Treat the AI like a new employee

Looking back, Nate said the biggest misconception was that the team could plug in an AI tool, turn it on, and let it do the work.

“It’s more like onboarding an employee, a brand new employee.”

A new employee needs clear responsibilities, access to the right knowledge, examples of good work, rules for difficult situations, and guidance on when to ask for help. An AI agent needs the same things, except it cannot fill in organizational gaps by casually asking five coworkers what an internal term really means.

Nate made another observation that will sound familiar to anyone building AI for analytics:

“Sometimes we blame it as a hallucination when in reality we just weren’t clear on our training.”

For a support agent, the required context includes product documentation, troubleshooting steps, release information, customer language, emotional-response guidance, and escalation rules.

For an AI analyst, the context is different, but the problem is the same. It includes metric definitions, joins, filters, business terminology, validated queries, permissions, and the logic hidden across the data warehouse and BI environment.

The model can generate language or SQL. The context tells it what those words and numbers mean inside your business.

We have written before about using AI to create and maintain the context required by other AI systems. Nate’s experience reinforces the point: this context is not setup work you finish once. It is part of the production system and must evolve with the business.

Your frontline team becomes the AI team

The technical implementation was only one part of the work. The other part was helping employees understand what would change.

Support representatives naturally had questions about job security and the future of their roles. The successful response was not to ignore those concerns or make vague promises. It was to involve the team in the implementation.

The human support agents became essential to the system. They handled the more complex cases. They reviewed interactions the AI could not resolve. They identified gaps, corrected the knowledge, and trained the agent to perform better.

Their work moved up the stack, from repeatedly answering simple questions to becoming expert operators of the support system.

With the benefit of hindsight, Nate said he would spend even more time with the people affected by the project. They understand the exceptions, the customer language, and the practical reality of the workflow. They are also the people required to make the AI improve after launch.

There is no production AI without someone owning that last mile.

Once the operating model works, it can travel

After the support project showed results, Henry Schein One began applying the same approach to inbound sales.

The AI can learn the company’s product portfolio, answer initial questions, qualify a lead, and schedule a meeting with the right account executive. This allows the existing inbound sales development team to spend more time on outbound work.

At the time of our conversation, the pilot had already contributed to its first closed-won opportunity. The rollout remained cautious. A bad support interaction creates frustration, but a bad sales interaction can also lose revenue.

What is interesting here is not simply that Henry Schein One found another agent use case. It is that the company had built reusable organizational capabilities: vendor evaluation, knowledge ingestion, testing, monitoring, human handoffs, ROI measurement, and clear ownership.

Once those capabilities exist, the next use case is no longer a completely new experiment.

The lesson for AI analytics

A voice support agent and an AI analyst look very different to the end user. Underneath, they share the same production challenge.

Both need accurate, current business context.

Both need evals that test more than a happy path.

Both need observability, escalation, and a human owner.

Both can lose user trust with one confident, incorrect answer.

And both become more valuable when the people closest to the work help train and govern them.

This is the work of semantic engineering: turning scattered organizational knowledge into governed context that an AI can use and humans can maintain.

The visible model is only a small part of the system. The operating model around it is what turns an AI capability into a trusted product.

That is the real journey from pilot to production.

I thank Nate for sharing the practical details, including the parts that did not work immediately. You can listen to the full episode here: YouTube, Apple Podcasts, Spotify

I recently recorded an episode of Building with AI: Promises and Heartbreaks with Lindsay King, a consumer-facing marketer with about 15 years of experience across agencies and major consumer tech brands, including Uber and Instacart. (listen to it on YouTube, Spotify, Apple Podcasts)

Lindsay is not the typical guest we bring onto the podcast. Most of our conversations are with data leaders, AI builders, analytics practitioners, and people working deep inside the modern data stack.

That is exactly what made this one interesting.

Lindsay has spent years in brand marketing, but she also spent time in data observability and became fascinated by the modern data stack. So she sits in a very useful place: close enough to marketing to understand the creative and emotional side of the work, and close enough to data to see how much of the system still runs without the infrastructure it needs.

The core question we explored was simple:

What happens to brand marketing when AI makes creative production almost free?

Subscribe now

Production is no longer the hard part

Marketing has always had two broad buckets.

Performance marketing is easier to measure. You put messages into channels, test variants, track clicks or conversions, and optimize quickly.

Brand marketing works higher up the funnel. It is about awareness, perception, favorability, trust, and eventually preference. It is the reason you choose one app, product, or service over another before you can fully explain why.

Historically, brand campaigns moved slowly. A team would define the audience, write the brief, align stakeholders, bring in an agency, develop the creative, buy media, launch the campaign, and wait for results.

For a big campaign, this can take months. It can also cost millions.

AI changes the production side of that equation. It is now much easier to create more campaign ideas, more images, more videos, more variants, and more personalized messages for different audiences.

But Lindsay made an important point: that is only the first-order effect.

If everyone can produce more content faster, production efficiency stops being the moat.

The real advantage shifts to effectiveness.

In other words: who can figure out what will actually work?

Brand marketing has a memory problem

This is where the conversation started sounding very familiar to anyone who works in data.

Brand marketing has a lot of information, but much of it is fragmented.

There are campaign briefs, creative assets, media plans, audience segments, brand lift studies, agency learnings, performance reports, research decks, and post-campaign analysis. Some of it is structured. Much of it is unstructured. Some of it sits with agencies. Some of it sits in dashboards. A lot of it lives in people’s heads.

The problem is not that there is no data.

The problem is that the data is not connected into a reusable system of context.

A campaign launches. The team gets results. Maybe it drove lift. Maybe it changed perception. Maybe it taught the team something important about an audience, a message, a channel, or a creative direction.

But does that learning get structured and fed back into the next strategic brief?

Usually, not really.

Lindsay described brand as “the dark matter of the universe.” You know it is powerful. You can see the effect. But it is hard to measure directly.

That may have worked when campaigns were slower and creative production was expensive. It will not work as well in a world where AI lets teams produce many more options, much faster.

More content without better memory just creates more noise.

The next step is campaign simulation

One of the most interesting ideas Lindsay raised was campaign simulation.

Today, some companies already test concepts against synthetic audiences. But Lindsay’s point was that the next step is bigger than synthetic audiences.

It is synthetic channels.

Imagine taking a campaign concept, audience data, media plan, creative assets, past campaign performance, and external market context, then simulating how different options might perform before spending the media budget.

Not perfectly. Not as a replacement for judgment.

But directionally.

Which creative concept is likely to create the most lift? Which audience may respond best? Which channel mix looks weak? Which message is bold but risky? Which campaign is safe but probably forgettable?

That would change the role of the brand marketer.

The marketer would not just be managing a slow, linear campaign process. They would become more of a systems designer: defining the audience, business objective, constraints, message, context, and evaluation criteria, then using AI systems to generate and test more options.

The human still matters. Taste still matters. Judgment still matters.

But the human gets better infrastructure.

This is a semantic context problem

The connection to Solid’s world is pretty direct.

In AI analytics, we see the same pattern all the time. Giving an AI model access to data is not enough. It needs to understand the business context around the data.

What does this metric mean? Which tables are trusted? Which joins are valid? Which definitions are outdated? Which dashboard reflects accepted business logic? Which examples should the AI learn from?

Without that context, Text2SQL breaks. Data agents hallucinate. Business users lose trust.

Marketing is heading toward a similar wall.

If AI is going to help marketers make better decisions, it cannot only generate more assets. It needs to understand the context behind those assets.

What was the brief? Who was the audience? What was the message? What emotional response were we trying to create? Where did it run? What did we expect? What happened? What did we learn? Which assumptions should change next time?

That is not just a content-generation problem.

It is a semantic engineering problem.

Marketing teams need a structured memory of their own business logic. They need to turn briefs, assets, channels, results, and learnings into context that future systems can use.

Otherwise, AI will help them make more things without helping them make better decisions.

The future is not just more AI-generated ads

I do not think brand marketing will become purely scientific.

It should not.

Culture, emotion, timing, humor, and taste will always matter. A great brand campaign will always have something inside it that is hard to fully explain.

But “hard to explain” should not mean “impossible to learn from.”

The future of brand marketing is not just more AI-generated creative. It is a marketing system with memory.

A system that connects strategy, creative, audience, media, results, and business context. A system that helps teams move faster without becoming reckless. A system that helps marketers defend bold ideas with better evidence.

In AI analytics, trustworthy answers require semantic context.

In brand marketing, trustworthy creative decisions may require the same thing.

Thanks to Lindsay for joining me on the podcast. You can listen to the full conversation here.

If you would like to learn more about Solid, reach out to us.

If you want to listen to the podcast, find it on YouTube, Spotify, Apple Podcasts

I recently recorded a podcast episode with Preeti Vaidya, who works on AI at Morgan Health, part of JPMorganChase. Morgan Health focuses on improving outcomes for people covered by employer-sponsored insurance, which is a much larger population than many people realize. Roughly 160 million Americans receive health insurance through their employers, which means this is not a niche healthcare problem. It includes employees, families, benefits teams, clinicians, insurers, and the companies trying to help them all make better decisions.

Preeti sits at the intersection of healthcare, AI, data, clinical workflows, and employer-sponsored benefits. Naturally, we spent a lot of the conversation talking about AI in healthcare: care navigation, clinical decision support, radiology, fragmented health records, claims data, and the shortage of physicians. But the more we talked, the more I felt that healthcare was exposing a much broader enterprise AI problem.

Most AI programs still assume that the user wants to chat, and that’s wrong

Listen to the episode now on YouTube, Apple Podcasts, and Spotify.

That assumption works well for a very specific kind of user. It works for the person who is already curious about AI, already trying new tools, already comfortable writing prompts, and already willing to inspect the answer before trusting it. Preeti called these people “canary users.” Every organization has them. They are the early adopters who will test anything, find useful patterns, and prove that something is possible.

The mistake is treating their success as proof that the average user is ready.

Subscribe now

Canary users are not the average user

Every company has people who are excited about new technology. They are not just the first users of AI. They are the first users of almost every new tool. They ask for access before the tool is formally rolled out. They try things outside the organization and then come back asking whether they can use them internally. They are motivated, curious, and willing to tolerate friction because they believe there is value on the other side.

These users are incredibly important. They help teams discover what a model can do, where the workflows might change, and which use cases are worth pursuing. But they can also create a false signal. When an AI pilot works with canary users, it can look like the organization is ready for broad adoption. In reality, all you have proven is that motivated users can make a flexible AI interface useful.

That is not the same thing as building a product the average user will actually use.

The average user usually does not want to become good at prompting. They do not want to understand model behavior. They do not want to learn how to inspect every answer for hallucinations. They do not want another destination they need to visit during the day. They want help doing the job they already have.

This is where many AI programs start to break. They give everyone access to a chatbot, point to the success of the power users, and then wonder why the rest of the organization does not adopt it with the same energy.

The wrong default is “here is a chatbot”

The original GenAI experience was ChatGPT, so it makes sense that a lot of people still imagine AI through that interface. There is an empty box. You type something. The model answers. You refine. You ask again. Sometimes it works incredibly well.

But that is not how most operational work happens.

Preeti gave the example of a nurse in an emergency room. A nurse does not need a blank chatbot in the middle of a clinical workflow. She does not have time to stop, describe the situation, ask the model what to do, evaluate whether the model is hallucinating, and then figure out how to translate the answer back into care. The better experience is much more embedded. If a marker changes, surface it. If a patient needs attention, alert the right person. If relevant context exists, put it in front of the clinician at the point of decision.

That distinction matters. The value is not in making the nurse “use AI.” The value is in helping her deliver care more effectively inside the workflow she already has.

I see the same pattern in analytics. Giving a business user a blank “chat with your data” box sounds powerful, but it often creates more work. The user asks a vague question. The model makes assumptions. The answer looks confident. Then someone has to verify what happened, inspect the SQL, check the metric definition, and decide whether the answer can be trusted.

At that point, AI did not remove work from the system. It moved the work somewhere else.

This is why the product question cannot be, “How do we get everyone to chat with AI?” The better question is, “Where in the existing workflow would better information, surfaced at the right moment, change the outcome?”

Start with the workflow, not the model

Preeti’s approach was very practical. Before deciding where AI belongs, you need to understand the workflow as it exists today. Not the idealized version. Not the version described in a strategy document. The actual workflow, with all the handoffs, shortcuts, delays, repeated checks, and moments where people rely on judgment.

That means talking to the people doing the work. What does the nurse do today? What does the radiologist do today? What does the benefits administrator do today? Where do they lose time? Where do they need more context? Where are they overloaded with alerts? Where are they forced to make a decision with incomplete information?

Only after that mapping does it make sense to decide where AI should enter.

This sounds obvious, but a lot of enterprise AI programs do the opposite. They start with a model, vendor, or platform, then go looking for places to apply it. That usually produces impressive demos and disappointing adoption. The demo shows what is possible in a controlled setting. The workflow reveals what is useful in the real world.

The better path is less flashy but much more durable. Understand the workflow. Find the decision point where better context matters. Insert AI there. Make the output understandable to the person doing the work. Do not force that person to become an AI operator just to benefit from the technology.

In healthcare, that might mean clinical decision support. In analytics, it might mean surfacing the right governed metric, explaining a dashboard, generating SQL against an approved semantic model, or warning the user that the question is ambiguous before producing a number.

The pattern is the same: AI works better when it meets people where they already work.

Subscribe now

AI should support decisions, not pretend to make them

A lot of AI conversations eventually turn into the same question: will AI replace people?

In healthcare, that question often becomes: will AI replace doctors?

Preeti’s answer was clear. Healthcare is deeply personal. There are things AI can do extremely well, and there are things it should help with immediately. It can process large bodies of medical literature. It can help radiologists identify subtle changes in images. It can summarize relevant context. It can reduce administrative burden. It can help clinicians spend less time typing notes at the end of an exhausting shift.

But the human still matters.

A doctor may notice that something feels off even when the lab results look normal. A nurse may see a patient in a way the data does not capture. A clinician brings judgment, empathy, and accountability to the room. AI can support that work, but it should not be framed as a replacement for it.

That framing is useful outside healthcare too. In analytics, the goal should not be to pretend the AI is the analyst, the data team, and the business decision-maker all at once. The goal should be to help people make better decisions with less friction. That means giving them relevant context, governed definitions, clear assumptions, and answers that can be inspected and trusted.

Decision support is a stronger product promise than decision replacement. It is also a more honest one.

Data fragmentation is not one problem

At one point in the conversation, I asked Preeti about something I personally feel as a patient: my healthcare data is everywhere. I have multiple portals across different providers. There are labs, clinical notes, insurance claims, wearable data, family history, and information I may or may not remember to share. Even if I personally want all of it connected, the system does not make that easy.

So the question was simple: how can AI work if the data is so fragmented?

Preeti’s answer was basically yes, data fragmentation is a huge problem, but no, we do not need to solve all of it before AI can be useful.

I liked that answer because it avoids two bad extremes. One extreme is pretending the data problem does not matter. The other is waiting for a perfectly unified data world before building anything useful. Neither is realistic.

Preeti broke the problem into several layers. There is system fragmentation, where different providers and systems hold different parts of the patient picture. There is temporal fragmentation, where claims data may be useful but delayed by 30 to 90 days. There is also semantic fragmentation, where different systems describe things differently, and where clinical notes, claims codes, lab results, and provider judgment do not automatically line up just because the data has been collected.

That last layer is especially important.

Semantic fragmentation is not just a healthcare problem. Every enterprise has it. One team says “customer,” another says “account,” and another says “workspace.” A revenue metric means one thing in finance and something slightly different in sales. A dashboard contains logic that is not fully documented. A dbt model contains logic that is technically correct but not obvious to a business user. A Slack thread explains the exception, but the AI cannot see it.

Then someone asks an AI analyst a question and expects it to know what the business means.

This is why AI for analytics cannot just be a Text2SQL problem. Generating SQL is not enough. The system needs business meaning. It needs definitions, relationships, assumptions, governance, and context about how the data is actually used. Without that semantic layer, the AI may produce an answer that is syntactically correct and operationally wrong.

Build for incomplete context

One of the most important points Preeti made was that AI builders need to understand not only what their data can do, but what it cannot do.

That is easy to say and hard to implement. Most AI demos are built around the happy path. The model has the right data. The user asks a clean question. The answer exists. The output looks impressive.

Production is not the happy path. Production is missing context, ambiguous language, conflicting definitions, lagged data, undocumented business logic, and users who do not know exactly how to phrase what they need. The system has enough information to sound confident, but not always enough information to be right.

A useful AI system needs to recognize that gap.

In healthcare, that might mean knowing that it has today’s lab result and the current visit notes, but not the full patient history. In analytics, it might mean knowing that there are two definitions of gross margin and asking which one should apply. In both cases, the system becomes more trustworthy when it can explain what it knows, what it does not know, and which assumptions it is making.

This is one of the places where I think semantic engineering becomes essential. The goal is not to create a perfect data world. The goal is to make enough meaning explicit so that AI can behave responsibly in an imperfect one.

That includes knowing when to answer, when to ask a clarifying question, when to use a governed definition, and when to say that the data does not support the request.

The best AI may feel less like AI

One of the interesting patterns in the conversation was that the best healthcare AI examples did not sound like standalone AI tools. They sounded like better workflows.

Care navigation that appears inside the channels people already use. Radiology support that helps clinicians review images more effectively. Clinical context that appears when it is needed. Documentation support that reduces after-hours work. These are not experiences where the user goes somewhere else to “do AI.” They are experiences where AI makes the existing work easier.

That may be a good test for enterprise AI products.

If adoption depends on users leaving their workflow, learning a new behavior, and becoming good at prompting, the bar is very high. If AI is embedded into the place where the work already happens, the bar is different. The user does not need to believe in the AI transformation story. They just need the next step to be easier, faster, or more reliable.

This also changes how we should evaluate success. The question is not how many people logged into the AI tool. The question is whether the workflow improved. Did the clinician save time? Did the patient find the right provider? Did the analyst avoid repetitive work? Did the business user get a governed answer instead of creating another ad hoc spreadsheet? Did the data team reduce the amount of manual clarification required before answering a question?

AI adoption is not the outcome. Better work is the outcome.

The real promise is less friction and better decisions

The optimistic version of AI in healthcare is not that we all get an AI doctor and never see a human again. The optimistic version is more practical and, I think, more compelling.

Clinicians spend less time on administrative work. Patients get better navigation. Benefits become easier to understand. Radiologists get support spotting subtle changes. Doctors get faster access to relevant research. People in underserved areas get more consistent support. The shortage of clinicians becomes less painful than it otherwise would have been.

AI does not need to replace the system to improve it. It needs to reduce friction in the right places.

That is the lesson I took from the conversation with Preeti. It is also the lesson I think applies to almost every enterprise AI effort. The future will not belong to the flashiest chatbot. It will belong to systems that understand the workflow, respect the limits of the data, expose their assumptions, and help humans make better decisions.

That is true in healthcare.

It is true in analytics.

And it is probably true everywhere AI actually needs to work.

Listen to the episode now: YouTube, Spotify, Apple Podcast.

Over the last couple of weeks, three things happened that are worth putting together.

Snowflake announced Cortex Sense, a foundational context layer for CoWork that “automatically learns how a business defines its data and operations.” Cortex Sense uses signals from query history, metadata, dashboards in Power BI and Tableau, and enterprise data outside Snowflake to understand things like revenue definitions, fiscal calendars and snapshot tables. Snowflake also shared a pretty incredible internal test: CoCo and CoWork reached 83% accuracy with Cortex Sense, compared to 47% without it and 23% for frontier coding agents with Snowflake MCP. (snowflake.com)

Anthropic published a post on how they use Claude for self-service data analytics. The point was not “Claude can write SQL now.” The more interesting point was that analytics accuracy is mostly a context and verification problem. Their team wrote that the central challenge is mapping a user’s question to the right entities in the data model and knowing how to work with them. Once that context exists, writing the SQL becomes the easier part. (claude.com)

And Marc Benioff recently made the point that AI needs a semantic layer to work well. Salesforce has also been pushing this publicly through its Open Semantic Interchange work, arguing that agentic AI requires a realignment around the semantic data model, and that agents need trusted semantic definitions to translate intent into accurate outputs. (atscale.com)

Enterprise AI needs semantics. Which is exactly what we’ve been saying for over a year.

Subscribe now

Recap: why semantics are needed (feel free to skip)

The problem with AI and enterprise data is not just Text2SQL. It is not enough for a model to generate a syntactically valid query. The real challenge is understanding what the user actually meant, which data should be trusted, and how the business defines the thing being asked about.

When someone asks for ARR, do they mean booked ARR, billed ARR, active ARR or ending ARR? When someone asks for churn, are they asking about logo churn, gross revenue churn, net revenue churn, voluntary churn or something else? When someone asks about “active users,” what actually counts as active?

The answer is rarely obvious from the schema. A table name can help. A column name can help. But the real meaning is usually spread across dashboards, SQL queries, dbt models, BI tools, tickets, documentation, Slack threads and people’s heads.

That is why AI needs semantics.

We wrote about this in Autogeneration of a semantic layer - the key for AI/BI, where we argued that AI needs a business-aware layer between users and data.

Manual creation of semantics will get you nowhere

We wrote about it in Semantic layer for AI: let’s not make the same mistakes we did with data catalogs: the obvious failure mode is already familiar: companies will try to manually document everything, and the documentation will immediately start going stale.

That is the important part. The semantic layer AI needs cannot be another manual project.

Enterprises have too much data, too many systems, too many dashboards, too many metric definitions and too many edge cases. A small team can manually define the top 20 metrics. Maybe the top 100. But enterprise AI needs much more than that. It needs to understand the full structured data estate, including cloud and on-prem systems, old and new platforms, BI and warehouse layers, operational systems and the business workflows around them.

This is where auto-generated semantics become necessary.

Not because an LLM should blindly invent your business definitions. That would be a terrible idea. Anthropic is right to point out that auto-generated metric definitions can look plausible while preserving the exact ambiguity you were trying to remove.

The answer is not “let the model guess.”

The answer is to generate semantics from the enterprise’s existing evidence: actual queries, dashboards, lineage, metadata, transformations, documentation, usage patterns, tickets and business context. Then score that evidence, reconcile conflicts, surface uncertainty and let humans review, approve and own the definitions.

That is very different from asking AI to hallucinate a metric layer from raw tables.

But building an automated ain’t easy (we should know…)

A heavily used executive dashboard connected to a curated model is a strong signal. A random query from three years ago is a weak signal. A dbt model with tests and lineage is a strong signal. A stale dashboard nobody opens is probably not. A ticket explaining why a metric changed last quarter may be more useful than a column description written five years ago.

The semantic layer should learn from all of this.

And it should keep learning, because the business keeps changing.

This is where Solid is leading.

Solid was built around the belief that enterprise AI needs an enterprise-wide semantic foundation. Not a narrow layer inside one warehouse. Not a semantic model locked to one BI tool. Not a manual spreadsheet of metric definitions. A semantic layer that spans the full structured data set of the enterprise, across cloud and on-prem systems, and connects technical metadata with business meaning.

That is the future Snowflake, Anthropic and Salesforce are all pointing toward.

Snowflake is saying agents need context. Anthropic is saying analytics needs trusted mappings between questions and data models. Salesforce is saying agentic AI needs a semantic layer.

We agree.

We would just add one more thing: in the enterprise, that semantic layer has to be generated, continuously maintained and connected to the full data estate. Otherwise it becomes another partial artifact that starts clean and slowly drifts away from reality.

AI will not succeed in the enterprise by guessing what the business means. It will succeed when it has a trusted semantic foundation.

That is what we’ve been saying for over a year. And that is what Solid is building.

Semantic models need the same shift.

If semantic models are going to power AI agents, Solid’s Text2SQL, BI tools, Snowflake Cortex, Databricks Genie, Looker, dbt, and every other analytics interface, “I think this model is right” is not good enough. We need Test Driven Semantic Modeling. And this is one of the most important innovations we’re building into Solid.

Subscribe now

Semantic models are becoming production software

Semantic layers used to be mostly about consistency. Define ARR once. Define active customer once. Make sure dashboards don’t contradict each other too badly. That is still important, but AI changes the stakes.

Now the semantic model is the thing standing between a business question and the SQL an AI system generates. If the model understands the business correctly, the AI has a chance. If it doesn’t, the AI confidently generates nonsense.

This is why everyone is suddenly talking about semantic models again. The AI needs business context. But there is a hard truth underneath all the excitement: creating the model is hard, testing it is harder, and maintaining it over time is brutal.

Generation helps, and we’ve written before about why autogeneration of a semantic layer is key for AI/BI. But generation alone is not enough. A generated model still needs to be validated.

The old way does not scale

Today, most semantic model work is painfully manual. A data team starts with warehouse tables, ambiguous columns, dashboard logic, dbt models, old SQL queries, and business definitions scattered across the company. Then they manually choose entities, define joins, write metrics, add descriptions, test a few questions, fix obvious issues, and ship.

Then production happens.

A user asks a question the model doesn’t understand. A column changes. A metric definition changes. The AI picks the wrong join path. The generated SQL returns the right-looking number for the wrong reason. Now the team has to debug everything manually.

This is not a scalable process. It is also not how we build serious software.

The TDD idea, translated to semantic models

In software, a test defines expected behavior. For semantic models, the equivalent is a business question with expected SQL.

For example: “Which customers have high average daily balances but few open product types?”

That question is the test. The expected SQL is the ground truth. A semantic model passes only if the Text2SQL engine (such as Solid Analyze), using that model, generates SQL that returns the expected result and uses the right business logic.

That last part matters. It is not enough to accidentally return the same number. The SQL needs to use the right tables, joins, filters, and metric definitions. Otherwise, you don’t have correctness. You have luck.

This is what Solid Benchmarking does.

For every semantic model, Solid maintains a benchmark suite: business questions paired with expected SQL. These benchmarks can be generated automatically from historical SQL, added manually, imported in bulk, or promoted from real production questions asked through Solid’s MCP server. Solid then runs the questions, compares generated SQL to expected SQL, and shows what passed, what failed, and why.

That changes the workflow completely. Instead of building the model, hoping it works, finding out later, and fixing it manually, you get a loop: generate the model, run the benchmark, see what failed, apply fixes, run again, and save a better version.

That is red, green, refactor for semantic models.

Optimize turns failures into fixes

A failing benchmark is useful, but only if it helps you improve the model. A question might fail because the model is missing a column, because the wrong join was selected, because a metric is ambiguous, or because the expected SQL is outdated.

This is where Solid’s Optimize capability matters.

Benchmark failures feed into Optimize. Solid analyzes the failures, identifies root causes, groups related issues, ranks them by impact, and recommends fixes. In most cases, the modeler can review and apply the fix with AI.

This turns semantic modeling into an operational loop, not a one-time project. You are not manually debugging generated SQL for hours. You are reviewing prioritized recommendations based on actual failures.

Production becomes part of the benchmark

The best test suite is not only based on what you thought users would ask. It also learns from what users actually ask.

This is why the MCP usage loop is so important. When an AI agent uses Solid’s MCP server to answer business questions, Solid shows the questions asked, the SQL generated, and the answers returned. A modeler can take a real production question and add it directly to the benchmark suite.

Production teaches the model. The benchmark suite remembers.

This is how semantic models improve over time instead of slowly becoming stale.

Versioning makes it safe

Optimization without versioning is scary. What if a fix improves one question and breaks five others? What if accuracy drops after a change? What if the model worked last week and nobody knows what changed?

Solid’s Versioning closes that gap.

Benchmark runs are tied to specific model versions. If accuracy improves, you know which change helped. If accuracy drops, you can see which version introduced the regression and restore the previous one.

That creates the safety net semantic models have been missing. Benchmarking tells you whether the model is correct. Optimize tells you how to improve it. Versioning lets you make changes without fear.

This is the real innovation

A lot of people focus on Solid’s ability to generate semantic models automatically. That makes sense. Seeing a model created in minutes is impressive.

But the deeper innovation is the lifecycle.

A semantic model is not a YAML file. It is not documentation. It is not a one-time artifact. It is production infrastructure. And production infrastructure needs tests, observability, optimization, versioning, and rollback.

Software engineering learned this lesson decades ago. Data teams should not have to relearn it the hard way.

The future of semantic modeling will look much more like modern software development: tests before trust, benchmarks before production, optimization based on failures, versioning for safety, and production feedback as fuel.

That is Test Driven Semantic Modeling.

And if AI analytics is going to move from impressive demos to trusted production systems, this is not a nice-to-have. It’s the foundation.

If you want to see this in action, contact us.

A few months ago, I started recording a podcast with leaders and influencers in our space, specifically ones who have worked to build something of substance with AI.

The latest episode I recorded is with Nondas Virvikatis. Nondas and I have known each other for a while, and I always enjoy speaking with him about AI, where things are going, what is actually working, and what is still painful. You can jump straight into the episode NOW: YouTube, Spotify and Apple Podcasts.

Nondas leads a team of product managers at UBS Wealth Management Americas, building AI capabilities for financial advisors in the US. Before that, he spent years across product, analytics and data, including in financial services and growth-stage startups.

Subscribe now

AI is not replacing the financial advisor

When most people hear “AI in wealth management”, their mind probably goes to ChatGPT managing their money, buying and selling stocks, and deciding what they should do with their portfolio.

That’s not what Nondas and his team are building.

Wealth management is a relationship-heavy business. Financial advisors understand the client’s financial goals, tax situation, estate planning needs, personal context, risk appetite and life goals. In the US, Nondas explained, financial advisors often operate almost like entrepreneurs. Their client relationships are deeply tied to them.

So the goal is not to replace the advisor.

As Nondas put it, “AI is not going to replace the humans in this industry, but people who use AI will replace the people who don’t use AI.”

That framing is important. AI is not the product. The financial advisor is still the product, in many ways. AI is the leverage.

This is similar to how I think about AI in analytics. We’ve written before that throwing away BI is a bad idea, and that blindly giving business users an unchecked chatbot can be dangerous. The better path is usually augmentation: help humans do the work faster, better and with more confidence.

That same idea applies here.

A good advisor still needs to sit with the client, understand them, calm them down when markets are chaotic, explain tradeoffs, and help them make decisions. AI can help the advisor prepare better, find information faster, personalize service and operate more effectively.

But you still want the human in the loop.

Generic AI gets you part of the way. Specialized AI gets you further.

We also talked about the difference between generic AI tools, like Microsoft 365 Copilot, and more specialized tools built for a specific workflow.

Nondas sees both as valuable.

Generic AI tools can help with productivity. They can write, summarize, help create questionnaires, draft interview questions, prepare for internal presentations, process information and generally accelerate knowledge work.

But in a regulated, high-stakes environment, “good enough” is not always good enough.

There are cases where a generic tool gets you 80% of the way there. That’s useful. But there are also cases where you need much higher certainty, stronger control, better data integration, better evaluation and workflow-specific behavior.

This is very aligned with what we see in data and analytics. General-purpose AI is amazing, but when you want to connect it to enterprise data, definitions, dashboards, metrics and permissions, you need more than a generic chatbot. You need context, governance, evals, observability and a workflow that understands the job to be done.

That’s why we spend so much time thinking about semantic layers for AI, evals and how to keep a close eye on AI behavior with tools like LangSmith.

The same is true in wealth management.

Copilot is useful. But if you want AI to support financial advisors in specific client workflows, with firm-specific data, policies, expectations and risk controls, you need specialized products.

The Chief AI Officer is not just a fancy title

We talked about the rise of the Chief AI Officer role. Someone on LinkedIn once joked that having a Chief AI Officer today is like having a Chief Electricity Officer 100 years ago.

Funny. Also, not completely wrong.

But in a large enterprise, especially a regulated one, the role makes sense.

Nondas described it as a hub-and-spoke model. The central AI organization helps with tooling, training, infrastructure, responsible AI, governance, communities of practice and avoiding 20 teams building the same thing 20 times.

Then each business area figures out how to apply those capabilities to its own workflows.

That balance matters.

If everything is centralized, AI becomes too detached from the business. You get platform teams building impressive capabilities that no one uses.

If everything is decentralized, every team reinvents the wheel, risk management becomes impossible, and the enterprise ends up with chaos.

The winning model is somewhere in the middle: central enablement, local execution.

That feels right to me.

We’ve seen similar patterns in data. A central data team can build the foundation, governance and core assets, but the business value usually happens closer to the workflow. The people who understand the work need to be close to the AI.

Adoption is a product problem

One of the most interesting parts of the conversation was not technical at all.

Nondas emphasized that building the technology is not enough. You need communication. Training. Repetition. Champions. Use cases. Feedback loops. Follow-up.

Financial advisors, as he said, are wired to sell. They are not wired to adopt new technology just because a product team shipped it.

This is such an important point.

Many enterprise AI efforts fail not because the model is bad, but because the rollout is bad.

People need to understand:

• Why should I use this?

• How are my peers using it?

• What does “good” look like?

• When should I trust it?

• When should I not trust it?

• What happens if it fails?

Nondas and his team spend a lot of time close to the field, speaking with users, collecting feedback, identifying champions, and sharing use cases across teams.

This is exactly the mindset I think enterprise AI needs. You cannot just launch and hope for the best. AI adoption is not a Slack announcement.

It’s a product motion.

And the users of internal AI tools should be treated like customers. You look at usage. You segment users. You interview power users. You interview people who dropped off. You ask what changed. You figure out what triggered adoption or abandonment.

That applies whether you’re building AI for financial advisors, analysts, salespeople, product managers or engineers.

The technical problems are still very real

Of course, there are also plenty of technical challenges.

Nondas called out a few:

• Data availability

• Data quality and readiness

• Performance

• Evals

• Feedback loops

• Fallback processes

• Risk controls

That list should sound familiar to anyone building AI inside an enterprise.

The application layer may look simple. A user asks a question. AI gives an answer. Magic.

But underneath, there is a lot of plumbing. Do you have the right data? Is it permissioned correctly? Is it fresh? Is the answer good enough? How do you know? What happens when the model is wrong? How do users report issues? Can they fall back to a traditional process?

This is why I keep saying that AI for enterprise workflows is much harder than demos make it look.

A demo can be built in a weekend.

A production-grade enterprise AI system needs to survive contact with messy data, real users, security teams, compliance teams and business expectations.

As we wrote in (Almost) no AI in production, there’s still a huge gap between experimentation and production. Nondas’ experience reflects that gap, but also shows how serious organizations are starting to cross it.

Regulated environments need levels of freedom

One of the tensions we discussed is the gap between what you can do at home and what you can do inside a large bank.

At home, you can vibe code freely. Open Claude, Cursor, Gemini, ChatGPT, connect things, upload files, build little tools, burn tokens, break stuff, fix stuff, start over.

Inside a major financial institution, that freedom has to be constrained.

And rightfully so.

Nondas described a useful mental model: the broader the impact, the stronger the controls should be.

If you are using AI for yourself, the risk is smaller. You can have more freedom.

If you are building something that affects your team, you need more structure.

If you are building something that affects clients, advisors or the broader firm, you need real controls, approvals, model risk management and governance.

That makes sense.

The trick is not to eliminate freedom. If you do that, you kill innovation.

The trick is to match the level of control to the level of risk.

Small blast radius, more freedom.

Large blast radius, more control.

That is probably the right way to let enterprise AI grow without letting it blow up.

Copilot, Gemini and the “can I do this faster?” habit

Nondas also shared a very practical habit that I loved.

Every time he is about to do something on his computer, he asks himself whether he could do it faster or better with AI.

That’s it.

That habit alone probably separates people who really benefit from AI from people who just occasionally open a chatbot.

He uses AI to draft questionnaires, prepare interview questions, work on internal presentations, process data and support both professional and personal workflows. In his personal life, he uses Gemini heavily because his personal ecosystem is already in Google. At work, his professional context lives more in Microsoft tools.

His comment was great: “your imagination and your connectors, I think, are your two limitations.”

That is exactly right.

The model matters. But the connectors matter too.

If AI has access to your calendar, files, documents, messages, data and workflows, it becomes dramatically more useful. If it is disconnected from everything, you spend your life copying and pasting.

This is why the next few years will likely involve a massive amount of work around APIs, connectors and workflow integration.

The chatbot window is useful. The connected AI system is much more useful.

Vibe coding is addictive

We ended up talking about something I think many of us are experiencing, but maybe don’t always admit.

Vibe coding is addictive.

Nondas described building personal tools with Claude and Claude Code, including an asset aggregator across accounts in Greece and the US, and even a networking database that could help match people in his network based on what they need.

I immediately understood the feeling.

You sit there. You prompt. It builds. It fails. You prompt again. It gets closer. You prompt again. Suddenly your wife is telling you that you need to leave, and you’re saying, “just one more prompt.”

It feels like an arcade machine, except instead of putting in coins, you’re burning tokens.

There is something very powerful about having a machine work with you in real time. It is not passive software. It responds. It tries. It improves. It sometimes frustrates you. It sometimes surprises you.

And yes, sometimes it goes too far.

Nondas made a great point about models overthinking or continuing too long before checking whether they are on the right path. His approach is to constrain the work: do this for three profiles first, let me review it, then scale it.

That’s a good lesson for all of us.

Treat AI like a junior teammate. Give it enough room to be useful, but don’t let it run 500 miles in the wrong direction before checking in.

AI is not cheating

In the lightning round, I asked Nondas for an AI myth he would bust.

His answer was that people should stop feeling uncomfortable admitting they used AI to get to an outcome.

I strongly agree.

There is still a weird stigma around using AI, as if using it means you didn’t really do the work.

That’s nonsense.

Of course, don’t plagiarize. Don’t blindly submit something you didn’t review. Don’t pretend you wrote every word by hand if you didn’t.

But using AI to think, draft, analyze, improve, summarize, structure, code, research or prepare is not cheating.

It’s using a tool.

In a few years, I think this stigma will look silly. Just like no one says you cheated because you used Excel instead of doing math on paper.

The question won’t be whether you used AI.

The question will be whether the output was good.

The next five years: connectors, connectors, connectors

When I asked Nondas for one AI prediction for the next five years, he made it conditional.

If we invest more in APIs and connectivity between applications, AI will really take off.

I think he’s right.

Models will keep improving. But for enterprise use, the bigger unlock may be access. AI needs to connect to the systems where work happens.

CRM. BI. Data warehouses. Documents. Email. Calendar. Ticketing systems. Product tools. Financial systems. Workflow engines.

Once AI can securely and reliably operate across those systems, the value changes.

It stops being a chatbot.

It becomes a worker.

Not a replacement for humans, at least not in most of the enterprise workflows we discussed. But a worker that can help humans move faster, make better decisions and spend less time fighting the machinery around them.

That’s the direction we’re all heading.

I thank Nondas for spending the time with me on the podcast. You can listen to our conversation on YouTube, Spotify and Apple Podcasts.

In the meantime, if you would like to learn more about Solid, reach out to us.

Thanks for reading Building an AI-powered Analytics Startup! Subscribe for free to receive new posts and follow our journey.

I’ve been around long enough to remember when semantic layer meant Universe Designer. In the 1990s BusinessObjects introduced a “universe” — a metadata model that hid SQL and let business people drag and drop their way through a relational database. If you worked in BI back then you probably spent months building one: picking tables, defining joins and measures, and praying the business wouldn’t change its mind.

Three decades later that old idea has resurfaced. LinkedIn feeds and Gartner reports are buzzing about semantic layers. Google Trends shows a hockey‑stick curve for searches on the term. Why now? What changed? And what can the past teach us about the future?

Subscribe now

Universes: hiding SQL from business users

The first commercially successful semantic layer was the Universe. BusinessObjects’ Universe Designer let developers build a metadata layer on top of relational databases. It mapped tables into business‑friendly objects and defined measures and dimensions so that users could query without writing SQL. The Universe acted as a bridge between raw data and people, abstracting complexity, providing governed access and enabling drag‑and‑drop analysis. For many enterprises in the 1990s and 2000s it became the backbone of reporting and budgeting.

But the early universes had limitations. They were tightly coupled to the BusinessObjects stack; building and updating them required specialised tooling; and each universe was tied to a single database. When SAP acquired BusinessObjects it introduced the UNX universe and Data Foundation to federate multiple sources. Even so, universes remained proprietary and slow to adapt. The lock-in was a major detractor. Teams needed faster answers, and self‑service BI promised agility.

Cubes and OLAP: pre‑aggregated semantics

At the same time, OLAP platforms like Microsoft Analysis Services and Hyperion Essbase defined hierarchies, measures and dimensions in cubes. Cubes pre‑aggregated data so queries were fast, and they embodied a kind of semantic layer by encoding business logic (e.g., calendars and product hierarchies). Yet they were also vendor‑specific and required ETL to flatten data into a multidimensional structure. They thrived in finance and supply‑chain planning but were less suited to the messy, evolving schemas of modern analytics.

Self‑service BI and the semantics vacuum

By the late 2000s a new mantra emerged: self‑service. Tools like Tableau and Qlik let analysts connect directly to databases, build their own dashboards and share insights without waiting on IT. This speed came at a cost: every team defined metrics differently. The shared context encoded in universes and cubes evaporated. Analysts wrote SQL with business logic sprinkled throughout; dashboards proliferated; definitions drifted. In other words, the semantic layer went away because it was too slow and too proprietary to keep up.

LookML, HANA and semantics as code

The next generation tried to marry agility with governance. Looker introduced LookML, a developer‑friendly language for defining dimensions, measures and joins in YAML. LookML treated semantics as code: version‑controlled, reusable and composable. But it was still locked inside the Looker ecosystem (now, the BigQuery ecosystem within Google Cloud).

SAP took a different approach with HANA. Its calculation views and Core Data Services (CDS) moved semantic modelling into the database itself, pushing down business logic for real‑time analytics. HANA’s semantic layer supported federated queries across universes, BW (BEx) queries and external systems. Despite these advances, semantics remained tied to particular vendors, and adoption was limited to enterprises deep in those stacks.

Semantics for purists, not for the business

One of the biggest failures of semantic layers was that no one, outside the data / BI team, cared about them. Data leaders would debate which semantic approach is best, and where you should manage your entities and their relationships.

But, the business didn’t care, and that meant that financially these endeavors would never take off. Companies built to fix the semantic layer problem struggled to get real traction, and saw their growth flatten out.

The context crisis: AI exposes the gap

Enter generative AI. Large language models (LLMs) can answer questions and generate analyses, but they don’t know what revenue means. When analysts ask natural‑language questions directly against a warehouse, LLMs often guess incorrectly; one study found that natural‑language queries against raw data were wrong in 80 % of cases. Without shared definitions, AI amplifies ambiguity and hallucinations.

On the flip side - well built semantic layers can generate massively reliable results, as dbt recently showed.

This context crisis is reigniting interest in semantic layers. Instead of hiding SQL from humans, the new goal is to teach machines what data means. AI‑driven analytics require deterministic definitions, machine‑enforceable relationships and real‑time governance. The old universes were built for reporting. The new semantic layers are built for agents.

What’s different this time?

Several things make today’s semantic layers more than a repeat of the past:

• Open and headless. Modern semantic layers are decoupled from any one BI tool. They expose definitions via open APIs (MCP, REST, GraphQL) so that agents, dashboards, notebooks, AI-assisted coding and chatbots can all consume the same context.

• Machine‑enforceable semantics. Definitions are not just documentation; they are executed automatically. When a query references net revenue, the layer applies the correct joins, filters and security policies without the analyst having to remember them, nor the business user needing to know them.

• Federated and composable. Unlike early universes, modern layers can sit on top of multiple warehouses, vector stores and knowledge graphs. They let you define semantics once and use them everywhere, even across platforms.

• Driven by AI. The killer application is no longer drag‑and‑drop reporting; it’s AI copilots and agents that need trustworthy data to automate tasks and decisions. Research shows that grounding LLMs in a semantic layer increases accuracy by more than 3×.

In short, we’re not going back to 1992. We’re building something new: a universal, open semantic layer that acts as the control plane for both humans and machines.

The new role: semantic engineering

All of this raises a people question. In the old days the Universe designer sat in IT and rarely interacted with the business. Today, someone needs to own your company’s semantic layer and continuously align it with reality. That person might be a finance lead who knows the nuances of ARR, an operations manager who understands edge cases, or an analytics engineer who curates dbt models. Naming them semantic engineers gives them a mandate: codify the business logic that AI depends on.

Their job isn’t just to build dashboards; it’s to maintain a living semantic model that exposes metrics, relationships and policies to every tool and agent. Success is measured by how reliably AI operates across the enterprise, not by how many reports they ship.

Writing SQL code by hand will be gone by the end of 2026. Maintaining semantic models will be the new norm.

Lessons from history — and how to start

The history of semantic layers teaches us three things:

1. Abstraction matters. Universes and cubes were successful because they hid complexity and empowered non‑technical users.

2. Coupling kills adoption. They fell out of favor because they were tightly bound to proprietary tools. As the stack diversified, semantics didn’t keep up.

3. Context is infrastructure. The AI era demands deterministic, governed semantics. Without them, LLMs hallucinate and agents do the wrong thing.

If you want to build a semantic layer today, start small. Pick a domain where metric drift causes pain; nominate a semantic engineer; generate a first model using existing SQL and BI assets (Solid can automate this for you); then iterate. Treat semantics as code, version it, and share it via MCP. Learn from the history and avoid the coupling mistakes of the past.

If you’d like to see a modern semantic layer in action, get a demo. We’ll show you how Solid generates and maintains a universal semantic layer on top of your data so that humans and AI can finally speak the same language.

Check out Solid

Last week, I did a very startup-y thing.

I took a bunch of our recent customer and prospect interactions, anonymized the sensitive stuff, and asked ChatGPT a simple question:

Why are organizations coming to Solid right now?

Not what do we think they want. Not what our pitch says they should want. Not what the analyst reports say the market wants.

What are they actually trying to solve?

This is not a scientific survey. It’s not a Gartner Magic Quadrant. It’s not statistically significant. It’s a qualitative analysis of the last couple of months of conversations, pilot plans, solution proposals, and meeting summaries.

Still, the pattern was surprisingly clear.

Companies are not coming to us because they woke up one morning and decided they need a semantic layer.

They’re coming because they want AI to do real work with their data, and they’re discovering that the missing piece is the context layer, and semantics, that makes their data understandable, governed, and reliable.

In other words: AI has created the urgency. Messy enterprise data has created the blocker.

Subscribe now

The short version: everyone wants agents, but agents need trusted data

A year ago, most conversations about AI and data were still vague.

“We want to use GenAI.”
“We want chat with data.”
“We want business users to self-serve.”
“We want to reduce analyst bottlenecks.”

Those conversations still happen. But they’re getting sharper.

Today, the question is much more concrete:

How do we let an AI agent query our data, understand our metrics, use the right joins, respect our business definitions, and return something people actually trust?

That’s a very different conversation.

It’s no longer about a chatbot sitting on top of a warehouse and hoping for the best. We wrote about that early on in Data Chatbots: what people are really doing, and the reality hasn’t changed much. If the AI doesn’t understand the data, the chat experience breaks very quickly.

What has changed is that organizations are now trying to move beyond demos. They want AI agents and AI-powered workflows in production.

And production is much less forgiving than a demo.

Trend #1: “Chat with your data” is still the door opener

The most common entry point is still some version of “chat with your data.”

Business users want to ask questions in natural language. Data teams want to reduce ad-hoc requests. Executives want answers faster. Everyone wants the magic box where you type a question and get a reliable answer.

This is not surprising. Snowflake, Databricks, Microsoft, Google, and others are all pushing the market in this direction. Snowflake has Cortex Analyst, Databricks has Genie, and the broader AI ecosystem is converging around standards like MCP.

The surprise is that “chat with data” is often not the end goal.

It’s the easiest way to describe the pain.

The real pain is that business users don’t know where the data is. Analysts are tired of being the routing layer between every business question and every database table. Data engineers are tired of maintaining fragile logic across dashboards, notebooks, semantic layers, and half-forgotten SQL files.

So the user says “chat with data,” but what they often mean is:

Please make our data understandable enough that humans and machines can use it without opening five Slack threads and asking the one analyst who has been here since 2017.

That’s a much bigger problem.

Trend #2: the analyst bottleneck is getting worse

Another strong pattern: data teams are overwhelmed.

Not “busy.” Overwhelmed.

In many organizations, analysts are still spending too much time finding the right table, reverse-engineering old dashboards, understanding metric definitions, and figuring out which version of a query is trusted.

We’ve written before that nobody cares about the efficiency of the data analyst. That headline was intentionally annoying, but the point was real: the business usually doesn’t buy “analyst efficiency” as the main outcome.

What they do care about is speed to decision.

What we’re seeing now is that the analyst bottleneck is becoming a business bottleneck. The business wants campaigns optimized faster. Sales wants better account signals. Finance wants cleaner performance views. Operations wants alerts that trigger action, not another dashboard to check.

The analyst is sitting in the middle of all of that.

So organizations come to us with a technical problem, but underneath it is a business problem: the demand for data work has exceeded the human capacity to fulfill it.

AI is supposed to help, but without a trusted semantic layer, it creates more review work, not less.

Trend #3: manual semantic modeling does not scale

This one comes up again and again.

Enterprises understand the need for a semantic layer. They know that metrics, dimensions, relationships, definitions, and business logic need to be captured somewhere. They know that without this layer, Text2SQL and agents will hallucinate.

The issue is not awareness.

The issue is labor.

Manual semantic modeling is slow. Evaluation is slow. Maintenance is slower. Every new domain requires people to identify the right tables, understand the logic, define metrics, document relationships, test the model, and keep it current as the data changes.

That may work for a few high-value BI dashboards. It does not work for enterprise AI.

We wrote about this in Semantic layer for AI: let’s not make the same mistakes we did with data catalogs. The core mistake is believing humans will happily document and maintain everything if only the project is important enough.

They won’t.

Humans hate documenting data. They especially hate maintaining documentation they didn’t need in the first place.

So when organizations come to Solid, they’re often looking for a way to generate the semantic layer automatically, evaluate it automatically, and maintain it with much less manual effort.

The important nuance: they don’t want a black box.

They want automation with control.

The model should be generated automatically, but reviewed by humans. Accuracy should be measured, not assumed. Fixes should be recommended, not hidden. The data team still owns the truth, but AI does the heavy lifting.

Trend #4: trust is becoming the real production blocker

Many organizations have already tried some version of “AI on top of data.”

The demo worked.

Then someone asked a slightly different question, and the answer was wrong.

Or the SQL looked plausible but used the wrong metric.

Or the agent joined tables in a creative way.

Or the business user got a number that didn’t match the dashboard.

That’s when the excitement turns into skepticism.

We wrote about this in Everyone wants Text2SQL, but the pros don’t trust it. The problem is not only whether AI can generate SQL. The problem is whether the people responsible for the data trust the generated SQL enough to put it in front of the business.

The market is now asking for evidence.

Not “our model is accurate.”
Not “the LLM is better now.”
Not “trust us.”

They want benchmarks. They want expected SQL. They want side-by-side validation. They want to know where the model failed, why it failed, and what needs to change.

This is a healthy shift.

AI in production requires measurable trust. Vibes are not enough.

Trend #5: the use cases are becoming more operational

Another interesting shift: the use cases are moving from “answer my question” to “run my workflow.”

The early data chatbot dream was mostly analytical. A user asks a question. The system answers. Maybe it creates a chart.

Now we’re seeing more operational use cases:

An agent that analyzes account signals and recommends next actions.
An agent that reviews campaign performance and suggests changes.
An agent that helps prepare customer-facing teams before a meeting.
An agent that monitors operational issues and pulls the relevant context automatically.
An agent that turns business signals into prioritized workflows.

This matters because operational workflows raise the bar.

If an AI assistant gives you a wrong answer in a sandbox, that’s annoying.

If an AI agent takes the wrong action in a production workflow, that’s dangerous.

So the underlying data logic needs to be more governed, not less. The more autonomous the agent, the more important the semantic layer becomes.

The ROI from AI will not come from buying a generic assistant and hoping employees find uses for it. It will come from improving specific workflows.

That’s exactly what organizations are trying to do now.

Trend #6: customers want flexibility, not another locked-in layer

One theme I didn’t expect to be so strong: flexibility.

Organizations are very aware that the AI stack is changing quickly. Today they may be testing one agent framework. Tomorrow it may be another. One team may be using a cloud-native AI tool. Another may be experimenting with an orchestration platform. Another may want to export semantic models into dbt, Snowflake, Looker, or something else.

They do not want to rebuild their context layer every time the AI tool changes.

This is why the semantic layer needs to be decoupled.

It should serve many consumers: BI tools, AI agents, Text2SQL engines, internal apps, and whatever comes next. That’s also why MCP is interesting. Whether or not MCP becomes the standard, the direction is obvious: AI systems need a structured way to access tools and context.

The enterprise buyer is increasingly allergic to anything that feels like a closed semantic island.

They want one governed source of business logic that can travel.

Trend #7: the market is accepting that data will stay messy

This may be my favorite one.

For a long time, companies would start the conversation by apologizing for their data.

“Our data is a mess.”
“Our documentation is bad.”
“Our warehouse is complicated.”
“Our metric definitions are inconsistent.”
“We’re not ready yet.”

Recently, the conversation is changing.

Companies still know their data is messy, but they’re less interested in pretending they can clean everything before AI arrives.

Good.

We wrote about this in “Sorry for the mess” - everyone’s data is messy, and it’s Okay. The point is not that messiness is ideal. The point is that messiness is reality.

The winning approach is not to wait until the warehouse is perfect.

It’s to build AI that can learn from the messy signals that already exist: schemas, query logs, BI dashboards, documentation, tickets, Slack discussions, Confluence pages, metric definitions, and usage patterns.

That’s where the real organizational knowledge lives.

Not in one pristine document called “Final_Final_Official_Metric_Definitions_v7.”

What this means for the market

My takeaway from this qualitative analysis is simple:

The market has moved from curiosity to implementation.

In 2024 and early 2025, many companies were asking: “Can AI help with data?”

Now they’re asking: “What do we need to put in place so AI can safely work with our data?”

That’s a much better question.

It’s also a much harder one.

The answer is not just a better LLM. It’s not just a data catalog. It’s not just a BI tool with a chat box. It’s not just a manually built semantic layer.

It’s an AI-native context layer that can create, evaluate, expose, and maintain business semantics across the organization.

That sounds a bit buzzwordy, so let me say it more simply:

AI needs to know what your data means.

Not just what tables exist. Not just what columns are called. What the business means when it says revenue, active customer, churn risk, qualified lead, suspicious transaction, campaign performance, account health, or pipeline sufficiency.

And it needs to know that in a way that is testable, governable, and usable by agents.

The funny part: most companies don’t come asking for that

This is the part I find most interesting.

Very few organizations start the conversation by saying:

“We need an AI-native semantic layer that can be automatically generated, benchmarked, maintained, and exposed through MCP to downstream agents.”

No one talks like that. Thankfully.

They come with symptoms.

They want chat with data.
They want agents.
They want fewer ad-hoc requests.
They want faster dashboards.
They want to migrate off old BI.
They want better campaign analytics.
They want better sales signals.
They want to reduce manual analyst work.
They want to stop hallucinations.
They want to make AI useful.

Our job is to help connect those symptoms to the root cause.

The root cause is not that the LLM isn’t smart enough.

The root cause is that the organization’s data knowledge is scattered across systems, people, dashboards, SQL queries, documents, and tribal memory.

Solid’s job is to turn that scattered knowledge into something AI can actually use.

Urgency found the market

A few months ago I wrote that we found urgency, or maybe it found us.

After looking at the last two months of conversations, I think that’s even more true now.

The urgency is not “semantic layers are cool again.”

The urgency is that enterprises want AI agents in production, and they’re realizing that agents without semantics are interns with database access.

Very enthusiastic. Very fast. Occasionally useful. Not someone you’d leave alone with important business decisions.

The next phase of enterprise AI will be less about demos and more about trust.

Less about asking questions and more about automating workflows.

Less about connecting AI to data and more about helping AI understand what the data means.

That’s the market pulse we’re seeing right now.

And honestly, it’s a pretty exciting one.

Talk to us

In a live environment, Text-to-SQL isn’t a simple translation task - it’s an agentic system. It’s a dynamic loop of generation, execution, and self-correction. To push the boundaries of what our agent can do, we conducted a comparison between our current baseline and the Snowflake/Arctic-Text2SQL-R1-7B.

The results represent a significant achievement for specialized AI: the Snowflake/Arctic-small 7B model expert didn’t just keep up; it exceeded expectations and painted the road for database-finetuned models.

Subscribe now

Agentic Framework for Text2SQL

The agentic workflow, partially illustrated in Figure 1, represents the end-to-end process for generating reliable SQL. The process starts with retrieving a semantic model that encapsulates the context of the question, i.e., the tables, columns, joins, metrics, and additional relevant data to answer the question. Then, with this context, an LLM based Initial SQL Generation Module (Figure 1a) creates the query that answers the question, which serves as the critical point of comparison between the GPT-based baseline and the Arctic SLM. Once the initial SQL is generated, a cycle of quality control and verification with automated checks assess the syntax, schema adherence, content and executability of the query, and attempts to fix the query if necessary (Figure 1b). The output of this process is a textual response including: the SQL query, the generation process explainability and detailed reasoning.

Building the Agentic Toolkit

Arctic-7B is a fine-tuned SLM specialized for generating SQLite queries given a semantic context. Incorporating it into a production agentic workflow requires controlling the information flow in and out of the model. Specifically, the prompt structure of the semantic context must be organized as the input that the model was trained on. In addition, the SQLite queries must be transformed to the specific SQL dialect of the environment database.

To this end, we encapsulated the model with the following pre- and post- processing procedures (Figure 2):

1. Prompt Standardization: We based our inputs on Solid semantic models, extending the OmniSQL standard, augmenting it with enriched descriptions for each column and table, extended metric and query examples, and a schema summary. This provided the model with a clear and valuable interpretation of the schema.

2. The Transpilation Layer: SQL dialects have distinct functions and syntax. Since the Arctic SLM was trained on a specific SQLite dialect, generated SQL code requires transpilation (conversion) to the dialect of the target data warehouse. In this set of experiments we used Snowflake, so we developed a custom layer to handle query transpilation to Snowflake dialect.

Results

For this experiment, we replaced the initial SQL generation module based on GPT models to a module based on the open source Snowflake/Arctic-Text2SQL-R1-7B small model (Figure 2). We used an internal evaluation benchmark comprising 137 couples of free text business questions and their related queries. Each agent performance was measured by executability, whether the generated SQL was able to execute successfully at the source system (Snowflake, in our case), and returned values alignment of the generated query to a reference (ground truth query).

The Arctic and baseline initial SQL generation was able to produce an executable SQL for 65% and 77% of questions, respectively. Understandably, most of the initial failures for the Arctic model were syntax errors, which were fixed after a single correction step. After completing the correction loops, both the Arctic and baseline models generated executable SQLs in a Snowflake dialect for 97% of questions. Additionally, there was not a significant difference in the generated SQLs retrieved values between the versions.

Conclusions and Future Work

Ultimately, this experiment demonstrates that specialized SLMs can rival industry giants when embedded within a robust agentic framework. By leveraging a modular loop of self-correction, the Snowflake/Arctic-7B model exceeded expectations, proving its performance is comparable to much larger models once initial errors are mitigated.

A key factor in this success was our use of Solid semantic models, which condensed large database schemas into enriched, high-signal summaries; this reduced the required context window and allowed the SLM to navigate complex data structures efficiently. Furthermore, using a SQLite-specialized base proved highly effective for generalizability; with the help of the transpilation layer, we narrowed the initial performance gap to just 12% due to syntax. This remaining delta was successfully closed during the fixing loop, resulting in nearly identical final executability. Moving forward, the true potential lies in database-specific fine-tuning, enabling these models to learn the unique nuances of individual datasets from the ground up.

dbt’s recent post, Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update, is useful precisely because it moves the conversation forward. Instead of arguing from ideology, it argues from failure modes. And that is the right way to think about enterprise AI on top of data.

Their conclusion is also directionally right: both approaches matter. Text2SQL has gotten dramatically better. Semantic layers remain far more deterministic when the question is within scope. If accuracy matters, semantics win. If flexibility matters, Text2SQL still has an important role to play.

I agree with that.

But I think there is a more important conclusion hiding underneath their benchmark:

The real challenge is not Text2SQL or semantic layer. The real challenge is semantic model production.

Subscribe now

The benchmark proves more than it says

The most interesting part of dbt’s post, in my opinion, is not just that the semantic layer performed better on modeled data.

It is that when coverage was missing, they added just three additional models, reran the benchmark, and suddenly the semantic layer could answer every question in scope, while Text2SQL improved too. That is a very important signal. It tells us that the bottleneck is not only reasoning quality. It is the system’s ability to create the right semantic structure, at the right time, for the right business question.

That is exactly where most enterprise projects get stuck.

On a whiteboard, “just build the semantic layer” sounds reasonable. In the real world, that means defining metrics, joins, dimensions, business definitions, tests, ownership, and maintenance processes across a living data stack that keeps changing. New tables appear. Existing logic shifts. Teams rename KPIs. Dashboards drift. Query patterns evolve.

So yes, the semantic layer is incredibly valuable. I wrote about that in The Two Souls of a Semantic Layer: A Tale of Governance and Insight. But the moment you say “semantic layer,” you also have to ask: who is building it, who is testing it, who is publishing it, and who is keeping it alive six months later?

That is where the theoretical debate becomes an operational one.

This is the part Solid is focused on

At Solid, our view has been consistent for a while.

In Data, Meet Business: Why You Need JIT Semantic Models, Instead of a Static Semantic Layer, I argued that static semantic models are too slow for a business that changes every week.

In The Ghost in the Machine: How Solid drastically accelerates semantic model generation, I explained part of the “how”: the logic of the business is often not fully captured in the schema itself. It lives in usage patterns, query history, BI assets, and the actual paths analysts take through the data.

That is why I think the future is not “Text2SQL alone,” and it is not “semantic layer alone,” either.

It is Text2SQL plus semantics, connected by an engine that continuously creates, tests and maintains the semantics themselves - even as data changes, usage evolves.

Publicly, that is exactly how we describe Solid Build: automatically turning warehouse usage, BI semantics, SQL query history, and metadata into semantic models; keeping them current as the data changes; letting teams review and approve them; and validating model logic before publication.

That matters because the enterprise problem is not proving that semantic layers work in a benchmark.

The enterprise problem is making enough of them, fast enough, with enough quality, and keeping them fresh enough, that the benchmark becomes your everyday reality.

The real architecture is not “vs.” It is “and, plus automation”

So when I read dbt’s post, my reaction is not disagreement. Quite the opposite. I think it confirms where the market is going.

How do you produce and maintain that modeling at enterprise scale?

If the answer is “a heroic manual effort by a small team of experts,” then the system will help in the demo and decay in production.

If the answer is “use AI to marry exploration and semantics, and automate the lifecycle of the semantic models themselves,” then you start to have something much more durable.

That is the architecture I believe in:

Text2SQL for exploration.
Semantics for determinism.
And an always-on semantic engineering layer that generates, tests, publishes, and maintains the bridge between them.

That, to me, is where this market is actually headed.

And that is why I think the most important question in 2026 is no longer “semantic layer or Text2SQL?”

It is:

Who is building the semantics, and how fast can they keep up with the business?

I recently sat down with Ryan Wexler on the podcast, and I enjoyed this one a lot. It’s available to listen to now: YouTube, Spotify, Apple.

Usually, on Building with AI: Promises and Heartbreaks, I speak with builders, operators, and data leaders who are deep in one company’s AI journey. Ryan brought a different angle. As a Principal at SignalFire, he gets a horizontal view into the market: what founders are building, what enterprises are actually buying, where the hype is ahead of reality, and where something very real is already happening.

That made for a particularly interesting conversation.

Ryan is in a unique position to see patterns early. SignalFire is not a traditional VC that just writes checks and hopes for the best. It has built a real technology and data muscle internally, including Beacon AI, to support sourcing, recruiting, GTM, and portfolio support. So when Ryan talks about how AI is changing work, he’s not speaking only from the investor seat. He is also seeing how these systems get used in practice.

Subscribe now

Most enterprises are not refusing AI. They are stuck in the middle.

One of the most useful parts of the conversation was how Ryan broke down the market.

In his view, there are roughly three groups right now:

1. The disbelievers, the people who still think AI is mostly a parlor trick.

2. The big middle, where organizations are using AI tools and seeing value, but are not yet rethinking work around them.

3. A much smaller group that is fully bought in and actively redesigning workflows, teams, and products around AI.

That lines up pretty well with what I’m seeing too.

The market narrative is often too binary. Either “AI is changing everything” or “AI is overhyped and barely works.” Reality, as usual, is messier. Most large enterprises are somewhere in the uncomfortable middle. They have Copilot, ChatGPT Enterprise, Cursor, Claude Code, or some equivalent floating around the organization. Some people are getting real value from it. But the company itself is still not fully operating in an AI-native way.

This is also why I keep coming back to themes I wrote about in (Almost) no AI in production and Throwing away BI is a Bad Idea. People want the new experience, but they still need the old safety rails. They want AI, but they still need trust. They want agents, but they still keep a human in the loop.

That is not failure. It is transition.

Ryan’s point was that the biggest blocker is not disbelief. It is change management. The tools are still early. The models keep changing. The workflows are not yet stable. So even when people believe, it still takes time to move.

Founders should stop asking “what’s hottest?” and start asking “what can I win?”

Another part of the conversation that stuck with me was Ryan’s framing of the types of AI companies selling into enterprises today.

He described three broad buckets.

The first are transformation plays. These are the companies trying to help enterprises fully rethink how work gets done. Huge opportunity, obviously. Also hard. These deals involve process change, redesign, consulting, buy-in, internal champions, and patience.

The second are next-generation SaaS companies. These are applications where AI is built into the product itself. The buyer does not need to reinvent the whole company to get value. A customer support platform that now resolves tickets with agents is an example. Same budget owner, same category, new capabilities.

The third are bottoms-up prosumer tools. This is where an individual gets a tool that makes them more productive: ChatGPT Enterprise, Copilot, Cursor, Claude Code, ElevenLabs, and so forth. Ryan’s view was that this is where a lot of the fastest adoption has happened because the value is immediate and the buying motion is much simpler.

Then I asked him the natural founder question: if you were starting something new, which of the three would you build in?

His answer was excellent: “Whichever one you can win in.”

I liked that a lot.

Founders sometimes over-rotate to whatever category is currently generating the most noise on X or LinkedIn. But not every founder is built for the same motion. Some are excellent at bottoms-up adoption. Some are great at enterprise solution sales. Some can navigate long cycles and help customers rethink processes. Others cannot. And that’s okay.

The goal is not to build in the hottest category. The goal is to build in the category where your team has a real shot at winning.

That also ties directly to a point I made in Nobody cares about the efficiency of the data analyst. Buyers do not wake up wanting AI in the abstract. They want impact. They want better outcomes. They want a wedge that matters.

ROI is still messy, and that’s part of the story

We also dug into a question many founders and buyers still struggle with: how do you prove ROI?

If AI is automating a specific workflow, the math is easier. If an agent resolves a support ticket, runs a recruiting screen, or handles a voice interaction, you can start counting minutes, throughput, coverage, and outcomes.

But when AI is helping an individual employee do their job better, the math gets fuzzier.

How exactly do you measure the ROI of giving 10,000 employees access to a co-pilot? Lines of code are a bad metric. “Time saved” is often hand-wavy. Yet, clearly, something is happening.

Ryan was very honest about this. A lot of organizations are still buying these tools before they have perfect measurement. They know the direction is right, even if the spreadsheet is still weak.

That matches what I’ve been hearing from leaders as well. In my conversation with Meenal Iyer, one of the recurring themes was that the winning teams are not waiting for the world to become perfectly measurable. They are building AI literacy, experimentation muscle, and the right foundation now, so they can scale what works later.

Two areas Ryan is especially bullish on

The first is voice AI.

Ryan’s reasoning here is very pragmatic. With coding assistants or general-purpose copilots, ROI is harder to pin down. With voice, it is often much clearer. Humans can only speak so many words per minute. If AI handles some of those minutes, the value is much easier to quantify.

More importantly, voice does not only replace labor. It enables new interactions that were previously too expensive to staff.

Ryan shared a great example of a behavioral treatment organization that automated initial recruiting conversations with voice AI. The impact was not just efficiency. They were able to run more interviews, keep human recruiters focused on the more valuable parts of the process, and improve outcomes.

That’s a big deal.

The second area is self-improving agents.

This one is especially exciting to me.

Ryan described a future where one agent does the work, another evaluates the trace, another improves the prompt or rules, and the system improves over time through usage. In other words, not just agentic systems, but systems that get better at being agentic.

We’re already starting to see the early versions of this pattern. I think we’ll see much more of it over the next couple of years.

And frankly, that is where things begin to get really interesting. Not when AI merely answers a question, but when it learns how to make its own future answers better.

So where is this all going?

Ryan does not believe AI is a bubble in the simplistic sense.

Sure, individual companies will go up, down, miss expectations, or disappear. Some categories will get overcrowded. Some valuations will correct. That’s startup-land. But the broader shift is not going away.

On that, I agree with him.

We are moving from co-pilots to colleagues. From tools that wait for a prompt, to systems that can take initiative, execute tasks, and eventually coordinate with other systems. Some of this will first happen inside enterprises. Some of it will show up in consumer software. But all of it will require better context, better data, and a lot more trust than most people realize.

That’s what made this episode fun for me.

Ryan brought the market view, but the takeaways were refreshingly grounded: most organizations are still in the middle, ROI is real but often messy, founders should pick the motion they can actually win, voice AI is much bigger than many think, and agents improving agents may become one of the most interesting product patterns of the next couple of years.

I thank Ryan for taking the time to join me on the podcast. You can learn more about him on SignalFire’s team page, read about Beacon AI, and check out the AI Circle community he mentioned in our lightning round.

Or - hop straight into the episode now: YouTube, Spotify, Apple.

I first saw Rittika Jindal speak at Gartner in Florida a little over a year ago. She was presenting OpenSETU, a data agent for non-technical users, and I remember walking out thinking: this is someone who is not treating AI like a demo. She is treating it like engineering.

That distinction matters more at Thomson Reuters than in most places. This is a company serving legal, tax, risk, compliance and journalism professionals. In those environments, the wrong answer is not just annoying. It can affect real work, real customers, and in some cases real lives.

One sentence from our conversation captured that bar perfectly:

“accuracy is the key.”

I think that line is worth sitting with for a minute, because it explains almost everything that came next in our conversation (available on YouTube, Spotify and Apple Podcasts).

Subscribe now

In high-stakes AI, accuracy is not a metric. It is the product

Rittika works in Thomson Reuters’ Content Innovation group, an R&D team focused on how content is processed, enriched and delivered into products used by professionals. In practice, that means building AI and agentic workflows into pipelines that need to be fast, scalable, and above all trustworthy.

That emphasis on trust strongly echoed what we heard in my recent conversation with Joel Hron, Thomson Reuters’ CTO. The setting is different, but the core idea is the same: when AI enters high-stakes workflows, you do not get to hide behind a cool demo. You need systems, guardrails, evaluation and clear accountability.

That is also why I liked Rittika’s framing of the problem so much. She did not talk about hallucinations as a quirky model behavior that we all laugh off on X. She talked about them as a product failure.

OpenSETU was a bridge, not a magic trick

The first project I knew Rittika for was OpenSETU. “Setu” means bridge in Sanskrit, which is a fitting name.

The idea was simple and powerful: let non-technical teams ask natural-language questions about their data, and have an agent translate those questions into Python or SQL, run the code, and return the result. In other words, bridge the gap between business users and the analysts they usually wait on.

But the interesting part was not the interface. It was the discipline behind it.

Rittika was very clear that the team did not stop at “the code runs, so I guess it works.” They built with ground truth from day one. They knew what correct answers looked like for representative questions, and used that to evaluate whether the agent was producing the right output.

This rhymes strongly with how we think about evals at Solid, and with a point I keep seeing across strong teams: if AI is part of the product, evaluation cannot be something you bolt on at the end. It has to be part of the design.

Or, as Rittika put it:

“We make sure that we have the ground truth data. We are evaluating whatever we are building, starting from day one.”

That is not flashy. It is also probably why her work feels real.

Then the job of software development moved

A lot of people still talk about AI in software as “the thing that writes code faster.” That is true, but it is also the least interesting part of the story.

What Rittika described is much bigger. Her team has moved from AI-assisted development toward AI-native engineering. The code-writing step is just one piece. AI is now involved across planning, coding, testing, debugging, observability and review.

The workflow she described was one of my favorite parts of the episode:

A developer starts by creating a plan document with AI. That plan gets reviewed by humans. It goes into GitHub. Then code gets generated against that plan. Before a PR is opened, an internal review skill checks the plan, code, tests and architectural alignment. After the PR is opened, a bot reviewer runs automatically and comments on what it finds. The developer then feeds those comments back into their coding agent and iterates. Only after that does the PR land in front of a senior engineer.

So yes, Claude writes code. But Claude also reviews Claude.

If you have been following Rittika’s writing, especially posts like How We Taught Claude to Review Claude and Reviewing AI Code at Scale: A Principal Engineer’s Dashboard, this pattern will sound familiar. What I liked hearing on the podcast is that this is not a thought experiment. It is how her team is actually working.

It also pairs nicely with what we wrote when Solid moved to an agentic platform and later on how we observe that system in production with LangSmith. Once your development process becomes more agentic, observability and evals stop being “nice to have.” They become part of the development loop itself.

Yes, teams are shipping more. No, that does not mean quality stops mattering

One of the more striking parts of the conversation was the sheer change in pace.

Rittika described a world where some repos that used to merge five or six PRs a week are now seeing much more than that every single day. In her words, “we are merging 5 to 10 PRs every single day.”

That kind of compression changes expectations fast. It also creates a fear that I hear from a lot of people: are we just generating more slop, faster?

Her answer was more balanced than the hot takes you usually see online. Yes, AI slop is real. Yes, dead code shows up. Yes, review gets harder. But humans were never exactly writing slop-free code either. The real question is whether the team has built the mechanisms to catch the things that matter: wrong business logic, broken architecture, lack of tests, poor observability, missing evals.

That is why I found her emphasis on testing, logs, and review so important. In her team’s world, the answer to faster code generation is not hand-wringing. It is a stronger system around the code.

The future engineer may look more like a software thinker

The part of the conversation that stayed with me most was not about tools. It was about roles.

Rittika argued that engineers are now spending much more of their time on planning and design, and less on manually typing code. The work is shifting upward. You ask better questions earlier. You decide what should be built, whether it should be built, how it should be evaluated, and how the workflow should be instrumented once it is live.

That sounds subtle, but I think it is a major shift.

It also echoes what we heard in my conversation with Meenal Iyer: the winning teams are the ones building foundations first, then experimenting hard, then moving into production with clarity about trust and business value.

Rittika took the idea one step further when we talked about junior engineers and the next wave of talent. Her point was that the people arriving now are more AI-native than the rest of us. They are not “adopting” AI. They barely know how to work without it. That will change not just how teams build, but who teams hire for.

She said, almost offhandedly:

“Maybe after five years we’ll hire software thinkers.”

Maybe that is too provocative. Maybe it is exactly right.

After hearing how her team works, it no longer sounds crazy to me. The scarce skill is moving away from typing syntax and toward framing problems, building evals, setting constraints, and deciding when humans need to stay in the loop.

What I took from the conversation

If I had to reduce the episode to four points, it would be these:

1. In high-stakes domains, accuracy is not a dashboard metric. It is the product.

2. AI-native engineering is not about faster code completion. It is about rebuilding the whole SDLC around plans, review, evals and observability.

3. Shipping more PRs only helps if your quality system gets stronger at the same time.

4. The software engineer role is shifting upward, from code producer to system designer, reviewer and decision-maker.

Rittika is one of the more thoughtful voices I have heard on this topic because she is not speaking in abstractions. She is building inside a high-bar environment, writing openly about the lessons, and grounding the entire thing in practice. Her Medium is worth following, and so is Thomson Reuters’ broader engineering transformation story. For people trying to level up quickly, I also liked her recommendation to start with the docs from the model companies themselves and with DeepLearning.AI.

And yes, it will surprise no one that the AI tool she said she cannot live without is Claude Code.

Want to hear the full episode? Check it out on YouTube, Spotify, and Apple Podcasts.

On March 25, we hosted a webinar on a topic I hear about almost daily: everyone wants AI on top of their data, but very few teams are fully happy with the results. I got amazingly positive feedback on it, and wanted to share with you.

I had the pleasure of leading the discussion with Meenal Iyer of SurveyMonkey and data leader Solomon Kahn. We spoke candidly about what’s working, what’s failing, and where data leaders should focus if they want AI to drive real value inside the enterprise.

If you’d like to watch the full webinar, you can do that here.

A few ideas stood out to me.

Subscribe now

1. AI literacy does not happen in a slide deck

One of my favorite parts of the discussion was Meenal’s story about trying to bring AI into her organization.

Her first instinct was a very reasonable one: train the team. Create an internal AI academy. Upskill people. Teach the concepts.

It did not work the way she hoped.

What did work was much more hands-on. She had the team do a mini AI hackathon, focused on real problems in their own workflows. Suddenly, the conversation changed. People were no longer thinking abstractly about models, prompts, or which tool was hottest that week. They were thinking about friction in their day-to-day work, and whether AI could actually remove it.

That shift matters.

I keep seeing the same pattern across companies: people understand AI much better once they build with it. Not when they hear about it. Not when they read another thought piece on LinkedIn. When they actually try to solve something with it.

This is also very much in line with what I wrote in Building an AI-powered Intelligent Enterprise, where I shared more of Meenal’s broader journey and how her team has been evolving from experimentation toward production.

2. The problem is not “the data” alone. It’s the missing context around it

Another major theme in the webinar was one I think many teams learn the hard way.

You cannot just put a GPT on top of a dataset and expect magic.

Even if the dataset is clean.
Even if the columns are named well.
Even if there is a semantic layer underneath it.

That still does not mean the AI understands the business.

It does not know the internal logic behind the metrics. It does not know which edge cases matter. It does not know which definitions changed last quarter, which exceptions live in Confluence, which nuance is buried in Jira, or what your executives actually mean when they use a certain term.

Without that context, AI fills in the blanks on its own. That is where hallucinations and bad decisions start showing up.

This connects directly to what I wrote in “Sorry for the mess”. Enterprise data is messy. Everyone’s is. The answer is not to wait for some mythical moment where everything is perfectly documented and pristine. The answer is to build systems that can understand the reality of how businesses actually work.

3. “100% accuracy” is usually the wrong conversation

A big part of the Q&A ended up centering on accuracy, and understandably so.

If an AI system is 92% accurate, what do you do with the other 8%? Can you trust it? Should you use it for decision-making? Is that good enough?

My view is that we often hold AI to a strange standard.

Human analysts are not 100% accurate either. In many cases, they get to a high-confidence answer through back-and-forth, clarification, validation, and iteration. AI should be judged with similar realism. Not every use case requires perfection. Some do. Many do not.

The more useful question is: what is the use case, what is the risk, and what level of validation is needed?

For financial reporting, compliance, and tightly audited workflows, you still want highly controlled systems and dashboards. For exploratory work, data literacy, pattern finding, and helping people navigate large datasets, AI can already be very valuable even if it is not perfect.

That is one of the reasons I still believe throwing away BI is a bad idea. AI and BI should work together. One is not replacing the other anytime soon.

During the conversation I also brought up Waymo’s safety work as an analogy. The standard is not “never make a mistake.” The standard is whether the system performs well enough, consistently enough, in the context it is being used for.

4. Data leaders should be measured on business impact, not just on perfection

Solomon made a point that I think many data leaders feel deeply.

Data teams are often only noticed when something is wrong.

When the number is right, silence.
When the dashboard works, silence.
When the business gets what it needs, silence.

But when something breaks, or a metric is off, or an answer is confusing, that is when everyone suddenly remembers the data team exists.

That is a very hard way to operate.

The better framing is business impact. Are you helping the organization close deals, reduce churn, improve workflows, support better decisions, and move faster with more confidence? That is the real scoreboard.

This also ties back to something I wrote last year in Nobody cares about the efficiency of the data analyst. Efficiency is nice. Happier analysts are nice. But the main thing the business cares about is results. If AI helps deliver those results, people will forgive a lot. If it does not, no amount of cleverness will save it.

5. Start small, but start

The closing advice from the panel was refreshingly practical.

Do not boil the ocean.

Start with a smaller domain. Start with a use case where the data is relatively well understood. Start with a workflow where the upside is clear. Let people experiment. See what breaks. See what works. Learn quickly.

There is far too much pressure right now for companies to show they “have an AI strategy.” That pressure often creates giant top-down initiatives that look impressive in a deck and disappoint in reality.

A smaller, grounded, iterative approach is much less glamorous. It is also much more likely to get you somewhere real.

Final thought

If there is one message I would want people to walk away with, it is this:

Enterprise AI is not struggling because there is not enough excitement. It is struggling because real business data is nuanced, fragmented, and deeply contextual.

The teams that succeed will not be the ones with the flashiest demo.

They will be the ones that help AI understand how their business actually works, earn trust over time, and stay relentlessly focused on business outcomes.

Thanks again to Meenal and Solomon for such a sharp and honest discussion.

If you missed the webinar, you can watch it here. And if you want more on Meenal’s journey, this earlier post is a good place to start.

First, kudos to Jason Cui and Jennifer Li for their a16z post, Your Data Agents Need Context.

It’s a strong piece. More importantly, it puts the conversation in the right place.

A lot of the market still talks about data agents as if this is mainly a model problem. Better reasoning, better SQL generation, better interfaces, and suddenly everyone in the business can ask whatever they want and get back a trustworthy answer.

That’s not the main blocker.

Jason and Jennifer make the important point: the real blocker is context.

Not generic context. Real business context. Which definition of revenue is the right one. Which dashboard people actually trust. Which table is technically available, but should not be used. Which metric changed meaning last quarter. Which exception “everyone knows,” but no system has ever written down.

This is exactly why, in Data Chatbots: what people are really doing, I argued that enterprise data is not just a SQL problem. Every company has its own business language, and often multiple business languages. The model is not just translating English into SQL. It is trying to navigate an organization’s internal logic.

That is much harder.

Subscribe now

Context is the problem. Static context is the trap

Where I’d push the a16z argument a step further is this:

Context is necessary. But static context is not enough.

Most enterprise context is messy. It lives across BI tools, dbt models, semantic layers, dashboards, documentation, tickets, query history, Slack threads, and people’s heads. Some of it is useful. Some of it is stale. Some of it conflicts with other pieces. Some of it was right six months ago and wrong today.

That’s why I keep coming back to a simple point: everyone’s data is messy. No one has perfect documentation. No one has a fully clean warehouse. No one has a beautifully maintained source-of-truth map that always reflects reality.

And that’s okay.

The mistake is thinking the answer is to pause everything and go build a pristine context layer by hand.

That sounds good in theory. In practice, it can turn into the next documentation project that starts with great intentions and slowly falls out of date. We’ve seen this before in catalogs, governance projects, and semantic initiatives. It’s also why I think Tal’s point in Semantic layer for AI: let’s not make the same mistakes we did with data catalogs is so important: if humans have to manually maintain every part of the system forever, the system usually loses.

So yes, agents need context.

But the bigger requirement is that context has to keep up with a real business. New product lines get added. Definitions shift. Dashboards become stale. Teams change how they operate. The “trusted” table from last year may no longer be the one finance relies on today.

If your context layer cannot adapt to that, it is not really a context layer. It is a snapshot.

Enterprises already have more context than they think

The good news is that companies are not starting from zero.

A lot of their context already exists inside the work they’ve done over the past decade. It’s embedded in BI, in semantic models, in modeling layers, in warehouse metadata, and in query patterns. It may not be organized perfectly, but it’s there.

That’s one reason I wrote Throwing away BI is a Bad Idea. There is this temptation right now to act as if AI will replace everything that came before it. I think that’s the wrong frame.

Your BI stack contains years of accumulated business logic. Your dashboards, LookML, dbt models, Power BI reports, Tableau workbooks, and saved queries all encode decisions the organization has already made. Not perfectly, of course. But they are still among the highest-signal assets you have.

The right move is not to discard those systems.

It’s to extract from them.

That’s also why posts like Autogeneration of a semantic layer - the key for AI/BI and Data, Meet Business: Why You Need JIT Semantic Models, Instead of a Static Semantic Layer resonate so much with what Jason and Jennifer are saying. The market does need context. But it needs context that can be built and refreshed from the assets companies already have, not just from a manual curation exercise.

My takeaway

So my takeaway from the a16z post is pretty simple:

They are absolutely right that data agents need context.

I’d just add that context also needs infrastructure. It needs a way to be generated, refined, corrected, and updated without depending on humans to keep every rule and every definition perfectly current by hand.

That is the harder challenge.

And that, in my opinion, is where the next real wave of value will be created.

This is not some far off vision of how analytics might work one day. It is a pretty good description of what we are already building and deploying with customers.

And if you want a concrete example, we have a public case study with SurveyMonkey that shows these ideas in production today:
SurveyMonkey x Solid Case Study

Subscribe now

What Gartner actually means by “agentic analytics”

Let me translate the report into four simple ideas.

1. Start from specific, repeatable scenarios

Agentic analytics is not “ask anything in a chat box.” (we already discussed how that’s not an important use case)

The successful patterns start from well defined, high value workflows, for example:

• Weekly business reviews

• Account health and renewals

• Campaign performance

• Operational anomaly detection and follow up

These are things that already happen, on a schedule, with some manual collection of numbers and spreadsheets.

Gartner’s point is that you get real value when agents make these workflows faster and better, not when you show a cool one time demo.

2. Semantics and policy are the foundation

The report is very clear. If you point agents at raw tables and column names and hope they “figure it out,” you are going to get confident nonsense.

Organizations that succeed:

• Invest in semantic layers, ontologies and knowledge graphs for their important domains.

• Align metric definitions across tools, dashboards and teams.

• Make sure agents and models are grounded in that shared layer.

In Journey we have been calling this the “context layer” between your warehouse and your AI. Posts like “The Two Souls of a Semantic Layer” and “Behind the scenes: how we think about semantic model generation” are basically a deep dive on this point.

3. Human and AI roles need to be explicit

Gartner spends a lot of time on delegation, escalation and human review.

The questions they ask are simple:

• Which tasks can an agent do alone?

• Which tasks must always be reviewed by a human?

• When should an agent suggest rather than act?

The answer will be different for forecasting revenue, sending emails to customers, or building a one off dashboard for a team. The key is to make the rules explicit instead of letting each team invent its own.

4. Governance and cost are first class

Agentic systems can run expensive queries and call expensive models.

That means you need:

• Observability into what agents are doing.

• Guardrails around which data they can touch.

• Controls on how much they can spend for a given type of insight.

Gartner talks about this as managing the “cost per useful insight” rather than just tracking model usage. It is a good mental model.

So that is the theory. Now let us talk about how Solid is positioned inside that world.

How we position Solid in the agentic analytics landscape

When people ask what Solid is, I usually describe it like this:

Solid is the context layer between your data and your AI systems.

More concretely, Solid:

• Learns semantic models automatically from your warehouse, query history, BI and other sources.

• Keeps those models governed, tested and up to date as things change.

• Exposes that logic through APIs and MCP so that any agent, LLM or workflow can use it.

This is not a generic chatbot. It is an an enabler for all of your AI projects - those that leverage your structured data.

Here is how that maps to Gartner’s four themes.

Semantic foundations, automated

We agree with Gartner that semantic and policy alignment is foundational. The problem is that almost nobody has time to build and maintain a semantic layer manually.

That is why Solid:

• Mines query logs, BI dashboards and recurring reports to see how the business already uses data.

• Auto generates candidate semantic models that reflect real joins, metrics and entities.

• Lets humans review and refine those models instead of starting from a blank file.

We wrote about this in “The Ghost in the Machine: How Solid drastically accelerates semantic model generation” and in “End-to-end: generating semantic models for Snowflake Cortex Analyst/Intelligence in two weeks”.

The result is very similar to what Gartner describes: a selective semantic layer for the most important domains, aligned with how the business actually works.

Analysts first, “AI for everyone” second

In “Stop saying ‘Garbage In, Garbage Out’, no one cares”, we argued that the business will not wait until all your data is perfect before they try AI. You need a safe way to get value now and improve quality over time.

Our approach is to start with the analysts and data team:

• Give them an AI copilot that understands their warehouse and their metrics.

• Let them supervise and correct what the AI does.

• Only then open up the same capabilities to broader audiences in a governed way.

This is very close to Gartner’s picture of humans as supervisors and orchestrators of agentic systems rather than bystanders.

Embedded, not “sidecar chat”

The value of a “chat with your data solution” is extremely limited. That should NOT be your goal.

Solid is designed to show up where work is already happening:

• Inside recurring analytics rituals like QBRs and weekly reviews.

• Right next to existing BI tools and documents.

• Triggered by events in the data, not only by user questions.

This maps to what Gartner calls conversational and perceptive analytics. We just think of it as embedding AI where it can actually help people do their jobs.

How we run pilots in an “agentic” way

The other part of the Gartner guide that felt familiar was their advice to “start with concrete scenarios” and “treat pilots as the first step toward production, not proofs of concept.”

That is almost exactly how we run pilots.

1. Pick one high value, bounded scenario

We never start a pilot with “ask anything about your data.”

We start with something like:

• Help marketing run weekly campaign reviews without manual data collection.

• Give customer success a reliable view of at risk customers before renewals.

• Make it trivial for product managers to answer their five most common usage questions.

These scenarios have three things in common:

• They happen all the time.

• People are already doing them, but with a lot of manual effort and spreadsheets.

• There is clear business value in making them faster and more reliable.

This is very close to Gartner’s advice to anchor agentic analytics initiatives in specific analytical scenarios and measure cycle time and quality of decision, not just “AI usage.”

2. Build the semantic slice that matters

Once we have a scenario, we narrow down the data to the actual slice that powers it.

Solid then:

• Learns from historical queries and dashboards which tables and joins really matter.

• Proposes a semantic model with the relevant entities, metrics and relationships.

• Generates human readable documentation so business stakeholders can sanity check it.

You do not need a company wide semantic layer on day one. You need a good semantic model for the scenario in front of you, that can then grow step by step.

3. Define human and AI roles up front

Before any agent does anything beyond drafting and suggesting, we agree with the customer on clear guardrails.

For a given workflow we decide:

• What the agent is allowed to do automatically.

• What requires explicit human approval.

• What the agent is not allowed to touch at all.

For example:

• An agent may be allowed to draft commentary for a business review slide, but not to send that slide to an executive without someone reading it.

• An agent may flag unusual patterns in the data, but not create or modify alerts in production systems.

• Anything that touches pricing, discounts or regulated metrics always goes through human review.

This is our practical version of the delegation and escalation frameworks Gartner describes.

4. Measure outcomes, not just demos

Finally, we define what success looks like in terms that matter to the team.

Typical success metrics include:

• Reduction in analyst time spent on repetitive requests.

• Faster turnaround for key decisions (for example weekly reviews prepared in hours instead of days).

• Fewer “shadow” spreadsheets and one off queries.

• Reuse of the semantic model in new workflows after the pilot.

The goal is that a pilot creates a durable asset, not just a one time demo. The semantic models, guardrails and workflows we build in the first month are designed so that they can power the next set of use cases.

A quick example from the field

One short quote from our public case study captures the spirit of what “agentic analytics” feels like when it works.

In that case study, Meenal Iyer says:

“Our goal is for AI to remain dependable,” Meenal explained. “Teams should be able to use it with confidence, knowing the answers reflect how the business actually works.”

This is exactly why we care so much about semantics, governance and human in the loop design. Agentic analytics only matters if teams can depend on it.

If you want to see the full story, including the impact on accuracy, time to production and maintenance, you can download the case study here:
Download the case study

Where we are going next

Gartner also talks about “perceptive analytics” and agents that:

• Monitor events and changes continuously.

• Understand goals and constraints.

• Adapt their behavior based on feedback.

In this blog we explored similar ideas in “Vibe Analytics: The new era of data experiences”. The short version is that we want AI to feel less like a tool you occasionally consult and more like a reliable colleague who taps you on the shoulder when something important happens.

A lot of our current roadmap is about:

• Learning not just from schemas, but from how people actually use data over time.

• Bridging the “two souls” of the semantic layer, so that governance and insight are not in conflict.

• Making it easy for customers to go from one well defined scenario to a network of agentic workflows that reuse the same trusted foundation.

If you care about agentic analytics, here is what I would do next

If your leadership is asking “What is our agentic analytics strategy?” or “How do we move beyond pilots?”, here is a simple plan.

1. Read the Gartner Market Guide
   Grab it here and read it with your own organization in mind:
   Analyst Report: Gartner Market Guide for Agentic Analytics

2. Read the case study in parallel
   Look at how the ideas in the report show up in a real deployment:
   SurveyMonkey x Solid Case Study

3. Skim a few Journey posts to see how we think about this internally
   For example:

   • “AI for AI: how to make ‘chat with your data’ attainable within 2025”

   • “Behind the scenes: how we think about semantic model generation”

   • “(Almost) no AI in production”

You will see the same themes repeated again and again.

Start from concrete scenarios.
Invest in semantics where it matters.
Keep humans in the loop by design.
Treat governance and cost as product features, not afterthoughts.

That is what Gartner calls “agentic analytics.” For us, it is just how we build. We’d love to hear from you.

Contact Solid

The dream: anyone can ask anything

If you spend even a few minutes on LinkedIn these days, you will see some version of the same pitch:

“Imagine every business user in your company can just chat with your data.”

The demo is always similar:

• Someone opens a chat window

• Types “What were Q4 revenues in EMEA versus target?”

• A nice chart pops up

• Everyone nods. “This is the future.”

I get why this is compelling. We have spent a decade building dashboards, training people on BI tools, arguing about metric definitions. The idea that we can put a natural language interface in front of all of that, and suddenly “everyone becomes data driven”, is very appealing.

We have written before about why enabling this is so hard on the technical side, from Text2SQL accuracy issues in “Everyone wants Text2SQL, but the pros don’t trust it” to the heavy lifting needed in “Semantic layer for AI: let’s not make the same mistakes we did with data catalogs”.

This time I want to focus on something else:

Even if you solve all the technical problems, “chat with your data” has two very real business problems.

Subscribe now

Problem 1: usage is not ROI

When companies roll out a “chat with your data” experience, here is what they tend to measure:

• How many people tried it

• How many questions they asked

• How often they came back

• Satisfaction scores like “this answer was useful”

All of those are fine. None of them are ROI.

It is very hard to answer questions like:

• Did this chat experience help us win more deals?

• Did it reduce churn?

• Did it increase revenue per customer?

• Did it reduce cost per ticket in support?

You can say “Marketers now get answers faster” or “Sales can self serve their questions”, but that is not the same as proving material business impact.

We see a similar pattern to what we saw with early BI rollouts:

1. A big launch, internal roadshow, a few great demo moments

2. A couple of power users adopt it seriously

3. Everyone else goes back to:

   • Asking the same data people in Slack

   • Looking at the same dashboards as before

   • Exporting to Excel

You can show a nice adoption chart. You can say “we had 3,000 questions asked this quarter”. But when the CFO asks “What did we get for this?” it is very hard to go from “3,000 questions” to “X million dollars”.

That is not a problem unique to AI. It is the same problem BI has had for years. The difference is that with AI, the expectations are much higher.

Problem 2: after the novelty, no one knows what to ask

The second pattern we keep hearing in conversations with data and analytics leaders is even more worrying:

“The first month, people played with it. After that, usage dropped. People did not know what to do with it.”

We saw a similar behavior in the work behind “When users don’t chat with your chat”. Most users are not “trained” to interact with AI in a rich, structured way. A typical prompt is closer to a search query than to a thoughtful analytical question.

In practice that looks like:

• “Show me sales last month”

• “Top 10 customers”

• “Pipeline by region”

Those are fine questions, but they are not very far from what you already had in dashboards or reports. And once people have tried a few of these, they hit a wall:

• They do not know how to decompose fuzzier questions

• They are not sure what the system “knows”

• They do not want to get something wrong in front of their manager based on a chat answer

So they fall back to the pattern they trust:

• Open the dashboard that “everyone uses”

• Ask an analyst to “just pull something for me”

• Copy paste into a deck

Chat becomes another tool they occasionally open, not something that fundamentally changes how the business runs.

Why this is happening: chat is a pull interface

If you zoom out, there is a structural reason for this:

Chat is a pull interface. It relies on the human to:

• Notice there is a question worth asking

• Translate it into words the system understands

• Decide what to do with the answer

• Then go and do the work

That is a lot to ask from already overloaded business stakeholders.

The people you are most excited to “put AI in front of” are usually the ones with the least time and attention. They are managing teams, customers, numbers. Asking them to become “prompt engineers” on top of that is not realistic.

So you get:

• Some cool one off wins

• Good internal marketing material

• But not a durable change in how decisions are made and work gets done

Which leads to the core point of this post:

The real promise of AI on structured data is not “chat with your data”. That is a nice bonus. The real promise is agents that take work off people’s plates.

What agents do that chat never will

When I say “agents”, I do not mean a fancy wrapper around chat that calls a few tools. I mean systems that:

• Run on a schedule or trigger from events

• Pull from multiple sources:

  • Data warehouse and metrics

  • Product events

  • CRM

  • Support tickets

  • Even free text notes

• Apply reasoning and business logic

• Make a decision or propose one

• Then take action in the real systems of record

Some concrete examples.

Marketing budget reallocation

Instead of a marketer asking “How did Campaign X perform?”, an agent:

• Monitors performance of all campaigns

• Understands targets for CAC, LTV, and budget constraints

• Spots under performing campaigns and over performing ones

• Proposes a reallocation plan in the marketing tool

• Either executes it automatically, or prepares it for one click approval

Now the ROI is clear:

• Improved ROAS

• Faster reaction time

• Less manual spreadsheet work

Sales pipeline hygiene

Instead of a sales manager asking “What is my real pipeline?”, an agent:

• Scans open opportunities

• Cross checks product usage, email activity, last touch, win rates

• Flags deals that are clearly stale

• Updates fields, nudges reps, or creates summary views for forecast

Again, ROI is measurable:

• Forecast accuracy

• Time saved in pipeline review meetings

• Higher win rates because teams focus on the right deals

Support and product feedback loop

Instead of product managers asking “What are customers complaining about?”, an agent:

• Reads tickets, call transcripts, NPS comments

• Clusters them into themes

• Ties those themes to product areas, customer segments, and revenue

• Proposes a prioritized backlog and shares it with the team

Here the ROI is:

• Faster detection of issues

• Reduced churn

• Better prioritization

Notice what is common across all of these:

• No one typed a prompt

• The agent started the interaction

• Humans are in the loop for judgment and approvals, but not doing the mechanical work

In “Solid’s Leap to an Agentic Platform” we shared how we are reorienting our own product around this kind of behavior, not just a smarter chat window on top of the warehouse. That shift in architecture is what unlocks these use cases.

Why agents make ROI obvious

If you build the right agents, the ROI conversation becomes much simpler and much faster.

Instead of talking about value over “the next 18 to 24 months”, you can ask:

• In the next 1 to 3 months, how much time can we save per week for this role?

• In the next quarter, how many errors can we avoid?

• How much faster can we react to key events once an agent is in place?

• What measurable uplift in revenue or margin do we expect if we react in days instead of quarters?

You can usually tie that to:

• Reduction in manual steps

• A measurable change in a metric

• A clear before and after comparison on a specific workflow

You do not have to convince anyone with “We had 3,000 chats this month”. You can show:

• “We recover 2 percent more revenue on renewals because at risk customers get flagged and handled earlier.”

• “We uncovered 5 percent more business opportunities that are high quality.”

• “We cut time to resolution by 25 percent on a certain ticket type.”

That is a different level of conversation with your CFO or CEO.

Validating the quality of what agents actually do

There is a catch. Once agents are doing real work, they also have real power to mess things up.

So there are two questions you have to answer at the same time:

1. Are we getting business outcomes?

2. Is the agent’s behavior reliable enough that we trust it on those outcomes?

That is where evals and benchmarks come in.

In our post on how we do evals for Solid’s chat, we talked about building a proper evaluation harness instead of relying on “it feels good in the demo”. The same idea applies to agents, but the bar is higher.

A good agentic platform should let you:

• Define representative tasks
  Real prompts or events that mirror how the agent will be used in production. Not toy examples.

• Specify what “good” looks like
  Ground truth outputs, constraints, and checklists. For agents this can include not only the final answer, but also which tools they should or should not touch.

• Run benchmarks continuously
  Every time you change a model, a tool, a prompt, or business logic, you rerun the benchmark and see what got better and what regressed.

• Track the right metrics
  Inspired by the metrics we use for chat evals:

  • Accuracy: did the agent take the correct action, on the right entities?

  • Consistency: do we get similar behavior across similar scenarios?

  • Latency: is it fast enough for the business context?

  • Cost: are we comfortable with the per task cost at scale?

  • Safety and governance: did it respect permissions and policies?

Most serious agentic platforms are now building some flavor of this. If you are evaluating tools, I would put “What is your eval and benchmarking story?” very high on the list of questions.

Without this, you are left with anecdotes, screenshots, and a lot of “seems fine”. With it, you can treat agents like any other production system: you ship, you measure, you improve.

So is “chat with your data” useless?

No. It is useful, but you have to put it in its proper place.

Here is how I would think about it:

• Nice bonus, not the hero feature. Chat is great for exploration, long tail questions, and debugging. It is not where your big business outcomes will come from.

• Power tool for experts. Analysts and data savvy users can use chat to move faster, prototype, and inspect. But they are a small portion of your total user base.

• Support tool for agents. Agents will sometimes need to “explain themselves”, show their work, or help a human dive deeper. Chat can be the interface for that.

If you start with chat as the core story, you end up optimizing for demos that impress in 3 minutes. If you start with agents and real workflows, you optimize for systems that deliver clear value in the next quarter and keep delivering after that.

How to shift your roadmap from chat to agents

If you are a data and analytics leader planning your next couple of quarters, here is a simple way to reframe.

1. Start from business workflows, not from data access.
   Pick two or three high value workflows where:

   • The inputs are mostly digital and visible

   • Decisions are repeatable

   • There is clear business ownership

2. Map the “last mile” to action.
   For each workflow:

   • Who actually clicks the buttons today?

   • In which systems?

   • What information do they look at before deciding?

3. Design the agent as the “doer”, not just the “answerer”.
   The agent should:

   • Monitor conditions

   • Propose actions with reasons

   • Execute in systems of record, with audit and guardrails (with or without human-in-the-loop)

4. Bake in evals from day one.
   Before you roll out widely:

   • Build a small but sharp benchmark set for the workflow (Solid automates this for you)

   • Decide which metrics matter most (accuracy, safety, speed, cost)

   • Make every change to the agent go through this benchmark
     We learned this discipline the hard way on our own chat engine, and captured some of it in “The curse and promise of the white box”.

5. Add chat around the edges.
   Once the agent exists, surround it with:

   • Chat for “why did you do this?” explanations

   • Chat for “can you also do X?” refinements

   • Chat for analysts to debug and improve behavior

6. Measure the right things fast.
   Track:

   • Outcomes (revenue, cost, speed, quality) every week or month

   • Volume of work automated

   • Human time saved for the specific team
     Use “number of chats” only as a secondary signal, not the headline.

If you do this, “chat with your data” becomes part of a bigger, agentic strategy instead of the main show.

If you are evaluating vendors, or designing your own internal platform, I would encourage you to ask a different question:

“Over the next 3 to 6 months, what real work will AI be doing for our business, how will we know it is working, and how will we know it is working reliably?”

If the answer revolves mostly around a chat window, you are probably leaving most of the value on the table.