"Chat with your data" is NOT what you should be aiming for

Everyone wants a chat with their data - allow non-technical users to interact with it. That's wrong; the real value shows up when AI stops just answering questions and starts doing the actual work.

The dream: anyone can ask anything

If you spend even a few minutes on LinkedIn these days, you will see some version of the same pitch:

“Imagine every business user in your company can just chat with your data.”

The demo is always similar:

• Someone opens a chat window

• Types “What were Q4 revenues in EMEA versus target?”

• A nice chart pops up

• Everyone nods. “This is the future.”

I get why this is compelling. We have spent a decade building dashboards, training people on BI tools, arguing about metric definitions. The idea that we can put a natural language interface in front of all of that, and suddenly “everyone becomes data driven”, is very appealing.

We have written before about why enabling this is so hard on the technical side, from Text2SQL accuracy issues in “Everyone wants Text2SQL, but the pros don’t trust it” to the heavy lifting needed in “Semantic layer for AI: let’s not make the same mistakes we did with data catalogs”.

This time I want to focus on something else:

Even if you solve all the technical problems, “chat with your data” has two very real business problems.

---

Problem 1: usage is not ROI

When companies roll out a “chat with your data” experience, here is what they tend to measure:

• How many people tried it

• How many questions they asked

• How often they came back

• Satisfaction scores like “this answer was useful”

All of those are fine. None of them are ROI.

It is very hard to answer questions like:

• Did this chat experience help us win more deals?

• Did it reduce churn?

• Did it increase revenue per customer?

• Did it reduce cost per ticket in support?

You can say “Marketers now get answers faster” or “Sales can self serve their questions”, but that is not the same as proving material business impact.

We see a similar pattern to what we saw with early BI rollouts:

1. A big launch, internal roadshow, a few great demo moments

2. A couple of power users adopt it seriously

3. Everyone else goes back to:

   • Asking the same data people in Slack

   • Looking at the same dashboards as before

   • Exporting to Excel

You can show a nice adoption chart. You can say “we had 3,000 questions asked this quarter”. But when the CFO asks “What did we get for this?” it is very hard to go from “3,000 questions” to “X million dollars”.

That is not a problem unique to AI. It is the same problem BI has had for years. The difference is that with AI, the expectations are much higher.

---

Problem 2: after the novelty, no one knows what to ask

The second pattern we keep hearing in conversations with data and analytics leaders is even more worrying:

“The first month, people played with it. After that, usage dropped. People did not know what to do with it.”

We saw a similar behavior in the work behind “When users don’t chat with your chat”. Most users are not “trained” to interact with AI in a rich, structured way. A typical prompt is closer to a search query than to a thoughtful analytical question.

In practice that looks like:

• “Show me sales last month”

• “Top 10 customers”

• “Pipeline by region”

Those are fine questions, but they are not very far from what you already had in dashboards or reports. And once people have tried a few of these, they hit a wall:

• They do not know how to decompose fuzzier questions

• They are not sure what the system “knows”

• They do not want to get something wrong in front of their manager based on a chat answer

So they fall back to the pattern they trust:

• Open the dashboard that “everyone uses”

• Ask an analyst to “just pull something for me”

• Copy paste into a deck

Chat becomes another tool they occasionally open, not something that fundamentally changes how the business runs.

---

Why this is happening: chat is a pull interface

If you zoom out, there is a structural reason for this:

Chat is a pull interface. It relies on the human to:

• Notice there is a question worth asking

• Translate it into words the system understands

• Decide what to do with the answer

• Then go and do the work

That is a lot to ask from already overloaded business stakeholders.

The people you are most excited to “put AI in front of” are usually the ones with the least time and attention. They are managing teams, customers, numbers. Asking them to become “prompt engineers” on top of that is not realistic.

So you get:

• Some cool one off wins

• Good internal marketing material

• But not a durable change in how decisions are made and work gets done

Which leads to the core point of this post:

The real promise of AI on structured data is not “chat with your data”. That is a nice bonus. The real promise is agents that take work off people’s plates.

---

What agents do that chat never will

When I say “agents”, I do not mean a fancy wrapper around chat that calls a few tools. I mean systems that:

• Run on a schedule or trigger from events

• Pull from multiple sources:

  • Data warehouse and metrics

  • Product events

  • CRM

  • Support tickets

  • Even free text notes

• Apply reasoning and business logic

• Make a decision or propose one

• Then take action in the real systems of record

Some concrete examples.

Marketing budget reallocation

Instead of a marketer asking “How did Campaign X perform?”, an agent:

• Monitors performance of all campaigns

• Understands targets for CAC, LTV, and budget constraints

• Spots under performing campaigns and over performing ones

• Proposes a reallocation plan in the marketing tool

• Either executes it automatically, or prepares it for one click approval

Now the ROI is clear:

• Improved ROAS

• Faster reaction time

• Less manual spreadsheet work

Sales pipeline hygiene

Instead of a sales manager asking “What is my real pipeline?”, an agent:

• Scans open opportunities

• Cross checks product usage, email activity, last touch, win rates

• Flags deals that are clearly stale

• Updates fields, nudges reps, or creates summary views for forecast

Again, ROI is measurable:

• Forecast accuracy

• Time saved in pipeline review meetings

• Higher win rates because teams focus on the right deals

Support and product feedback loop

Instead of product managers asking “What are customers complaining about?”, an agent:

• Reads tickets, call transcripts, NPS comments

• Clusters them into themes

• Ties those themes to product areas, customer segments, and revenue

• Proposes a prioritized backlog and shares it with the team

Here the ROI is:

• Faster detection of issues

• Reduced churn

• Better prioritization

Notice what is common across all of these:

• No one typed a prompt

• The agent started the interaction

• Humans are in the loop for judgment and approvals, but not doing the mechanical work

In “Solid’s Leap to an Agentic Platform” we shared how we are reorienting our own product around this kind of behavior, not just a smarter chat window on top of the warehouse. That shift in architecture is what unlocks these use cases.

---

Why agents make ROI obvious

If you build the right agents, the ROI conversation becomes much simpler and much faster.

Instead of talking about value over “the next 18 to 24 months”, you can ask:

• In the next 1 to 3 months, how much time can we save per week for this role?

• In the next quarter, how many errors can we avoid?

• How much faster can we react to key events once an agent is in place?

• What measurable uplift in revenue or margin do we expect if we react in days instead of quarters?

You can usually tie that to:

• Reduction in manual steps

• A measurable change in a metric

• A clear before and after comparison on a specific workflow

You do not have to convince anyone with “We had 3,000 chats this month”. You can show:

• “We recover 2 percent more revenue on renewals because at risk customers get flagged and handled earlier.”

• “We uncovered 5 percent more business opportunities that are high quality.”

• “We cut time to resolution by 25 percent on a certain ticket type.”

That is a different level of conversation with your CFO or CEO.

---

Validating the quality of what agents actually do

There is a catch. Once agents are doing real work, they also have real power to mess things up.

So there are two questions you have to answer at the same time:

1. Are we getting business outcomes?

2. Is the agent’s behavior reliable enough that we trust it on those outcomes?

That is where evals and benchmarks come in.

In our post on how we do evals for Solid’s chat, we talked about building a proper evaluation harness instead of relying on “it feels good in the demo”. The same idea applies to agents, but the bar is higher.

A good agentic platform should let you:

• Define representative tasks
  Real prompts or events that mirror how the agent will be used in production. Not toy examples.

• Specify what “good” looks like
  Ground truth outputs, constraints, and checklists. For agents this can include not only the final answer, but also which tools they should or should not touch.

• Run benchmarks continuously
  Every time you change a model, a tool, a prompt, or business logic, you rerun the benchmark and see what got better and what regressed.

• Track the right metrics
  Inspired by the metrics we use for chat evals:

  • Accuracy: did the agent take the correct action, on the right entities?

  • Consistency: do we get similar behavior across similar scenarios?

  • Latency: is it fast enough for the business context?

  • Cost: are we comfortable with the per task cost at scale?

  • Safety and governance: did it respect permissions and policies?

Most serious agentic platforms are now building some flavor of this. If you are evaluating tools, I would put “What is your eval and benchmarking story?” very high on the list of questions.

Without this, you are left with anecdotes, screenshots, and a lot of “seems fine”. With it, you can treat agents like any other production system: you ship, you measure, you improve.

---

So is “chat with your data” useless?

No. It is useful, but you have to put it in its proper place.

Here is how I would think about it:

• Nice bonus, not the hero feature. Chat is great for exploration, long tail questions, and debugging. It is not where your big business outcomes will come from.

• Power tool for experts. Analysts and data savvy users can use chat to move faster, prototype, and inspect. But they are a small portion of your total user base.

• Support tool for agents. Agents will sometimes need to “explain themselves”, show their work, or help a human dive deeper. Chat can be the interface for that.

If you start with chat as the core story, you end up optimizing for demos that impress in 3 minutes. If you start with agents and real workflows, you optimize for systems that deliver clear value in the next quarter and keep delivering after that.

---

How to shift your roadmap from chat to agents

If you are a data and analytics leader planning your next couple of quarters, here is a simple way to reframe.

1. Start from business workflows, not from data access.
   Pick two or three high value workflows where:

   • The inputs are mostly digital and visible

   • Decisions are repeatable

   • There is clear business ownership

2. Map the “last mile” to action.
   For each workflow:

   • Who actually clicks the buttons today?

   • In which systems?

   • What information do they look at before deciding?

3. Design the agent as the “doer”, not just the “answerer”.
   The agent should:

   • Monitor conditions

   • Propose actions with reasons

   • Execute in systems of record, with audit and guardrails (with or without human-in-the-loop)

4. Bake in evals from day one.
   Before you roll out widely:

   • Build a small but sharp benchmark set for the workflow (Solid automates this for you)

   • Decide which metrics matter most (accuracy, safety, speed, cost)

   • Make every change to the agent go through this benchmark
     We learned this discipline the hard way on our own chat engine, and captured some of it in “The curse and promise of the white box”.

5. Add chat around the edges.
   Once the agent exists, surround it with:

   • Chat for “why did you do this?” explanations

   • Chat for “can you also do X?” refinements

   • Chat for analysts to debug and improve behavior

6. Measure the right things fast.
   Track:

   • Outcomes (revenue, cost, speed, quality) every week or month

   • Volume of work automated

   • Human time saved for the specific team
     Use “number of chats” only as a secondary signal, not the headline.

If you do this, “chat with your data” becomes part of a bigger, agentic strategy instead of the main show.

---

If you are evaluating vendors, or designing your own internal platform, I would encourage you to ask a different question:

“Over the next 3 to 6 months, what real work will AI be doing for our business, how will we know it is working, and how will we know it is working reliably?”

If the answer revolves mostly around a chat window, you are probably leaving most of the value on the table.