The Two Souls of a Semantic Layer: A Tale of Governance and Insight
Semantic layers are in vogue again, but this time for a different reason. Tal Segalov, Solid's CTO and Co-Founder, explains the relationship between the new semantic layer and the old one.
Have you ever been in a meeting where two departments present different numbers for the same metric? It’s a frustratingly common scenario that chips away at the trust we place in our data. This is where the concept of a semantic layer comes in, and it's rapidly becoming the most critical component of the modern data stack. But here's the interesting part: the semantic layer is developing a split personality, two distinct souls wrestling for the heart of our data strategy. When you say “semantic layer” to a data practitioner, they may have one of two images in mind.
On one side, we have the "Governance-as-Code" paradigm. Think of this as the meticulous librarian of your data world. Championed by tools like dbt Semantic Layer and Google's Looker, this approach treats your business logic as mission-critical code. Every metric, every definition is version-controlled and rigorously tested. The primary audience here is the analytics engineer, the detail-oriented professional who ensures that when someone asks for "customer lifetime value," the calculation is consistent, accurate, and trustworthy across the entire organization. This is the bedrock of a data-driven culture – a single source of truth that eliminates the "he said, she said" of data debates. Leaders in the space thought this would be the future, the “missing piece”, then discovered that there wasn’t enough need for this on its own.
On the other side, we have the "Conversational Insight" paradigm. This is the charismatic storyteller who can effortlessly weave narratives from complex data. The semantic layer is a critical part of this AI engine - the output of your AI agent is only as good as the documentation you feed it. Platforms like Snowflake's Cortex Analyst and Databricks Genie are the stars of this show. They leverage the power of Large Language Models (LLMs) to provide a natural language interface for business users. The goal here is speed-to-insight. A marketing manager can simply ask, "Which campaigns had the best ROI last quarter?" and get an immediate, data-backed answer without writing a single line of SQL.
Do we really have to choose?
Now, you might be thinking, which side should I bet on? The rigorous librarian or the engaging storyteller? Here’s the twist: it's not a competition. In fact, these two paradigms are becoming increasingly symbiotic. The Conversational Insight engines, for all their AI-powered brilliance, are prone to "hallucinations" without a solid foundation. They need the meticulously curated definitions from the Governance-as-Code layer to provide accurate and reliable answers.
At Solid, we are working to build the best semantic models to feed the Conversational Insight platforms. All these platforms require you to first provide them with the right model, that includes granular documentation, data filters, metrics, aggregations and relations (BigQuery, Databricks and Snowflake all have tips for how to do this well). It also requires you to provide validated uses of the model with queries that serve a business purpose. Conversational engines then rely on this data to construct a query based on a business question, and the result is as good as your model is. That is why the entire industry is investing in those models - they are the key to generating significant business value from your data.
This creates a powerful partnership: the engineering-centric layer provides the trusted, governed "what," and the business-centric layer delivers the accessible, intuitive "so what." For enterprise data leaders, investors, and fellow entrepreneurs, understanding this duality is key. It's not about choosing one over the other, but about architecting a data strategy that embraces both souls of the semantic layer. By building a strong foundation of governed data, we empower the storytellers to unlock its true value, turning raw numbers into the compelling narratives that drive our businesses forward.
Essentially, making all those cool demos work in reality.