The next generation of Databricks Genie

Databricks Genie is no longer just a data chatbot. With the recent updates announced by Databricks, the product is positioning itself as something much more ambitious: a conversational data agent with real reasoning capability, native integration with the Lakehouse ecosystem, and support for Digital Twins use cases. Anyone who has followed the evolution of Databricks’s portfolio over the last two years knows that the company has been consistently accelerating its bet on generative AI applied to data, and Genie is one of the most visible products of that strategy.

In this article, I will explore what changed in the new generation of Genie, what it means in practice for engineering and analytics teams, and why this move is worth paying attention to, especially in the context of the Solution Accelerator Series focused on Digital Twins.

I have worked with Databricks for a few years and have closely followed how the platform evolved from a Spark-focused environment into a unified data and AI platform. Genie is a product that sums up this transition well: it connects what Databricks built in infrastructure with what the market now demands in terms of accessibility and intelligence.

What Databricks Genie is and what changed

Genie was initially launched as a feature within Databricks SQL that allowed business users to ask questions in natural language about data, receiving automatically generated SQL queries as a response. The goal was to democratize data access without requiring analysts and managers to know how to write SQL.

The new generation significantly changes the scope. Genie now operates as a data agent, not just a natural-language-to-SQL translator. This means it can chain reasoning across multiple steps, interpret business context provided by the data team itself, and generate responses that combine explanatory text with visualizations and tables, all within a conversational interface.

The central points of the update include:

• Custom instructions and business context: engineering and analytics teams can configure Genie with internal documentation, metric definitions, glossaries, and business rules. This drastically reduces the ambiguity problem that compromised responses in previous versions.

• Support for multiple tables and complex joins: the model now handles richer relational schemas better, being able to infer relationships between tables without the user having to make them explicit in the question.

• Integration with Unity Catalog: Genie respects Unity Catalog permissions, which means a user will only see data they have access to, with no need for additional security configuration at the agent level.

• Traceability of responses: each response generated by Genie includes the underlying SQL query, allowing the user to validate the model’s reasoning.

Digital Twins and the Solution Accelerator context

The announcement of the next-generation Genie is directly linked to the Digital Twins Solution Accelerator Series, which is one of the most interesting use cases for understanding the product’s potential.

Digital Twins in the context of industrial or infrastructure data involve virtual models of physical systems, fed by real-time sensor data, operational history, and simulation parameters. The volume and complexity of this data make it practically impossible for operational users, such as field engineers or plant managers, to consume insights directly from traditional dashboards.

This is where Genie fits in. Instead of building dozens of specific panels for each user profile, the data team configures Genie with:

1. The Delta Lake tables that contain the twin's data (sensor time series, maintenance logs, equipment parameters).
2. Business instructions that explain what each metric means, what the relevant operational thresholds are, and how to interpret anomalies.
3. Examples of questions and expected answers, which work as few-shot prompting for the model.

An operator can then ask things like “Which equipment on line 3 had the largest temperature variation in the last 24 hours?” or “Compare this week’s energy consumption with the average of the last 30 days per shift.” Genie generates the query, runs it against Delta Lake via Databricks SQL, and delivers the answer formatted with explanatory context.

A practical example

I will show a simplified example of what configuring a Genie Space would look like for an industrial Digital Twin case.

• Table structure in Unity Catalog

-- Sensor readings table
CREATE TABLE manufacturing.digital_twin.sensor_readings (
  equipment_id    STRING,
  sensor_type     STRING,  -- 'temperature', 'pressure', 'vibration'
  reading_value   DOUBLE,
  unit            STRING,
  event_timestamp TIMESTAMP,
  plant_id        STRING,
  line_id         STRING
)
USING DELTA
PARTITIONED BY (plant_id, DATE(event_timestamp));

-- Equipment table
CREATE TABLE manufacturing.digital_twin.equipment_master (
  equipment_id     STRING,
  equipment_name   STRING,
  equipment_type   STRING,
  line_id          STRING,
  plant_id         STRING,
  install_date     DATE,
  nominal_temp_max DOUBLE,
  nominal_temp_min DOUBLE
);

-- Maintenance orders table
CREATE TABLE manufacturing.digital_twin.maintenance_orders (
  order_id        STRING,
  equipment_id    STRING,
  order_type      STRING,  -- 'preventive', 'corrective'
  open_date       TIMESTAMP,
  close_date      TIMESTAMP,
  root_cause      STRING
);

• Business instructions for Genie (configuration example)

You are an assistant specialized in industrial operations for the manufacturing plant.

Context:
- sensor_readings contains real-time readings from all equipment.
- equipment_master contains the nominal operating limits of each piece of equipment (nominal_temp_max, nominal_temp_min).
- A temperature anomaly is defined as any reading above nominal_temp_max or below nominal_temp_min.
- Questions about "efficiency" refer to the `reading_value` field of sensor_type = 'efficiency_index'.
- Always filter by plant_id = 'PLANT_BR_01' as the default, unless the user specifies another plant.
- When the user asks about "last week", use the interval of the last 7 days from today.
- When identifying problematic equipment, cross-reference with maintenance_orders to check whether there are open orders.

• Typical query generated by Genie

Given the context above, a question like “Which equipment is operating outside the nominal temperature right now?” would generate something close to:

WITH latest_readings AS (
  SELECT
    sr.equipment_id,
    sr.reading_value AS current_temp,
    sr.event_timestamp,
    em.equipment_name,
    em.line_id,
    em.nominal_temp_max,
    em.nominal_temp_min
  FROM manufacturing.digital_twin.sensor_readings sr
  JOIN manufacturing.digital_twin.equipment_master em
    ON sr.equipment_id = em.equipment_id
  WHERE
    sr.plant_id = 'PLANT_BR_01'
    AND sr.sensor_type = 'temperature'
    AND sr.event_timestamp >= NOW() - INTERVAL 15 MINUTES
),
ranked AS (
  SELECT *,
    ROW_NUMBER() OVER (PARTITION BY equipment_id ORDER BY event_timestamp DESC) AS rn
  FROM latest_readings
)
SELECT
  equipment_id,
  equipment_name,
  line_id,
  current_temp,
  nominal_temp_min,
  nominal_temp_max,
  ROUND(current_temp - nominal_temp_max, 2) AS deviation_above_max,
  event_timestamp
FROM ranked
WHERE
  rn = 1
  AND (current_temp > nominal_temp_max OR current_temp < nominal_temp_min)
ORDER BY ABS(current_temp - nominal_temp_max) DESC;

This query would be executed automatically and the result delivered to the operator with an explanation in natural language, without them needing to know any SQL.

Trade-offs and limitations

The new-generation Genie is a much more capable product than the previous version, but there are important limitations that need to be considered before putting it into production for critical cases.

• The quality of the business instructions is decisive.

The model relies heavily on the context provided by the data team. If the metric definitions are ambiguous or incomplete, Genie will generate plausible but incorrect responses. This requires a real investment from the engineering and analytics team in curating these instructions, work that many organizations underestimate.

• It does not replace well-done data modeling.

Poorly designed schemas, without good naming conventions, with ambiguous or duplicated columns, will make Genie’s job much harder. The product performs best when the Lakehouse underneath is well organized.

• Inference costs need to be monitored.

Each question asked to Genie consumes LLM tokens. In high-frequency usage scenarios, this can generate significant costs that need to be on the radar.

• It is still not suitable for critical decisions without human validation.

In industrial contexts, a wrong query can lead to an incorrect interpretation of the state of a piece of equipment. Genie exposes the generated query precisely to allow this validation, but it requires operational users to have the maturity to question the answers.

Databricks’s bigger move in AI

Genie does not exist in isolation. It is part of a broader strategy that Databricks has been executing over the last few years: turning the Lakehouse into a complete platform for AI applied to data.

The acquisition of MosaicML, the launch of DBRX, the investments in MLflow and Unity Catalog, and now the new generation of Genie with agentic capabilities, form a cohesive portfolio. Databricks is betting that the future is not having a generic AI model answering questions about public data, but rather having specialized agents, fed by each organization’s proprietary data, with native governance and traceability.

This is different from what most traditional BI players are doing, which is essentially putting a chatbot in front of an existing dashboard. Genie operates directly on Delta Lake, respects Unity Catalog, and can be extended with custom logic via Databricks Apps and the Genie Spaces API.

Conclusion

The new generation of Databricks Genie represents a genuine evolution, not just a product refresh. The ability to configure business context, the deep integration with Unity Catalog, and the support for use cases like Digital Twins show that Databricks is taking seriously the goal of making data accessible to non-technical users without giving up governance and reliability.

For data engineering teams, the work does not disappear: it transforms. Instead of building dashboards for every possible question, the focus shifts to ensuring the Lakehouse is well modeled, that the business instructions are precise, and that the data pipelines are delivering enough quality for Genie to work.

The next practical step for those who want to explore this is to create a Genie Space with a subset of real data, write business instructions carefully, and test with real domain users. Results vary a lot depending on the quality of the schema and the context provided, and this controlled experimentation is the best way to calibrate expectations before a broader rollout.