Enabling Guided, Intelligent Lakehouse Construction and Management
Modern lakehouse platforms have historically required significant technical expertise to operate effectively. Connecting data sources, modeling clean layers, applying governance semantics, creating views, and troubleshooting query failures have been tasks reserved for skilled data engineers. The learning curve has been a meaningful adoption barrier, particularly for organizations building their first lakehouse or expanding beyond a small core team.
Dremio's embedded AI Agent addresses this barrier directly. It does not merely advise; it acts. The agent connects sources, writes and executes SQL, creates views, applies semantic annotations, configures governance policies, and delivers analytical results, all on behalf of the administrator. It understands Dremio's architecture, catalog state, and query history, enabling it to take context-aware actions rather than offer generic instructions.
The agent is also reactive, not just proactive. When queries fail, when jobs produce unexpected results, or when users encounter errors, the AI Agent automatically analyzes what went wrong, explains the root cause in plain language, and offers to remediate the issue directly. This closes the loop between detection and resolution in a way that traditional monitoring and documentation cannot.
When a user opens the AI Agent panel within Dremio, they are presented with a simple question: “What can I do for you?”
This interface presents four primary intents as starting points:
Figure 1b. The AI Agent’s response to “Show me what Dremio can do,” demonstrating its ability to execute contextual orientation for any user role.
Figure 1. The AI Agent entry screen presenting four goal-oriented starting points.
This goal-oriented entry point meets users where they are, rather than presenting a blank command interface. A new administrator who has never built a lakehouse before and an experienced data engineer who wants to connect a new source can both start from the same screen, and in both cases, the agent takes over from there, executing the work rather than simply explaining what to do.
After a user selects “Build a lakehouse on my sources,” the AI Agent asks a scoping question: “Which approach suits your needs?” Two options are presented:
Minimal setup. The user connects a single source or uploads a file and gets started immediately. This path is optimized for time-to-first-query and is appropriate for proofs of concept, initial exploration, or users who want to validate a source before committing to a broader implementation.
End-to-end execution of a production-grade, multi-source lakehouse. The agent generates a complete implementation plan and then carries it out, connecting sources, writing transformation SQL, building silver and gold layers, applying semantic annotations, and configuring governance policies. The administrator reviews and approves at key decision points; the agent does the work.
Figure 2. The AI Agent scope-setting interface, presenting two implementation paths.
This branching design reflects a core principle in Dremio’s AI Agent architecture: the agent does not assume a single user type or a single use case. By surfacing the choice explicitly and describing each option in plain terms, the agent helps users self-select into the right workflow without requiring prior knowledge of lakehouse architecture.
Whether a user chooses the quick start or multi-source path, the next action is connecting data. The AI Agent surfaces an “Add Data” interface that consolidates all supported ingestion and connection modes into a single, organized view.
Users can drag and drop CSV, JSON, or Parquet files directly into the interface without any configuration. Dremio automatically ingests the file, infers the schema, and stores the data as a managed Apache Iceberg table in Dremio’s Open Catalog. The AI Agent handles format detection, column type inference, and table creation without user intervention.
Uploaded files are stored as first-class Iceberg tables with the same governance, versioning, and query optimization capabilities as any other table in the lakehouse.
For live data connections, the AI Agent presents a categorized source catalog:
Figure 3. The Add Data interface showing file upload and data source connection options.
In each case, the AI Agent executes the connection: it validates connectivity, registers the source in the catalog, and confirms success, presenting only the fields relevant to the selected source type. The administrator provides credentials; the agent handles everything else.
When a user selects the multi-source implementation path, the AI Agent generates a complete, structured 7-step implementation plan tailored to their project.
Figure 4. The AI Agent’s 7-step implementation plan for a multi-source lakehouse.
The agent begins by anchoring the lakehouse build to a specific analytical goal, such as “What are our support ticket trends by region?” It recommends choosing a question that spans at least two sources to ensure the multi-source architecture delivers value immediately.
The agent presents three methods for bringing data into the lakehouse: load into Open Catalog via Fivetran, dbt, or Airbyte; connect existing sources for zero-ETL federated query; or upload local files stored as Iceberg tables.
The agent generates the transformation SQL, executes it, and creates the silver-layer views directly in the Dremio catalog. The administrator describes the goal in natural language. For example, “remove nulls from the customer_id column and standardize the date format.” The agent then produces the SQL, runs it, and saves the result as a reusable view. No SQL authoring required.
On top of the silver layer, the agent generates and creates the gold views: pre-aggregated, business-aligned datasets exposing the KPIs relevant to the business question from Step 1. The administrator reviews and approves the proposed structure; the agent executes the creation.
The AI Agent writes the semantic annotations itself, analyzing the structure and content of the data and producing plain-language descriptions for every view, table, and column. What would take a data engineering team weeks of manual documentation is executed by the agent in minutes, at a consistency level that manual processes cannot match. These annotations also directly improve AI query accuracy: Dremio uses them to disambiguate terms and generate correct SQL from natural language.
With a governed, semantically annotated lakehouse in place, the agent executes analytical queries and surfaces findings directly, producing inline visualizations and natural language summaries without requiring the administrator to write SQL. For organizations using external BI tools, the agent configures the connection to Tableau, Power BI, or any JDBC/ODBC-compatible client via Dremio’s Arrow Flight SQL endpoint. Users can ask free-form questions and receive answers backed by live SQL execution against the actual lakehouse, with the agent showing its work.
The final step focuses on production readiness:
Beyond the guided construction workflow, Dremio’s AI Agent provides continuous operational intelligence: when queries fail or produce unexpected results, the agent automatically analyzes the error and explains what went wrong in plain language.
Consider a representative example. A user submits the SQL statement:
| DROP TABLE dremio_samples.customer360 |
Figure 5. The Dremio SQL Runner showing a failed DROP TABLE statement with validation error.
The job fails in 186 milliseconds at the validation and planning phase with the error:
| VALIDATION ERROR: Table [dremio_samples.customer360] does not exist. (com.dremio.exec.store.iceberg.exceptions.IcebergTableNotFoundException) |
The user hits “Explain” then the AI Agent’s response provides three distinct layers of value:
Rather than surfacing the raw exception, the agent explains the context: this was a destructive DDL statement that failed at validation before any execution operators were created, meaning no data was affected.
The agent explains possible causes ranked by likelihood: a misspelled table name, a schema or catalog path mismatch, a table already dropped in a prior session, or a permissions issue.
The agent presents actionable next steps and offers to execute corrective actions on behalf of the user, such as running a catalog lookup to identify similarly named tables.
Figure 6. The AI Agent’s automated error analysis for the failed SQL job, showing Query Summary, Query State, and root-cause diagnostics.
This capability represents a meaningful reduction in mean time to resolution. The path from error to understanding to resolution is contained within the same interface where the work is happening. The AI Agent applies this diagnostic pattern to a wide range of failure types: permission violations, schema mismatches, pushdown failures, source connectivity issues, and syntax errors in generated SQL.
One of the most powerful aspects of Dremio’s embedded AI Agent is its ability to go far beyond guidance: it executes end-to-end data analysis from a single plain-language request. Users do not need to know SQL, understand schema relationships, or manually construct queries. They simply describe what they want, and the AI Agent does the work.
In the example below, a user types: “create a graph with the top 10 product orders for all months of September.” No additional context is provided. The AI Agent takes over from there, autonomously executing a multi-step plan to fulfill the request.
Figure 7a. A user submits a plain-language analysis request: “create a graph with the top 10 product orders for all months of September.”
The AI Agent immediately begins executing a structured plan visible to the user in real time. It searches the catalog to identify relevant datasets, inspects schema definitions for the orders, line items, and product tables, checks existing SQL recipes for reusable patterns, validates the aggregation logic, and finally runs the query. The entire process, from natural language to finished visualization, happens without the user writing a single line of SQL or navigating a single menu.
Figure 7b. The AI Agent delivers a completed bar chart visualization, having autonomously searched the catalog, inspected schemas, validated SQL, and executed the query.
This capability removes one of the most persistent barriers in enterprise data platforms: the gap between the people who have business questions and the people who know how to query data. With Dremio’s embedded AI Agent, any user in the organization can ask a question in plain language and receive a fully executed result. The AI Agent does not suggest how to build the query; it builds and runs it. The output lands directly in the conversation as a chart, table, or summary, ready for use.
The AI Agent also provides full transparency into how it arrived at its answer. Users can inspect the exact SQL it generated, view the underlying data table, and read an “Agent Insight” summary that interprets the results in plain language. In the example below, the Agent identifies that Small Marble Computer is the clear September leader at 38.5M distinct orders, and notes that the remaining top 9 products cluster tightly around 24.4M, indicating broadly uniform September demand. This level of autonomous interpretation means the AI Agent does not just fetch data; it delivers a complete, ready-to-act answer.
Figure 7c. The Query tab shows the AI-generated SQL and an Agent Insight interpreting the results in plain language, no manual analysis required.
Figure 7d. The Table tab presents the raw query results alongside the catalog the AI Agent searched to fulfill the request, giving users full visibility into the data and sources behind the answer.
The capabilities described in this document represent a coherent architecture for AI-executed data platform operation. The distinction from conventional AI assistants is fundamental: Dremio’s AI Agent does not produce instructions for an administrator to carry out. It carries them out. The key characteristics of this approach are as follows.
| Characteristic | Description |
| Embedded, not external | Operates within the Dremio product with full access to catalog state, job history, user permissions, and source metadata, enabling recommendations specific to the user’s environment. |
| Executes, not just advises | The agent performs the work, connecting sources, writing SQL, creating views, applying governance, generating documentation, rather than explaining what steps a human should take. Users retain control at key decision points; the agent carries out the tasks. |
| Lifecycle-complete | Covers the full operational lifecycle from first source connection through production governance, semantics, insight delivery, and error remediation. |
| Proactive and reactive | Initiates guided workflows when users express intent and responds automatically when the system detects failures, both within the same conversational interface. |
| Standards-based | All AI-generated assets are built on open standards including Apache Iceberg and Arrow Flight. No proprietary lock-in at the data layer. |
For organizations evaluating AI-enabled data platforms, Dremio’s embedded AI Agent represents a substantive and differentiated capability: the ability to reduce the expertise required to build, govern, and operate a production lakehouse without reducing the control or standards compliance that enterprise data teams require.
For additional information on Dremio’s AI Agent capabilities, you can sign up for a free trial at https://www.dremio.com/get-started/
Deploy agentic analytics directly on Apache Iceberg data with no pipelines and no added overhead.
By unifying data from diverse sources, simplifying data operations, and providing powerful tools for data management, Dremio stands out as a comprehensive solution for modern data needs. Whether you are a data engineer, business analyst, or data scientist, harnessing the combined power of Dremio and Apache Iceberg will undoubtedly be a valuable asset in your data management toolkit.
We're always looking for ways to better handle and save money on our data. That's why the "data lakehouse" is becoming so popular. It offers a mix of the flexibility of data lakes and the ease of use and performance of data warehouses. The goal? Make data handling easier and cheaper. So, how do we […]
This blog helps you learn about table-driven access policies in Dremio Cloud and Dremio Software v24.1+.
