Explore this interactive demo and see how Dremio's Intelligent Lakehouse enables Agentic AI
The current Dremio Software architecture often uses a fixed pool of executor engines. While this provides stability for baseline workloads, it struggles to handle predictable spikes in demand, leading to performance bottlenecks during peak periods, and overallocation of engines during quieter periods. Overallocation can have an impact on Cloud costs.
This document’s examples were tested using Dremio Software, running on v26.0.6.
To accommodate predictable peak workloads, the proposed design introduces the concept of Time-Segmented Engine Pools.
| Feature | Current Architecture | Proposed Random Engine Design |
| Scaling for peak/medium workloads | Manual or relies on external scaling (if configured). Slow reaction. | Automatic activation of pre-configured replica engines during scheduled times. Fast reaction. |
| Engine utilisation | Potentially underutilised during off-peak. Overloaded during peak. | Consistent utilisation of E-Base, with supplemental capacity (E-Medium, E-Peak) only when scheduled. |
| Predictability | High variability in performance during peak. | Stable and predictable query performance during known peak windows. |
The following table explains the available engine capacity over a 24-hour period, comparing the current fixed architecture with the proposed Round Robin design using scheduled Peak and Medium Engines.
| Time Slot (24h) | Proposed Engines | Proposed Capacity (Nodes) |
| 08:00 - 12:59 (Medium) | E-Base, E-Medium | 8 |
| 13:00 - 13:59 (Low) | E-Base | 4 |
| 14:00 - 17:59 (Peak) | E-Base, E-Medium, E-Peak | 12 |
| 18:00 - 07:59 (Low) | E-Base | 4 |
This section will describe how to configure the additional engines and rules, to satisfy the use case described earlier.
Assumptions
Two new engines will be added: E-Medium and E-Peak. They will be sized identically to the existing engine E-Base.
We need to ensure that our new engines are enabled to “Automatically start/stop” (see Engine Settings documentation):
NB. If the “E-Base” Engine is required to be always on (24/7), then this engine’s setting for “Automatically start/stop” should be disabled. Alternatively, if the E-Base engine should be shutdown after a period of inactivity, then the “Automatically start/stop” feature should be enabled, but with an appropriately-defined Idle Time.
We need to add 2 extra queues and rules to handle Medium and Peak workloads.
(see also Dremio Queues documentation)
The new queues will be based on the existing E-Base queue, with the difference that they will route to the new Engines.
| Queue Name | Engine Name |
| Medium | E-Medium |
| Peak | E-Peak |
(see also Dremio Rules documentation)
We need to add our new rules so that they are only activated during required time periods. We will also need to apportion the workload randomly during those allotted time periods.
The new rules:
Any existing “E-Base” routing rules will also have to be included into our new rules.
query_cost() >= 300000In these examples:
query_cost() >= 300000 is an existing rule condition.EXTRACT (HOUR FROM CURRENT_TIME) feature to determine the allotted time periods for medium / peak workloads
CURRENT_TIME function returns a result in GMT / UTC.| Priority | Rule Name | Rule Conditions | Queue | Comment |
| 1 (new rule) | Peak | query_cost() >= 300000 | Peak | Peak Activity engine will only be active between 14:00 -17:59. |
| 2 (New rule) | Medium | query_cost() >= 300000 | Medium | Medium Activity engine will be active between both 14:00 -17:59 and between 08:00 -12:59. |
| 3 (existing rule) | Base | query_cost() >= 300000 | Base | Base Activity engine will be active all day. |
The result of this configuration will be:
| Time of Day | Engine Name | Proportion of Traffic |
| 0800 - 1259 (Medium) | E-Base | ~ 50% |
| E-Medium | ~ 50% | |
| 1300 - 1359 (Base) | E-Base | 100% |
| 1400 - 1759 (Peak) | E-Base | ~ 33% |
| E-Medium | ~ 33% | |
| E-Peak | ~ 33% | |
| 1800 - 0759 (Base) | E-Base | 100% |
If you would only like the replica engines to be active only on weekdays, add the following rule to the replica engines’ routing:
AND extract(DOW from CURRENT_DATE) between 2 and 6
NB. the DOW function extracts day-of-week as follows:
Eg., If we decide to add an extra band, such as “SuperPeak”, to cater for higher concurrency, only active between 17:00 and 17:59:
| Queue Name | Engine Name |
| SuperPeak | E-SuperPeak |
Note that for the new rule which is adding a 4th time band, we will allocate ¼ of the queries to the new Engine, during the time it is required (17:00 - 17:59). This means that 25% of the queries are allocated to SuperPeak and the remaining 75% will be allocated to the other engines.
| Order | Name | Rule | Queue |
| 1 (new rule) | SupPeak | query_cost() >= 300000AND (EXTRACT(HOUR FROM CURRENT_TIME) = 17)AND RANDOM() < 0.25 | SuperPeak |
| 2 (existing rule 1) | Peak | query_cost() >= 300000AND (EXTRACT(HOUR FROM CURRENT_TIME) BETWEEN 14 AND 17)AND RANDOM() < 0.33 | Peak |
| 3 (existing rule 2) | Medium | query_cost() >= 300000 | Medium |
| 4 (existing rule 3) | Base | query_cost() >= 300000 | Base |
Dremio engines can be enabled to automatically start and stop - see Adding an Engine.
Description of autostart feature: When a query is routed to the engine, then the engine will start (if it is not already running).
Impact. If an engine is not already running, then there will be a delay before the query is executed. End users will be impacted with slower query times.
Workaround: Start the engine before it is scheduled to run. This can be achieved using the Dremio REST API - see starting engines - REST API documentation.
Description of autostop feature: After a period of idle time (default 1 hour), the engine will automatically stop running.
Impact of low idle time: If the engine stops too frequently, then subsequent queries will be delayed from executing.
Workarounds:
Impact of high idle time: The engine will remain running for a longer period. This will increase the cost of running your instance.
Deploy agentic analytics directly on Apache Iceberg data with no pipelines and no added overhead.
Column nullability serves as a safeguard for reliable data systems. Apache Iceberg's capabilities in enforcing and evolving nullability rules are crucial for ensuring data quality. Understanding the null, along with the specifics of engine support, is essential for constructing dependable data systems.
Clustering is a data layout strategy that organizes rows based on the values of one or more columns, without physically splitting the dataset into separate partitions. Instead of creating distinct directory structures, like traditional partitioning does, clustering sorts and groups related rows together within the existing storage layout.
The Polaris benchmarking framework provides a robust mechanism to validate performance, scalability, and reliability of Polaris deployments. By simulating real-world workloads, it enables administrators to identify bottlenecks, verify configurations, and ensure compliance with service-level objectives (SLOs). The framework’s flexibility allows for the creation of arbitrarily complex datasets, making it an essential tool for both development and production environments.
