Introducing Elastic Engines


Table of Contents

Introducing Elastic Engines - Dremio


Dremio AWS Edition supports the ability to provision multiple separate execution engines from a single Dremio coordinator node, start and stop based on predefined workload requirements at runtime. This provides several benefits, including:

  • Workloads are isolated within their own set of CPU, Memory & C3 resources and not impacted by other workloads
  • Time-sensitive yet resource intensive workloads (nightly jobs, reflections refreshes, etc.) can be provisioned with the appropriate amount of resources to complete on time, but remain cost effective by only running when required
  • Track cost by team by running workloads on their own resources
  • Right size execution resources for each distinct workload, instead of implementing a one size fits model
  • Easily experiment with different execution resource sizes, at any scale

In this article we walk you through the steps to provision and manage Elastic Engines, we also show you the steps to manage workloads using queues and rules.

How to Launch and Manage Elastic Engines?

Step 1. Elastic engines is one of the easiest features to use in Dremio. To provision a new engine, in the Dremio UI go to** Admin -> Elastic Engines**

image alt text

Step 2. You will see a default engine already deployed. Now click on Add New

image alt text

Step 3. The Set Up Engine popup page by default automatically selects the EC2 configuration options used to launch the engine (Security Group, EC2 Key Pair, etc). To continue the configuration, add a name, select the instance count and then click on Advanced Properties(optional)

image alt text

Step 4. Review the Advanced Properties menu. Here is where we will have the option to enable Auto Start and Stop.

  • When Auto Start is enabled: The engine will start as soon as it receives the first query.
  • When Auto Stop is enabled: The engine will stop after 5 minutes of inactivity.

Once you are ready selecting your options, click on Save & Launch

image alt text

Workload Management

Creating a Queue

Now, we need to tell Dremio which workloads we want it to process using any of the existing engines. To do so follow these steps:Step 1. Navigate to the Queues option within the Workload Management **in the **Admin settings. Then click on Add New.

image alt text

Step 2. Give the new queue a name, then from the Engine Name drop-down menu select the engine that you just created (or the engine that you want to assign to this queue). Here you will also see options related to time, memory and CPU management. When done, click Save.

image alt text

Creating a Rule

Now, we need to create a rule where we will specify that if a workload (or query) meets certain criteria, it will be handled by a certain engine. To create a new rule follow these steps

Step 1. From the Admin menu, click on Rules and then Add New.

image alt text

IMPORTANT: The priority of the rules is defined by their order i.e rule 1 will be parsed first and can supersede any other rules that follow. To change the order of the rules, click on three bar icon next to its name and drag it to the desired position.

Step 2. Give the rule a name and conditions. You will see several condition examples that you can use i.e. User, Group Membership, Job Type, Query Plan Cost, or a combination of all. Once you add the condition, you have the option to reject or assign the workload to the queue that you created in the previous section.

This means that any workload that meets the condition requirements, will be assigned to a queue, and then that queue will be executed by a specific engine.

image alt text

Step 3. When done, simply save the queue and launch a query to test out the new configuration. Visit our documentation to learn more details about how to properly configure workload management queues and rules within Dremio.

Wrapping Up!

To learn more about Dremio visit our tutorials and resources, also if you would like to experiment with Dremio in your own virtual lab, go ahead and checkout Dremio University, and if you have any questions visit our community forums where we are all eager to help.

Ready to Get Started? Here Are Some Resources to Help

Whitepaper Thumb


Harness Snowflake Data’s Full Potential with Dremio

read more
Whitepaper Thumb


Simplifying Data Mesh for Self-Service Analytics on an Open Data Lakehouse

read more
Whitepaper Thumb


Dremio Upgrade Testing Framework

read more
get started

Get Started Free

No time limit - totally free - just the way you like it.

Sign Up Now
demo on demand

See Dremio in Action

Not ready to get started today? See the platform in action.

Watch Demo
talk expert

Talk to an Expert

Not sure where to start? Get your questions answered fast.

Contact Us

Ready to Get Started?

Bring your users closer to the data with organization-wide self-service analytics and lakehouse flexibility, scalability, and performance at a fraction of the cost. Run Dremio anywhere with self-managed software or Dremio Cloud.