Dremio Jekyll

Introducing Elastic Engines

Introduction

Dremio AWS Edition supports the ability to provision multiple separate execution engines from a single Dremio coordinator node, start and stop based on predefined workload requirements at runtime. This provides several benefits, including:

  • Workloads are isolated within their own set of CPU, Memory & C3 resources and not impacted by other workloads
  • Time-sensitive yet resource intensive workloads (nightly jobs, reflections refreshes, etc.) can be provisioned with the appropriate amount of resources to complete on time, but remain cost effective by only running when required
  • Track cost by team by running workloads on their own resources
  • Right size execution resources for each distinct workload, instead of implementing a one size fits model
  • Easily experiment with different execution resource sizes, at any scale

In this article we walk you through the steps to provision and manage Elastic Engines, we also show you the steps to manage workloads using queues and rules.

How to Launch and Manage Elastic Engines?

Step 1. Elastic engines is one of the easiest features to use in Dremio. To provision a new engine, in the Dremio UI go to** Admin -> Elastic Engines**

image alt text

Step 2. You will see a default engine already deployed. Now click on Add New

image alt text

Step 3. The Set Up Engine popup page by default automatically selects the EC2 configuration options used to launch the engine (Security Group, EC2 Key Pair, etc). To continue the configuration, add a name, select the instance count and then click on Advanced Properties(optional)

image alt text

Step 4. Review the Advanced Properties menu. Here is where we will have the option to enable Auto Start and Stop.

  • When Auto Start is enabled: The engine will start as soon as it receives the first query.
  • When Auto Stop is enabled: The engine will stop after 5 minutes of inactivity.

Once you are ready selecting your options, click on Save & Launch

image alt text

Workload Management

Creating a Queue

Now, we need to tell Dremio which workloads we want it to process using any of the existing engines. To do so follow these steps:

Step 1. Navigate to the Queues option within the Workload Management **in the **Admin settings. Then click on Add New.

image alt text

Step 2. Give the new queue a name, then from the Engine Name drop-down menu select the engine that you just created (or the engine that you want to assign to this queue). Here you will also see options related to time, memory and CPU management. When done, click Save.

image alt text

Creating a Rule

Now, we need to create a rule where we will specify that if a workload (or query) meets certain criteria, it will be handled by a certain engine. To create a new rule follow these steps

Step 1. From the Admin menu, click on Rules and then Add New.

image alt text

IMPORTANT: The priority of the rules is defined by their order i.e rule 1 will be parsed first and can supersede any other rules that follow. To change the order of the rules, click on three bar icon next to its name and drag it to the desired position.

Step 2. Give the rule a name and conditions. You will see several condition examples that you can use i.e. User, Group Membership, Job Type, Query Plan Cost, or a combination of all. Once you add the condition, you have the option to reject or assign the workload to the queue that you created in the previous section.

This means that any workload that meets the condition requirements, will be assigned to a queue, and then that queue will be executed by a specific engine.

image alt text

Step 3. When done, simply save the queue and launch a query to test out the new configuration. Visit our documentation to learn more details about how to properly configure workload management queues and rules within Dremio.

Wrapping up!

As you can see, Dremio’s elastic engines allow you to use your cluster more efficiently in a fully automated way. Now each workload can make use of one or many coordinator nodes depending on the workload management settings that you configured, and once the workload is complete the nodes will shut down, thus you won’t have to pay for idle resources anymore. This will avoid over provisioning and over paying, and under-provisioning and performance compromise.

To learn more about Dremio visit our tutorials and resources, also if you would like to experiment with Dremio in your own virtual lab, go ahead and checkout Dremio University, and if you have any questions visit our community forums where we are all eager to help.