Introducing Elastic Engines - Dremio
Dremio AWS Edition supports the ability to provision multiple separate execution engines from a single Dremio coordinator node, start and stop based on predefined workload requirements at runtime. This provides several benefits, including:
In this article we walk you through the steps to provision and manage Elastic Engines, we also show you the steps to manage workloads using queues and rules.
Step 1. Elastic engines is one of the easiest features to use in Dremio. To provision a new engine, in the Dremio UI go to** Admin -> Elastic Engines**
Step 2. You will see a default engine already deployed. Now click on Add New
Step 3. The Set Up Engine popup page by default automatically selects the EC2 configuration options used to launch the engine (Security Group, EC2 Key Pair, etc). To continue the configuration, add a name, select the instance count and then click on Advanced Properties(optional)
Step 4. Review the Advanced Properties menu. Here is where we will have the option to enable Auto Start and Stop.
Once you are ready selecting your options, click on Save & Launch
Now, we need to tell Dremio which workloads we want it to process using any of the existing engines. To do so follow these steps:Step 1. Navigate to the Queues option within the Workload Management **in the **Admin settings. Then click on Add New.
Step 2. Give the new queue a name, then from the Engine Name drop-down menu select the engine that you just created (or the engine that you want to assign to this queue). Here you will also see options related to time, memory and CPU management. When done, click Save.
Now, we need to create a rule where we will specify that if a workload (or query) meets certain criteria, it will be handled by a certain engine. To create a new rule follow these steps
Step 1. From the Admin menu, click on Rules and then Add New.
IMPORTANT: The priority of the rules is defined by their order i.e rule 1 will be parsed first and can supersede any other rules that follow. To change the order of the rules, click on three bar icon next to its name and drag it to the desired position.
Step 2. Give the rule a name and conditions. You will see several condition examples that you can use i.e. User, Group Membership, Job Type, Query Plan Cost, or a combination of all. Once you add the condition, you have the option to reject or assign the workload to the queue that you created in the previous section.
This means that any workload that meets the condition requirements, will be assigned to a queue, and then that queue will be executed by a specific engine.
Step 3. When done, simply save the queue and launch a query to test out the new configuration. Visit our documentation to learn more details about how to properly configure workload management queues and rules within Dremio.
To learn more about Dremio visit our tutorials and resources, also if you would like to experiment with Dremio in your own virtual lab, go ahead and checkout Dremio University, and if you have any questions visit our community forums where we are all eager to help.
