
Whitepaper
2024 State of the Data Lakehouse
Benchmark your organization with Dremio's State of the Data Lakehouse Survey Report!
read moreOne of the many advantages of Dremio, is its deployment flexibility. You can deploy Dremio on any of your favorite cloud flavors, and also on Prem using different methods such as Yarn, Docker and Kubernetes. In this article I will walk through the steps of evaluating Dremio by deploying it through Kubernetes using MicroK8s on an Ubuntu Server environment.
NOTE: The steps on this tutorial are meant to be used only for the purpose of testing Dremio on a single-node Ubuntu MicroK8S environment.
Let’s get started!
If you already have an Ubuntu machine (or VM) on your environment, you can skip this step. Otherwise, let’s go through the following checklist:
Follow the standard Ubuntu install on your machine or favorite VM platform. In this case, I will use VirtualBox.
Install the SSH Server when prompted, to do this, select the “INSTALL OPENSSH SERVER” option and click “Done”
Next, from the list of options, select MicroK8s
Select “Done” and wait for the installation to finish, this will take just a few minutes depending on your bandwidth.
Once the installation is done, select “Reboot”
Now, let’s SSH into the server and enable storage, dns and helm using the following command
microk8s.enable dns storage helm
NOTE: it is possible that you will need to add “sudo” to the command to avoid permissions issues.
If everything runs correctly, you should see the following messages
Now we need to install Docker, to do so, simply run the following command
snap refresh
Then
snap install docker
Then let’s login to Docker Hub using the following command
docker login
Provide your user name and password when prompted
You should see the following message once the connection is created. If for some reason you get an access denied error, trying “sudo” before logging in.
Take note of the secret output location (config.json) from the command we just ran. In my case it is /root/snap/docker/423/.docker/config.json
Now, add a secret to kubectl running the following command
microk8s.kubectl create secret generic regcred \ --from-file=.dockerconfigjson=<path/to/.docker/config.json> \ --type=kubernetes.io/dockerconfigjson
If everything runs correctly, you should see the following result
Now, let’s download Dremio Cloud Tools, run the following command
sudo git clone [https://github.com/dremio/dremio-cloud-tools](https://github.com/dremio/dremio-cloud-tools.git)
Then, navigate to the templates directory
cd dremio-cloud-tools/charts/dremio/templates/
Then, edit the following files to add the imagePullSecrets:
Add the following lines under the containers section
imagePullSecrets: - name: regcred
Additionally, you can modify your values.yaml based on your required configuration. For more details take a look at the dremio-cloud-tools readme file.
NOTE: Dremio Enterprise Edition users, edit the image name to docker/docker-ee
Deploy Dremio using helm using the following command
microk8s.helm install --wait dremio --name dremio
NOTE: run this command from the /dremio-cloud-tools/charts directory
Once the deployment is completed, use the IP address + port 9047 to on your browser to navigate to the Dremio UI
These are the most common troubleshooting scenarios that you might encounter:
Timeout on helm - If you receive a timeout or a failure when doing an install from helm
microk8s.kubectl get pods
Pods will be listed, look for the master pod name
microk8s.kubectl describe pod ‘master-pod-name-here’
In the output look for the failure, typically on the last line under events
microk8s.kubectl get pvc
pvc will be listed, look for the master pvc name
microk8s.kubectl describe pvc ‘master-pvc-name-here’
In the output look for the failure, typically on the last line under events
Benchmark your organization with Dremio's State of the Data Lakehouse Survey Report!
read moreTDWI senior research director James Kobielus will engage data industry experts in an in-depth discussion of data integration trends and best practices
read moreWie bringt ein Data Lake House die Smart Factory auf ein neues Level?
read more