If you prefer videos over written text, here’s a recording of a presentation of this content In this article, we’ll go through: ✅ What Iceberg is ❌ What Iceberg is not – A table format specification- A set of APIs and libraries for engines to interact with tables following that specification – A storage engine- An […]
AWS Glue Architecture AWS Management Console Defines AWS Glue objects such as crawlers, jobs, tables, and connections Sets up a layout for crawlers to work Designs events and timetables for job triggers Searches and filters AWS Glue objects Edits scripts for transformation scenarios AWS Glue Data Catalog AWS Glue Data Catalog provides centralized uniform metadata […]
A cloud data lake is a cloud-hosted centralized repository that allows you to store all your structured and unstructured data at any scale, typically using an object store such as Amazon S3 or Microsoft Azure Data Lake Storage (ADLS). Its placement in the cloud means it can be interacted with as needed, whether it’s for […]
Apache Arrow combines the benefits of columnar data structures with in-memory computing. It delivers the performance benefits of these modern techniques while also providing the flexibility of complex data and dynamic schemas. And it does all of this in an open source and standardized way. Over the past few decades, databases and data analysis have […]
Data is at the center of every business today. Companies use data to answer questions about the business, such as: In most organizations, many different systems generate data, and each system typically uses different technology and has a distinct owner within the organization. For example, consider data about customers: Together, this data provides a comprehensive […]
At its simplest, data warehouse is a system used for storing and reporting on data. The data typically originates in multiple systems, then it is moved into the data warehouse for long-term storage and analysis. This storage is structured such that users from many divisions or departments within an organization can access and analyze the […]
Businesses generate massive amounts of data that must be analyzed in order to derive business value. Unfortunately, analyzing data within the systems where the data is created is not ideal for the following reasons: These two systems (data generating vs. data analyzing) are often called OLTP (online transaction processing) and OLAP (online analytical processing). Here’s […]
ETL stands for extract, transform, and load, and ETL tools move data between systems. If ETL were for people instead of data, it would be akin to public and private transportation. Companies use ETL to safely and reliably move their data from one system to another. ETL was created because data usually serves multiple purposes. […]
