In this excellent and informative Q&A session, Jason Nadeau - VP of Marketing at Dremio, explains what is happening what the IT community is doing to manage the challenges that comes with collecting massive amounts of valuable data.
This is a great takeaway:
“Here at Dremio, we’ve expanded upon the SQL engine concept to create a purpose-built data lake engine, which uses multiple highly-efficient query acceleration technologies to deliver the necessary interactive performance. For example, Dremio is the co-creator of Apache Arrow, Apache Gandiva, and Apache Arrow Flight, which are a set of columnar data processing and data exchange technologies. Our data lake engine is deeply architected around these technologies, and they deliver dramatic performance gains vs. traditional SQL engines.”
