Our CEO and co-founder, Tomer Shiran, was asked to weigh in on the role of the data engineer in 2018 for a Datanami article. You can read more in Why 2018 Will Be The Year Of The Data Engineer.
One interesting section on the shortage of data engineers:
‘To run a successful data science project, it typically takes a ratio of one data engineer for every data scientist, Shiran says. “But even the hottest Silicon Valley companies are unable to achieve a one-to-two ratio,” he says. “You don’t have enough engineering talent out there. It’s very expensive.”
The complex technical nature of distributed data stores like Hadoop, Amazon S3, and Azure BLOB has increased the demand for data engineers because “by and large, really only the engineers have been able to get value out of the system,” Shiran says. (Dremio’s software was designed to enable analysts “be their own engineer,” he says.)’
