Dremio Supports Moonfare’s High-Performance Culture with a High-Performance Lakehouse

Summary

Moonfare replaced a PostgreSQL-based data warehouse on Amazon Web Services (AWS) with an Dremio data lakehouse to offer data engineers, analysts and business users a high performance platform for business intelligence and predictive analytics empowering them to make better data-driven decisions.

The Business - Opening the Private Equity Market to Individual Investors

In contrast to public equity, private equity is a stake in any company that is not publicly traded on a stock exchange. The focus is not on making quick gains. Instead, private equity gives companies access to alternative funding for a longer strategic horizon while offering investors particularly high returns that are less correlated to public markets and are therefore an attractive way to diversify and mitigate the risk of a portfolio.

Until recently, access to private equity has been limited to institutional investors as investment minimums are usually in the tens of millions. Now, with Moonfare, individual investors and their advisors can access the private equity market with as little as €50,000*. It works by aggregating individual demand into a feeder fund structure, which then invests directly into the underlying target funds.

Founded in 2016, Moonfare is headquartered in Berlin, Germany, and operates in 24 countries. The company has offices in New York, Hong Kong, London, Zurich, Singapore, Paris and Luxembourg. Over 3,500 clients have invested more than €2 billion on its digital platform.

The Challenge – High-Speed Access to Insight at a Reduced Cost

Moonfare’s mission is to lead a new era for private equity investing and to open the door to higher returns for more people. To achieve this, the company has created a powerful platform offering a unique experience to their investor community. 99% of the technology and the complete infrastructure is cloud-based.

Moonfare launched their first fund in 2018. At that time, a PostgreSQL-based data warehouse on AWS was chosen as data platform. But with increasing data volumes and growing numbers of investors, it became a bottleneck, delaying access to vital information.

When looking for a replacement, Moonfare’s decision makers had the advantage that they did not have to worry about any legacy systems. They could start from scratch and look at the most 3 innovative technology on the market concentrating on best of breed solutions that could meet their exacting requirements regarding scalability, speed, functionality and costs.

The Solution – A Cloud Platform Ready for Growth

At first, Moonfare’s decision makers looked at alternative data warehouses and examined Snowflake and Firebolt, but both systems did not offer what they had in mind. Firebolt was lacking a connector to Tableau that was vital for their use cases and Snowflakes proved to be too inflexible. Its proprietary data format meant that Moonfare would have no control over their own data, resulting in a potential vendor lock-in.

While the data warehouse idea was discarded, the benefits of the new, open data lakehouse architecture immediately appealed to the evaluation team. This approach offered them the best of both worlds: the data management features and structures of a data warehouse combined with the flexibility, scale and cost-efficiency of a data lake.

As a customer of AWS, Moonfare initially evaluated Amazon S3 together with Amazon Athena, an interactive query service that enables SQL queries on S3 data. But in a proof-of-concept runoff Dremio put Athena out of contention. Dremio convinced Moonfare with its user-friendly interface, a variety of documentation features including data lineage graphs and table wikis, and – most importantly – its speed.

With query engine Sonar and data management service Arctic, the Dremio Cloud data lakehouse platform ticked all the boxes, offering

• Scalability
• An open architecture
• Super fast performance
• Role-based security and governance
• A highly functional user interface

Once the data lakehouse platform was chosen, Moonfare management onboarded dedicated experts to support and maintain it. “This may seem rather unorthodox,” says data engineer Angelo Slawik. “But that way, no HR time and money was wasted and Moonfare enlisted the right talent at the right moment while I was offered the fascinating opportunity to work with a new technology right from the start. We all are rather proud of the swift implementation. After the first contact with Dremio it took us only 3 months to get the first use cases up and running. Considering that we started from scratch, this is pretty good timing.”

In the new, Dremio-centered architecture the data lake in S3 is populated with data from three main sources. Custom Python scripts store data from external sources like Twitter or Facebook while user tracking data is synced via Segment. Data from MySQL databases that represent the multiple back-end-services of the Moonfare platform itself is streamed to S3 using AWS Database Migration Service (DMS).

In the data lakehouse, data is stored in Iceberg, Parquet and CSV formats. Here data engineers use Python or PySpark via AWS Glue to manipulate and prepare the data before staging.

Angelo Slawik: “Dremio’s view-centered architecture enables us to execute SQL-based data transformations without the need to write ETL jobs for each and every table. Furthermore, we can store our data in our formats in our own S3 buckets. We have complete control and are not dependent on a vendor.”

With Dremio, data analysts and skilled business users are now able to explore data at speed using either a tool of their choice or querying it directly. Normal end users can tap into their knowledge in the form of reports and dashboards, that are available for them in Tableau.

Nexts Step

So far, customers using the Moonfare site have to pick and chose their funds themselves. While experienced investors welcome this self-service, it presents an obstacle to those who are new or used to having their stocks and funds managed. Currently, Moonfare is looking into new service offerings to better assist these investors. Whatever the solution, the Dremio data lakehouse is ready to support it and future use cases, like providing the investment team with a faster, more transparent and structured process for fund analytics.