Data Mesh in Practice

Video Synopsis

• How Was Data Mesh Approached at Shell
• Organizational Aspect of Data Mesh at Shell
• Challenges Overcome at Shell
• Other Challenges While Implementing Data Mesh
• Data Mesh Issues With Agile Lab
• Getting Started With Data Mesh
• Key Capabilities Customers Want With Building Data Mesh
• Difficulty of Operating Data Mesh at Shell
• Agile Labs Experience With Data Mesh
• Advice For Building Data Mesh

Transcript

Note: This transcript was created using speech recognition software. While it has been reviewed by human transcribers, it may contain errors.

Ben Hudson:

Thank you everyone for joining us for day two of Subsurface Live 2023. We’ve got another day of fun-filled talks lined up for you all today. And to kick it off, we’re very excited to have a panel to talk about data mesh. So why are we here? Data mesh is a very hot topic in the data ecosystem, but one of the questions we hear most from practitioners, anyone undergoing data mesh or interest in data mesh is, how do I implement data mesh? How do I go, about, on this journey to build a data mesh at my organization? So to answer those questions today, we’ve assembled a star studded panel to talk through that, and I’d like to introduce the first panelist, Zhamak Dehghani, CEO and founder of Nextdata. Thank you so much for joining us.

Zhamak Dehghani:

Great to be here.

Ben Hudson:

All right. Our second panelist, I’d like to introduce Raja Peru, who is an architect and technical manager at Shell.Thanks for joining us, Raja.

Zhamak Dehghani:

Good to see you Raja.

Ben Hudson:

And last, but actually not last, but not least I’d like to introduce Tomer Shiran, who is the founder of Dremio. And last but not least, joining us all the way from Italy. So happy to have him here is Ugo Ciracì, who is engineering leader and business unit leader at Agile Labs. So everyone give a big round of applause so you can hear us from across the internet. Ugo. All right.

Ugo Ciracì:

Hi.

Ben Hudson:

Hey.

Ugo Ciracì:

Hi everyone.

Ben Hudson:

So we have a range of expertise on the panel today. We’ve got founders, we’ve got builders, we’ve got consultants here who advise companies on their data mesh journey. We’re very fortunate to have a range of expertise of people focused on different areas. So let’s get to know everyone. So each of the founders or each of the panelists here, I’d like to ask you to tell us about who you are, what you do, and how you’re related to the data mesh conversation. Zhamak?

Zhamak Dehghani:

All right. I’m Zhamak. I guess my official title is a troublemaker. I created the concept of data mesh. I spent a few years implementing it with customers. And at the moment, I’m trying to build a technology to serve the developers with their experience of building data mesh. I’m the founder, CEO of Nextdata.

Ben Hudson:

Awesome. Thanks for Zhamak. Raja?

Raja Perumalsamy:

Hi, I’m Raja. I’m a technical manager and architect at Shell and I have like 20 years of experience working on data versioning and data related projects. And in our recent journey in Shell, we have worked on the data mesh for our project. So I’ll be discussing that here today.

Ben Hudson:

Tomer?

Tomer Shiran:

Hey, Tomer, I’m the founder and chief product officer of Dremio. And if you’re at this conference, you’re probably familiar at least, to some extent with Dremio. We’re a data lakehouse platform and helping hundreds of enterprises and other companies on their data mesh journeys.

How Was Data Mesh Approached at Shell

Ben Hudson:

Thanks. Tomer. Panel is about data mesh and practice talking about challenges to overcome, how to approach building a data mesh organization. And we’re very fortunate to have Raja from Shell who is undergoing this journey today. So, to kick us off on this data mesh and practice panel, Raja, I’d love to ask you why you undertook this journey and how you approached it at Shell and where you are as well.

Raja Perumalsamy:

Yeah. So hi. So for us, we started with this project. When we initially started it was, we didn’t really think of it like a data mesh, right? So there, we started with a, there is this recommendation. And our, we had a Shell energy team who needed forecasting data and they tried to get this data from different varieties of sources, and they had their own account, everything. And we are. Our team is a mission learning team. So load for the forecasting team where we provide, enable them with machine learning, load forecasting, and provide the results. When we started the journey, “Okay, how do we get this data from these various sources?” Then the first step is to write, “Hey, do I want to write all this CTL bring in?” Right? And then again, the ownership of the data. Who is going to own this data?

And how are we going to get it here? Is this the right approach? Right? And that is where we started the journey with Dremio too, right? So when we explored them, we started exploring it as a lakehouse platform, right? So then it was easy for us to onboard our Shell Energy team, like bringing in different data sources. Like they have SQL Server, MongoDB, and they even have their own AWS account, S3 data, all that stuff. Now it was very easy for us to onboard them into Dremio and then we were able to access this data from them. This is where our journey started. And then currently where we are at, is we are, like, towards almost there. I mean, I shouldn’t say that. We have a mini data mesh. Maybe I’ll put it in that way. So we have this data, which is completely controlled by solar energy team. And we use that data, we add machine learning to our thing, and then we share back the forecast back to them. So, with this domain, we share back the data to them. So this is where we stand, actually. So I can go into more details, but maybe that’s good.

Organizational Aspect of Data Mesh at Shell

Ben Hudson:

That’s a great start. Thanks so much for giving us the context and where you are at the data mesh journey right now. I know a lot of folks don’t go from zero to 100 all in one go, right? Raja, from what you described, there is a, some sort of mesh like architecture that you need to achieve along the way. And I’m going to ask Zhamak and Ugo and tell her about this in a bit. So the one thing I wanted to ask about before we move forward is, you’ve talked a lot about the technical underpinnings of why you wanted to undergo the data mesh journey. You have a lot of data in different places, and you want to bring that all together and help teams onboard in a self-service manner. So this resonates with the self-service data platform principle of the data mesh post from Zhamak. That’s, I think, technical underpinnings is just one side of the data mesh coin. Data mesh also talks about organizational structure and enabling that side as well. So before we get to the rest of the panelists here, can you touch upon the organizational aspect of your journey at Shell as well? So you’ve done the technical side. Did you have to convince someone to undergo this journey as well as self-service at Shell?

Raja Perumalsamy:

No. Actually at least the self-service part was the good thing about it, right? So honestly, to say, that was a quick win for us. So in the way we used Dremio for bringing in the sources, right? And it’s big and all our data was SQL oriented, right? And we have this big data in SQL and users were able to leverage Dremio for running all these queries and stuff. Couple of points there, right? So when we brought this data, it was very easy for us to onboard non-technical users. That was a good thing about the whole journey. So we had business analysts who now they can go in, they can write the queries and take a look at the data lineage in Dremio and then find out, “Hey, where is this data coming from? Where is that data coming from? “And when our team wants some data, we don’t have to worry about how to get this data and all this stuff, we can go back and talk to them saying, “I need this data.” They are the experts in that data ownership. So it’s very easy for them to figure out where this data is coming, getting that and for us to use.

Challenges Overcome at Shell

Ben Hudson:

Yeah. Thanks. So there’s a lot of elements of the self-serve experience with data products for your data teams at Shell. So what are a couple of challenges that you had to overcome in order to get to where you are right now?

Raja Perumalsamy:

Yeah, the challenges, right? One of the key challenges, I think many people will face this too. So because as I said, like, we have this, the data is provided by another team, right? So, any issue with the data, right? Then the cycle of it, right? To make sure we go back to them and they have to fix the data coming back to us. That takes some time. And also the standardization. Because like when we talk to different teams, each team might try to do their own way of things, right? And we need to ensure that that is a standard across here and then making sure we are able to get data in a standard way. The other thing is the accuracy and timeliness of the data. So when I need to run the job today, “Okay, do I have all the data that I need and do I get it in a timely way and in an accurate way.”

So those are the things, and governance is another aspect to it, right? So when we talk about governance here, like with Dremio, we were able to achieve most of the things like we have this find and access control. So we are able to set this access control at dataset level, all these things. But still because it is a distributed architecture, like, so again, we have, so sometimes when people move from centralized to a distributed way of things, right? They, we don’t know if the right governance is applied, and is it handled properly? So that is another challenge I see coming up there.

Other Challenges While Implementing Data Mesh

Ben Hudson:

Awesome. Thanks for outlining those challenges. There’s challenges all around the computational governance perspective and that principle of data mesh. There’s the data lineage and also the standardization of data as well, and how to enforce those uniform access controls and rules as you have different data sets. So that brings me to the rest of the panelists, Raja talked about some of the challenges that they have faced at Shell in practice, boots on the ground, implementing and building data mesh at Shell, and there are some specific challenges… Sounds like a lot around the technology and the governance side that you’ve had to overcome to get to this first step at Shell. So going from a monolithic to distributed architecture and making sure you approach that in a safe and governed manner. So Zhamak and Ugo. So starting with Zhamak, how does Raja’s challenges and that experience compare to the companies that you work with? And what are, if they’re similar or if they’re different, what are the other challenges that companies need to be aware of when implementing data mesh?

Zhamak Dehghani:

Yeah, I mean the, what’s. I was smiling a little bit. From a place of empathy, of course. When he was mentioning standardization on the issue of the data. So what data mesh, just going back to the origin of data mesh, why this matters at all. We are moving toward decentralization of data because we have to act on data at the speed of the market. With automation, automated decision making, AI and ML needs to be as close as possible to the business. So this idea that we get the data from different places, we put it in one place, nice, and we have a data team governing it, and then serving that data to other people creates this long lead time from where the data is to where the value is. And in the middle of it, we have put a bottleneck, right?

So that, and we’ve seen this before, I think the reason data mesh as a concept was so overwhelmingly received, was because we had seen this before, early days of digitization, moving to different teams, building applications, and creating these digital touchpoints. We had no choice but to break down the monolith and giving responsibility of different services to different teams. So I think that’s the foundation of it. That’s what this nirvana we’re trying to get to. The challenges going from this very monolithic, centralized thinking about the data, the domain people that actually know the data best often are oblivious to even the data that needs to be provided and shared and bring the business domains along. So that human factor of now you are responsible for your data, you’re using it for yourself, for your business objectives, and also you’re sharing it with the rest of the organization.

That’s quite a mindset to change. Cause before they were just expecting that somebody else down the stream will take care of the data. So, I think the mindset change, and with any mindset change often we say the words. It’s easy that we’re doing data mesh, we have a data product, but shifting and really changing the behavior, that’s really hard. And that’s where you need technology to help you reshape behavior, right? So, I think the top challenges that I see and we aren’t focusing on in terms of enablement organizationally is getting business teams to come along with that distributed ownership and accept that distributed ownership from the technology perspective, how do we enable them? What is this? What does the codification of data products look like? What is this long standing thing that we’re going to maintain?

So that creating a standard or common way of creating data products, sharing data products, and how do we make sure that these common standards or APIs for accessing and using data products for analytical use case, they are governed, right? So have your cake, decentralized ownership so you can move faster, and can make decisions closer to the business. Have those data products discovered and used in a common way, eat it too and don’t get fat. Have governance on top of it as well, right? Build things. So these are like the top three problems that we see. And it is part of it, those reorganizational parts of a technology and enablements.

Data Mesh Issues With Agile Lab

Ben Hudson:

Zhamak. Thank you. Ugo do you want to add some thoughts on the Agile Lab side?

Ugo Ciracì:

Yeah, I mean, very interesting to hear that talking about organizational issue. When talking about standardization, actually the teachings are more connected than we realize and reflect the organization, including the technical implementation. And this is also in the context of like one of the top utilities worldwide, the energy industry. It’s a big organization solution, structure center. You can handle business line, global functions and the regional meets together several ways in that case, bigger is going to different technologies and of course the organization and the way they work will be different for each unit. Like working in Agile, with working in Waterfall. Let’s say the so-called maturity is not even normalized across all those big enterprises. So it’s very difficult to find a standard in this situation. The way you can break the way for standards is of course data mesh. As a big opportunity to get into these problems and solve this problem.

And finding a standard is of course an organizational issue because you put together and make them agree about some standards, very different contexts and environments. This is a nice point. Other around organizations are of course politics because anyone that can have an area influence where to decide which kind of technologies can be part of the platform implementation of a data mesh. Of course, they want to be involved, but they want to influence the decision to, I don’t know, make, say some honored technologies or say some different criteria than the business enablement that can come from the development of the data mesh platform. Regardless of the technology and other things I can mention. You are of course at the level of the organization at the different model modalities, I was mentioning that.

And of course we are building a platform team that is going to develop the entire platform. This platform team will not be independent from other teams or other departments or even areas of the company. Just to mention some examples, IT infrastructure security and cybersecurity or whatever cross carting concerns that are usually silver in some of the company departments, they will influence and they will be work as dependencies for a platform team to work. All this is a bigger organizational issue that must be addressed somehow. And I mean our experience and my experience is that there is no single pattern that works. Linking many different departments to make a platform team work well is very difficult and challenging. You must address every single channel in a very careful way.

Getting Started With Data Mesh

Ben Hudson:

Thanks very much. So I think you’ve touched upon a lot of different things from technology, but mostly a lot of your response there was centered around the organization, right? So, and also your note about everyone, there’s no single view or no single right answer to implementing data mesh. So Raja’s got their own view or own implementation of data mesh. Ugo, I’m sure the customers that you’ve worked with all have their own shape or form for data mesh, right? Technically it fits within the bounds of data mesh. It looks like a mesh-like thing. So that differs from what’s in the text, right? So my question to each of the panelists here is, in your experience, how does, what does this mesh like first step look like for anyone looking to achieve a full data mesh vision? What is the core set of competencies from a technical side that you need to overcome and you need to achieve if you want to get started on this journey? So maybe Zhamak.

Zhamak Dehghani:

Do you want me to start?

Ben Hudson:

Yeah.

Zhamak Dehghani:

Okay. So to your point, I’m really bad at answering straight questions, sorry. Maybe I will just go back a little bit. Wind back. You mentioned that the incarnation of, I guess, data mesh different organizations looks different, and that was intentional. So when I put the concept out there, I articulated the problem and then the principles behind a solution, there was never a particular reference implementation to say, “That’s a north star, that’s what it looks like.” And that was intentional to invite people to come up with different practices, different technologies, different ways of implementing it. So that’s why we see this diversity. And I think like any new concept, we go through a hockey stick of adoption, things would look ugly and maybe hard to implement.

And then we converge, we diverge, and then we converge, and then we get those efforts somewhat converge and we standardize certain things. So we don’t reinvent the wheel in every single implementation of data mesh, and we harmonize in some places and we diversify in others and that life would get better. So that, again, that’s why we see that diversification, we’re still in the bottom of that hockey stick, in my opinion. To answer your question now, okay, if you want to get started, what does that first step look like? I would say that you really need to think about this as an end twin thin slice. The thing about the pillars, if they’re nice, because cognitively help us unpack a problem to say, “Okay, domain ownership, data is a product .Self platform computational governance.”

But the challenge with those is that people say, “Okay, I can’t do the domain ownership. We’ll do the data product later and competition, governance, oh, I don’t even understand what that means. So we will do that some other time.” And the challenge with that is that you don’t really get that in, the impact you are looking for, which is and the little mistakes you should be looking for, which is how long does it take for someone, for a data analyst, for a business that has a analytical question or they want to apply a machine learning? How long does it take from that hypothesis, from that question to get access to data to apply that data into the business? So if you are not doing all those pillars, perhaps a thin slice of it, I’m not sure if you’re moving that fast.

We might be moving fast, we might, but we might be breaking things because we didn’t put policy in place. So my observation and how we, at least in the past, I’ve implemented the data mesh’s, it was pick a thin slice end to end and implement a hello world version of domain ownership, a data as a product. So, and start with business use cases. Start with some of the use cases that require access to analytical data to one or two domains. Start with one or two domains, walk backward, and then identify those domains, identify the data as a product that needs to be shared, and then build enough of the platform, enough of those technology pieces that creates that from use case accessing the data products and applying it into the business. And just build enough of that.

So do not go, I guess pillar by pillar, go thin across and broad. And when you do that, then the type of type of technology solutions you need to have in place are around decide what does part data product mean to you? What does it look like, right? When you touch it like this? Is this a service with APIs? This is a file and a disc. Is this a view? Everyone has a different, I guess, interpretation. Get an agreement on that and enable access to it. Build just enough of the policy, like the very first important policies, access control, right? Retention and quality metrics and so on. They will come back, they will come later. And then just thin slice, end to end, rinse and repeat to expand your mesh and go deeper in terms of technical capabilities.

Ben Hudson:

So start with a thin slice that encompasses all four principles and pillars, and then you can go deeper if you’d like. Let’s get to that Hello, world state first and build upon.

Zhamak Dehghani:

Exactly. And get use, use existing real business use cases, real people, real domains as a vehicle for execution of those thin slices. Do not build something in abstraction, or in isolation. What I’ve seen a lot of companies do, they say, okay, we shall go and build the platform and they will come and use this platform. And you go and spend years of millions of dollars building a platform that nobody wants. I mean, I absolutely ever agree with Uber that this, even the idea of a single platform is flawed because you, when you go to organizations half of the organization on Snowflake, the other half on something else, and you never get people, humans agree. So find the absolute minimum that we all have to agree upon, right?

If I go back to having seen this in, I guess in my career before, when we went from monolithic applications to microservices, the reason that technology movement was successful was a very basic minimal thing that we agreed upon, which was to use the web for intercommunication, and that was it. We use HCTP and that liberated us to use any technology for implementation we wanted. So again, when you are starting, don’t start, go big, build a big platform and then they will come and use it. No, let’s find out what’s the absolute minimum to connect those seams between the data products, build that and use those use cases for execution.

Tomer Shiran:

Yeah. I think though that the. When you get started. You’re right. Getting started with one specific use case or a few specific use cases is important because if you don’t have quick wins, then ultimately people get tired of management. gets tired of the initiative. Why are we spending all this money? But I do think it’s important to make technology decisions that are future-proofing you, right? That you’re not going to get locked in. And at this conference we’ve been talking a lot about open data architecture and things like Apache iceberg as what’s looking now, like the new standard way of representing data where you have different, many different systems whether it’s Dremio or Snowflake or the AWS stack or the Google stack or so forth all supporting that same standard.

So I do think you, if you make the choices that allow you to then scale up in the future, cause the data mesh isn’t about one group at the end of the day, right? It’s all about building something that scales throughout the organization, right? And so making technology choices, ideally that will allow you to scale that to the rest of the organization. And I think also because it’s such a strategic platform that you’re building a strategic architecture, something that you know will be okay in the future down the road, right? Where potentially some of the things you’ll use to process data don’t even exist today. It’s a startup that is just being created today or will be created next year, right? You want to make sure that that’s going to be something you’ll be able to use.

And so yeah, in my opinion, making those kinds of choices such as Iceberg, such as Parquet, right? Really standards that are supported broadly and will be supported for many years to come, I think that’s really important versus “Okay, choosing something whether it’s a specific data warehouse,” right? That today’s popular tomorrow won’t be popular, right? You know, this isn’t like developers choosing MongoDB for this app and Postgres for that app and MySQL for that app, which is okay, right? If you’re just building LTP apps, every developer can do whatever they want and you’ll be fine, right? But this is a little bit different, I think.

Zhamak Dehghani:

Okay. Plus a hundred to that one. Data mesh is about interoperability and doing AI and analytics in a distributed way and you start with decisions that are bound to an organization let’s say. But very quickly the data doesn’t like even the simplest and today modern data driven organizations. When you look at their data landscape, and I’m sure Shell would be the same as well, data doesn’t come even from the same organization, it comes from across the boundaries of organizations. So I think data mesh is just a stepping stone toward this very collaborative ecosystem of responsible data sharing for analytics and ml. So I absolutely agree that decisions, if we want to make, it can’t be, oh, as long as you are on this one vendor and technology, it will work. Otherwise you’re a mesh break. So I think we’re just putting the building blocks in place to have this global data sharing the same way that if I build the application today, I use APIs from many different organizations that don’t even belong to me, right? Through APIs and through those API, basis standardization, I think the same. We can learn similar lessons from the past with regard to data mesh.

Key Capabilities Customers Want With Building Data Mesh

Ben Hudson:

So Tomer, you bring a good point about technology. A lot of the discussion so far has been organizational challenges with approaching data mesh, but Tomer, you bring a spotlight onto the technology side and talking about future proofing from the technology side. When you talk with customers, what are some of the key capabilities that they’re looking for as they’re trying to build data mesh as they relate to things like self-service and governance?

Tomer Shiran:

Yeah, you’re right. I mean, at the end of the day, data mesh is, it’s not just technology. It’s not just organizational, right? It’s, it’s both, right? In fact I was having many conversations yesterday here in San Francisco at Subsurface about companies and fortune 10 companies having challenges around the organization, and how do you make sure that the different businesses really take ownership of the quality of their data, right? and so there’s only so much you can do as a platform if you’re not able to convince the organization to act that way. But there are also very important requirements when it comes to the platform which we talked about, some yesterday at the keynote. So if you haven’t watched that, definitely recommend that.

But at the end of the day, the platform has to provide that ability for people to operate and have domain ownership and manage data as a product. And for example, the things we’ve done at Dremio, right? Being able to connect to a variety of different data sources and all the different BI tools and being able to create a semantic layer in the middle, which makes it much simpler to expose the right data to the right user with all the permissions, row level/column level permissions, all those kinds of things. You know, the organization has to take advantage of these technologies and apply them because permissions don’t set themselves. Then somebody has to decide what is sensitive data and what’s not sensitive data and who’s allowed to see that sensitive data.

That’s a business policy decision. We’re very excited about the work around Dremio architect and Project Nessi bringing the ideas of GitHub to data, right? Because when you think about data as a product, how do we build products today when it comes to software and source code, right? We don’t, we don’t just put source code in some file repository in some NetApp filer and say, “Oh, everybody can collaborate on that source code,” right? We use things like GitHub where we have the ability for people to work in isolation with branches and we have the ability to recover very easily from mistakes. And we have the ability to go back in time and see who changed what source code and when, right? All these things have allowed much more agility, much more much much better hygiene and governance around source code.

And all of that really applies to data as well, right? We think about data and data engineering work, whether it’s the ingestion, the, the data quality, the data governance, you need all those capabilities. And so that’s the reason that we created Dremio Arctic to allow that and to allow that in a very open way. Again, going back to the idea of data mesh, it’s an organizational wide thing, and some people are going to maybe use Dremio for sequel, but other people will use Flink for streaming or spark for batch processing. And they should all be able to see the same, same tables, same version control, the same branching capabilities that should span all of that, right? And so that’s, that’s the reason we created that. And iceberg fits very nicely into that because it’s a standard table format that can be plugged into all these different technologies.

Difficulty of Operating Data Mesh at Shell

Ben Hudson:

Thanks, Tomer. I’m excited about Arctic. It brings a whole new meaning when you think about data as a product, right? So thinking about software products and bringing that to the paradigm of data, right? Adding something else you can think about when you’re going through the data mesh journey and thinking about what does it mean for data to be a product? So Tomer, you’ve shared your views on technology and technology is absolutely essential in order to enable data mesh, right? So technology needs to enable teams to undergo data mesh. There’s the organization that you have to convince to go down this journey from a structural perspective. But I think a third component of data mesh really is the people and the teams that are part of the domains, people who are running the self-service infrastructure of data mesh. So, Raja, you’ve talked about the technology perspective, talked about some of the challenge that you’ve talked about organizational as well and where you are right now, but how much of a challenge is enabling the actual teams at Shell to get to this mindset and to operate this technological foundation and this platform for data mesh? Can you say a couple words about that?

Raja Perumalsamy:

Yeah. There, this is a very good point. Surprisingly there is an open mindset now because, right? So even in Shell, our top leadership, there is a shift in the way we are approaching. So earlier we were in a platform team, so yesterday I was talking about the PDC team. We are a platform team now. We are migrating to a different way of working, which is called product line and product based teams. So now what this product based team is actually a combination of business users and technical uses. So we are shifting to that pattern, like rather than just trying to be in the platform now we focus on the product from the business perspective and from technical perspective. So with this, it’s really going to help us to speed up this journey and then make sure, “Okay, as a product, I can really focus on making sure I have the right technology in place, right processes in place, right? Right. Governance, everything in place, and then make sure this product is self-sufficient, right? So it can be self-served.

Ben Hudson:

Yeah. Cool. Thanks. I like that structural change going from that platform team to now you have domain teams that combine business and technical users all working off that centralized data platform.

Raja Perumalsamy:

So it’s not going to be centralized. That’s something I want to correct here. So one of the journey. When we started it was that whole monolithic, centralized thing. Like we really want to get away from that, right? So the whole distributed way and then each team is responsible for their own technology choice, the way they maintain the data. Again, still it has to be an open standard, but still the, each team will have their flexibility to meet in whatever they want, from data perspective to technology perspective to the product. So they define what it is.

Ben Hudson:

So going from that change at Shell, how big of an ask was that to make this structural change? So bringing it back to the organization, you had to go through this transition. How big of a lift was that? Did you?

Raja Perumalsamy:

No, we are still in the process. So it does not happen yet. So, but what I’m saying is, there is a shift happening everywhere. So that’s what I was talking about, so, or at least from my perspective.

Agile Labs Experience With Data Mesh

Ben Hudson:

Got it. Okay. So you’re still undergoing this change. Yeah. Ugo. I want to ask you a similar question, right? With the customers you’ve worked with, there’s the platform technology, but as Agile Lab, you do a lot of consulting and making data mesh possible, especially in the European region. So how much of a lift is the technology enablement for these domain teams as they make a shift to this data mesh mindset?

Ugo Ciracì:

Well, the technology plays an important role, but I mean, I want to bring a different perspective on what technology can enable. Because actually what is more important is to preserve practices. I believe. I mean, datalakes is like launching many data engineering practices to develop practices, such practices, and even a framework to make them work very well in an integrated manner across different partners, different cultures, in same, this is very different. So technology is good. If a technology can reinforce a practice that is good by itself. If you can, you have to achieve a point where you can keep the practice and replace the technology. Because the practice is stronger than the technology. And until you get to this very important point you ought to reach a certain level of maturity. You may arrive at this point, and, but this is the right point to achieve.

If you want to survive a big techno, technological changes or bad events like companies failing or whatever is a disrupting technology that is entering the market, you want to survive and adopt the new technology to don’t, don’t lose the, don’t lose the, the, the, the weight. So in this sense, open standards and technologies, the work on open standards would enabler cause they can survive to specific technologies, give the opportunity the company to make a long-standing decision, long-term decision, and all the vendor locking features that maybe they can solve a specific problem, they can enable specific capabilities that moment, they will make you rely on that feature forever and maybe that you are going to rely too much on that feature. and this will bring you in the so-called lock that it’s actually a big issue when you want to get away from that technology. So I mean, my, let’s say recommendation is to go for practices, understand which are the upper standards that make your practices work, and then choose the technologies on the basis of those practices.

Ben Hudson:

Got it. Thanks Ugo. So start off with open standards, then get onto that technology that will enable that and make that a reality.

Tomer Shiran:

By the way, I think a lot of the, a lot of the things we’re talking about here are things that also exist in the world of software engineering, right? When you think about all the, all the responsibility that developers and developer teams are taking on themselves, right? These days may be different from 20 years ago around doing their own testing, right? And having all the testing be automated. I know Dremio, for example when we think about our product and our Dremio cloud so much automation goes into that, right? Where each team that’s building a microservice is responsible for making sure that’s well tested they’re responsible for being on call and when something goes wrong, they take ownership of that. It’s not one isolated, centralized team that’s responsible for these things.

As it was maybe a couple decades ago when I was working at Microsoft, things were very different, right? It was a completely different team that was doing the testing and operations and things like that. So I think in some ways data mesh is bringing these same principles to the world of data, right? The, the data team is the, the business team that owns the data is responsible for testing it and checking that things are still okay and the pipeline is still working. And versus the centralized team that might not, not, might not even understand that data certainly not, not as well. And so that’s I think a lot of the same kind of similarities we’re seeing and a lot of this is common sense, right? It’s how we have to solve this tension that we have between centralized teams demanding rightful governance and security and business teams wanting to move fast and be agile. And so bringing these principles is a lot of common sense and [inaudible] thank you for making this top of mind for most companies in the world now. It, it’s made, made things a lot easier for you know for, for companies and for, for technology providers as well.

Zhamak Dehghani:

Yeah. And thank you for supporting it. I think if I had a magic wand and I could wish for one thing it would’ve actually been next time, maybe in five years, when we come to these sort of conferences, it’s not a data conference and then there’s an app software conference. It is just digital technology conference and embedded software building, software and applications or building AI and ML. This is all just one thing. These are not separate bifurcations of branches of technology and data mesh indeed attempted to do that. So creating these ex developer experiences that are, they resonate with an average or a typical. And we don’t look for a specialization anymore, and everyone can work with data or build apps, or run AI and ml. It’s just the same technology landscape that we work with and similar experiences we’re creating.

Advice For Building Data Mesh

Ben Hudson:

I hope we get to that vision in five years at the digital technology conference. So we have two minutes left, folks. Thank you so much for all your insights here. I want to end this panel with a mini lightning round. So for the audience tuning in across the globe, each panelist, what is one piece of advice you would have to anyone wanting to build a data mesh? Could be anything where you start, what you look for, what is your piece of advice Ugo?

Ugo Ciracì:

I will say to don’t trivialize data initiative with quick wins that don’t bring to any results and go for to focus on entry point that can really leverage the initiative look for people that wants to change, and for people that needs to change, that need to change and work on the minimal viable data platform that can deliver the minimal viable data. 

Ben Hudson:

Thank you. Tomer.

Tomer Shiran:

Sign up for Dremio Cloud in five seconds. No, I think I agree with you. Get started with, with a project, something that you can demonstrate a quick win. I think that’s always the best the best way to get started and allows you to do a lot more afterwards,

Ben Hudson:

Quick wins and change agents. Raja?

Raja Perumalsamy:

For me, I would say from my perspective, it is the business users. So like, try to involve the business users, try to see what is their quick value, they want to look at it, right? Try to build that quick and then share it, right? In our case, what we did is we brought in these data sources quickly and enabled the users and we shared the data across. That was a quick win, which enabled us more in this journey. So I would recommend involving business users and getting the key point and going in the,

Ben Hudson:

So bring everything back to the business. Zhamak

Zhamak Dehghani:

I agree with all those points. Maybe I’ll just add something to those. Think about the largest population of technologies that you have to mobilize to build, use, share data, products, and those are developers. So enabling and empowering developers with technologies that is native to them to use. Its key. and, and come and talk to us if you’re interested in doing that.

Ben Hudson:

Awesome. Well, everyone, thank you so much for taking the time to share your expertise with the world here. Anyone tuning in we hope you have a great rest of the conference. Have a great day, and please feel free to reach out to any of our panelists if you want to learn more about data mesh. Thank you.

Zhamak Dehghani:

Thank you.

Ben Hudson:

Thank you.