A growing number of companies realize the value of data and have some sort of data architecture to store, manage, make sense of, and utilize data for various business purposes. However, the monolithic data architectures that most companies rely on can be quite limiting. The concept of data mesh was devised in response to these challenges. Read our short guide to data mesh below, and download our latest eBook to understand the concept and benefits of this novel approach to data architectures!
Becoming a data-driven company is on the strategic agenda of most digitally mature companies today. Using data to inform strategic goals, product innovation, and various other areas of business operation is a key success factor: not only can it help tackle KPIs, it is also the primary way to future-proof a company operates in an increasingly competitive digital market landscape. Data helps optimize operational costs, drive personalization for a better user experience, and can also equip teams with insights that help build a more profitable business overall.
However, data can be hard to manage. Today’s data architectures that most businesses rely on can create scalability bottlenecks and technological debt, both of which render the company unable to fully realize the value of data.
Most enterprises rely on either of the two established types of data architectures to ingest, manage, and process data. Both data warehouses and second-generation data lakes have certain limitations that make them suboptimal solutions for a company aiming to future-proof its data-centric operations. These classic data architectures are monolithic, requiring the business to invest in creating and maintaining an overly complex central data system. Scalability is a common concern, as is the unwilling creation of silos between disconnected data sources and data consumers.
To help overcome these limitations of existing data architectures, the concept of data mesh was created by Zhamak Dehghani at Thoughtworks. In her book, released in April 2022 and titled Data Mesh: Delivering Data-Driven Value at Scale, Zhamak guides readers on their journey from monolithic big data architectures through the paradigm change that building a distributed data architecture requires.
In short, data mesh is a new enterprise data architecture concept founded on domain-driven design principles. Companies adopting the data mesh approach are engaged in creating a data architecture that mirrors the dispersed data environment that exists in their businesses. Rather than building and maintaining a monolithic, centralized data architecture, they adopt a distributed way of thinking where each business domain is responsible for treating data as a product and owning valuable data products. They offer up these data products in a self-serving system for data customers across the organization, with the aim to deliver insights that directly translate to domain-specific business benefits. Overall, data mesh helps decentralize and democratize data across the enterprise.
The data mesh approach is founded on four key principles. Diving into each of these will help understand the basics of the data mesh concept – let’s tackle them one by one!
Data mesh decentralizes data management, also distributing the responsibility of managing the data architecture across various business domains. This way, teams own domain-relevant data sets, preserving the context of data as they offer it up to other stakeholders. In a data architecture based on the data mesh concept, each domain is responsible for hosting and serving data that originates in that specific business domain.
In a decentralized data mesh, each team (domain) owning its respective dataset is responsible for offering up that data to other stakeholders in the business. The creation of high-quality and easy-to-consume data products helps democratize data across the company, unlocking meaningful business insights.
As data mesh also distributes the management of the data architecture, accessibility is a key requirement. In a data mesh environment, it should be easy to create and maintain data products without hyper-specialized data engineering knowledge. Tools like Google’s Dataplex help ensure accessibility for all domain data owners.
While the data architecture itself is distributed, maintaining data quality is a crucial concern so that integrity is maintained. That’s why data mesh requires practitioners to adopt a central, shared data governance model. These shared quality rules help ensure interoperability across data products.
Through the decentralization and democratization of data, data mesh unlocks a range of important benefits for the business, including:
Interested in learning more about the concept of data mesh and putting it into practice? Download our eBook for key insights, including an introduction to Dataplex and a case study of Delivery Hero, an innovative company that leveraged data mesh to accelerate its transition to data-driven operation!
