Title Image

Blog

Building a data mesh with Collibra and Google Cloud’s Dataplex By Koen Van Duyse

  |   RSS Feeds

The explosion of data across organizations has led to a wide variety of ways to store, catalog and use that data. One common organizational theme is a central data lake, which is useful for centrally maintaining as much data as possible and having corporate data engineering store it securely. But modern data-consuming organizations are discovering…
The post Building a data mesh with Collibra and Google Cloud’s Dataplex appeared first on Collibra.

The explosion of data across organizations has led to a wide variety of ways to store, catalog and use that data. One common organizational theme is a central data lake, which is useful for centrally maintaining as much data as possible and having corporate data engineering store it securely.

But modern data-consuming organizations are discovering that central data lakes have their downsides. The data lake itself is a monolithic, centralized platform, which requires its own set of managers and administrators to apply rules and governance to the data it stores. That means those administrators can become bottlenecks in the flow, transfer and availability of data. Further, due to the fact that the central IT team managing the lake becomes a bottleneck, it is likely “shadow IT” organizations will grow, creating their own lakes, hampering discoverability from the wider organization. 

Data mesh: A self-serve data storage design

A more modern solution to the challenge of storing and sharing data within an organization is the data mesh. The data mesh is a decentralized data architecture that leverages a domain-oriented, self-serve design. Data mesh reflects a shift away from a centralized data platform to a modern distributed architecture where organizations can truly unify data from disparate systems and sources. 

The result can be an intuitive, scalable self-serve data model. In a data mesh architecture, each organization can cater to their own data, while at the same time benefiting from a framework of overarching governance and the corporate wide infrastructure. The fact that each team can directly produce their own data means that important context remains with the data, and that consumer teams can directly interact with the data-producing teams. The central IT team will cater for a governance framework which will put guardrails around the corporate data, preventing misuse. 

A data mesh architecture allows domains to be owned by subject matter experts, and to let the data contained to be more accessible than data lake data managed by a single small team.

3 steps to data mesh

Google Cloud offers sophisticated tools for quickly onboarding data streams, pre-processing streaming data and data visualization. Google Cloud Dataplex is a data management platform that offers seamless integration with Google-native and open source tools, allowing an integrated analytics experience to curate, secure, and analyze data at scale.

Google Cloud partnered with Collibra to build a data mesh architecture. Through a three-step process, Collibra and Google Cloud built a seamless, scalable, self-service data system.

Step 1: Avoiding the clogged data lake

The first step is to identify and absorb all the data sources coming into the data lake. The data is registered and grouped according to identified taxonomies. Those can include data policies, usage, permissions, ownership, related BI reports and more.

Step 2: Applying the correct policies

Once the data is registered and absorbed, Collibra metadata helps to validate the data, and Collibra Data Quality is used to make sure the right data gets to the right zones. Collibra Data Quality ensures that the right governing policies are applied to the data so that it can be organized with full context.

Step 3: Building a robust, self-service  catalog

After the data is analyzed and organized with business context, the solution is pushed to Google Cloud Dataplex. The result: A robust  catalog that enables self-service for data users across an organization. The resulting data mesh provides an intuitive experience across the enterprise, and its users can rely on data quality and lineage, as well as have peace of mind that the data aligns to all necessary policies. 

The benefits of Collibra and Google Cloud’s data mesh platform 

Benefit 1: Facilitate faster time to insight by accelerating time to access and analyze decentralized data

With Collibra and Google Cloud’s data mesh platform, managers of the data lake are no longer a roadblock, as data consumers across the enterprise are empowered to shop for and access the data they need. They can access the data more quickly, and that means they can use, analyze and act on it more quickly. 

Benefit 2: Enable data owners and producers to respond quickly and scale access with centralized data management  

In this system, Collibra and Dataplex allow for standards to be governed centrally, but the implementation of these standards – including policy enforcement, access rights, data quality glossary definitions and more – is pushed upstream. The data owners or subject matter experts can decide the rules about all of these standards, removing the roadblock of always going through the data lake management team.

Benefit 3: Optimize the central management of data to foster data literacy across the enterprise 

As the volume of data grows and the number of consumers grows, the solution also allows for scaling into the future, so that new data and policies can be ingested under the same decentralized rules, for more self-service data options.

Watch the on demand webinar to learn more details of how the Collibra platform integrates with Dataplex to combine data from across Google Cloud data products, giving organizations the ability to build a scalable, modern distributed data mesh architecture.

The post Building a data mesh with Collibra and Google Cloud’s Dataplex appeared first on Collibra.