Manage your information at scale with a data mesh approach to unlock added value for your business.

Finding the exact data you need exactly when you need it drives valuable actionable insights. But what's the best approach? A recent hot topic, let’s look at how the ‘data mesh’ concept can help your company make the most of all the information it already owns.

Data mesh is an evolving approach to dealing with the challenges of data management at scale. Moving away from the traditional approach of storing data in centralised structures, the data mesh concept embodies a far simpler decentralised model. That’s it in a nutshell! But let’s take a deeper dive to find out why and how this approach can be applied to your business.

Centralised structures vs a decentralised data mesh

As your business embarks on its data journey, gathering data from across your enterprise, the prevailing mindset may well be to move all your data into one centralised location. From there, you can access, model, and build reports and dashboards from it, deriving real insight. That’s why your data teams routinely build central structures, such as data warehouses, data lakes, or repositories.

Although these monolithic data stores have their use, setting up and supporting these centralised structures comes with three primary challenges:

• Time and money: it takes time for your business to bootstrap or plug new data sources into your centralised data lakes and data warehouses. And it takes time to model that data to incorporate interlinked relationships and reflect the business context. All this work calls for specialist skill sets, and we all know that data engineers don’t come cheap!
• Evolving need: for your traditional workloads, such as reports and dashboards, centralising data in one place does make sense, but for your modern workloads, like data science and machine learning, your teams need to explore data, build experiments, prove hypotheses and get their models into production at speed. They just don’t have the time to hang around, waiting for all that data to be centralised. Or, in some cases, your teams don’t even need to extensively model that data to meet their project’s needs.
• Beyond traditional data sources: although modern data warehouses and data lakes do support your unstructured data, key information and data are nevertheless lost within your content and record management systems. From Excel files within SharePoint folders, and Word and PDF documents on shared drives, to emails in shared inboxes, your semi-structured and unstructured data sources have huge analytical potential when properly classified and labelled. But they are often ignored because they fall outside the realm of traditional data sources, or your data teams simply don’t know they exist.

A data mesh makes sound business sense

Although the traditional centralised method does work, given these challenges, it’s far more straightforward and judicious to simply leave your data where it already lives, within its own business domain. From there, you can apply a data mesh architecture to interconnect all your data and information for a far more effective decentralised approach. The data mesh concept features four core principles:

1. Your data is organised and owned by its business domains.
2. These business domains produce their own data products (analytical and predictive datasets, models, reports, dashboards, and data stories).
3. These business domains need a self-served data infrastructure to create those data products.
4. And all three of these are underpinned by federated governance to standardise your data products.

How to put the data mesh approach in place – and govern it

First, it’s important to understand that the data mesh is not a solitary product, service, or capability – it’s a joint approach with an accompanying change in mindset. You need to focus on the key capabilities needed to enable a data mesh – and put them firmly in place. You need:

1. A data infrastructure to run and manage your data products
2. A set of capabilities to enable your data product developers to create those data products
3. The ability to search and discover your data products for use in specific use cases

Given the decentralised approach and ability for domains to do their own thing with data products, you need to be mindful that standard governance approaches won’t work. So, your business needs to onboard a federated governance model that successfully balances interoperability, maintenance, and standardisation against a degree of autonomy and flexibility at the domain level.

The first step is to get your data platform and product owners to agree on a set of shared rules that work for everyone. The next step is to continuously develop those rules to constantly drive maximum value from your data. There’s also a compelling case for automating aspects of this governance as the ecosystem matures, reducing manual intervention.

Aiimi Insight Engine powers a data mesh to unlock value for your business

At its core, the data mesh centres on business domains that build and own your data products – and to build those data products, you need to find the right data. This is where our Aiimi Insight Engine comes into its own. The Aiimi Insight Engine powers a data mesh to crawl, search, and discover your data products – regardless of format, platform, environment, or type – to supply one single unified user experience across your enterprise.

And by enabling your data product developers to search across all your data and content sources – and then enrich their findings with common entities, labels, and classifications – you can construct data products at speed. Your users can now search and discover the exact data products they need to carry out tasks on specific use cases, easily and quickly. By promoting the cycle of productivity, the Aiimi Insight Engine enables your team to unlock even more business value from your data assets.

Taking this one step further still, the Aiimi Insight Engine’s data map feature automatically discovers relationships between multiple data products. So, your users can also navigate interconnected information through a graph user interface to find the answers they need, driving even deeper insight.

Aiimi Insight Engine takes care of your governance, compliance, and security

Depending on the use case, the Aiimi Insight Engine also creates an abstract layer of metadata to support your workflows and enhance the user experience for your data product developers. Combined with the features already discussed, this metadata serves as a solid foundation for federated governance.

How? Well, it shines a light on your data product types, the domains they belong to, how they interconnect, and who’s using them to drive value – and, most importantly, who isn’t. This ‘vehicle’ can steer federated governance, highlighting non-compliance around interoperability, standardisation, security, and privacy.

What does the future look like for this data mesh approach to business?

Sitting at the heart of the Aiimi Insight Engine is the conviction that it’s best to leave your data and content in place, and then supply an abstract layer of capabilities to search, discover, govern, and build data and digital experiences across your entire information landscape. It’s only a matter of time before Insight Engines become an essential business tool, delivering this decentralised data mesh approach to support all businesses in realising the massive potential of their data assets.

Like any new concept, every iteration fine tunes the combined capabilities, tools, and services creating every layer of the data mesh paradigm. As this joint tour de force becomes more universally adopted, we predict that Insight Engine technology and the data mesh approach will steadily evolve side-by-side to predict your users’ data needs and foresee their next steps – and then deliver that information directly to their doors.

Schedule your one-on-one demo to find out exactly how the Aiimi Insight Engine can help your organisation adopt a data mesh approach.