Top Trends in Data & AI for 2021: data mesh.

Aiimi Data Scientist James Robinson shares his prediction for 2021's top trend in data and AI. Find out what a Data Mesh is – and why it will be important for business in 2021.

Enjoyed this video? Discover all our predictions for 2021's Top Trends in Data & AI.

What do you think will be the top trend in Data and AI for 2021?

There’s been an interesting trend to come about over the last year or so called a Data Mesh. It’s essentially a decentralised approach to your current data platform, where you have a common standard and common elements to the way that you go around your ETL (Extract, Transform, Load) pipelines. But, essentially, you allow each business domain to have their own store of data and they manage their own processes.

This means that rather than having all the requests from both the users and the data owners come through that central data engineering team, each domain – so, say customer data or finance data – has its own individual data owners, data engineers, and data users. And they can run their own custom pipelines, ingestion, and transformations.

There are a couple of good points around Data Mesh. Firstly, as I briefly touched upon, is its democratic use. For instance, if your department has a data need, they go to their specific department engineers and they have their own process of doing it. That also means you have data engineers who are clued up on your specific department and your specific data, so you have that level of expertise as well. It makes it a bit more scalable, and it removes the bottleneck of an entire process going through a centralised data team. So, while a Data Mesh allows each team to have a bit more control over their ETL pipelines, adopting a common framework and common standards, it still allows the complete interconnectivity of all your data sets. So, if you're in the customer section and you need some finance data, then you have the same common standards and connectivity to be able to bring that data into your own reporting needs.

This allows a bit more detail, a bit more ownership and granularity, and just a bit more room for being data experts within that field.

What’s so great about it?

There are a couple of good points around Data Mesh. Firstly, as I briefly touched upon, is its democratic use. For instance, if your department has a data need, they go to their specific department engineers and they have their own process of doing it. That also means you have data engineers who are clued up on your specific department and your specific data, so you have that level of expertise as well. It makes it a bit more scalable, and it removes the bottleneck of an entire process going through a centralised data team. So, while a Data Mesh allows each team to have a bit more control over their ETL pipelines, adopting a common framework and common standards, it still allows the complete interconnectivity of all your data sets. So, if you're in the customer section and you need some finance data, then you have the same common standards and connectivity to be able to bring that data into your own reporting needs.

Forgetting about data and AI, what’s the one thing you’re hoping for in 2021?

Oh gosh, where to begin. There are so many things I'm looking forward to doing. Obviously seeing my family and seeing my friends again would be lovely. Going to the office, the Aiimi office. That was so much fun the last time I went, going on those hoverboards. I’m looking forward to that. Going to meet the client team that I've been working with for the last six months – I’m looking forward to that as well. Plus a couple of other things like being able to play my sport again and being able to meet up with family and friends.