RIP Data Lake, hello Data Mesh?

‘Data Mesh’ describes a way of providing targeted data products to domain-specific teams, using a distributed architecture rather than a centralized Data Lake. And while that sounds splendid, public sector organisations are left wondering whether it matters to them. In this article, we discuss how it didn’t use to, but it soon will. Most organisations […]

‘Data Mesh’ describes a way of providing targeted data products to domain-specific teams, using a distributed architecture rather than a centralized Data Lake. And while that sounds splendid, public sector organisations are left wondering whether it matters to them. In this article, we discuss how it didn’t use to, but it soon will.

Most organisations recognise the unique power data has to unlock their vision and strategy but many struggle and fail – often repeatedly – to put the right data in the right place at the right time to become truly data-driven.

‘Big Data’ was once heralded as the solution. Now, over a decade old, most organisations are yet to achieve the kind of data revolution they thought was promised. What happened? What was the problem? Well, there were a few, one of which concerns a logical flaw in traditional thinking that is so entrenched it goes unnoticed and unchallenged.

Traditional thinking goes: ‘we gather all the data from our sources and bring it together in the centre, sort it out, and supply useful cuts of it back to our organisation’. The flaw – why deliberately engineer in this gap between data and the people who understand it best? Surely it is better to keep the data next to the experts.

Data Mesh is different, designed specifically to close the gap between data and the people who understand it. It is a decentralised data and analytics strategy – first termed by Zhamak Dehghani of ThoughtWorks.

By democratising data into distributed products, a Mesh empowers both end-users and data scientists to directly analyse data and operationalise their insights. Mesh also scales immeasurably better than a central Data Lake approach whilst supporting event-driven, real-time, and streaming-based analytical needs that traditional warehouses were never designed to solve.

As the public sector seeks once more to use data to design citizen-centric services – keeping children safe, society functioning, and patients recovering – attention turns again to the sharing and distribution of data across organisational boundaries, rather than simply collecting it in the centre. No one solution solves everything, but Mesh certainly outperforms a monolithic Data Lake when we look at the agility needed to address these overwhelmingly significant societal goals.

From designing data products through data-sharing models and distributed data architecture to engineering real-time microservices that empower end-users directly, Agilisys is pushing the boundaries of the cloud-based data mesh so organisations can finally unlock that data.

If you’d like to talk more about it, we’d love you to get in touch.