Data Sprints – why run a data project when you can sprint.

Ryan Moore, Head of Data & Analytics at Aiimi, discusses his experience of using Sprint methods to tackle real-world data challenges - and why this approach is so much more effective than your typical data project.

At last year’s Innovate East festival (think Glastonbury for data scientists), I was in awe of how the hack teams were able to produce viable digital and data solutions to real-world business and operational challenges in just a few days. Following up on some of these hacks, I was glad to see the solutions were taken through the machine of governance, the quality gates, and on their way to production. Which made me question, how were these teams able to achieve this in a few days, while larger data projects and programs were stuck in development hell for months and sometimes years?

In a rousing closing speech on the final day of the festival, our COO Tom O’Farrell called out that it’s easier to feel energised and enthused in environments like hackathons, festivals, and incident rooms, where Dashes and Design Sprints are the norm and the rules are eased (and in some cases, lifted). The true challenge is how do we then take this energy and mindset back to our organisations and apply them to data and digital projects in the real world?

Challenge accepted. Working with Aiimi’s creative team (our Design Sprint gurus), we looked at how they were able to discover, define, prototype, and deliver in just five days. True, they didn’t have to deal with the complexities of data security, discovery, acquisition, and quality – but more on that later.

Many Zoom calls, Miro boards, and Slack conversations later, we found that along with clearly defined roles and goals, discipline played a huge part in Design Sprints and hackathons – AKA sticking to the plan. In fact, my colleague David Ford has written all about how to run a successful Remote Sprint, while my colleagues Michelle Hall and Jo Finch have shared what they learnt from running two fully-virtual hacks this year.

Taking the approach of a Design Sprint, merging it with the mindset of a hackathon, and considering all the constraints of real-world data projects, we came up with Data Sprints. Put simply, Design Sprint + Data Hack = Data Sprint, a methodology and framework for delivering the value of data science and AI into the business.

Starting the process with a Data Dash, we spend two days defining the challenge, understanding the data required to solve that challenge, and producing what we call a ‘feasibility matrix’. This is a crucial go/no-go checkpoint, enabling the sponsor or product owner to make an informed and pragmatic decision on whether to move forward to the next stage, based on six key factors.

In the 10-day Data Sprint that follows, our data engineers, scientists, and visualisers work with the team to build out a prototype using a combination of tools, techniques, and accelerators. One of the key accelerators is our Aiimi Insight Engine platform, which helps us discover, catalogue, and make data available to our data scientists in a secure way. Find out how Aiimi Insight Engine helps organisations discover and catalogue data.

At the end of the 12 days, you get a set of clearly defined problem/solution statements, a working prototype, and a clear path to production. This methodology works well for Proof-of-Concept/Proof-of-Value activities, where we need to prove the business case before seeking additional funding, but it’s also a viable alternative to traditional data service delivery.

Want to find out more about our Sprints and how we work? Check out our Data Sprint services.