Taking an Agile Approach to Data Science

Taking an Agile Approach to Data Science

by Alexandrea Ridden, Senior Data Scientist at Altius

These days most software development teams practice Agile methodology, or a version of it. Its framework and practices suit the unpredictability of constructing software.

Data science teams are understandably more cautious about adopting an Agile project management methodology and approach. By its nature, data science is an experimental process. Quite often you’re trying out and testing something that won’t directly lead to the required solution. If you’re not always going to be delivering into a working solution, then surely an Agile approach, where, after an agreed period, you have to pause and demonstrate work and results, isn’t the right tactic?

Many data scientists believe that Agile will stifle their creativity and restrict their freedom to solve the problem in the way they think best. This might be the case if it’s not done correctly. However, I believe there are many merits to subscribing to an Agile methodology when working in a data science team. Here are some of them…

It provides focus and direction

Agile project management methodology uses short, iterative development cycles called “sprints”.  The purpose is to allow for feedback after each sprint so that the project under development can be refined. Just because data science is experimental and not as tangible as software development, doesn’t mean it can’t benefit from the direction that this approach affords.

The key is for the data science team to estimate what they are going to achieve during each sprint. My team prefer two-week sprints, but the timescale can be determined according to how the team will work most efficiently. The sprint structure provides the project with focus and direction. That’s because when each sprint is completed, the project pauses and the data science team demonstrate what they have achieved or discovered. This provides a very valuable window into the data and the insights, which allows the project’s stakeholders an opportunity to provide input and guidance on what to research or test next.

Whilst you might not deliver something that someone will use during each sprint, you should always be building and working towards the project’s end goal. With more stakeholder interaction and more direction, the end goal can usually be achieved faster and more effectively.

It should be empowering 

In almost all workplaces there is some kind of management structure – whether it’s a project manager who sets project deadlines or a client asking for progress reports.

Typically, data science projects are run by managers who dictate the timeframes. This usually necessitates an ‘all guns blazing’ approach to deliver in by a specified deadline.

A refreshing aspect to an Agile approach is the fact that as a team you get to define what you’re going to deliver, and when. For my team, the benefits we have felt by working agile is that we define what it means to finish a piece of work. We also decide how much we can get done during a sprint, whilst keeping our stakeholders happy through regular touch points. Overall this gives us autonomy and helps us feel empowered by delivering work based on criteria that we have pre-defined.

It’s actually less disruptive 

Many data scientists I’ve spoken to are put off the Agile approach because they assume that the regular meetings will be disruptive. They see the meetings as an unwanted overhead, but the reality is, Agile gives you a clear window (which in the case of my team, is a two week window) with no unplanned distractions. Imagine the joys of being able to conduct your research, safe in the knowledge that you won’t receive any ad hoc requests for updates on where you are at or what you are doing. You start each sprint knowing that at the end of the period you will have to present your results, but aside from that, you are left to your own devices. Now, doesn’t that sound appealing?

Admittedly the Agile approach is not for every team, but for teams trying to create an impact to the business in shorter time frames, this approach can really work. It’s highly collaborative and interactive, which can be a foreign concept to some data scientists, but it certainly works for us and we’ve seen clear benefits to our team, the projects we work on and the project stakeholders.

Making progress is a positive thing and with Agile, you’re continuously progressing towards building a solution to a business problem and deciding with the stakeholders when a project is ready for the next phase. How you embed an Agile culture into your data science team – well that’s another challenge and another topic for another blog.