by Ryan Lamb, Business Consultant
Originally posted on his Medium blog here
What do you do in your spare time when your two great passions are data science and rugby? You use machine learning to try and predict the 2019 Rugby World Cup scores of course.
I love all things data science, and I scratch that itch in my role as a Business Consultant at Altius. But my other passion is rugby, and as a born and bred South African I was desperate to see the Springbok’s repeat their 1995 and 2007 successes in this years Rugby World Cup.
What better way to expand my machine learning and Python skills than by building a model to try and predict the outcomes of the tournament’s matches? And that’s exactly what I did. Over the course of a few evenings in September I put my skills to the test to try and predict the winners and scores of the World Cup’s group stages. I did it using Python, Jupyter notebooks, PyCharm and Power BI. And here’s how I got on…
Step 1: Sourcing the data
I found the basic data that I needed on ESPN Scrum – an extensive database of international rugby matches. I was after data relating to all international rugby matches from 2003, when the World Rugby Rankings were first introduced. The problem was, this equated to over 175 pages of data. That’s where BeautifulSoup came to my rescue. It’s a library for pulling data from HTML and XML files. Here’s a look at the raw scraped data relating to a Wales match on 7 September 2019:
After a little bit of work I had 16 years worth of data in my dataset — over 8,700 rugby matches!
Next I wanted to add the World Rugby Rankings as a feature in my model, so I performed a similar scraping exercise to obtain each team’s ranking since 2003.
Step 2: Prepare the data
Before it was ready for machine learning to be applied, the scraped data needed cleaning to make sure it was in a format suitable for my model. Pandas makes this task relatively straightforward. I also performed a few preparation steps, such as accounting for errors and blanks in the raw data and formatting dates correctly.
Step 3: Features
The next stage was to merge the World Rugby Rankings on to the main dataset so that each team’s ranking at the date closest to the date of each match was applied. While I had the World Rugby Rankings, I still wanted to calculate a separate, simpler skill score for each team. I calculated relative skill using a generalisation of the Elo rating system. In this system, a player’s rating changes depending on the match result and the relative difference in rating between the two teams.
For example, if a team ranked 10th beat a team ranked 2nd, the 10th ranked team will receive more rating points than if they beat a 15th ranked (weaker) team. Similarly, the losing 2nd ranked team will lose more rating points for losing against a weaker opponent, than if they lost to the top ranked team. This ensures that the algorithm is self-correcting and ‘learns’ the skill of each team over time.
The World Rugby Rankings take various match features into account, such as match status (more points for a World Cup final), home advantage, as well as points scored in the match. The calculated skill score ignores all of that and simply looks at match result and relative skill between the two teams at the time of the match.
Here’s how the rankings compare at a snapshot in time:
When I looked at how the rankings compared there were some interesting results. Australia ranked higher than expected, and Ireland and Wales lower than the true rankings. I suspect Australia’s high ranking was heavily influenced by their initial high starting point – in 2003 they were ranked 3rd best team in the world.
Below you can see how the skill scores of some teams changed as the algorithm learns. Notice the steady rise of England, Ireland and Wales from about 2014 onwards.
In summary, the data for each historical match included:
- Match date
- Match result
- Points for
- Points against
- World Rugby rankings for both teams
- Relative skill score for both teams.
Step 4: Train and test
For score prediction I used Keras, a high-level neural networks API. The simplest model in Keras is the Sequential model. I performed hyperparameter tuning and experimented with both wide (one layer with a lot of neurons) and deep (more layers but fewer neurons per layer) networks. I ended up using a first layer of 15 neurons and a second layer of 8 neurons, both with the Rectified Linear Activation Unit (ReLU).
You can see below a plot of the loss (Mean Squared Error) over epochs.
I also tried the fan favourite XGBoost, but found I obtained slightly better results with the neural network, at the expense of a longer training time.
Step 5: Predict!
For model input I created a dataset of the upcoming World Cup group stage matches and assigned each team their latest skill score and ranking.
So how did the predictions perform? I’m pleased to say, much better than I expected (at least for the first 15 games)! Of the 30 games my model predicted, it correctly predicted the winner in 27 of them. Admittedly, predicting the Japan versus Ireland upset would have been astounding!
The model performed particularly well in the first 15 games, being within less than 5 points difference of the actual score difference in 10 of those.
I didn’t retrain the model or update the features during the group stages because by then I was too engrossed in watching the tournament. The results above were obtained right at the beginning of the World Cup.
It was interesting to note that once most teams had played at least one game, the predictions became less accurate. This makes sense as the relative rankings and skill scores would have changed, so one could assume that the predictions would have improved if I had updated these features and retrained the model after the first round of games.
Either way, not a bad result for a couple of evenings of work. Machine learning is a fascinating field, and there are many of ways I could improve my model and interesting features that could be added. I’ve got plenty of time. It’s another four years until the 2023 Rugby World Cup in France.