Building a Way to Measure NFL Coach Value

Despite being a trope that is often used to excuse inefficient decision making, there are indeed multiple ways to win in the NFL. In this article, I want to compliment the work we did in our first ever article at SumerSports, which investigated the in-game decisions that coaches had to face and how they did relative to their peers.

To build off that, I want to take more of a top-down approach to the question of measuring coach value.  

SumerSports In-Game Coaching Model: An Introduction | SumerSports 

The NFL betting markets incorporate as much available information as possible; team and player strengths, injuries, weather conditions, and rest differentials are just a few data points considered. Coaching strength is certainly taken into consideration as well, either directly through the ratings used by originators to move the line or indirectly through player-level data (e.g., EPA per pass attempt for a quarterback).  

However, in my experience, coaching is the most difficult piece of the pie to quantify. While its effects are incorporated into the markets, its signal may be one of the least-understood elements considered.

Thus, to model coaching value, we look at the outcome of the game and build a mixed-effects regression using the pre-game point spread, with random effects equal to the quarterback and head coach. From this, we can derive a point-spread value for each coach

In practice, one should do this over intervals to capture the effect of an ascending or declining head coach. However, for the purposes of this introductory article, we will look at all of the data from nflreadr from 1999 to the present and provide a point spread estimate from that data. Here are the leaders during that time: 

This list mostly checks out, with arguably the best coach of all time sitting at the top while two top coaches currently in the NFL are second and third.

It might be surprising to find Mike McCarthy’s name up there, but a decent amount of that is from his Green Bay days, where he won a Super Bowl despite being down multiple starters by the time the season ended. 

Also note that there is a combination of old-school coaches, like Bill Belichick, Pete Carroll, Bruce Arians, and Bill Cowher, and guys that have leaned into evidence-based approaches to in-game decision making, like John Harbaugh and Doug Pederson. Billick, who was an offensive wunderkind in Minnesota, adjusted to a defense-centered team in Baltimore and brought the franchise its first Super Bowl victory. 

It might also be surprising that these point spreads are a) a bit low, and b) have credible intervals that are quite wide. I think this is evidence of what was said at the outset: great coaching is hard to quantify, and with the exception of a few coaches that we all agree are great, there is plenty of uncertainty.  

To close this short article out, let’s look at the top 12 active coaches in this measure: 

As I said in my Friday column a few weeks ago, there are coaches like Tomlin that clearly move the needle for their teams but don’t score well when looking at in-game decisions. The Tomlins, the Carrolls, and the Mike Vrabels of the world show up in this analysis, which is comforting. Now, would it be nice if they had all the motivational chops in the world and went for the right fourth downs? Yes, but there are no gods in football. 

In this article we discussed a top-down way of measuring coaching effectiveness and contrasted it with the in-game decision model we built last season. The results of the model appear reasonable and will be a part of the work that we put forth on sumersports.com in the coming months.