Symmetry is a beautiful thing in life and in statistics, and what’s more is it makes life so easy if you’re a data scientist. Ah, the beautiful simplicity of declaring something is normal, slapping on a t-test derived confidence interval, and going out for calzones with your local team owner. A dream life if there ever was one.
Unfortunately, it can also be the exact wrong thing to do if you’re not careful. If you ask anyone in the football analytics team at Sumer there is literally nothing that drives me nuts more than not checking assumptions before creating a metric. It drives me batty. This usually happens when someone makes a new metric and then throws a z-score around it or uses a t-test to define a confidence interval. My first question is, are you sure it’s symmetric let alone normal?
Why does it matter?
Bad assumptions make for bad predictions, even if you’re doing something good like trying to depict possible error around your prediction. The problem comes when you use the wrong tools or assumptions when doing so. Let’s talk through an example.
The above chart was lovingly stolen from Open Source Football
Let’s say you were making a draft value chart and wanted to express the error around the value of each pick in the draft since draft picks can be highly uncertain. That’s a worthy goal! Now, let’s think about what that error looks like.
In keeping with the assumptions of the above chart, let’s say we are near the top of the draft and there is a non-QB we really like and we’re considering trading up to acquire him much like our President of Football Operations, Thomas Dimitroff, did when he traded up for Julio Jones. What is the shape of the error around our estimate? “What the heck is this guy talking about?” you may be asking yourself, and I don’t blame you, but I promise it’s an important question. Let’s phrase it a different way, assuming the 1st overall pick is deemed to be worth about $16 million a year for 4 years by the draft chart in question (plus whatever cap inflation assumptions you want to bake in) is the chance that player is a Tyreek Hill level player (~$32M) the same as the chance they are a bust (no second contract, i.e. $0M)? I will argue emphatically no, let me explain by talking through a simulation of talent entering the NFL.
In the United States, there are approximately 1 million high school football players, 12,000 Division I players, and for the sake of easy math let’s say ~300 players drafted and undrafted who enter the NFL each year. For the sake of simplicity, we will ignore player development in the following very simple simulation as there are a number of assumptions we could make there as well.
From that population we’ll assume college directors of player personnel are recruiting their incoming class (3000 players) each year using traditional scouting and analytics and from that pool the NFL drafts/signs 300 players a year.
To start I’m going to assume the talent of high school players is shaped like a bell curve (normally distributed mean 50 standard deviation of 10 for you nerds). We define talent as their scouting grade assuming positional importance is not baked in, this is how our grading scale is defined here at SumerSports. Additionally, I will assume no player has negative football talent as I have removed myself from the dataset. To simulate how players are selected from each pool to the next I defined the weight of a player to be their talent divided by the maximum talent in their respective talent pool raised to an exponent. That exponent is a scaling factor meant to represent how effectively we can assess talent when selecting players i.e. the higher the scaling factor the more likely we will sample players from the right tail of our normal distribution up to D1 and then again from the right tail of the D1 distribution up to the NFL.
As you can see above, the more information we have about a player’s talent be it from scouting or analytics the more likely it becomes that our resulting NFL talent pool is asymmetric. This should be an intuitive finding. Even if we are recruiting and/or drafting players from a population which is normally distributed we are trying our best to only sample from the right tail of that bell curve, i.e. we want to recruit and draft the best players. The better we become at identifying talent through scouting, analytics, or a combination of the two the more skewed our result will be suggesting that the distribution itself could be non-stationary.
To demonstrate this fact lets look at the skewness difference just between our Medium and High information plots.
Medium Information
High Information
Now, hypothetically if you’ve never seen a skewness and kurtosis chart before, not you dear reader of course we are speaking hypothetically, how would we read this chart? Well, the farther away from the ✳ our red dot appears to be the farther away from a normal distribution our data appears to be. In the high information graph above we can see we are right on the line between a Beta and Gamma distribution and while both of these distribution types can be symmetric we observe enough skewness (the data is off to one side) that we know it isn’t symmetric around the mean (average). Even if we assume we have different levels of information between the NCAA recruiting process and the NFL draft process we still end up with a high asymmetric player talent pool.
Basically, the higher the scaling factor the better our “scouting” is in this scenario. Now let’s look at some real scouting grades. Below is the distribution of the bins of SumerScout grades from the 2023 college season. Note: These bins have been grouped i.e. a 5.6 grade was put into the 5, a 6.8 into the 6.
Look familiar? Now, this isn’t to say that scouting grades are a perfect measure of player talent. They aren’t. (Obligatory: Neither is analytics, they work best when combined. Like peanut butter and chocolate.) Now let’s return to our draft example. At each pick in the draft from 1 thought 256 plus comp picks each team is trying to select from the extreme right tail of the remaining players. Even if they aren’t drafting Best Player Available (BPA) they are at least trying to find the player who best matches their fits and/or needs who is as close as possible to the BPA. It’s right tail sampling all the way down.
So, what should we expect the distribution of players selected near the top of the draft to look like? If you answered asymmetric with a long right tail, then maybe I’ve done a good enough job convincing you. Or maybe you were smarter than me to begin with. Through the love of ball all things are possible. In general, we see this pattern repeatedly in professional sports, when a player is selected by a trait that trait tends to be asymmetric for exactly the reasons we’ve shown above, when a player just happens to have a trait but it isn’t strongly baked into the selection criteria it tends to appear more symmetrical and normal. So, if you are plodding along in the Big Data Bowl and find yourself with a metric appearing more log than normal fear not, you may have actually stumbled upon something that scouts are actively measuring!
So why does this matter? Let’s say our hypothetical WR in the draft is a scouting grade 7 plus or minus 1 because we assume our risk is symmetric. Well, if our player turns out to be a SumerScout 6 they are backup quality, and if they end up an 8 they are a first year starter and likely Offensive Rookie of the Year candidate. But have we truly captured the downside risk? Is the lower bound of our error really just that he’s a high-end backup? This is the risk of symmetrical assumptions. While we could possibly transform our distribution into something more symmetrical we lose ease of interpretation for our decision makers. Decision makers who aren’t drafting for average results but trying to hit the extreme right tail, if our explanation of risk is lacking we may entice them into the wrong decisions. Additionally, since our n is relatively small (and the distribution itself possibly non-stationary!) in many cases in pro sports it may make more sense to simply bootstrap your confidence intervals.
Point estimates are better than nothing, and measures of error even better still, but if we don’t take great care in the construction of our error estimates and simply throw around Z-scores we may end up offering recommendations that are biased or nonsensical.
Another implication of this asymmetric property of player talent is on allocating spending during your team construction. However, you have to blend your estimates of player talent with positional value and scarcity. For NCAA teams especially that in essence hit free agency every year, having accurate estimates of their player’s talent is the first step in the budgeting process and crucial given the uncertain landscape of judicial or congressional input into the revenue sharing and name image and likeness rules of the future. These are all problems that SumerSports is uniquely positioned to assist in solving.
This article was meant to be approachable and in non-academic language as I am not a teacher nor a PhD. Just an old Army guy with an axe to grind against symmetry who loves ball. If you would like another look at symmetry in professional sports from someone infinitely smarter than myself please read here. If you as a lover of ball (or puck) learned something here or think I’m wildly wrong let me know. Remember, I’m pulling for you. We’re all in this together.
Source(s):
Baldwin (2024, March 28). Open Source Football: NFL Draft Value Chart. Retrieved from https://www.opensourcefootball.com/posts/2023-02-23-nfl-draft-value-chart/
