Which stat should you use when trying to see how good a quarterback is?
Not long ago, football fans and writers were content with evaluating quarterbacks based on the ol’ reliables: win-loss record, completion percentage, passing yards, touchdown-interception ratio, and so on.
If your completion percentage was high, you were accurate, and if it was low, you were inaccurate. If you had a lot of passing yards, you were explosive, and if you didn’t, you were a game manager. This is how simple the statistical discussion was in the NFL discourse.
But here in the great year of 2024, the football world is blessed with a seemingly endless pool of advanced statistics to choose from when evaluating quarterbacks. We have more context than ever before. Hidden aspects of the game that were previously lost to history are now being vigorously documented. It’s a win for the football community, as it allows us to collectively get a much more accurate grasp of which quarterbacks are thriving and which are not.
With so many options, though, it can be difficult to know which stats are the most telling.
That’s what we’re here to talk about today. What metrics should New York Jets fans focus on when evaluating Aaron Rodgers this season?
Breaking down the quarterback metrics
In a pinch, I don’t think old-fashioned passer rating is a bad metric. It’s a decent catch-all metric that accounts for four factors: completion percentage, yards per attempt, touchdown pass rate, and interception rate. If you just want a quick, simple synopsis of how efficient a quarterback’s box-score numbers are, this is the way to go.
However, passer rating leaves out a lot of context. Down and distance, field position, game situation, fumbles, sacks, penalties, and rushing production are all left out. It also suffers from the usual No. 1 flaw of box-score metrics: assigning full responsibility to the passer without accounting for the performance of his supporting cast.
Expected Points Added (EPA) per play has arguably become the most widely used QB metric. It accurately captures the efficiency at which the QB helps his team score points after accounting for down and distance, field position, game situation, fumbles, sacks, penalties, and rushing production. It doesn’t account for the supporting cast’s performance, though, so a QB with bad receivers or a bad offensive line will still be dealt an unfavorable hand in this metric.
Pro Football Focus’ grading system seems to do the best job of accounting for a QB’s surroundings. Unlike any other metric, a QB can earn a well-deserved positive grade if his receiver drops a great pass, and he can be let off the hook for a negative result that isn’t his fault due to poor protection or route-running. It goes the other way, too. QBs will get the negative grade they deserve for throwing a dropped interception or getting bailed out by a great catch on a bad pass.
However, the flaw with this metric is that PFF’s grading system is subjective and conducted by anonymous humans through an undisclosed grading system, so it’s anybody’s guess as to what’s happening behind the scenes. Its methodology is far more vague than something like EPA. Everyone knows what EPA is measuring, how it’s measured, and what it means. PFF grades remain a mystery. Still, the system tends to spit out fairly decent conclusions that align more closely with the film than box-score stats do.
Completion percentage over expected (CPOE) is a metric that’s growing in popularity. It attempts to contextualize accuracy by accounting for the difficulty of each pass attempt, using tracking data to estimate the expected completion percentage of each attempt (accounting for factors like depth, window tightness, pressure, etc.). To get CPOE, you subtract a QB’s expected completion percentage from his actual completion percentage.
In theory, this is a great idea, but in my personal opinion, I don’t think the execution is there yet. It doesn’t do the best job of separating the QB from his receivers. For example, if a QB chucks up a 50-50 jump ball, the expected completion percentage will be very low; these are the plays that tend to give the biggest boosts to CPOE. Yet, if the pass is bad but the receiver makes an insane catch, it counts all the same for the QB, giving a huge boost to his CPOE just because his receiver made a great play.
Look no further than this iconic play. In CPOE’s eyes, it goes down as Zach Wilson completing a pass with a 34.5% completion probability, making it seem like he threw a dot. But we all know what really happened.
Zach Wilson & Garrett Wilson (3-yd TD)
🔹 Coverage Defender: Tre'Davious White
🔹 Target Separation: 0.6 yds
🔹 Completion Probability: 34.5%**Wilson's most improbable TD reception of his career#BUFvsNYJ | #TakeFlight pic.twitter.com/lsBQFutYAV
— Next Gen Stats (@NextGenStats) September 12, 2023
If you look at the 2023 CPOE leaderboard, Patrick Mahomes was only 10th in the league out of 30 qualifiers, one spot behind Derek Carr and one ahead of Geno Smith. Anyone who watched the Chiefs knew that Mahomes was being hung out to dry by one of the worst receiver units in the NFL, laying an undeserved dent in his statistical production compared to his usual standards. CPOE’s methodology is designed to account for those types of factors, but it doesn’t always accomplish that goal.
Brock Purdy led the NFL in CPOE (by a wide margin) even though everyone who watched the 49ers could see how easy his job was. That’s not to say Purdy was bad – there’s no question that he was very good in his role – but did he really exceed the expectations of his supporting cast more than any other QB in football? It’s hard to claim that based on the eye test.
Here’s a play where Purdy chucks a poorly thrown ball into double coverage but lucks out that the field-side safety plays it horrendously. Brandon Aiyuk makes a smart adjustment to his route and secures the ball. This goes down as a completion with a 27.8% probability in CPOE’s eyes.
2Q: 1 & 10 at the SF36 (7:43) Purdy to Aiyuk for 42-yds on ‘X-Rider.’ However, this did not look like a typical ‘rider’ play. There was no play action, and a slightly altered route from Aiyuk. But, I’ll take 42 yards any time. pic.twitter.com/RscS7nxUtG
— Bret Rumbeck (@brumbeck) October 4, 2023
Brock Purdy & Brandon Aiyuk (42-yd reception)
Purdy's pass traveled 46.8 yards in the air, Purdy's longest completed pass by air distance of his career.
💡 Completion Probability: 27.8%#AZvsSF | #FTTB pic.twitter.com/cdF39fSB3C
— Next Gen Stats (@NextGenStats) October 1, 2023
Even more shockingly, C.J. Stroud was 18th, two spots behind… Justin Fields. I could go on and on. It’s hard to trust a metric with conclusions like these when its whole purpose is to account for the context that our eyes can see. Not one person with eyes would tell you that Fields was more accurate relative to expectations than Stroud was.
CPOE has the right idea, but to be perfected, it must get to the point where it solely evaluates the accuracy of the QB. Right now, all it does is track the odds of a pass being completed and whether or not it was completed. That doesn’t tell me anything about how accurate the QB was.
What I would like to see is some sort of formula that compares a) the expected completion percentage of a pass attempt at the moment it is attempted versus b) the expected completion percentage at the moment the ball arrives at the catch point. This would isolate the QB’s accuracy from what the receiver and defender do at the catch point.
Let’s say Aaron Rodgers attempts a 40-yard go route to Garrett Wilson along the sideline, while under pressure from the interior, with Wilson’s defender trailing him by one yard in coverage. My proposed formula would evaluate the expected completion percentage of this pass at the moment it leaves Rodgers’ hands. Let’s say it’s 25%.
Then, it would evaluate the expected completion percentage when it arrives in Wilson’s vicinity. If Rodgers throws a perfect lob out in front that hits Wilson in stride, this could be a 75% gimme. We’d give Rodgers credit for boosting the expected completion percentage by 50%. If Rodgers underthrows the ball and forces Wilson to make a circus grab over the defender, the odds of a completion might go down to 15%, in which case Rodgers would be knocked for reducing the expected completion percentage by 10%.
That’s how it should work: isolating the effect of the QB’s accuracy on the play’s odds of success. Unfortunately, I don’t have the tech or the expertise to create that formula. Get on it, whoever makes these things. I’ll be over here praying to the stat-nerd gods, trying to manifest the QB metric of my dreams.
So, yeah, long story short, I’m not a huge CPOE fan.
I don’t think the perfect QB stat exists yet. The best stat for QBs depends on what you’re looking for.
If you’re focused on statistical output, use EPA per play. It does the best job of capturing the QB’s impact on the scoreboard.
With that being said, fans should understand what EPA per play is measuring. It solely looks at the results of the QB’s plays and how they affect the team’s chances of winning. It doesn’t account for the process that leads to the result. The QB’s supporting cast remains unaccounted for. So, EPA per play is still not a perfect measure of how well a QB is actually performing as an individual. It’s the best metric for evaluating production, not performance.
If you want a contextualized metric that is designed to evaluate the QB’s individual performance rather than his production, PFF’s grading system is the best place to look. Still, it’s PFF, so be aware of the caveats that come with it.
Ultimately, if you want to know how well a QB is truly performing within his role, the best measure remains the old-fashioned eye test. When watching Rodgers this season, if you trust your eyes with an unbiased approach, you should get the true story.
