clock menu more-arrow no yes

Filed under:

DVOA ranking of Rams defense exposes more flaws in analytics

Which, factually speaking, should actually not be called “analytics” at all

Philadelphia Eagles v Los Angeles Rams Photo by Sean M. Haffey/Getty Images

Last week’s contest against the Seahawks setup the Rams’ highly effective (and efficient) defense for a matchup worthy of testing their mettle. Seattle ranks first in points and up until maybe 10 days ago, quarterback Russell Wilson was meant to be the league’s most valuable player and DK Metcalf, his long-sought after weapon of cultivating mass destruction.

What LA’s defense lacked in marquee opponents faced (Carson Wentz, Daniel Jones, Dwayne Halex Smith, Big Pick Nick), it made up for in dominance: through nine games, the Rams rank second in points allowed, second in yards allowed, first in passing touchdowns allowed, first in points per drive allowed, second in yards per drive allowed, first in net yards per pass attempt allowed, third in first downs allowed, fifth in rushing yards allowed, fifth in third down defense, ninth in red zone percentage.

Also, their slate may not be parmesan, but it hasn’t quite been camembert either.

Other than when he was injured, Dak Prescott’s worst start came against the Rams; Josh Allen made his mark against Los Angeles, same as he did the week before against another good defense in Miami, and he leads the NFL in passing yards with a strong all-around resume; Tua Tagovailoa has played remarkably better in his last two starts than his 93 yards on 22 attempts against the Rams; oddly enough, it was Jimmy Garoppolo who may have done the best this season against LA and yet all that production came on San Francisco’s first four drives. As they did against the Bills, the Rams defense eventually shut things down.

Every team is allowed midseason lapses. The great ones know it’s only a lapse and I’m confident that the Rams have a great defense this season.

It hasn’t helped the defense that the offense is 23rd in turnovers and they have at least one interception or fumble in all nine games this season. But they rank first in points per drive allowed and Wilson had his worst game since the last time he faced Aaron Donald and Jalen Ramsey and some would say that Sunday was the worst game of the quarterback’s career when you factor in experience and the complementary pieces at his disposal in 2020.

This was meant to be the Seahawks offense that had rebuilt its offensive line, added a premier number one target in the mold of Calvin Johnson so that it could finally solve the kryptonite that is the NFL’s best pass rusher and Wilson’s worst nightmare. They didn’t.

In fact, has anybody solved the riddle of how to consistently contain Donald without in turn allowing sack after sack from their other defensive gems like Leonard Floyd and checks / double checks / triple checks / Troy Reeder? Though sometimes it is a random name like Reeder that has three sacks in a game, the Rams have obviously talent components on that side of the ball other than Donald, Ramsey and Floyd, such as Darious Williams (first in passer rating allowed, according to some), Troy Hill, John Johnson, Jordan Fuller, Michael Brockers and Micah Kiser.

Donald, Floyd and Kiser have all been honored as an NFC Defensive Player of the Week this season.

The LA Rams defense is nothing if not efficient, so why do they rank ninth in “efficiency”?

What is DVOA and why is it being misused and incorrectly termed as analytics?

I have been using DVOA in my writing and research of the NFL for over a decade but the flaws of “analytics” — a catch all term that has no real meaning in the sports world other than the fact that the word refers to “systematic computational analysis of data or statistics” and because it makes some people feel smarter when they make references to “analytics” — are not difficult to find when you search for them.

The problem is that few seem to challenge these numbers, let alone ask what they mean or where they came from. Few seem to understand data analysis like DVOA and the rest don’t seem to care, even those who are self-proclaimed “analytics experts.”

Case in point: the word “analytics” literally refers to the process of understanding what happened in the past in order to predict the future. But DVOA is not predictive of anything and I do not see where it seeks to understand the data that Football Outsiders pulls from NFL play-by-plays for its handy dandy single digit output.

As you can find in the Football Outsiders glossary where DVOA is oddly defined by three different lengths of explanations depending on how legitimately interested in understanding it each reader tends to be (which, if a site has to have an “ultra-short” explanation and a “short” explanation tells me that years of feedback ultimately led to: “Explain this to me like I’m five or I’m leaving”), DVOA “is a method of evaluating teams, units, or players.”

“It takes every single play during the NFL season and compares each one to a league-average baseline based on situation. DVOA measures not just yardage, but yardage towards a first down: Five yards on third-and-4 are worth more than five yards on first-and-10 and much more than five yards on third-and-12.”

Setting aside the argument against five yards on first-and-10 being less valuable than five yards on third-and-4, which is perfectly acceptable to take some issue with, DVOA’s own definition does not entail an attempt to understand those five yards or to predict what will happen on the next down. That’s not something that DVOA or any other “advanced stat” that I know of can really do because accurate predictions in sports are not something anyone can really do.

What DVOA and EPA and even something as rudimentary and cryptically useless as PFF grades fall under is “data analysis.”

Analysis can be considered an in-depth review and sorting of the current facts, which may be a more-than-sufficient assessment for many decision-making scenarios.

DVOA can’t predict outcomes of games or champions of seasons any better than Vegas or uncle Dave or a laser-pointing dolphin can. Last season’s DVOA champion, by a wide margin, was the playoff-winless Baltimore Ravens. In fact, 2019’s top three teams in DVOA went 0-3 in the playoffs.

The Kansas City Chiefs ranked fourth and while it is true that KC was first a year earlier, the team they beat in the Super Bowl, the San Francisco 49ers, ranked 30th. No, the New England Patriots, ranked seventh in DVOA that season, won the Super Bowl. In the postseason, they beat teams ranked first, third and fourth by DVOA.

There’s nothing more predictive about DVOA or EPA or whatever FiveThirtyEight is throwing at the wall today than simply being a football fan and having an opinion.

If you come away with any message from me today, let it be that you’re as much of an analytics expert as somebody who got a blue checkmark and saw it as an invitation to drag somebody for questioning if running backs might matter or for believing it’s fine to punt on fourth and one sometimes.

Anybody can become a self-proclaimed “nerd,” all you gotta do is proclaim it.

Anybody can throw team logos onto an X-axis and Y-axis with four quadrants and call it “analytics.”

Anybody can criticize a coaching decision, you don’t even have to know anything about coaching or football anymore. That’s obvious. The difference is that people used to criticize coaches or people who disagreed with them and all they could do is hope they were right. Now with numbers that are approximately as useful as any stats we had 20 years ago, people actually starting to believe they are right.

Well, you have as much of a right to your opinion as they do to their opinion, and DVOA is no more than a number that tells you what happened — though I must be clear that because DVOA is a formula developed by a human (in this case, Aaron Schatz), it is more opinion than fact — and you are allowed to question its validity.

Especially in cases like the Rams defense, where DVOA is obviously missing something. A lot of something.

Despite LA’s strong play on defense over most of the last 36 quarters, the Rams’ defense ranks ninth overall by DVOA, holding steady at ninth against the pass and 12th against the run. By DVOA, the Rams have supposedly had a worse pass defense than the Buccaneers, Steelers, Bears, Colts, Football Team, Saints, Chiefs and Dolphins.

Even if all I’ve done with this post is create new enemies, surely I won’t find a passionate argument against the proposition that the Rams could have a better pass defense than most, if not all of these teams. The opinion-based DVOA says that LA is -3.3% better than the average defense and yet Chicago is -12.8% better, which is a significant gap. The Bears defense has been remarkably consistent (every game this season has resulted in them giving up somewhere between 13 and 26 points) but Chicago ranks 11th in net yards per pass attempt allowed, with 12 touchdowns allowed and six interceptions.

LA has allowed nine touchdowns, picked off eight passes and their NY/A allowed is nearly a full yard better than what the Bears are giving up. The Rams have 31 sacks in nine games, the Bears have 21 sacks in 10. Yet the difference in DVOA between them is the same as the difference between the Rams and the Buffalo Bills, who rank 19th in NY/A allowed and 25th in points per drive allowed.

Any modern analysis expert would tell you that passing matters and rushing is practically “take it or leave it.” Setting aside the problems with that argument, as well as the one that says all downhill running backs are interchangeable, having an elite pass defense should catapult any team to a formidable overall ranking. Of course, DVOA does weight pass defense heavier than run defense, but as already mentioned, Football Outsiders is not as keen on the Rams defense as an objective eye may find them to be.

And if the idea that LA’s pass defense is being underrated by DVOA isn’t as surprising to you as it is to me, then try to swallow this cheeseburger: the Rams are ranked fifth by DVOA on offense. According to DVOA, Sean McVay’s offense has been carrying the team more so than their defense.

The Rams are 18th in scoring and 22nd in points per drive.

“Some people try to find things in this game that don’t exist but football is only two things: blocking and tackling.” - Vince Lombardi

So then, should Rams fans take it as a personal insult that the defense is ranked far too low by DVOA?


As Schatz has said many times over the years while defending his formula over defending his favorite NFL team (the Patriots), DVOA is simply reading the data. How that data is manipulated was mostly decided years ago, though there have been tweaks along the way. The insult is not to the Rams or any other football team.

The insults come by way of the people who use the data. The ones who don’t realize or care that DVOA is still an opinion-based number (much like the much-derided passer rating, which is also an opinion: a guy who went through life with the name Seymour Siwoff is mostly responsible for inciting those arguments by deciding that this —

would somehow be more indicative of quarterback passing quality than say, dividing that sum by five or seven instead of six, or any other number of other directions they could have gone with in 1971) and that they hold power over you because you choose to believe that DVOA or PFF grades do not have significantly more meaning than your own version of “common sense.”

I am all for more data and I will continue to cite DVOA when it is appropriate to do so. I think perhaps some modern tools that could be more useful though would be things like route trees provided by NextGen. Or passing stats broken down by direction and distance at the SportsInfoSolutions DataHub. Or full articles written by Schatz at that explain why his numbers are what they are even when he has to come to the defense of them:

Other advanced analytics may disagree, but the Football Outsiders DVOA ratings continue to drive the New Orleans Saints bandwagon this week.

Or great work by Vincent Verhei at Football Outsiders, who also must come to the defense of DVOA and DYAR early and often, a common occurrence with many of these sites, including when PFF gives out a grade that most do not agree with:

If you’re a Tampa Bay fan, you probably opened Quick Reads expecting to see Jones’ name near the top of our list of the week’s most valuable running backs, but you won’t find him there. OK, you’re thinking, some other guys had better days catching the football than Jones did (almost all of them, actually), but surely Jones will make the top five in rushing numbers. But no, Jones didn’t make that list either — he finished with 29 rushing DYAR, eighth-best among running backs going into Monday Night Football. What’s going on here?

But there is a difference between not understanding something and not agreeing with something. You should be allowed to disagree without being told that you don’t understand.

Advanced “analytics” has to take a lot of time each week to explain why their numbers are “right” and the rest of the world must be wrong and what I’m finally saying is:

You are not necessarily wrong.

And I don’t believe the Rams have the had eighth-best defense.