The new NET (and, yes – I’m absolutely positive they came up with the acronym first and worked backwards to get to what it should stand for) ranking system implemented by the NCAA for this year was supposed to use game results, strength of schedule, game location, scoring margin, net offensive and defensive efficiency, and the quality of wins and losses to rank teams.
“Great,” everyone said, “the RPI is a horribly flawed tool and the NCAA has (publicly) expressed an unwillingness to rely on other, proprietary metrics, so we’re excited to see some changes for the better. Anything that will improve upon what teams are selected and how they are seeded for the NCAA tournament, and provides a transparent method for how teams are being judged, will be a fantastic change.”
Now that the season is a few weeks old, the NCAA released the first NET rankings, and, well…here’s your Top 10:
Some accompanying information from the NCAA’s site about the initial rankings: “You’ll see a lot of differences between this ranking and the AP Top 25. For example, AP No. 1 Gonzaga is at No. 5 here, and AP No. 2 Kansas is at No. 11. These discrepancies are pretty understandable, as the ranking systems take a very different approach. The AP ranking is compiled from the individual votes of 65 sportswriters and broadcasters. The NET ranking, however, is based entirely on statistics, and is a representation of how well a team has played up to this point. It takes no predicted trends or results into account.”
Reading this, I can’t help but think the NCAA knew there might be some blowback, and this was their way of trying to acknowledge and mitigate it. I think they severely underestimated the response that was coming, however, as it seems that most people…well…hate it.
From the Twitter feed of respected analytics guru Nate Silver, of FiveThirtyEight.com:
“These are the worst rankings I’ve ever seen in any sport, ever. NCAA needs to go completely back to the drawing board. This is incompetently designed. It’s worse than RPI. Worse than the eye test. It could make a total shitshow of the NCAA tourney for a couple years and cost the NCAA millions by devaluing its most valuable franchise. It’s why you never want to design an algorithm by committee.”
Well, then. Can’t say I like seeing the word “shitshow”. Is it too late to bring the RPI back, or did we already put a bullet in its head out behind the barn?
Let’s set aside the fact that if anyone had half a brain and looked at the Top 10 and saw Ohio State as #1 and Loyola Frickin’ Marymount as the #10 team, they should have raised their hand and asked the rest of the team to revisit the algorithm – or at least delayed the release of the rankings for a bit while it was re-evaluated. If for nothing else, even if the algorithm & methodology are perfect – and knowing how the NCAA will do the wrong thing every. Single. Time. it wasn’t going to be – you’d think they would have thought more about the PR aspect of the rankings and adjusted their accompanying commentary appropriately.
Obviously any “objective” metrics are going to differ from the AP Top 25. I can even understand how the NET and commonly used metrics such as KenPom, Sagarin, ESPN’s BPI and others can differ significantly – they have a slightly different emphasis, weight things differently, etc. But these four metrics (NET, KenPom, Sagarin, BPI) incorporate many of the same dimensions for measurement: wins and losses, performance at home versus on the road, quality of the opponent, and offensive & defensive efficiency. So they should all fall somewhat in line, right?
Um…in a word, no.
Here is a table that lists all AP and NET top 25 teams and their respective rankings; their KenPom, Sagarin and BPI rankings; and a handful of other teams (all rankings as of November 27):
|North Carolina State||–||31||35||21||14||-25%|
For fun, you’ll notice I also included a column for “NET Variance”. I took the average ranking from KenPom, Sagarin and BPI, and calculated the percentage difference from the NET ranking. There are some real, eye-popping variances.
Again, let’s forget the “eye test” of Loyola Marymount being in the Top 10 to begin with – their average KenPom, Sagarin and BPI ranking is 131. 131! I know this method is hardly complete or scientific, and not something Nate Silver would probably approve of using in any sort of “clinical” environment, but if I worked at the NCAA I would have gone through a similar exercise as a sanity check. I would have then seen the radical differences between my brand-spankin’-new metric and the well-established & trusted ones and at least thought to myself that I ought to see if a 1 wasn’t carried somewhere, or something.
NET apologists predictably said things like, “It’s a new metric, it will work out as the season goes along, don’t get too hung up on the current rankings.” Don’t get too hung up on the current rankings?! Then why release it?! Why all the talk about how much better things will be by using this calculation to rank teams?! Why have a splashy graphic and press releases about the NET Top 10?! I’m sorry, but if your shiny new toy – a toy that can literally cost schools millions and coaches their jobs depending on which side of the metric they fall – has a Top 25 that includes Loyola Marymount, Belmont, Redford and Pittsburg, it’s not an effective tool. Forget about OSU being #1, KU being #11, and some other, eyebrow-raising rankings of acknowledged & consensus “good” teams that you might be able to explain via emphasis of the data and calculations: In no way, shape or form should those four schools sniff the Top 25.
From Deadspin: “The quickest glance at Kenpom shows how out of wack this evaluation is. Ohio State (26th in Kenpom) has two impressive wins and four cupcake wins, but is somehow considered better than Maui Invitational winners Gonzaga or the behemoth they defeated in the final, Duke. Kansas—who’s undefeated with wins over Michigan State, Marquette, and Tennessee—is somehow a spot below Loyola Marymount, whose best win is against lowly Georgetown. Belmont even sneaks in at number 12, over teams like Auburn and North Carolina, thanks to victories against fearsome schools like Lipscomb and, uh, Kennesaw State.”
So essentially, the problem seems to be that strength of schedule isn’t factored correctly in the NET calculation and that teams with good records from playing weak teams are rewarded. This doesn’t seem like something that will get worked out as the season continues to move forward. If anything, it seems that the problem could be compounded by teams in weaker conferences.
From Matt Norlander of CBS Sports: “Another lingering issue is adjusted winning percentage, which seems to have overlap in the overall formula. Teams are getting double credit for its wins and losses, and taking some of the data accrued in the team value index’s criteria. There is also the issue of the 1.4/1.0/0.6 weighted values and how those can’t possibly represent difficulty depending on specific venues (specifically on the road). Bringing even more concern into adjusted winning percentage is the fact that quality of opponent isn’t accounted for.”
Oh. So not only is the strength of schedule not accounted for correctly, but performance against weak teams can count for double? Great (he said with great sarcasm), I can’t wait to watch this year’s tournament – I’m really looking forward to mediocre teams getting in when they shouldn’t and being rewarded for their weak schedules…