The NET is Nuts

Ds72JOCWwAUvPIK.jpg

The new NET (and, yes – I’m absolutely positive they came up with the acronym first and worked backwards to get to what it should stand for) ranking system implemented by the NCAA for this year was supposed to use game results, strength of schedule, game location, scoring margin, net offensive and defensive efficiency, and the quality of wins and losses to rank teams.

“Great,” everyone said, “the RPI is a horribly flawed tool and the NCAA has (publicly) expressed an unwillingness to rely on other, proprietary metrics, so we’re excited to see some changes for the better. Anything that will improve upon what teams are selected and how they are seeded for the NCAA tournament, and provides a transparent method for how teams are being judged, will be a fantastic change.”

Now that the season is a few weeks old, the NCAA released the first NET rankings, and, well…here’s your Top 10:

Ds8ejgRWoAIhxX7.jpg

Some accompanying information from the NCAA’s site about the initial rankings: “You’ll see a lot of differences between this ranking and the AP Top 25. For example, AP No. 1 Gonzaga is at No. 5 here, and AP No. 2 Kansas is at No. 11. These discrepancies are pretty understandable, as the ranking systems take a very different approach. The AP ranking is compiled from the individual votes of 65 sportswriters and broadcasters. The NET ranking, however, is based entirely on statistics, and is a representation of how well a team has played up to this point. It takes no predicted trends or results into account.”

Reading this, I can’t help but think the NCAA knew there might be some blowback, and this was their way of trying to acknowledge and mitigate it. I think they severely underestimated the response that was coming, however, as it seems that most people…well…hate it.

From the Twitter feed of respected analytics guru Nate Silver, of FiveThirtyEight.com:

“These are the worst rankings I’ve ever seen in any sport, ever. NCAA needs to go completely back to the drawing board. This is incompetently designed. It’s worse than RPI. Worse than the eye test. It could make a total shitshow of the NCAA tourney for a couple years and cost the NCAA millions by devaluing its most valuable franchise. It’s why you never want to design an algorithm by committee.”

Well, then. Can’t say I like seeing the word “shitshow”. Is it too late to bring the RPI back, or did we already put a bullet in its head out behind the barn?

Let’s set aside the fact that if anyone had half a brain and looked at the Top 10 and saw Ohio State as #1 and Loyola Frickin’ Marymount as the #10 team, they should have raised their hand and asked the rest of the team to revisit the algorithm – or at least delayed the release of the rankings for a bit while it was re-evaluated. If for nothing else, even if the algorithm & methodology are perfect – and knowing how the NCAA will do the wrong thing every. Single. Time. it wasn’t going to be – you’d think they would have thought more about the PR aspect of the rankings and adjusted their accompanying commentary appropriately.

Obviously any “objective” metrics are going to differ from the AP Top 25. I can even understand how the NET and commonly used metrics such as KenPom, Sagarin, ESPN’s BPI and others can differ significantly – they have a slightly different emphasis, weight things differently, etc. But these four metrics (NET, KenPom, Sagarin, BPI) incorporate many of the same dimensions for measurement: wins and losses, performance at home versus on the road, quality of the opponent, and offensive & defensive efficiency. So they should all fall somewhat in line, right?

Um…in a word, no.

Here is a table that lists all AP and NET top 25 teams and their respective rankings; their KenPom, Sagarin and BPI rankings; and a handful of other teams (all rankings as of November 27):

School AP NET KenPom Sagarin BPI NET Var.
Ohio State 16 1 23 13 17 1667%
Virginia 4 2 3 7 2 100%
Texas Tech 20 3 11 8 12 244%
Michigan 7 4 7 5 20 167%
Gonzaga 1 5 5 2 3 -33%
Duke 3 6 1 1 5 -61%
Michigan State 9 7 8 3 8 -10%
Wisconsin 22 8 12 16 7 46%
Virginia Tech 13 9 16 9 9 26%
Loyola Marymount 10 120 114 159 1210%
Kansas 2 11 2 6 4 -64%
Belmont 12 73 44 78 442%
Nevada 5 13 6 17 11 -13%
Nebraska 14 24 10 28 48%
Iowa 14 15 32 41 31 131%
Auburn 8 16 9 11 10 -38%
Maryland 24 17 33 35 46 124%
Houston 18 37 32 16 57%
Notre Dame 19 52 49 52 168%
Purdue 19 20 13 12 18 -28%
North Carolina 11 21 4 4 1 -86%
Radford 22 131 97 142 461%
Pittsburgh 23 118 96 164 448%
Kansas State 12 24 18 24 19 -15%
San Francisco 25 74 30 65 125%
Tennessee 6 27 10 15 6 -62%
Buffalo 21 30 40 34 35 21%
North Carolina State 31 35 21 14 -25%
Iowa State 32 17 18 21 -42%
Indiana 34 21 14 27 -39%
Texas 17 35 25 26 37 -16%
Villanova 23 38 15 20 13 -58%
Oregon 18 39 31 42 41 -3%
Florida State 15 44 14 19 15 -64%
Butler 45 26 22 29 -43%
Cincinnati 46 39 25 38 -26%
Central Florida 49 43 23 44 -25%
Kentucky 10 61 19 47 24 -51%
Mississippi State 25 66 28 37 34 -50%
Florida 80 30 38 23 -62%
Marquette 99 36 28 22 -71%
West Virginia 141 27 55 25 -75%

For fun, you’ll notice I also included a column for “NET Variance”. I took the average ranking from KenPom, Sagarin and BPI, and calculated the percentage difference from the NET ranking. There are some real, eye-popping variances.

Again, let’s forget the “eye test” of Loyola Marymount being in the Top 10 to begin with – their average KenPom, Sagarin and BPI ranking is 131. 131! I know this method is hardly complete or scientific, and not something Nate Silver would probably approve of using in any sort of “clinical” environment, but if I worked at the NCAA I would have gone through a similar exercise as a sanity check. I would have then seen the radical differences between my brand-spankin’-new metric and the well-established & trusted ones and at least thought to myself that I ought to see if a 1 wasn’t carried somewhere, or something.

NET apologists predictably said things like, “It’s a new metric, it will work out as the season goes along, don’t get too hung up on the current rankings.” Don’t get too hung up on the current rankings?! Then why release it?! Why all the talk about how much better things will be by using this calculation to rank teams?! Why have a splashy graphic and press releases about the NET Top 10?! I’m sorry, but if your shiny new toy – a toy that can literally cost schools millions and coaches their jobs depending on which side of the metric they fall – has a Top 25 that includes Loyola Marymount, Belmont, Redford and Pittsburg, it’s not an effective tool. Forget about OSU being #1, KU being #11, and some other, eyebrow-raising rankings of acknowledged & consensus “good” teams that you might be able to explain via emphasis of the data and calculations: In no way, shape or form should those four schools sniff the Top 25.

From Deadspin: “The quickest glance at Kenpom shows how out of wack this evaluation is. Ohio State (26th in Kenpom) has two impressive wins and four cupcake wins, but is somehow considered better than Maui Invitational winners Gonzaga or the behemoth they defeated in the final, Duke. Kansas—who’s undefeated with wins over Michigan State, Marquette, and Tennessee—is somehow a spot below Loyola Marymount, whose best win is against lowly Georgetown. Belmont even sneaks in at number 12, over teams like Auburn and North Carolina, thanks to victories against fearsome schools like Lipscomb and, uh, Kennesaw State.”

So essentially, the problem seems to be that strength of schedule isn’t factored correctly in the NET calculation and that teams with good records from playing weak teams are rewarded. This doesn’t seem like something that will get worked out as the season continues to move forward. If anything, it seems that the problem could be compounded by teams in weaker conferences.

From Matt Norlander of CBS Sports: “Another lingering issue is adjusted winning percentage, which seems to have overlap in the overall formula. Teams are getting double credit for its wins and losses, and taking some of the data accrued in the team value index’s criteria. There is also the issue of the 1.4/1.0/0.6 weighted values and how those can’t possibly represent difficulty depending on specific venues (specifically on the road). Bringing even more concern into adjusted winning percentage is the fact that quality of opponent isn’t accounted for.”

Oh. So not only is the strength of schedule not accounted for correctly, but performance against weak teams can count for double? Great (he said with great sarcasm), I can’t wait to watch this year’s tournament – I’m really looking forward to mediocre teams getting in when they shouldn’t and being rewarded for their weak schedules…

Fail.png

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s