NBA: Basketball Quotient

[Latrinsorm]

There are lots of "advanced" stats out there, and they all follow the same basic philosophy. We record various things about what basketball players do. Can we combine those things into an overall picture of winning games? If so, how? If a player scores 20 points, what portion of a win is that worth? Does it matter if he does so from the field or the free throw line, or if he does so more or less efficiently than an arbitrary level, and where should we set that arbitrary level?

I have selected four composite stats. I will describe their methodology, investigate their correlation with each other, and finally investigate their correlation with wins by summing the values for each individual player on each team.

.

Win Shares

One I have used often in the past, this method was developed by Justin Kubatko based on work in baseball by Bill James and in basketball by Dean Oliver. Like all four methods, it uses only traditional box score information, so only things like points, field goal attempts, etc. and not ± or play-by-play. Win Shares uses information both about the player and the team he plays on, so it can estimate granular things like how often the player is assisted on his field goals, how well he plays defense beyond steals and blocks, etc. All data is from http://www.basketball-reference.com

VORP Wins

Value Over Replacement Player is a recent addition to http://www.basketball-reference.com developed by Daniel Myers. In itself it isn't a representation of wins, but it can be converted easily by multiplying by 2.7, which is nice. Its direct inputs are rate stats like ORB% and AST% rather than offensive rebounds and assists, but the former are calculated from the latter so the baseline source is the same. It also uses information about the team but only in an overall way, by adjusting player ratings based on team rating. It is important to note that because VORP Wins are over replacement, they are the only metric where we have to adjust each team's summation because we would otherwise find the team's wins over a team of replacement players rather than 0. Empirically I found the value for the wins of The Replacements to be 14, and when I discussed this matter with the stat's creator he confirmed that the number was 14. All data is from http://www.basketball-reference.com

Estimated Wins Added

EWA was developed by John Hollinger using his (notorious) PER framework as a base line. Converting PER to EWA turns out to be really straightforward: subtract off a position-dependent figure, multiply by minutes played, and divide by 67. Thus if we have a point guard and center (for example) with the same PER and minutes played, they will not have the same Estimated Wins Added. PER itself uses no information about the team and therefore neither does EWA. EWA is published on espn.com but they do a pretty sucky job of it and it's really easy to calculate, so all of my team data is from http://www.basketball-reference.com Because it is position-dependent and basketball position is really a matter of opinion, there are probably differences between what my team totals say and what espn's would (if they weren't so sucky).

Wins Produced

This is the only metric developed by academics, which you can see by it being credited to David Berri, Martin Schmidt and Stacey Brook. It is most like Win Shares but it also corrects on a position-dependent basis, and does so even more dramatically than EWA as we will shortly see. All data is from http://www.boxscoregeeks.com/


.

To see how correlated these metrics were with each other, I took (almost) every player from the 2014 season and did a linear regression between each stat. This was a lot harder than it sounds because I used three different websites for player data, each of which used different naming conventions, and only one of which (basketball-reference) gave splits for players who were traded or otherwise played for multiple teams during the year. Also, for whatever reason boxscoregeeks didn't have Chris Johnson (789 minutes for the 76ers) at all.

First off, every metric should result in about 1230 wins because that's how many there are in an NBA season. It turns out that the actual totals for each metric were 1256.1, 1230.4, 1236.6, and 1267.9 so that all worked out pretty well.

Because there are four stats, each stat was involved in three regressions. An R^2 of 1.000 would mean the two stats correlated exactly, 0.000 would mean they were completely uncorrelated. The results:


0.8400 PS
0.8353 PV
0.7977 SV
0.6418 PE
0.8221 SE
0.7335 VE

2.3171 P
2.4598 S
2.3665 V
2.1974 E
Where S = Win Shares, V = VORP Wins, E = Estimated Wins Added, and P = Wins Produced. I was very surprised how good the correlation was given the wildly divergent methodologies, but it suggests that all these methods are measuring something. The best individual correlation was Win Shares to Wins Produced, the best overall correlation was Win Shares, the worst was Estimated Wins Added, and the worst individual was Estimated Wins Added to Wins Produced. This last one is super interesting because they are the two that do adjust for position, which sets them apart from the other two, but as said above the way they do so is not really that similar:

http://img.photobucket.com/albums/v456/johnnyoldschool/NBABQ1_zps1b7fb826.png

The way to read this graph is that 1 is point guard and so on up the line, and that a LOW number means it's HARDER to replace that position, so their production is really worth more: a shooting guard's point is more important, a power forward's rebound is less important. The two stats agree that shooting guard and small forward are tough and power forward is easy, but EWA thinks centers are valuable while WP thinks they're more overrated than the SEC, they flip on point guards, and EWA's adjustments are tiny compared to WP's... and after all that WP still comes out very strongly correlated with WS, which is position-agnostic!

.

We've learned that these four stats take four roads to pretty much the same place. Next post we will look at just what that place is, and if we can smash them together in some way so as to get closer to it.

[Latrinsorm]

We can look at correlation between stats' player wins, but there's no measurement we can make of those player wins directly. To do that we have to condense players onto teams. First I thought it would be interesting to see how each stat looked:

http://img.photobucket.com/albums/v456/johnnyoldschool/NBABQ2_zps59d10896.png

You can see the flaws of PER on full display: EWA hates teams really good at defense but not so much at blocks/steals (Pacers, Bulls) and loves those really good at blocks/steals but not so much at defense (Pelicans, Pistons). We can also see a systematic flaw of VW: "multiply by 2.7" is based on an approximation of the Pythagorean win-loss model that is only accurate near .500. Obviously if you put LeBron and Michael and Wilt and Shaq all on the same team, you can't just add up their individual stats and end up with an 80 win team, there are eventually diminishing returns. This means that the extremes are going to be deflected, but that may yet work in our favor...

We can take each stat and compare it to the actual wins the team had, and a common way to do that is to do root mean squared error. We subtract the actual from the predicted, square it, average that over all 30 teams, then take the square root. The first advantage to this method over just adding up all the errors is that if you get -1 and +1 adding them up gives 0 but you didn't have 0 error. Second, it makes it worse to have +1, +1, +1, +8 than +3, +3, +3, +3 because we don't want to be way off. When we look at each stat alone against wins, we get RMSE of 3.32, 5.02, 7.68, 3.69 for WS, VW, EWA, and WP. That's about what we'd expect given what we've learned so far. But what if we combined the stats to varying degrees? Suppose we had three parts WS, two parts WP, a dollop of VW, and a pinch of EWA? Can we reduce our RMSE with such a combination? As it turns out... yes!

I have found it useful to talk about orders of model. The first order will just be whichever stat had the best RMSE to start with, because if our combination is worse than that then it's clearly pointless. The second order will be adding the best other stat where "best" is defined as "reducing RMSE the most", and so on to the fourth order model. When we do that with 2014 our relative weights are these:


WS VW EWA WP RMSE
1 3.3198
51 10 0 0 3.2344
231 50 9 0 3.2282
2422 520 88 71 3.2280

Obviously the returns in RMSE improvement are rapidly diminishing, but each ingredient we add to the soup does help a little. Much more interesting is that WP turns out to be almost useless! even though on its own it is very competitive with WS.

.

But of course we know that actual wins aren't as predicative of wins as Pythagorean wins are, because close games are governed by random noise. The Pythagorean method is just points scored ^ 14 / (points scored ^ 14 + points allowed ^ 14), and when we compare the stats to that we get:


WS VW EWA WP RMSE
1 1.5470
22 3 0 0 1.2454
197 30 7 0 1.2097
1987 303 66 0 1.2095

Again we get diminishing returns, but they don't diminish as much and our fit is much better to start with, which is quite a double whammy. WP is completely useless here, which is pretty shocking.

.

I have already done a couple more years and these factors did not hold. WS and WP finished 1-2 and 2-1 for 2013 and 2011 against actual wins, so it looks like for whatever reason WP just had a bad year in 2014 and VW a good one. EWA finished least significant both years against both win types, though. What I'm going to do is get a five year sample, see how the fourth order factors go for individual years, then do a regression for all five years at the same time. My guess is that we'll end up with equal parts WS and WP, a little VW, maybe no EWA at all.

[Latrinsorm]

So I finished five years, and the results were pretty interesting. Factors continued to change considerably from year to year and actual to Pythagorean, but in general the relevancy continued to go WP, then WS, then a gap to VW, then a gap to EWA. The average reduction of first order RMSE was 95% for actual wins and 79% for Pythagorean.

When looking at the five year sample as a whole, the weights for actual wins were pretty close to the average weights: 30/25/5/40 to 35/20/5/40. For Pythagorean wins the shift was much more dramatic: 30/10/0/60 to 40/12/3/45. The weighted model or Basketball Quotient improved the RMSE of WP over the entire sample from 3.07 to 2.89 actual wins (94%) and 1.35 to 0.99 Pythagorean wins (73%). The actual wins isn't that impressive, but it is an improvement nonetheless.

I also summed the total number of wins allotted for every year and good thing I did, because there's a very interesting possibility in play...


WS VW EWA WP
1256.1 1230.4 1236.6 1267.9
1255.5 1232.2 1239.0 1233.2
1266.4 1231.6 1242.5 1234.4
1269.7 1232.6 1234.0 1231.0
1265.4 1227.9 1241.3 1233.3

...made more explicit when we subtract off 1230...


WS VW EWA WP
26 0 7 38
25 2 9 3
36 2 13 4
40 3 4 1
35 -2 11 3

WS is very reliably about 30 wins too high, so why don't we subtract off 1 win from each team prediction it makes? VW is quite good and the only metric to ever undershoot 1230 (which is a good sign), EWA is kind of crap anyway, and other than the 2014 debacle WP is pretty close too. I am going to try Adjusted Win Shares and see if it improves the regression, and if so I can figure out how to apportion that -1 win among the players (which is what we ultimately care about). I'm also going to look into whether VW's sterling average can be improved by bending the extreme values towards the middle, but that's a more delicate operation than -1 so it will take longer. With all that finished, it will be possible to calculate BQs for players and find out who really was the best, although as the stats are so well correlated we probably won't learn much new.

[Tgo01]

And people accuse me of afk scripting.

[Latrinsorm]

If not for the recent Troubles I would have reported that post. But my good friend Antie has enough on his plate.

[Tgo01]

If not for the recent Troubles I would have reported that post. But my good friend Antie has enough on his plate.

HMC!!!

[Latrinsorm]

WS - 1 is the truth.



WS * -
3.140 3.001 2.945
1.422 1.079 1.014


* is multiplying all team Win Shares by 123/126, - is simply subtracting 1. As you can see it is clearly the best option by RMSE. As we expect this also improves Basketball Quotient's RMSE and dramatically increases WS' relative weight: from 2.892 to 2.833 actual (35 to 55 weight) and 0.987 to 0.796 pythagorean (40 to 60 weight). The biggest loser in relative weight was WP, but it still maintains a very healthy lead over the other two.

I tweeted the stat's creator to ask him about this and he told me to stop being such a f---ing nerd, which I thought was a little harsh. I kid! He told me that he doesn't maintain the stat and I should talk to these other people, so I'll do that and see how it goes.

[Methais]

There are lots of "advanced" stats out there, and they all follow the same basic philosophy. We record various things about what basketball players do. Can we combine those things into an overall picture of winning games? If so, how? If a player scores 20 points, what portion of a win is that worth? Does it matter if he does so from the field or the free throw line, or if he does so more or less efficiently than an arbitrary level, and where should we set that arbitrary level?

I have selected four composite stats. I will describe their methodology, investigate their correlation with each other, and finally investigate their correlation with wins by summing the values for each individual player on each team.

.

Win Shares

One I have used often in the past, this method was developed by Justin Kubatko based on work in baseball by Bill James and in basketball by Dean Oliver. Like all four methods, it uses only traditional box score information, so only things like points, field goal attempts, etc. and not ± or play-by-play. Win Shares uses information both about the player and the team he plays on, so it can estimate granular things like how often the player is assisted on his field goals, how well he plays defense beyond steals and blocks, etc. All data is from http://www.basketball-reference.com

VORP Wins

Value Over Replacement Player is a recent addition to http://www.basketball-reference.com developed by Daniel Myers. In itself it isn't a representation of wins, but it can be converted easily by multiplying by 2.7, which is nice. Its direct inputs are rate stats like ORB% and AST% rather than offensive rebounds and assists, but the former are calculated from the latter so the baseline source is the same. It also uses information about the team but only in an overall way, by adjusting player ratings based on team rating. It is important to note that because VORP Wins are over replacement, they are the only metric where we have to adjust each team's summation because we would otherwise find the team's wins over a team of replacement players rather than 0. Empirically I found the value for the wins of The Replacements to be 14, and when I discussed this matter with the stat's creator he confirmed that the number was 14. All data is from http://www.basketball-reference.com

Estimated Wins Added

EWA was developed by John Hollinger using his (notorious) PER framework as a base line. Converting PER to EWA turns out to be really straightforward: subtract off a position-dependent figure, multiply by minutes played, and divide by 67. Thus if we have a point guard and center (for example) with the same PER and minutes played, they will not have the same Estimated Wins Added. PER itself uses no information about the team and therefore neither does EWA. EWA is published on espn.com but they do a pretty sucky job of it and it's really easy to calculate, so all of my team data is from http://www.basketball-reference.com Because it is position-dependent and basketball position is really a matter of opinion, there are probably differences between what my team totals say and what espn's would (if they weren't so sucky).

Wins Produced

This is the only metric developed by academics, which you can see by it being credited to David Berri, Martin Schmidt and Stacey Brook. It is most like Win Shares but it also corrects on a position-dependent basis, and does so even more dramatically than EWA as we will shortly see. All data is from http://www.boxscoregeeks.com/


.

To see how correlated these metrics were with each other, I took (almost) every player from the 2014 season and did a linear regression between each stat. This was a lot harder than it sounds because I used three different websites for player data, each of which used different naming conventions, and only one of which (basketball-reference) gave splits for players who were traded or otherwise played for multiple teams during the year. Also, for whatever reason boxscoregeeks didn't have Chris Johnson (789 minutes for the 76ers) at all.

First off, every metric should result in about 1230 wins because that's how many there are in an NBA season. It turns out that the actual totals for each metric were 1256.1, 1230.4, 1236.6, and 1267.9 so that all worked out pretty well.

Because there are four stats, each stat was involved in three regressions. An R^2 of 1.000 would mean the two stats correlated exactly, 0.000 would mean they were completely uncorrelated. The results:


0.8400 PS
0.8353 PV
0.7977 SV
0.6418 PE
0.8221 SE
0.7335 VE

2.3171 P
2.4598 S
2.3665 V
2.1974 E
Where S = Win Shares, V = VORP Wins, E = Estimated Wins Added, and P = Wins Produced. I was very surprised how good the correlation was given the wildly divergent methodologies, but it suggests that all these methods are measuring something. The best individual correlation was Win Shares to Wins Produced, the best overall correlation was Win Shares, the worst was Estimated Wins Added, and the worst individual was Estimated Wins Added to Wins Produced. This last one is super interesting because they are the two that do adjust for position, which sets them apart from the other two, but as said above the way they do so is not really that similar:

http://img.photobucket.com/albums/v456/johnnyoldschool/NBABQ1_zps1b7fb826.png

The way to read this graph is that 1 is point guard and so on up the line, and that a LOW number means it's HARDER to replace that position, so their production is really worth more: a shooting guard's point is more important, a power forward's rebound is less important. The two stats agree that shooting guard and small forward are tough and power forward is easy, but EWA thinks centers are valuable while WP thinks they're more overrated than the SEC, they flip on point guards, and EWA's adjustments are tiny compared to WP's... and after all that WP still comes out very strongly correlated with WS, which is position-agnostic!

.

We've learned that these four stats take four roads to pretty much the same place. Next post we will look at just what that place is, and if we can smash them together in some way so as to get closer to it.


We can look at correlation between stats' player wins, but there's no measurement we can make of those player wins directly. To do that we have to condense players onto teams. First I thought it would be interesting to see how each stat looked:

http://img.photobucket.com/albums/v456/johnnyoldschool/NBABQ2_zps59d10896.png

You can see the flaws of PER on full display: EWA hates teams really good at defense but not so much at blocks/steals (Pacers, Bulls) and loves those really good at blocks/steals but not so much at defense (Pelicans, Pistons). We can also see a systematic flaw of VW: "multiply by 2.7" is based on an approximation of the Pythagorean win-loss model that is only accurate near .500. Obviously if you put LeBron and Michael and Wilt and Shaq all on the same team, you can't just add up their individual stats and end up with an 80 win team, there are eventually diminishing returns. This means that the extremes are going to be deflected, but that may yet work in our favor...

We can take each stat and compare it to the actual wins the team had, and a common way to do that is to do root mean squared error. We subtract the actual from the predicted, square it, average that over all 30 teams, then take the square root. The first advantage to this method over just adding up all the errors is that if you get -1 and +1 adding them up gives 0 but you didn't have 0 error. Second, it makes it worse to have +1, +1, +1, +8 than +3, +3, +3, +3 because we don't want to be way off. When we look at each stat alone against wins, we get RMSE of 3.32, 5.02, 7.68, 3.69 for WS, VW, EWA, and WP. That's about what we'd expect given what we've learned so far. But what if we combined the stats to varying degrees? Suppose we had three parts WS, two parts WP, a dollop of VW, and a pinch of EWA? Can we reduce our RMSE with such a combination? As it turns out... yes!

I have found it useful to talk about orders of model. The first order will just be whichever stat had the best RMSE to start with, because if our combination is worse than that then it's clearly pointless. The second order will be adding the best other stat where "best" is defined as "reducing RMSE the most", and so on to the fourth order model. When we do that with 2014 our relative weights are these:


WS VW EWA WP RMSE
1 3.3198
51 10 0 0 3.2344
231 50 9 0 3.2282
2422 520 88 71 3.2280

Obviously the returns in RMSE improvement are rapidly diminishing, but each ingredient we add to the soup does help a little. Much more interesting is that WP turns out to be almost useless! even though on its own it is very competitive with WS.

.

But of course we know that actual wins aren't as predicative of wins as Pythagorean wins are, because close games are governed by random noise. The Pythagorean method is just points scored ^ 14 / (points scored ^ 14 + points allowed ^ 14), and when we compare the stats to that we get:


WS VW EWA WP RMSE
1 1.5470
22 3 0 0 1.2454
197 30 7 0 1.2097
1987 303 66 0 1.2095

Again we get diminishing returns, but they don't diminish as much and our fit is much better to start with, which is quite a double whammy. WP is completely useless here, which is pretty shocking.

.

I have already done a couple more years and these factors did not hold. WS and WP finished 1-2 and 2-1 for 2013 and 2011 against actual wins, so it looks like for whatever reason WP just had a bad year in 2014 and VW a good one. EWA finished least significant both years against both win types, though. What I'm going to do is get a five year sample, see how the fourth order factors go for individual years, then do a regression for all five years at the same time. My guess is that we'll end up with equal parts WS and WP, a little VW, maybe no EWA at all.

Cool.

[Latrinsorm]

I heard back from Mike at sports reference, who said WS is supposed to approximate team wins and left it at that, so I'm confident in my better approximation. I've done BQ leaderboards out to 2005 (the first NBA year with 30 teams and conveniently ten years) by doing 6 * (WS - .1) + 3 * WP + VW and there are some interesting results. I had worried that a method so heavily weighted towards Win Shares would simply duplicate the Win Shares leaderboards, but that does not turn out to be the case as we can see by comparing them...


Year BQ 1 WS 1 BQ 2 WS 2 BQ3 WS 3
2005 Garnett LeBron Dirk Dirk Amar'e
2006 LeBron Dirk Garnett LeBron Dirk Billups
2007 Dirk LeBron Nash Duncan
2008 Paul LeBron Billups Amar'e
2009 LeBron Paul Wade
2010 LeBron Durant Dwight
2011 LeBron Dwight Gasol Paul Dwight
2012 LeBron Paul Durant
2013 LeBron Durant Paul
2014 Durant LeBron Love
9 out of the 30 are different, although some of these are just order changes. By astonishing coincidence a metric I invented says LeBron is underrated. Who saw that coming? :D 2006 is the most interesting year not just because all three top spots change but because the BQ leader LeBron leads only in VW, the least important stat of the three. It so happens that WS leader Dirk was quite poor in VW and WP, and WP leader Garnett wasn't much better in WS and VW, so LeBron's one bests and two goods was arithmetically the best. It is therefore obvious that a player who leads VW and WP can overtake the WS leader, which is a good result.

It turns out that 200 is a pretty magic number. So far only LeBron and Paul have reached it: LeBron exactly 200 in 2010, Paul 204.5 in 2009 (the second best season of the last ten years) but LeBron 216.9 in the same year, wamp wamp.

As demonstrated earlier the stats are pretty well correlated, though, and five players in the sample won the triple: 05 Garnett, 10 12 13 LeBron, 14 Durant. I imagine it'll be tough sledding for both of these until we get to the Jordan years though. I'm going to continue with the regression for these years and into the sub-30 team years, hopefully the totals will go as 41 * team + team instead of 41 * team + 30.

[Methais]

I heard back from Mike at sports reference, who said WS is supposed to approximate team wins and left it at that, so I'm confident in my better approximation. I've done BQ leaderboards out to 2005 (the first NBA year with 30 teams and conveniently ten years) by doing 6 * (WS - .1) + 3 * WP + VW and there are some interesting results. I had worried that a method so heavily weighted towards Win Shares would simply duplicate the Win Shares leaderboards, but that does not turn out to be the case as we can see by comparing them...


Year BQ 1 WS 1 BQ 2 WS 2 BQ3 WS 3
2005 Garnett LeBron Dirk Dirk Amar'e
2006 LeBron Dirk Garnett LeBron Dirk Billups
2007 Dirk LeBron Nash Duncan
2008 Paul LeBron Billups Amar'e
2009 LeBron Paul Wade
2010 LeBron Durant Dwight
2011 LeBron Dwight Gasol Paul Dwight
2012 LeBron Paul Durant
2013 LeBron Durant Paul
2014 Durant LeBron Love
9 out of the 30 are different, although some of these are just order changes. By astonishing coincidence a metric I invented says LeBron is underrated. Who saw that coming? :D 2006 is the most interesting year not just because all three top spots change but because the BQ leader LeBron leads only in VW, the least important stat of the three. It so happens that WS leader Dirk was quite poor in VW and WP, and WP leader Garnett wasn't much better in WS and VW, so LeBron's one bests and two goods was arithmetically the best. It is therefore obvious that a player who leads VW and WP can overtake the WS leader, which is a good result.

It turns out that 200 is a pretty magic number. So far only LeBron and Paul have reached it: LeBron exactly 200 in 2010, Paul 204.5 in 2009 (the second best season of the last ten years) but LeBron 216.9 in the same year, wamp wamp.

As demonstrated earlier the stats are pretty well correlated, though, and five players in the sample won the triple: 05 Garnett, 10 12 13 LeBron, 14 Durant. I imagine it'll be tough sledding for both of these until we get to the Jordan years though. I'm going to continue with the regression for these years and into the sub-30 team years, hopefully the totals will go as 41 * team + team instead of 41 * team + 30.


http://youtu.be/RmwqnqL3Hbg

[Latrinsorm]

It turns out there are some bugs in the boxscoregeeks.com database that I've contacted them about, but I can't really proceed on the correlation analysis until they get back to me. Instead, I decided to find all the 200 seasons in history (78-present), and here they are:

90 208.6 Barkley - 53 wins, 2nd in MVP voting (but most 1st place votes), lost in 2nd round
94 202.3 Robinson - 55 wins, 2nd in MVP, lost in 1st round
09 205.1 Paul - 49 wins, 5th in MVP (smh), lost in 1st round

09 217.5 LeBron - 66 wins, MVP, lost in ECF
10 200.6 LeBron - 61 wins, MVP, lost in 2nd round

88 226.2 Jordan - 50 wins, MVP, lost in 2nd round
89 228.5 Jordan - 47 wins, 2nd in MVP, lost in ECF
90 205.4 Jordan - 55 wins, 3rd in MVP, lost in ECF
91 213.0 Jordan - 61 wins, MVP, Champion

200 is so hard to get that even if you use PEAK seasons (a player's best WS, best VW, best WP) nobody else gets to 200. Durant, Shaq, Garnett, Magic, Bird, nobody. Kobe's peaks (2006 for 15.3 WS, 2003 for 18.4 VW and 13.0 WP) only work out to 149.2, a total of one playoff round won, a 3rd and 4th place MVP finish. Also please note that the boxscoregeeks bug I referred to earlier INFLATES wins produced, so if anything it makes the case better for people than it would otherwise be.

4 Jordan
2 LeBron
Barkley, Robinson, Paul
end of file