[Discussion] is WAR a broken stat?

[The_Rocket]

Seen that some member of the analytics community have been posting historic WAR player cards based on requests from Twitter users. Both burrows and Kesler have been posted. 
 

somehow, burrows was one of the best penalty killers in the league, while Kesler was one of the worst.

 

it’s worth noting that PK WAR only considers DEFENSE. Scoring goals and generating rushes on the PK does not matter for this stat. 
 

how is it possible that Kesler and burrows, who spent most of their PK time TOGETHER are so drastically different?

 

how did burrows defensive game get so awful in 2011, his best year?

 

hiw is Burrows’s penalty differential so good, knowing how many minors he took?

 

unfortunately, these cards get passed around like gospel but I often find they leave far more questions than answers. And the data behind them isn’t easily reviewed, and their compiled by one guy. 
 

is this stat Broken, or am I missing something?

[Arrow 1983]

1 hour ago, The_Rocket said:

Seen that some member of the analytics community have been posting historic WAR player cards based on requests from Twitter users. Both burrows and Kesler have been posted. 
 

somehow, burrows was one of the best penalty killers in the league, while Kesler was one of the worst.

 

it’s worth noting that PK WAR only considers DEFENSE. Scoring goals and generating rushes on the PK does not matter for this stat. 
 

how is it possible that Kesler and burrows, who spent most of their PK time TOGETHER are so drastically different?

 

how did burrows defensive game get so awful in 2011, his best year?

 

hiw is Burrows’s penalty differential so good, knowing how many minors he took?

 

unfortunately, these cards get passed around like gospel but I often find they leave far more questions than answers. And the data behind them isn’t easily reviewed, and their compiled by one guy. 
 

is this stat Broken, or am I missing something?

Any single advance stat can only give a small piece to the Puzzle and in hockey we are talking about a 1000 piece puzzle.

 

with multiple stats and the eye test only then can you see what the bigger picture is.

 

A person just looking at a stat or multiple stats will never know what a player meant to their team like a fan does.

 

And a fan will never understand how good or bad a player truly was if they don't look at the stats.

 

Manny Malhotra is always my example. Look just at his point stats and one would think he did not play a big role on this team. But he freed Kesler up from his defensive duties.

 

Look at Malhotra only as a fan might and you might say he was good on faceoffs and PK but advance stats shows he was far more, than that. His possession time and linemate possession times where among the best in the league probably because he won so many faceoffs. 

Edited August 22, 2021 by Arrow 1983

[shiznak]

Well for one, Kesler had over 50+ more minutes than Burrows on the PK.

 

And two, Burrows primarily played with Malhotra on the PK, not Kesler.

Edited August 22, 2021 by shiznak

[ilduce39]

I’m more nuanced on the issue than my snide comments about them might imply… really breaking down stats using a system you understand in conjunction with a healthy dose of coaches/scouts eye (watching, breaking down and re-watching film - not just a one time live viewing)  is really the best way to evaluate a player. 
 

That said, what do these WAR cards tell us that a casual fan + Burrows/Kesler’s boxcar stats don’t?  Kesler scored 41 goals (72 points - and actually had 75 points the year before.)  He also won a Selke.  Burrows was ~30 goals, ~50 points the past few years too.  They were both in the +20s.  Key players on a strong team.  
 

Thanks Jfresh.  96%.  Sure. Whatever.

 

Edited August 22, 2021 by ilduce39

[King Heffy]

Perfect example of why these analytics dweebs shouldn't be taken seriously.

[IBatch]

The last time i bothered with WAR, was way back when they declared Alex Steen as the best player in the NHL.   Alex ... freaking ... Steen lol.   Ahead of Crosby?  Bergeron?  So yes, to me it's broken and has been for a long time.   I'm surprised it's still even around.   

[JM_]

9 hours ago, Arrow 1983 said:

 

Look at Malhotra only as a fan might and you might say he was good on faceoffs and PK but advance stats shows he was far more, than that. His possession time and linemate possession times where among the best in the league probably because he won so many faceoffs. 

funny thing is, you get stats people saying that face offs don't matter. 

[JM_]

55 minutes ago, King Heffy said:

Perfect example of why these analytics dweebs shouldn't be taken seriously.

in the year Kesler was supposedly really bad on the PK he had a monster differential in takeaways, 4 giveaways, 24 takeaways on the PK. But yeah the little card says he sucked so..

 

 

https://www.naturalstattrick.com/playerreport.php?fromseason=20082009&thruseason=20102011&stype=2&sit=4v5&stdoi=oi&rate=n&v=p&playerid=8470616

[BlastPast]

Within 5-10 years people will look back and cringe at how seriously they took a lot of these publicly available statistics. Some of them more than others.

[grandmaster]

42 minutes ago, IBatch said:

The last time i bothered with WAR, was way back when they declared Alex Steen as the best player in the NHL.   Alex ... freaking ... Steen lol.   Ahead of Crosby?  Bergeron?  So yes, to me it's broken and has been for a long time.   I'm surprised it's still even around.   

Exactly.

 

This WAR thing is stupid.

[JM_]

8 minutes ago, grandmaster said:

Exactly.

 

This WAR thing is stupid.

the funniest part is when you see it used to bash GMs. You see the term "replacement level" in many arguments and it stems from graphs like these. As if Chatfied would replace Myers (yes I've seen that). 

[GB5]

This is what happens when you take some that more or less works in a passive sport like baseball and try to tweak it to make it work in a highly combative action filled sport like hockey.

 

The two are apples and oranges and you will not get a good fit.

[SID.IS.SID.ME.IS.ME]

Huge missed opportunity by JFresh and company not to take some time to dig deeper into those Kesler numbers.

 

The best analysts would welcome a detailed discussion around Kesler’s PK WAR/GAR.

 

After all, when you have a Jack Adams winning coach deploying a Selke winning forward, with the highest PKTOI on the team, and among the highest PKTOI in the league, during that 2008-09 through 2010-11 period, and the model is only returning 2nd percentile results, that’s something worth looking at.

 

Hockey analysts interested in perfecting their models and increasing reliability would relish such an opportunity.
 

Whether it’s proving that the numbers are accurate and meaningful, and thereby demonstrating the value of analytics in teasing out useful data that’s missed by the “eye test,” or finding an error or weakness in the analytics, and using that information to continue to adjust and perfect the model, it’s really a win-win, if the goal is to better understand the game through numbers.

 

Disappointing not to see someone in the analytics community digging into this further, because I think there’s a really fruitful discussion to be had here.

Edited August 22, 2021 by SID.IS.SID.ME.IS.ME

[The_Rocket]

51 minutes ago, SID.IS.SID.ME.IS.ME said:

Huge missed opportunity by JFresh and company not to take some time to dig deeper into those Kesler numbers.

 

The best analysts would welcome a detailed discussion around Kesler’s PK WAR/GAR.

 

After all, when you have a Jack Adams winning coach deploying a Selke winning forward, with the highest PKTOI on the team, and among the highest PKTOI in the league, during that 2008-09 through 2010-11 period, and the model is only returning 2nd percentile results, that’s something worth looking at.

 

Hockey analysts interested in perfecting their models and increasing reliability would relish such an opportunity.
 

Whether it’s proving that the numbers are accurate and meaningful, and thereby demonstrating the value of analytics in teasing out useful data that’s missed by the “eye test,” or finding an error or weakness in the analytics, and using that information to continue to adjust and perfect the model, it’s really a win-win, if the goal is to better understand the game through numbers.

 

Disappointing not to see someone in the analytics community digging into this further, because I think there’s a really fruitful discussion to be had here.

Reminds me of when he posted a Nils Hoglander card a few months ago. Many pointed out the penalty impacts must be wrong. He said he would look into them but never ended up posting anything about why there is a discrepancy. 
 

that’s my only problem with these models. Data are collected, input, and displayed to be informative. However, so much information is lost along the way. Why is there a discrepancy between what we see and what the data indicate? Is it bad data? Input error? Are we missing something when we’re watching? How is an expected goal calculated? How does coaching impact the data and how is it accounted for? How do we know that the data be used are in fact the best metrics for determining how much a player contributes to their team winning?

 

even if these questions were answered, they aren’t answered in the player card that gets shared and retweeted around to make definitive points (by some) on which players are good an which are bad. I think there may be some merit to an argument that these cards cause more harm than good in informing discussions of player value 

[Josepho]

Hockey analytics are still in VERY early development stages and at this point I think it's fair to be skeptical of WAR models. These kinds of things need to be developed for years. 

 

However, any of the shot-based metrics and rate (per 60) stats definitely can be useful tools and shrugging them off is just being oblivious to reality. I also think analytics are probably significantly more accurate for goaltenders than they are for skaters.

Edited August 22, 2021 by Josepho

[Maddogy]

Car A is better than Car B because it's got more horse powers. 

[SID.IS.SID.ME.IS.ME]

1 hour ago, The_Rocket said:

Reminds me of when he posted a Nils Hoglander card a few months ago. Many pointed out the penalty impacts must be wrong. He said he would look into them but never ended up posting anything about why there is a discrepancy. 
 

that’s my only problem with these models. Data are collected, input, and displayed to be informative. However, so much information is lost along the way. Why is there a discrepancy between what we see and what the data indicate? Is it bad data? Input error? Are we missing something when we’re watching? How is an expected goal calculated? How does coaching impact the data and how is it accounted for? How do we know that the data be used are in fact the best metrics for determining how much a player contributes to their team winning?

 

even if these questions were answered, they aren’t answered in the player card that gets shared and retweeted around to make definitive points (by some) on which players are good an which are bad. I think there may be some merit to an argument that these cards cause more harm than good in informing discussions of player value 

Input error seems pretty likely.

 

For Höglander, it’s easy enough to look up his penalty stats, and I’m seeing 6 minors and 12 PIM at 5v5, with 8 penalties drawn, so a positive differential. All situations, he’s even better, with 8 minors taken and 13 penalties drawn.

 

Hard to imagine how a model using the correct data spits out a 3rd percentile result, for a player with a positive penalty differential, both at 5v5 and all situations.

 

Even if you go by per minute rates, Höglander still ranks in around the top third of Canucks, both in limiting PIM/60, and in the rate he’s able to draw penalties.

 

So there really looks to be something wrong with that 3% result.

 

I wouldn’t say this is a reason to “throw the baby out with the bath water,” but it’s definitely something they should look to clean up. A model is only as good as the data you put into it, and seems like Patrick Bacon (JFresh’s source) might have some errors slipping through.

 

And if not, then they really need to explain some of these numbers, and the methodology, because those results, taken at face value, certainly don’t seem to agree with the data you can easily pull off other sources.

Edited August 22, 2021 by SID.IS.SID.ME.IS.ME

[MrCanuck94]

5 hours ago, King Heffy said:

Perfect example of why these analytics dweebs shouldn't be taken seriously.

Or you know, using a balance of both analytics and the eye test to find the most effective conclusion instead of fully discrediting one method would be more credible.

 

As mentioned above by some, analytics is at an early stage in hockey and is progressing. It isn't perfect, but there are many helpful stats that can give vital information to certain aspects of a players game.

[brownky]

Hockey is an analytics-resistant sport.

 

There are too many variables to account for on a given play for the math to add up to a meaningful conclusion. "on averages" when you input the wrong data many, many times, the analytics data will also be flawwed.

 

Hence why different models yield different results with the same data. It's pretty much all made up so people can sell how smart they are to other people.

 

I laugh at anyone who uses WAR or Corsi at this point. There *are* analytics that show some interesting values... but neither of those two are that.

 

So to answer your question, yes, it is, at least in hockey. Other sports perhaps not, but in hockey, yes, it's very broken.

 

 

[IBatch]

2 hours ago, Josepho said:

Hockey analytics are still in VERY early development stages and at this point I think it's fair to be skeptical of WAR models. These kinds of things need to be developed for years. 

 

However, any of the shot-based metrics and rate (per 60) stats definitely can be useful tools and shrugging them off is just being oblivious to reality. I also think analytics are probably significantly more accurate for goaltenders than they are for skaters.

Well...it's being going on since 2011 right?   End of day this isn't baseball.