Evaluating Canuck Players using Expected Goals Above Replacement

[Canuck Surfer]

The best thing to draw from stats is just that. They are stats.

 

Not conclusions.  It is the danger of stats like Gar / plus-minus / Corsi.  If a Gar rating is positive or negative? It should just then point to studying the underlying metrics that drive the rating.

 

How many times a guy wins a puck. How many times he gets intercepted throwing a cross ice pass. Zone exits, pass completion rates are all more useful to me. As is the observation when the stat was recorded.

[iceman64]

18 hours ago, JamesB said:

It has been a slow month for Canuck news and I was looking for some excitement on a Saturday night so I decided to review Canuck player performance last year using the "expected goals above replacement" (xGAR) metric--which I think is the best metric.  In previous posts I have suggested that expected goals for and expected goals against are the best simple measures of performance, and I still think that is true. But the very best measures, in my view,  are complicated measures, particularly xGAR calculated by Evolving Hockey, which puts together PP, PK, and even strength performance and includes many factors in addition to  goals for and goals against, such as faceoffs, giveaways, penalties taken and drawn, hits, etc.  The weights on the different factors are not just someone's best guess but are based on regression analysis to infer the relative importance of these various factors. 

 

Last year there were 23 Canuck skaters who played more than 200 minutes. Here are the top 5 and bottom 5:

 

		xGAR
1	Elias Pettersson	14.9
2	Conor Garland	14.0
3	Quinn Hughes	14.0
4	J.T. Miller	12.2
5	Bo Horvat	10.1
19	Luke Schenn	-1.1
20	Juho Lammikko	-1.8
21	Kyle Burroughs	-2.8
22	Jason Dickinson	-4.4
23	Tyler Myers	-7.3

 

 

I have highlighted some outcomes that I think are very interesting and that explain part of what we have seen this summer.

 

There is a big drop-off after Horvat at #5. The names are the top are not a surprise, but the ordering is. I don't know how many times I have heard commentators refer to JT Miller at "the Canucks' best forward" or as "the Canucks best skater". But, according to this metric, the skater who has the best season last year was Petey. How is that possible?

 

The most important reason why Petey has a better xGAR than Miller is penalties. Last year Petey took 5 minor penalties and drew 29. Miller was 21 and 21. It turns out this is very important for expected scoring, which should not be a surprise as scoring rates on the PP are much higher that at even strength This is not just someone's opinion. It is based on careful statistical analysis.  This stat is kind of like walks in baseball: it is very important for scoring but does not get a lot of credit. Petey also has a much more favorable giveaway/takeaway ratio than Miller. In addition, this is based on expected goals, not actual goals. Miller had a great year for actual goals scored while he was on the ice but part of that was good luck (and that partially explains why he had a "career year" that he is unlikely to replicate).  Petey was also significantly better than Miller on the PK. In the analysis, these factors are much more important than Miller's edge in hits and slight edge in faceoff percentage.  

 

That said, JT Miller is a very good player and his xGAR of 12.2 is excellent, but he is not as good as a superficial look at his scoring numbers from last year would indicate. This might be a partial explanation for why the offers for Miller have not been as good as many people had hoped. (I would have taken the Ranger offer at the deadline.) General perceptions about Horvat and Hughes are probably about right but Garland tends to get underestimated. In terms of value per cap hit dollar (and that is what counts if you are trying to win a Cup) he was arguably the most valuable guy on the team after Miller last year. Calls to trade him to "clear cap space" make no sense. 

 

What about the bottom part of the list. This number is based on net goals expected relative to what would happen if a "replacement level" payer played instead. This is, by definition, a player just on the NHL/AHL margin earning about the league minimum.  So negative numbers are very bad. Any player earning significantly above the league minimum should have a positive number.

 

Dickinson's number is very bad, but Myers' xGAR was outrageously bad. However, his actual on-ice numbers were that bad,  What is going on. And didn't Schenn have a good year? How can his number be negative.  Part of the answer is that both Myers and Schenn were helped a lot by their regular D partners (OEL and Hughes). But the statistical methods can tease out the relative contributions of OEL v. Myers and Hughes v. Schenn. 

 

I have no problem with Schenn. His intangibles are worth something and he is only earning about the NHL minimum. And considering that he played top 4 last year, he was actually pretty good value. Myers, on the other hand is terrible value. This is probably why the Canucks have had trouble trading him given their determination not to give away assets to facilitate a trade  

 

On the good news front, the additions--Mikheyev, Lazar, and Joshua look like excellent value and Kuzmenko shoiuld also be very good. And, although the D "needs work" to quote JR, there is reason for some optimism with Rathbone, Dermott and maybe even Poolman. And even Myers would probably be okay if he were on in a third pairing role (although you hate to pay 6 million to a guy on your 3rd pairing).  

 

 

I'll keep adding to this but your mistaken on Petey over JTM. First the goals that JTM scored over Petey were GWG's. 

 JTM is also not afraid to mix it up and throw punches if he has to. 

 Plus JTM has the better slapshot by a wide margin, think it winw on Petey on wrist shot and I'm happy to give that but JTM is the better player because of intangibles Petey clearly doesn't have. 

 And when it comes to leadership abilities, JTM there too. 

 Nothing against Petey of course, the kid has a lot in his arsenal. 

 I still say I wouldn't go saying we're done though, we have trade bait that could afford us a upgrade or 2 but this team is way better than last year that went on 2 longer sized winning streaks. 

 The new guys are here to help make the team D better and that alone will help us immensely since we won't have to rely on D so much that helps us there whether we upgrade there or not. 

 That's not even counting Kuz, that guy is going to make quite the splash, albeit being older he hit Petey status in Europe, matched Petey's points and that takes a boatload of skill and he's a bigger version of Petey that helps even more. 

 Personally I can hardly wait for puck drop, googling "Canucks news" a few times a day is getting old... 

All I can is the coaching staff in Abby better be ready to work the kids hard to get them up to NHL speed because you know there will be injuries, fact of the game. 

Still short on depth and we have to work hard to get caught up on that before we can make a serious run at anything.

 We all know how this goes if we don't.. 

 Crosses my fingers for dark horse number 1 for me, Mr Klim... Hoping Podz and Kuz will help inspire him to greatness. 

 We could really use it. 

  

[D-Money]

If GAR gave Myers an extra -5 for dumping the puck to change in 3-on-3 overtime, that might explain the low rating…and I’m in complete agreement. 

[JM_]

14 hours ago, JamesB said:

In the second grouping you don't have OEL. Are you assuming he gets traded or maybe that he and Poolman play as the top shutdown pairing and these other three pairings given different options, or was dropping OEL just an oversight?

nope just a brain cramp 

[JM_]

13 hours ago, Viper007 said:

What happened To OEL if we pick up De Haan?

I happened. 

[kloubek]

No surprise there about Myers. He just doesn't really pass the eye test many games. Dickinson never really did pass the same test since he arrived, but I'm hopeful if we keep him that we'll see an improvement.

I think Schenn gets a pass, considering the competition he was put up against and what he otherwise brings to the lineup. Lammiko is gone, and Burroughs isn't likely going to be with the big club to start and will be used more for injury fill-ins.

So really, the only one that concerns me is Myers. But without a suitable replacement, I just don't see us moving on from him in any way.

[SID.IS.SID.ME.IS.ME]

16 hours ago, VancouverHabitant said:

There is nothing in those links that @SID.IS.SID.ME.IS.ME provided that talks about quality of competition.  

 

If I missed it, feel free to point it out, I did a quick Ctrl+F search through the documents. 

I don’t think Evolving Hockey’s model includes quality of competition.

 

But IMO this isn’t a weakness.

 

Firstly, QualComp is difficult to measure. At least in terms of creating an adjustment that would work well in this type of model. Current methods of measuring QoC are considered “problematic” by most analytics people. And it’s quite likely that including such a factor would actually make the model less accurate.

 

Secondly, while QoC is important, it’s not that important. There’s been a fair amount of work to suggest the the overall variance between players is not that significant, especially when comparing similar player types, eg. top-4 defencemen.

Generally, when comparing players within the same class, you’ll only find maybe something around a +/- 1 minute difference in the QoC TOI of their competition. And the vast majority of players with similar roles will have very similar QoC.

 

So you might get into a little trouble if you start comparing a fringe defenceman to a #1D, as their QoC will be significantly different, and this will have an effect (and should also effect how you read all their stats).

 

But in most cases, we’re looking to compare players in similar roles, and in the vast majority of those comparisons, QualComp is close to a non-factor. Certainly not enough to explain away any huge differences in GAR/xGAR etc.

 

Also, in developing and testing their models, the people behind GAR/WAR etc found that any effects of QoC were generally “washed out,” and also that the TOI% variable (which they do include) operated as a proxy for QoC:

 

“We do not specifically include a measure of Quality of Competition in our WAR model,” said the Younggrens. “This is for many reasons, but we feel the current methods are somewhat problematic. Also, the specific technique that we used - a statistical plus-minus approach - requires a long-term dataset (we used 11 years) for training the models. Basically, at that point, Quality of Competition effects have been significantly washed out.”

“We do include a TOI% variable (percentage of available minutes a player played),” they added. “This is similar to basketball’s Box Plus-Minus and can be interpreted as a proxy for Quality of Competition, we think. That variable was very significant in most of the models”

https://www.vancouverisawesome.com/courier-archive/news/why-quality-of-competition-doesnt-matter-to-analytics-experts-anymore-3082197

[Guest]

39 minutes ago, SID.IS.SID.ME.IS.ME said:

I don’t think Evolving Hockey’s model includes quality of competition.

 

But IMO this isn’t a weakness.

 

Firstly, QualComp is difficult to measure. At least in terms of creating an adjustment that would work well in this type of model. Current methods of measuring QoC are considered “problematic” by most analytics people. And it’s quite likely that including such a factor would actually make the model less accurate.

 

Secondly, while QoC is important, it’s not that important. There’s been a fair amount of work to suggest the the overall variance between players is not that significant, especially when comparing similar player types, eg. top-4 defencemen.

Generally, when comparing players within the same class, you’ll only find maybe something around a +/- 1 minute difference in the QoC TOI of their competition. And the vast majority of players with similar roles will have very similar QoC.

 

So you might get into a little trouble if you start comparing a fringe defenceman to a #1D, as their QoC will be significantly different, and this will have an effect (and should also effect how you read all their stats).

 

But in most cases, we’re looking to compare players in similar roles, and in the vast majority of those comparisons, QualComp is close to a non-factor. Certainly not enough to explain away any huge differences in GAR/xGAR etc.

 

Also, in developing and testing their models, the people behind GAR/WAR etc found that any effects of QoC were generally “washed out,” and also that the TOI% variable (which they do include) operated as a proxy for QoC:

 

“We do not specifically include a measure of Quality of Competition in our WAR model,” said the Younggrens. “This is for many reasons, but we feel the current methods are somewhat problematic. Also, the specific technique that we used - a statistical plus-minus approach - requires a long-term dataset (we used 11 years) for training the models. Basically, at that point, Quality of Competition effects have been significantly washed out.”

“We do include a TOI% variable (percentage of available minutes a player played),” they added. “This is similar to basketball’s Box Plus-Minus and can be interpreted as a proxy for Quality of Competition, we think. That variable was very significant in most of the models”

https://www.vancouverisawesome.com/courier-archive/news/why-quality-of-competition-doesnt-matter-to-analytics-experts-anymore-3082197

I do not like that comparison to basketball at all.  

 

In basketball, out of a roster of 12 you will generally have the starting 5 playing more then half the game (usually a lot more) and not see more then 8-9 players hit the floor in any given game unless it's a blowout.  

 

I actually think that quality of opposition has very little effect in basketball.  

 

I realize that adding another variable will add more noise to the calculation, however their explanation just reads like they don't have a good way to do it yet and are omitting it because of that.  

 

I think that this is a very important factor and is very amplified on our own team if you look at the sheltered deployment of Quinn Hughes vs OEL. While it would look like both played top 4 minutes (and they did), it was OEL who was always tasked with going up against the MacKinnons, McDavids and Crosbys of the league. 

 

 

[JamesB]

1 hour ago, SID.IS.SID.ME.IS.ME said:

I don’t think Evolving Hockey’s model includes quality of competition.

 

But IMO this isn’t a weakness.

 

...

“We do include a TOI% variable (percentage of available minutes a player played),” they added. “This is similar to basketball’s Box Plus-Minus and can be interpreted as a proxy for Quality of Competition, we think. That variable was very significant in most of the models”

https://www.vancouverisawesome.com/courier-archive/news/why-quality-of-competition-doesnt-matter-to-analytics-experts-anymore-3082197

 

50 minutes ago, VancouverHabitant said:

...

 

I think that this is a very important factor and is very amplified on our own team if you look at the sheltered deployment of Quinn Hughes vs OEL. While it would look like both played top 4 minutes (and they did), it was OEL who was always tasked with going up against the MacKinnons, McDavids and Crosbys of the league. 

 

 

Thanks to @SID.IS.SID.ME.IS.ME and @VancouverHabitant for very thoughtful comments on "QualComp". I would add a few things. 

 

1. The developers of Evolving Hockey (the Youngrens) said at one stage that they did not include QaulComp in the WAR, GAR and xGAR models. However, the model has been updated since then and, at a minimum, the model contains variables that can be regarded as proxies for QualComp and QoT (quality of teammates).  As Sid mentions, the TOI% variable is a good proxy for QualComp. (The D who is getting the most minutes is probably facing the toughest competition.)  And Zone Starts also provide information about QualComp. As for QoT, as I read the documentation, the rel_TM variables provide some correction for QoT effects.

 

2. However, as Sid says (and as I have commented in this thread already) QualComp is hard to do well and the xGAR model certainly does not fully adjust for QualComp.

 

3. So, how important is the model's weakness in area? (How much of Myers low xGAR can be explained by QualComp?) Sid provides a link to an article by Daniel Wagner that summarizes the opinion that not adjusting for QualComp is not a big problem. While I respect Wagner (and Sid), I think this argument is flawed. The data that is used to justify saying that QualComp does not matter much is subject to the same problem that measuring QualComp. Using that as evidence that QualComp does not matter is circular. 

 

4. One simple piece of evidence that I find compelling is that, on weak teams, the teams 3rd pairing D's often have a better plus/minus than the top pairing D's. Why? Because the top pairing D's are facing tougher competition that outclasses their teammates by a larger margin than for bottom pairing Ds. (Bottom 6 forwards vary less from team to team than top 6 forwards).  

 

5. Here is a thought experiment. Suppose you have two teams and each team has two lines made up of top NHL players and two lines made up of juniors. If the NHL players always play against each other and the juniors always play against each other, there would be no clear way of concluding that the NHL players were better, even though they are much better. The only possibility would be to use TOI% as the NHLers would presumably play more than the juniors. 

 

Bottom line: I think xGAR is a very good measure, but I think its biggest weakness is probably incomplete adjustment for QoC and QoT. I would also list failure to account for "toughness" and various intangibles as weaknesses. Taking these factors into account, Myers is better than xGAR suggests (but still not good) and Miller is almost certainly more valuable than Garland. But I think xGAR still provides a lot of insight that can be used to correct the "eye test" which suffers from significant biases even for experienced observers.  

[Baggins]

On 8/21/2022 at 5:29 PM, JamesB said:

You are right that Miller made a lot more hits than Petey and has a significantly higher face-off win percentage. 

 

Face-offs: What I should have said is that the face-off difference between Miller and Petey does not matter much. Face-offs are highly visible so we pay attention to them. But it has long been known that, in the data, they just don't have much impact on goals for and against or wins and losses,  especially at even strength play. Here is why. Last year Miller won 54% of his faceoffs and Petey won 44%. Petey took about 7 face-offs per game on average. He averaged about 3.1 wins per game. Suppose Miller had taken those face-offs instead of Petey. His expected wins per game would have been 0.54 x 7 = 3.8. That adds up to 0.7 extra possessions per game (3.8-3.1). I think I recall that a typical game has on the order of the 200 changes of possession (certainly way more than 100 although I cannot recall the exact number). One possession or half a possession out of 200 or so just does not move the dial much.

 

Faceoffs are more important on special teams, where possessions last longer. But Petey did not take special teams face-offs. Winning faceoffs is certainly a good thing. But its high visibility makes people overweight it when using the "eye test". 

 

Hits also do not get much weight in the model. That is not a judgement. It is determined by data. One factor is that you don't make hits when your team has the puck. Petey is actually very good at retaining possession of the puck and obtaining possession via takeaways and board battles rather than hits. Making hits instead is not necessarily good. 

 

I agree that you want some team toughness and having guys who can hit is good. But, once again, the high visibility of hits tends to make the "eye-test" over-rate their importance relative to, for example, winning a board battle to get possession. 

 

So, yes, Miller makes a lot more hits than Petey and has a significantly better face-off win percentage but, in the data, those factors do not contribute much to expected goals or expected wins. They contribute something, but less than most people think from the "eye test".

 

That said, I do think that the xGAR model probably underweights the intangible benefits of making hits (and of team toughness generally). So xGAR probably does under-rate Miller relative to Petey and Garland, But I don't think that effect is big.  

I think hitting is underrated. No it's not the be all end all and I don't think every player on the team needs to be highly physical. But there is more to it than just wearing opposition down. The secondary has real value but carries no stat line for the hitter leaving only the eye test. Hitting can separate a player from the puck or cause opposing players to rush plays and make mistakes. That can lead directly to a goal. The only stat is that opposing player getting credit for a giveaway. It can lead to a goal for, while the hitter gets no credit (assist or takeaway) and only gets credit for a hit. Hitting can play a large role.

 

Giveaways is an interesting stat. If you look at the best playmakers in the league you'll find they are also high up on giveaways. It's because they attempt plays that most wouldn't and they pass often. So how awful is their giveaway stat given how many high quality scoring chances they create? 

 

All in all, I stand by my comment this statline you're using as proof of the best player is more theoretical and quite imperfect. Stats are tools, but will never replace seeing what goes on. Here's another for you. There is no stat at all for what a player does away from the puck. But playing without the puck, knowing where to be, taking away passing and shooting lanes, forcing passes, is a huge part of the game and adds greatly to player value.