Evaluating Canuck Players using Expected Goals Above Replacement

[JamesB]

On 8/21/2022 at 2:35 AM, D-Money said:

I love Connor Garland, so no need to convince me that he has value. But if a metric states he’s more valuable than JT Miller, then that is an obvious flaw in the metric.

 

And I know Myers is not good, and that his counting stats last year were mostly an indication of how OEL covered for him. But THAT bad? Seems a bit extreme. 

 

On 8/21/2022 at 8:54 AM, 204CanucksFan said:

If you'd like some understanding as to why this metric weighs Garland above Miller, here are a few reasons (remember, as with most advanced stats, it's about the numbers as opposed to the eye test)...

 

 

On 8/21/2022 at 10:25 AM, SID.IS.SID.ME.IS.ME said:

What pushed Garland higher than Miller on GAR and xGAR is defence and penalties.

 

On offence, Miller has some very strong numbers: 19.7 GAR and 14.9 xGAR. Roughly doubling the offensive on-ice (team) impact of Garland over a season (11 GAR, 6.7 xGAR, for Garland’s offensive contributions)....

 

Great question from @D-Money  and great responses from @204CanucksFan and @SID.IS.SID.ME.IS.ME

 

I would add a few things. 

 

1. I agree that it is hard to believe that Myers was that bad and hard to believe that Garland contributed more value than Miller. There are two types of explanation. One type, as suggested by 204 and Sid, points out real factors that are important in generating and stopping goals but don't get much attention. Penalties drawn and penalties taken are the most important. So if Garland draws a penalty and Miller goes out on the PP gets a 2nd assist for making a routine pass to Hughes at the point or Petey on the sidewall, Garland gets no credit on the basic stat line and Miller does when, arguably, Garland actually played a more important role in generating that goal. In xGar, drawing the penalty is worth something significant and a PP second assist does not get a lot of value. (I am not sure which actually has a higher weight inside the xGAR model.)

 

2. The other big "real factor" is defence vs. offense. For most people the "eye test" is biased in favor of offensive contributions. We all notice when someone scores a goal or gets a nice assist. We don't all notice (or at least don't remember) a guy who played on the "right side of the puck" all night and did not allow the opposing team many good chances when he was on the ice. But stopping a goal is exactly equal in value to scoring a goal when it comes to actual winning and losing. And deployment is very important. A guy who gets a lot of PP time is going to score more points just by virtue of that PP time. Scoring on the PP is still good but, in comparing players, you need to account for the advantage provided by PP time, as xGAR does. 

 

3. The other type of explanation is that xGAR does not capture everything. One type of error is "model error". The underlying equations, most of which are linear equations, do not not replicate reality perfectly. That creates some random error and possibly some bias.  Frankly, I don't don't think the error from this source is high as the model is based on a big sample size by now (many years of data and many games per year). But some factors are hard to measure. Intangibles like "leadership", value "in the room", the ability to raise the emotional energy of the team in key situations, increasing the confidence level of other guys (or reducing it) cannot be quantified. The other big thing is quality of team-mates and quality of competition. The model makes some attempt to adjust for those factors but it is just very hard to do well. (I have tried on a couple of occasions.) So a D who plays shutdown against the other team's top line is probably going to look worse than he actually is, even though the model makes some attempt to correct for that. I am pretty sure that makes Myers look worse than he really is. He is still very poor value for his cap hit, but not as bad as it looks from xGAR. LIke I said before, he would probably be okay as a 3rd pairing D. 

 

4. Another factor that is not in xGAR is "toughness". Things like hits, blocked shots, and major penalties are in xGAR, but those are not particularly good measures of the kind of toughness that contributes to winning. What you want is a guy who deters the other team from taking cheap shots at your star players and makes guys on your team feel like he "has their back". Schenn does that and is one reason why I think he is good value despite his uninspiring xGAR. And Miller does that as well, which adds to his value and is one reason why I think he was more valuable than Garland despite having a slightly lower xGAR. 

Edited August 22, 2022 by JamesB

[204CanucksFan]

51 minutes ago, Elias Pettersson said:

Also, Garland's game is not as effective if he is on the PP as he is stationary as others mentioned

If you are referring to D-Money, what he actually said was:

 

3 hours ago, D-Money said:

Garland's game is based on speed/stamina, grinding, and taking shots when they are not expected. Putting him on a fairly stationary PP setup, when the defending team is focused on defending shots, takes away his strengths.

Which means Garland moves too much to be effective on a stationary PP.

 

Which is the only kind of PP Vancouver uses 

 

Edit: Didn't want to add it in a separate post, but Garland's most common centers at 5-on-5 were Miller and Pettersson so not sure how that factors in to your 'Garland is a sheltered 3rd liner whereas Miller and Petey play against top lines' argument.

Edited August 21, 2022 by 204CanucksFan

[JamesB]

5 hours ago, JM_ said:

Is there a way to look at this metric laid out over the course of the whole season? 

 

 

 

3 hours ago, SID.IS.SID.ME.IS.ME said:

Do you mean game-by-game?

 

Like a graph plotting how a player’s xGAR changed from game-to-game over the whole season?

 

I don’t know of any free source for this kind of data. It’s certainly possible to generate, but you’d need to pay someone for it, I’d think. 

I took a quick look at Evolving Hockey and could not find it. The underlying data is in the system, but there is no easy query to generate it,  

 

As for Myers, after Bruce took over, the team played a "quick up" style, which generated more chances at both ends of the ice. It helped the offensive forwards a lot but it put more pressure on the D and on Demko. Fortunately, Demko was up to the challenge and team benefitted a lot from this change. 

 

But it was probably tough  on Myers. As JR said as one point (I am paraphrasing) -- We have D who can be good if we play a more structured game in our own zone. I am pretty sure he was referring to Myers. Hughes and OEL have good enough vision and are good enough passers to to deal with a quick up style of play and make good passes to clear the D-zone. Myers is not in that category and would often give the puck away or ice the puck or just make a bad pass to nowhere. So I am guessing his play as measured by xGAR actually regressed under Bruce, even though the team as a whole improved a lot.  (And Demko was able to bail him out most of the time.)

[JM_]

10 minutes ago, JamesB said:

 

I took a quick look at Evolving Hockey and could not find it. The underlying data is in the system, but there is no easy query to generate it,  

 

As for Myers, after Bruce took over, the team played a "quick up" style, which generated more chances at both ends of the ice. It helped the offensive forwards a lot but it put more pressure on the D and on Demko. Fortunately, Demko was up to the challenge and team benefitted a lot from this change. 

 

But it was probably tough  on Myers. As JR said as one point (I am paraphrasing) -- We have D who can be good if we play a more structured game in our own zone. I am pretty sure he was referring to Myers. Hughes and OEL have good enough vision and are good enough passers to to deal with a quick up style of play and make good passes to clear the D-zone. Myers is not in that category and would often give the puck away or ice the puck or just make a bad pass to nowhere. So I am guessing his play as measured by xGAR actually regressed under Bruce, even though the team as a whole improved a lot.  (And Demko was able to bail him out most of the time.)

yeah I think you could be right on that. 

 

So... if we want to try pairing OEL and Hughes, who the heck is the right fit for Myers under a more structured system?

[ilduce39]

I’m sure an effort is made to parse this stuff out but I’d still wager these tools aren’t necessarily “who’s better” but rather “who was effective in a particular role.”  

 

Myers was logging 22 pretty hard minutes a night last year.  And, while I’ve dubbed Garland our, “best winger” I wouldn’t put him in the same league as Petey, Miller, Hughes or even Bo. And in a league full of analytics departments he wouldn’t hold their value. 

[JamesB]

3 hours ago, DSVII said:

Would you have the full list of metrics that go into xGAR? As much as I'd love to rag on Myers for his contract is the metric being fair to him as a defenseman who was matched against the other team's top lines (quality of comp)? What are the stats on the defensive side of the game included here if any or is this stat generally geared towards offensive side players?

 

I'm wondering as well what the league average xGAR for defensemen was. Whether it was a net positive or negative.

@SID.IS.SID.ME.IS.ME provided links to the pages on Evolving Hockey that explain the xGAR calculation. Frankly, those explanations could be better and they are not complete. To some extent, the model is a "black box". However, they say enough to convince me that they are doing an excellent job and are the best public source out there.  They include a lot of stats to try to get at the defensive side of the game and I don't think they under-rate defence.  So I don't think the stat is biased toward offensive players. The xGAR stat is actually highly correlated with xGF - xGA (expected goals for minus expected goals against), which I think is the best simple measure. 

 

As for D-men overall, there were 276 D-men who played at least 200 minutes last year. The median xGAR was 2.1. Here are the top 5, middle 5, and bottom 5:

 

Top 5.

 

1, Cale Makar (with a ridiculous xGAR of 35.8)

2. Hedman (xGAR = 25,7)

3. Fox

4, Ekblad

5. Toews

 

Middle 5:

 

Sergachev

Dermott

Murphy

Lindgren

McNabb

 

Bottom 5:

 

Myers

Dekeyser

Logan Stanley

Keith Yandle

Patrik Nemeth

 

The fact that Dermott is right in the middle and his low cap hit means thatat he is actually very good value. That is a theme of all the Canuck moves under the new management. All their moves have been good from an analytics points of view (and JR is of course well-known for taking analytics seriously).

 

This is a 180 degree shift  from the Benning regime, which was characterized by very bad moves from an analytical point of view -- not just in hindsight, but bad at the time. To trade Myers the Canucks need to find a GM like Benning who is impressed by the superficial aspects of his game (size, pretty good skater, likes to get involved in the offense). Unfortunately for the Canucks, there are not many GMs like Benning left (if any).  (I was going to try to avoid mentioning Benning in this thread but I just could not do it.)

 

 

[JamesB]

21 minutes ago, JM_ said:

yeah I think you could be right on that. 

 

So... if we want to try pairing OEL and Hughes, who the heck is the right fit for Myers under a more structured system?

Good question. I would try him with Dermott. That leaves Schenn and Rathbone as the "other" ( I hesitate to say "2nd") pairing.  OEL and Hughes would need to log a lot minutes in this world but it might be the best set-up. But if Poolman is healthy, he plays, probably with OEL as the top shutdown pairing. That leaves Hughes and Schenn, who are good enough as a 2nd pairing, and Myers with Rathbone or Dermott. 

[JamesB]

15 hours ago, gurn said:

Other than goals and plus minus, aren't most of the analytic numbers just eye test?

Is that shot from the slot, or is it just a bit to the left of the slot? In which case is it not a shot from the left side?

If so, who decided it was from the left side?- Somebody using their eyes?

Was it a hit? Well maybe to you, but to me that looked more like a push or did it look more like  he pinned him to the boards but didn't hit him?

Someone had to use their eyes to decide if the "hit" met the criteria.

Good question.

 

The most important numbers are pretty objective:

on ice goals for

on ice goal against.

on ice shots for 

on ice shots against.

location of the shot. I think that location is now determined by using precise on ice locations from video.

timing of the shot (a shot that comes quickly after another shot is more likely to score, with the precise value determined by regression analysis)

 

penalties taken and penalties drawn. 

 

Takeways and giveaways are bit more subjective but, in a large sample, are good measures.

Face-offs won and lost are pretty objective.

 

Zone starts are objective

Blocked shots are pretty much objective. 

 

Hits are probably the most subjective measure and they don't contribute much to xGAR in any case.  (While make a hit is good, if you are making a lot of hits that means your team does not have the puck, which is not good.)

 

Bottom line: The important things that get measured are relatively objective. If someone wants to criticize the xGAR measure, this is not the place to look. I think the bigger  problem is just that xGAR leaves out some important things altogether, as mentioned in other messages in this thread.  That said, xGAR and GAR are very good predictors of team performance.  

Edited August 22, 2022 by JamesB

[JamesB]

13 hours ago, Heretic said:

I'd like to see a Salary vs Performance metric, ie, best bang for the buck.

Good point. Evolving hockey reports salary value implied  by a given value of something called "Standing Points Above Replacement" (SPAR), which is pretty much exactly correlated with GAR. It does NOT report salary value as a function of xGAR.

 

Still, using SPAR is pretty good.  You can then subtract the cap hit from this value to get net value added, which is the right thing to look at when trying to assess contribution toward being a Cup contender. It still takes a lot of effort to extract these number for each player. But here are some interesting bits and pieces,

 

Most valuable guys relative to cap hit: Garland (net value added = 8,7) and Miller (7.1)

Least valuable guys relative to cap hit: Myers (-6.3) and Boeser (-4.8)

Pretty good value (Petey, Podz, Horvat)

Fair value: Hughes (he is very good, but he has a high cap hit), Schenn (low cap hit and he looks a bit better using SPAR or GAR rather than xGAR,)

 

OEL is a good player but not good enough for his cap hit, so his value added is negative. 

 

One possibly optimistic note is that if Boeser has a bounce-back year that could add a lot of value. Bounce backs from Hoglander or Dickie would also help. 

Edited August 22, 2022 by JamesB

[D-Money]

2 hours ago, 204CanucksFan said:

If you are referring to D-Money, what he actually said was:

 

Which means Garland moves too much to be effective on a stationary PP.

 

Which is the only kind of PP Vancouver uses 

 

Edit: Didn't want to add it in a separate post, but Garland's most common centers at 5-on-5 were Miller and Pettersson so not sure how that factors in to your 'Garland is a sheltered 3rd liner whereas Miller and Petey play against top lines' argument.

I said Garland was better on a match-up line, opposite of sheltered. He can disrupt offensive players, and transition quickly, when it’s unexpected.

[JM_]

1 hour ago, JamesB said:

Good question. I would try him with Dermott. That leaves Schenn and Rathbone as the "other" ( I hesitate to say "2nd") pairing.  OEL and Hughes would need to log a lot minutes in this world but it might be the best set-up. But if Poolman is healthy, he plays, probably with OEL as the top shutdown pairing. That leaves Hughes and Schenn, who are good enough as a 2nd pairing, and Myers with Rathbone or Dermott. 

I took a look at that on natural stat trick: https://www.naturalstattrick.com/pairings.php?fromseason=20212022&thruseason=20212022&stype=2&sit=5v5&score=all&rate=n&team=VAN&loc=B&toi=0&gpfilt=none&fd=&td=&tgp=410

 

very low # of minutes together (<60) but actually faired pretty well. Not sure if thats enough to go on but its all we have.

 

I feel like we really need to come out with something different to raise the bar.

 

OEL-Hughes

Rathbone-Poolman

Dermott-Myers

 

or if we do pick up De Haan:

 

Hughes-DeHaan

Rathbone-Schenn

Dermott-Myers

 

 

Edited August 21, 2022 by JM_

[Baggins]

14 hours ago, JamesB said:

these factors are much more important than Miller's edge in hits and slight edge in faceoff percentage.  

123 more hits isn't simply an edge in hits. Horvat being 2.9% better at faceoffs than Miller could be viewed as a slight edge. But Miller being a full 10% higher than Petey couldn't possibly be considered a "slight" edge. That a significant difference in faceoff ability. 

 

Weighted values, and theoretical repalcements sounds far more like a theoretical statistic rather than something concrete with proven accuracy. In the end I don't think any single stat, fancy or traditional, can be used on it's own to identify the best player on the team. 

[JamesB]

49 minutes ago, JM_ said:

I took a look at that on natural stat trick: https://www.naturalstattrick.com/pairings.php?fromseason=20212022&thruseason=20212022&stype=2&sit=5v5&score=all&rate=n&team=VAN&loc=B&toi=0&gpfilt=none&fd=&td=&tgp=410

 

very low # of minutes together (<60) but actually faired pretty well. Not sure if thats enough to go on but its all we have.

 

I feel like we really need to come out with something different to raise the bar.

 

OEL-Hughes

Rathbone-Poolman

Dermott-Myers

 

or if we do pick up De Haan:

 

Hughes-DeHaan

Rathbone-Schenn

Dermott-Myers

 

 

In the second grouping you don't have OEL. Are you assuming he gets traded or maybe that he and Poolman play as the top shutdown pairing and these other three pairings given different options, or was dropping OEL just an oversight?

[UKNuck96]

As with all these debates you get three types of people, those that love the stats and compare numbers and figures. Those that don’t trust them, and those that recognise that stats are meaningless without context and essentially playing top trumps. 
 

stats are useful but playing top trumps with them isn’t as there will never be one stat to rule them all. Context is still key 

 

using stats to work out day who is a good face off winner under pressure and then analyse why is good. Using them to identify trends and pattens of play also useful either to plan better plays or counter opponents.  Using to say x is better than why but not defining the context of comparison is meaningless. 

is mcdavid abetter player than Hedman because he scores goals than hedman? comparison is meaningless 

[Guest]

3 hours ago, JamesB said:

@SID.IS.SID.ME.IS.ME provided links to the pages on Evolving Hockey that explain the xGAR calculation. Frankly, those explanations could be better and they are not complete. To some extent, the model is a "black box". However, they say enough to convince me that they are doing an excellent job and are the best public source out there.  They include a lot of stats to try to get at the defensive side of the game and I don't think they under-rate defence.  So I don't think the stat is biased toward offensive players. The xGAR stat is actually highly correlated with xGF - xGA (expected goals for minus expected goals against), which I think is the best simple measure. 

 

There is nothing in those links that @SID.IS.SID.ME.IS.ME provided that talks about quality of competition.  

 

If I missed it, feel free to point it out, I did a quick Ctrl+F search through the documents. 

[JamesB]

36 minutes ago, Baggins said:

123 more hits isn't simply an edge in hits. Horvat being 2.9% better at faceoffs than Miller could be viewed as a slight edge. But Miller being a full 10% higher than Petey couldn't possibly be considered a "slight" edge. That a significant difference in faceoff ability. 

 

Weighted values, and theoretical repalcements sounds far more like a theoretical statistic rather than something concrete with proven accuracy. In the end I don't think any single stat, fancy or traditional, can be used on it's own to identify the best player on the team. 

You are right that Miller made a lot more hits than Petey and has a significantly higher face-off win percentage. 

 

Face-offs: What I should have said is that the face-off difference between Miller and Petey does not matter much. Face-offs are highly visible so we pay attention to them. But it has long been known that, in the data, they just don't have much impact on goals for and against or wins and losses,  especially at even strength play. Here is why. Last year Miller won 54% of his faceoffs and Petey won 44%. Petey took about 7 face-offs per game on average. He averaged about 3.1 wins per game. Suppose Miller had taken those face-offs instead of Petey. His expected wins per game would have been 0.54 x 7 = 3.8. That adds up to 0.7 extra possessions per game (3.8-3.1). I think I recall that a typical game has on the order of the 200 changes of possession (certainly way more than 100 although I cannot recall the exact number). One possession or half a possession out of 200 or so just does not move the dial much.

 

Faceoffs are more important on special teams, where possessions last longer. But Petey did not take special teams face-offs. Winning faceoffs is certainly a good thing. But its high visibility makes people overweight it when using the "eye test". 

 

Hits also do not get much weight in the model. That is not a judgement. It is determined by data. One factor is that you don't make hits when your team has the puck. Petey is actually very good at retaining possession of the puck and obtaining possession via takeaways and board battles rather than hits. Making hits instead is not necessarily good. 

 

I agree that you want some team toughness and having guys who can hit is good. But, once again, the high visibility of hits tends to make the "eye-test" over-rate their importance relative to, for example, winning a board battle to get possession. 

 

So, yes, Miller makes a lot more hits than Petey and has a significantly better face-off win percentage but, in the data, those factors do not contribute much to expected goals or expected wins. They contribute something, but less than most people think from the "eye test".

 

That said, I do think that the xGAR model probably underweights the intangible benefits of making hits (and of team toughness generally). So xGAR probably does under-rate Miller relative to Petey and Garland, But I don't think that effect is big.  

[Viper007]

1 hour ago, JM_ said:

I took a look at that on natural stat trick: https://www.naturalstattrick.com/pairings.php?fromseason=20212022&thruseason=20212022&stype=2&sit=5v5&score=all&rate=n&team=VAN&loc=B&toi=0&gpfilt=none&fd=&td=&tgp=410

 

very low # of minutes together (<60) but actually faired pretty well. Not sure if thats enough to go on but its all we have.

 

I feel like we really need to come out with something different to raise the bar.

 

OEL-Hughes

Rathbone-Poolman

Dermott-Myers

 

or if we do pick up De Haan:

 

Hughes-DeHaan

Rathbone-Schenn

Dermott-Myers

 

 

What happened To OEL if we pick up De Haan?

[JamesB]

9 minutes ago, VancouverHabitant said:

There is nothing in those links that @SID.IS.SID.ME.IS.ME provided that talks about quality of competition.  

 

If I missed it, feel free to point it out, I did a quick Ctrl+F search through the documents. 

Good point. The only thing Evolving Hockey has is at the following link:

 

Evolving-Hockey

 

I am not sure is this is available without a subscription. However it does not say much. The main thing it says is that an explanation of quality of competition (QoC) and quality of teammates (QoT) in the model will be available in the future. 

 

So right now, it is hard to know exactly how they handle it.

 

But I suspect this remains a weak spot in the model. They still do a great job, but probably do not fully account for QoC and QoT effects. That is how the results look to me and is consistent with my previous attempts to understand and work with player rating systems. I don't think this is a big problem but it is worth keeping mind. 

[Guest]

18 minutes ago, JamesB said:

Good point. The only thing Evolving Hockey has is at the following link:

 

Evolving-Hockey

 

I am not sure is this is available without a subscription. However it does not say much. The main thing it says is that an explanation of quality of competition (QoC) and quality of teammates (QoT) in the model will be available in the future. 

 

So right now, it is hard to know exactly how they handle it.

 

But I suspect this remains a weak spot in the model. They still do a great job, but probably do not fully account for QoC and QoT effects. That is how the results look to me and is consistent with my previous attempts to understand and work with player rating systems. I don't think this is a big problem but it is worth keeping mind. 

Myers and OEL were always our against the other team's top two lines, so I think that it might've played a bit of a factor in those numbers....  

 

Very interesting topic btw, thanks for starting it. 

[JamesB]

46 minutes ago, UKNuck96 said:

As with all these debates you get three types of people, those that love the stats and compare numbers and figures. Those that don’t trust them, and those that recognise that stats are meaningless without context and essentially playing top trumps. 
 

stats are useful but playing top trumps with them isn’t as there will never be one stat to rule them all. Context is still key 

 

using stats to work out day who is a good face off winner under pressure and then analyse why is good. Using them to identify trends and pattens of play also useful either to plan better plays or counter opponents.  Using to say x is better than why but not defining the context of comparison is meaningless. 

is mcdavid abetter player than Hedman because he scores goals than hedman? comparison is meaningless 

1. Your are certainly right that some people (like JR) basically like statistical analysis and some people (like Benning) either don't like it or don't understand it. One thing I like a lot about  Boudreau is that he is open-minded. He is a very good old-school players' coach but he has indicated that he wants to learn more about analytics and try to improve his coaching. For a guy his age (or any age) that is a great attitude. And in today's NHL I think it is important to keep moving forward.  

 

2. The point of modelling to try to incorporate as much relevant context as possible. The better the model, the less extra judgement you need to impose on top of the model. That is true whether you are trying to model the actions of a potential vaccine on some virus or hockey performance or anything else. You never eliminate the need for some judgement based on context (but who knows where AI-based models will take us).

 

3. GMs and coaches still need to make tough comparisons. (like your Hedman vs. McDavid comparison). With the Canucks, for example, Miller and Horvat are on expiring contracts this year and Petey comes up the year after. Due to cap issues it is unlikely the team can keep all three guys, so they have to make a call. The question is whether you use analytical methods as an important input (as JR does) or whether you rely on your "gut feelings" as Benning did.