Management by Baseball

In the last entry I briefly discussed Andy Pettitte's reputation as a big game pitcher and Thomas Ayers' analysis of the lefty's playoff and World Series performances. There's a good management lesson in using analysis to look past reputation because, as I stated:

In measuring performance, lazy management will frequently let a story with truth behind it or just good personal marketing overshadow the reality. Sometimes, the reputation was earned hundreds of events ago, with no comparable success since. Outside of baseball, this is very destructive. Unlike a lot of the bad metrics and bad interpretation of metrics and poor application of metrics discussions on this weblog over the last few months, this is the polar opposite: not allowing actual facts get in the way of good folklore.

Because a fair number of the letters that come in from this weblog are asking about how to develop metrics and analyze the data, I've already suggested you look at Ayers' analysis (I don't agree with all of it, but he's laid it out in a way that gives you a panoramic view of his very rational process and logic), and I'll show you the way I would look at his data to try and discern Pettitte's post-season performance for myself. I hope for those who have asked for how-to details, you'll be able to get an idea of both his pattern in this case and mine. It's less a technical manual (One, do this. Two, do that) than patterns you can try out. If you're no good at patterns and need a technical manual, you're going to probably struggle with practical metrics, though that doesn't mean you shouldn't try. People who don't have a "knack" for it can combine their efforts with someone else who does and add a lot to the mix.

THE OCTOBER PETTITTE

Is Andy Pettitte a good Playoff/World Series (PWS) starting pitcher? If you looked at Ayers' data, you can see roll-ups that indicate his consolidated output/statistics are essentially no different PWS than in the regular season. Since he's pitching against a subset of teams that qualified for PWS, and since few teams that make it to PWS have below-average offenses, I suggest the average opponent Pettitte faced in PWS was better than the average opponent he faced in the regular season. This is somewhat negated by the fact that the Yankees spend more on opponent scouting than any other team, so Pettitte and his catchers have been armed with great business intelligence, better than they would have had available during the regular season.

The conclusion that Pettitte is a better PWS pitcher than regular season pitcher is hard to reach. He's started 30 games and thrown 187 innings (just about a season's worth of starts) in the nine years he's pitched for Gotham in the PWS, so sample size is on the bubble, neither nailed nor silly to talk about.

HIS TEAM WHEN HE DIDN'T PITCH

His won-loss record in those 30 games is 16-5. Very cool. I'd like that. But what did his team do in gamkes he wasn't pitching in? The Yankees were 70-34 total in the PWS series in which Pettitte pitched. So we'll contrast games-he-got-a-decision-in against the rest of the Yankee games. Subtract 16-5 from the total 70-34, and the Yanks were 54-29 in games Pettitte didn't get a decision in. The Yanks performed at a very high level whether he was starting or not, but they performed better in games he started. Beyond baseball, this is a good way to start isolating individual performance: ask the question, "do we do better when X is in the project/group/task/effort than when she's not?".

How about the games Pettitte started and didn't get a decision in? What did the Yankees do in those? Back to the table.

I sorted the table by whether Pettitte Won or lost or had no decision, so we can isolate those games. First question:: Did the Yanks win or lose. Based on the kind of performance he put up in the No Decision (ND) games. The Yanks were 6-1 in games Pettitte started and got no decision in. Intuitively, this suggests he pitched decently enough (in the games he got no decision) to allow his team to win.

HIS TEAM WHEN HE DIDN'T GET THE DECISION

Let's take one more look at those games, because teams don't score the same number of runs every day. Sometimes they create an easy environment for a pitcher (even one having an off day) because they put up 11 runs, and sometimes they're as listless as a Faulkner character on a muggy summer day in Mississippi. Just as in your organization, many times a team's performance will cover up or hide an individual's performance.

I sorted these games by Ayers' Modified Game Score number (50 is average, higher is better). With Bill James' Game Scores (the metric Ayers modifed) I sometimes use my own thumbnail quickie standard: A score above 54 should be a win, below 46 should be a loss, and anything in-between is a toss-up. For the moment, I'm going to pretend Ayers' Modified Game Scores work the same way as James' (if I was doing this for a client, I'd run James' numbers, which I have a feel for, and use those rather than trust what I haven't mastered). The game scores in the no decision games are: 55, 49, 45, 44, 36, 23, 21. One "should have won", 1 toss-up, four "should have lost".

So those seven games in which Pettitte got no decision, and in which the Yankees went 6-1, were games Petittte generally didn't pitch well. They (offiense, relief pitching) saved his bacon by winning games he probably "shouldn't" have.

His set of performances here was masked to some degree by context, by what the rest of the team did. This figure, in part, tunes the 16-5 win-loss. If we chose to, we could add the 1-4 "should haves" to Pettitte's 16-5, which would give us 17-9, still awfully good. Now let's re-apply the games-in-which-he-didn't-start thumbnail: Subtract 17-9 from the total 70-34, and the Yanks were 53-25 in the non-Pettitte games. The Yanks essentially were as likely to win in games Pettitte didn't start as the ones he did.

This doesn't mean Pettitte is not a very fine pitcher in PWS (you have to be to beat the higher concentration of good opposition). It just means it appears to me Andy Pettitte is not significantly better than the other Yankee picthers in the measure of PWS performer.

The Reputation is Not the Guy.

In the next entry, I'll discuss reputation using Pettitte's example, and try to point out why he has the reputation as opposed to anyone else on the Yanks.