Saturday, October 04, 2003

Part Four -- The Seductions
(& Giant Sucking Sounds) of Metrics  

Team announcers for at least two teams that didn't make significant deadline deals (Houston and Seattle) and failed to make the playoffs when they tailed out of contention late in the season, defended their team's inaction by arguing deadline deals were overrated. Their poster boy for this argument was Sir Sidney Ponson, traded near the late July deadline from the out-of-it-this-year Os to the NL West division-leading Giants.

That argument was based on two fallacies that bedevil non-baseball organizations' attempts to evaluate employee performance, too: Confusing the average with the performance, and overlooking the context in which the employee was performing.

The Average is Not The Performance

The argument from the announcers went something like this: The Gigantes went to a lot of trouble to get Ponson, trading away two top pitching prospects and another young player. Ponson compiled a merely average ERA (Giants' ERA = 3.73, Ponson's = 3.71), and has a 3-6 won-loss record for a team with a strong win record during his time with the team. Given the Giants gave up good-looking prospects, their move was a failure.

Let look at his performance after the trade, courtesy of a table from ESPN's web site.

8/6 PIT L 0-2 7.1 7 2 0 3 3 14 8 31 109 56 L(0-1) 2.46
8/12 @NYM L 4-5 6.2 8 5 1 2 2 11 9 29 103 38 L(0-2) 4.50
8/17 @MON L 2-4 8.1 4 3 0 2 6 16 5 31 120 67 L(0-3) 4.03
8/22 FLA W 6-4 6.0 6 2 1 3 5 13 3 26 92 54 W(1-3) 3.81
8/27 @COL W 6-4 7.2 5 1 1 1 2 14 7 27 88 66 W(2-3) 3.25
9/1 @ARI W 2-0 8.0 6 0 0 1 3 13 9 30 107 72 W(3-3) 2.66
9/7 ARI L 6-9 6.0 8 4 1 2 3 9 9 27 94 41 -- 3.06
9/13 MIL L 4-5 6.0 5 4 2 0 3 11 7 23 76 49 L(3-4) 3.38
9/18 SD L 3-7 5.0 11 6 0 0 5 10 6 27 92 28 L(3-5) 3.84
9/24 @HOU L 1-2 7.0 4 2 0 4 2 11 3 25 83 59 L(3-6) 3.71

There's a column in bold towards the right edge I'm going to be referring to, labeled GSc. That stands for Game Score, and it's a way to evaluate a starter performance invented by sabermetrician Bill James. It's based on innings picthed, runs, hits and walks surrendered and strikeouts notched. It's an imperfect measure, but better than ERA. On this scale, the major league average start is a "50" and the worst approaches "0". Let's say for the purposes of this entry that scores below 47 are "bad" starts, 47 to 53 inclusive are "average" starts and 54 or higher are "good or good-enough" starts, that is, starts a team could expect to win more often than not when their starter throws a game like that.

His average game score over those 10 starts was 53, a little better than 50 but in the range I decided to call "average". He never threw a start that was his 53 average. His average was made up of 10 different starts. All three of his wins were in the "good or good-enough" category (72, 66, 54). In the start where he got no decision, he was bad (41). And in his six losses, he had 2 "bad" -- really awful -- starts (38, 28), 1 "average" (49) and 3 "good or good-enough" starts (56, 59, 67).

Out of 10 starts, Sir Sidney had 6 good or good-enough, 1 average and three bad starts. If his team had put up a normal number of runs behind him, he could easily have been 6-4 or even 6-3. If he had thrown all 53s, you might expect him to be 5-5.

Outside of baseball, it's a rare team member that is an average performer who puts up average accomplishments every day. If you look at her end of quarter sales figures or his breakage rate or her bids output, that summary is based on a lot of different events. Most team members will have put together a "season" of really effective and really less-effective days. It's important to look at the individual accomplishments that make up the average, as well as the average itself.

The Context is The Network

If you go back to Ponson's start chart above, look in the Result column in his losses. In the first game for example, 8/6 in Pittsburgh, the Giants lost 2-0...they didn't score a run. Hard to hold Ponson fully responsible for that loss. In his other losses he gave up 4, 5, 5, 7, and 2 runs. The first and last were games you might expect the Giants to win (those run totals are lower than average for a National League game) and in the three middle games, you'd expect them to lose.

If you just look at win-loss totals, you lose the context of what Ponson's team was doing the days he was pitching. In your own organization, you may have salesfolk struggling to sell (is the product line old? are the marketing materials ineffective? is the economy sucking wind?) or shooting the moon (did they inherit locked-up accounts? do they have plum territories? is the market booming?) and without context, you can misinterpret how effective the individuals are.

Keep context in mind. It's extra effort, but if you want to reward and keep the right contributors, you need to do it.

And keep in mind, team announcers are paid by the team...so if they are telling you it was wise of their team that failed to make the playoffs to not enlist reserves for a final assault on the pennant, they may not be unbiased observers

This page is powered by Blogger. Isn't yours?

free website counter