Management by Baseball

Part Two -- The Seductions (& Giant Sucking Sounds) of Metrics

As I mentioned yesterday, in business endeavors w/quantifiable results (manufacturing and sales, for example) there's a near-erotic fascination with "metrics". Sadly, too many of the folk who create them are not numerate. Those who have a "tin ear" for numbers are likely to grab onto metrics with their eyes closed, clenched tight, ignoring the realities, in favor of the numeric artifact that is a shadow of a dream of a reflection of reality.

The larger the organization (business, government, non-profit) the more the metrics are institutionalised, embraced without serious examination.

Seduction #2: One Size Fits All -- Dodger Stadium & Environmental Permits

Most people with management responsibilities who try to use numbers to support their evaluation of performance fall back on the simplest schemes they can. Many of them are not numerate and most are overworked (or lazy enough to behave as though they were), so it's simplest for them to embrace wholly a single number.

Baseball front-office types allow the one-size-fits-all mentality to undermine the usefulness of undermine their ability to evaluate player performance. Here's how.

Each ball park has a set of factors (including distance to fences, altitude, relative humidity, prevailing wind direction and strength) that affect performance of players. The Colorado Rockies' park amplifies run production (more for left-handed hitter than right-handed), and because baseball statistics are like double-entry accounting, this both increases apparent hitter proficiency and diminishes apparent pitcher proficiency. For a few years, before other teams' front-offices caught on, the Rockies were able to trade away hitters with fab-looking numbers that pretty quickly faded when they were no longer playing half their games in Denver.

Here are a pair of examples where you can compare hitters' Colorado output and what happened after: Jeff Cirillo and Dante "The Inferno" Bichette. It's not universal...there are some players who did as well not playing for the Rockies, but most of them were not-good in either setting.

The Dodgers' stadium has the opposite effect, draining run production and making the pitchers look good & the batters look weak. For a long time, the Dodger front-office seemed to be ignorant of this (remember that this was the team that in 1993 traded the most excellent contemporary pitcher, Pedro Martinez, for a never-was infielder), though they know now how their stadium affects the performance of average players.

You have to evaluate baseball numbers in context. Having an ERA of 3.00 playing your home games in Los Angeles is a humongous difference from having the same ERA playing your home games in Denver. The average baseball front-office is just starting to catch on to this, though non-baseball organizations have an awful time with this concept:

It's all about context.

I worked for a consulting firm that was doing work for the U.S. Environmental Protection Agency's division that handled enforcement of water discharge regulations. Each of the thousands of major facilities that discharged into waterways received a complex permit with many specifications, and then was responsible each reporting period, to properly fill out a form for each individual outfall pipe indicating what they were allowed to discharge and then what they actually had discharged. It wasn't working well because people unintentionally, or less frequently, intentionally, filled out the forms incorrectly. To oversimplify, the project was to take the thousands of permits and codify them so a computer would pre-print a report with everything printed correctly according to the permit's specs (except the current results, which were entered by the responsible person at the facility). Permits ranged from the tiny Peever, South Dakota sewage treatment plant with one outfall pipe and a six page permit, to the largest paint & chemicals factory in the world, National Lead in the St. Louis area, which had 17 outfalls and dumped dozens of chemicals, including hydrochloric acid, lye and mercury, and a permit so massive it made a Tolstoy novel look like Pat the Bunny in comparison.

There were about eight of us performing the codification of these permits, translating complex regulatory and chemical and biological data onto forms the computer could transform into the printed reports. The consulting firm had a contractual responsibility to measure group-wide and individual "performance" (a term not defined in the contract). The consulting company behaved as though it was venal and stupid. They decided to evaluate "performance" as how many permits per day you codified. Ignoring context, they gave you incentive to take the small simple ones and shirk the big, complex ones. In addition, because quality was not part of the criteria, a couple of the coders sped through permits that needed close attention.

Their payback? Everyone grabbed for the easy ones, increasing our output in the present, making it look like we were way ahead of schedule. The Consulting firm, not understanding what they hath wrought, reported to the EPA that we were way ahead of schedule and asked for a bonus. The EPA audited and, seeing what work had been done and what was left to be done, thought the company was trying to pull a fast one on them. And the audit discovered a higher-than-expected error rate.

It's all about context.

So later, when the EPA dumped the consulting firm, the group of us codifying formed our own company and bid for the rest of the job. I instituted a pay-for-perfromance system that had a low base and a high bonus based on a context formula that took into consideration quality, difficulty of permit and, yes, quantity. Our error rate was 54% lower and our volume (quantity) was 29% higher per hour.

But most big organizations won't consider context in their metrics. It's more work, and most managers feel (some rightly) they won't be recognized for creating better metrics.