What You Measure and How You Measure It
If you decide that you are measuring your law department’s performance in the correct context (see prior blogs), you then face two more important hurdles. The first, whether you are measuring something that accurately measures a relevant factor and second, that you are measuring the right thing.
Let me address the last issue first—are you measuring the right thing? Two measures seem to predominate the law department measuring game—customer satisfaction surveys and benchmarking. These are offered by the legal consultants in large part because it is easy to do and does not require the consulting firm to actually understand much about the delivery of legal services.
Customer surveys, or the happiness indicator as I refer to it, have two problems. First, it creates the wrong incentive for lawyers who will become reluctant to give unpleasant advice. Second, it measures happiness and not whether the lawyer is actually adding value in manner that benefits the overall corporation. The later is do able, but it far more complex and costly, and generally well beyond the skill and capability of the legal consultant firm.
The second method is benchmarking. The single biggest problem with that method is comparability. The categories which the consulting firms although superficially appealing could prove very misleading. Take for example my former employer, Alcan Aluminum Limited, and Alcoa, who at one time vied with each other as the largest aluminum producers in the world. From year to year who was the biggest shifted in terms of revenues and size of shipments, but they were always pretty close. Were they comparable for legal benching marking, say on legal expenses per ton or per revenue dollar? No, because Limited’s primary revenue was highly weighted to primary metal with large advantages in cheap power in Northern Canada whereas Alcoa dominated the higher revenue finished and semi-finished products that required more complex marketing, technological and patent issues (and far more product liability exposure). Differences in legal expenses actually described more about the difference in the businesses than in than the functioning of the business. I would suspect that other apparently similar businesses are also quite difference.
Benchmarking is offered is by the legal consultant industry because it is cheap for them to do and they can charge a lot for it. It also requires no real understanding of either the actual delivery of services or comparability between the businesses.
How you measure is also very important. Statistical significance and controls are critical. The classic demonstration of control is the Governor who pulls out a chart showing the downward sloping curve of highway fatalities starting in 1985 when he signed and implemented the slower speed limits on major highways. This he concludes demonstrates the effectiveness of his policy. The next day a major paper in the State published a Chart of highway fatalities from 1978 to 1984 and the slope was even steeper. So what was happening? Clearly, the Governor’s policy did not have the casual impact he claimed. Was it counter productive? Or was it wholly irrelevant?
Finally remember correlations are not statement of causation—they are statements of correlations. And remember statistical significance. That which is not statistically significant is noise and much of what you hear about the present recovery discussed on the business channels is nothing more that statistical noise. Do not make the same mistake in your studies. You might find somebody like me reading them.
So Fred Krebs is right--measure often, but measure the right thing, the right way at the right time.
