This article first appeared in the winter 2004 issue of the Expert Witness.
Expert witnesses often testify that their experience, or the latest research, leads them to believe that a plaintiff’s injuries will have certain long-term physical, educational, or employment consequences. For example, the plaintiff’s injuries are predicted to worsen, or improve, along some projected time line. Or those injuries are expected to affect the probability that the plaintiff will be able to complete a planned educational program or enter a preferred occupation. Or an opinion will be given concerning the effect that certain disabilities will have on the income that the plaintiff will be able to earn.
When listening to such testimony, I am often reminded of Benjamin Disraeli’s famous complaint that “there are three kinds of lies: lies, damn lies, and statistics.” Expert testimony – particularly expert testimony with respect to the application of medical statistics to the determination of damages in personal injury cases – gives rise to three alternative interpretations that might be attached to Disraeli’s adage.
First, there is the sense in which I suspect Disraeli himself meant his quote: as a complaint that laypeople – judges and lawyers in this case, politicians in his – often find the (statistical) testimony of expert witnesses to be so confusing that they have difficulty distinguishing fact from fiction. That is, his was a call for more clarity; less use of obscure, technical language.
A second version might be recast as: there are “liars, damn liars, and those who abuse statistics.” This is a complaint against those who intentionally twist the interpretation of statistics, hoping either that the opposing expert has insufficient statistical knowledge to be able to recognise the deception, or that the court will have insufficient expertise to be able to determine which of the experts is telling the truth.
Finally, the sense in which Disraeli’s dictum is of greatest relevance to legal advocacy might be restated as: there are “liars, damn liars, and those who misuse statistics.” By this I mean the situation in which “experts” have insufficient knowledge of statistical analysis to realize that they have misunderstood or misrepresented the data that they are citing. My experience suggests both that this situation occurs with depressing frequency in personal injury cases, particularly with respect to medical statistics; and that opposing counsel allow these “misused” statistics to go unchallenged far too often.
The purpose of this paper will be to assist the courts to recognise the sources of statistical “misuse” and to institute methods of responding to the errors that arise. In a second paper, to be published in the next issue of the Expert Witness, I will report a number of statistics concerning the impact of disabilities on earning capacity, taken from sources that use reliable statistical techniques.
Sources of Statistical “Misuse”
There are three reasons why statistical evidence concerning the impact of disabilities on earnings might not be reliable. First, the expert may be basing his/her conclusions on past experience treating patients similar to the plaintiff, without taking into account the statistical uncertainties inherent in such an approach. Second, the expert, due to inadequacies in his or her own statistical training, may have misinterpreted data produced by a third party. Third, the expert may not have recognised that the data he or she is using to develop a prognosis were themselves collected or reported using unreliable statistical techniques. Examples of each of these types of error are discussed in this section.
1. The Expert’s Experience May Not Be Reliable
In many cases, experts drawn from the specialties that treat plaintiffs’ injuries – doctors, psychologists, physiotherapists, etc – rely upon their past experiences dealing with patients similar to the plaintiff to predict the impact that the plaintiffs’ injuries will have on his/her future ability to earn income. There are many reasons why the court should be reluctant to rely on this experiential evidence:
1. The plaintiff may not be representative of the patients that the medical expert normally treats: thus, the expert’s experience may not transfer easily to the plaintiff’s situation. For example, if the expert lives in a large city and the plaintiff comes from a rural area, the expert may not be familiar with the impact that a particular type of injury will have on the plaintiff’s ability to work on a farm. Or if the expert normally treats working-age patients, he/she may not be familiar with the impact of a particular type of disability on a senior or a minor.
2. If the medical expert has treated only a small number of individuals like the plaintiff, the sample size may be too small to draw statistically reliable inferences. For example, even a doctor who specialises in spinal cord injuries may have treated only a small number of quadriplegics. His/her experience with such a small number will provide only limited information concerning the plaintiff.
3. Even if the medical expert has treated a relatively large number of individuals like the plaintiff over his/her career, if the recommended treatment for those individuals has changed significantly recently, the expert may have treated only a small sample since that change. Again, the number of patients receiving the new treatment may not be sufficient to draw reliable inferences.
4. Often, the medical expert has been asked to comment on the impact that a disability has on employment, schooling, or earnings. As these are non-medical outcomes, the expert may not have systematically monitored them. Thus, the sample on which his/her information is based may be biased. For example, those patients who have adjusted well to their injuries, and who have returned to work, may be less likely to return to a doctor or psychologist for further treatment than those who have had difficulty adjusting. In this case, the doctor/psychologist may have developed an overly pessimistic view of the effects of the injury.
5. If the harm to the plaintiff is expected to continue for decades into the future and the medical expert has not been in practice long enough to have experience with patients whose treatment has continued for that length of time, the expert’s experience may not be reliable for predicting long-term consequences.
2. “Expert” Interpretation of Statistical Studies May Not be Reliable
When the expert attempts to supplement information drawn from his/her own experience with information drawn from studies conducted by third parties, a new set of problems arises. Specifically, the expert may lack sufficient knowledge or experience to be able to interpret statistical studies correctly:
1. The expert may have insufficient experience in the field to recognise deficiencies in the data. Many medical studies, for example, use a definition of “unemployment” that differs from that which is used by agencies such as Statistics Canada. The unsuspecting reader, who tried to compare the statistics drawn from the former with those drawn from the latter, could reach erroneous conclusions. Indeed, even within a reliable agency, such as Statistics Canada, similar-sounding names are often used to refer to quite different concepts. One must be careful, for example, to distinguish between “constant” and “current dollar” wages, between “net” and “gross” income, and between “real” and “nominal” interest rates. Failure to recognise these differences can lead to serious errors.
2. Because there is a lack of reliable data for predicting the effects of disability on labour market outcomes, experts are often forced to rely on data that were collected for other purposes. In many cases, this leads to the inappropriate use of such data. For example, doctors often use the American Medical Association “Guides to the Evaluation of Permanent Impairment” to calculate an index of the percentage of “whole body function” that has been lost due to an injury. Loss of an eye, for example, might be considered to reduce the patient’s “whole body” physical capacity by 25 percent.
Although the AMA did not design this index as a method of predicting the impact of disability on earnings, in the 1960’s and 1970’s it became common for experts to argue that a 25 percent reduction in whole body functioning implied a 25 percent reduction in earning capacity. Yet the connection between, say, loss of an eye or loss of a foot on the one hand, and loss of earnings capacity on the other is a tenuous one at best. Whereas loss of an eye could end the career of a professional athlete, for example, it might have very little impact on the career of an economist or lawyer.
Similarly, the rating system developed by Statistics Canada to categorise disabilities as mild, moderate, or severe yields statistics that are of very little value for predicting the effect of disability on the earnings of individuals within specific occupations. Yet many experts are currently using these statistics to make predictions of this nature.
3. The expert’s training in statistical analysis may be insufficient to allow him/her to distinguish reliable studies from unreliable ones. For example, studies that attempt to draw a connection between disabilities and labour market measures (such as income and employment) commonly rely on unsophisticated statistical techniques, making their conclusions very unreliable. Expert witnesses who are not well trained in statistical analysis may be unable to distinguish reliable studies from unreliable ones.
3. Published Studies May Not be Reliable
The most important problem facing the expert who wishes to predict the effect that disabilities will have on earning capacity is that many (if not most) of the statistical studies that have been published on this topic are unreliable. Some of the most important problems of which the courts should be aware include:
1. Many studies of the impact of disability on employment rely on very small samples. For example, it is not uncommon for articles on medical issues to study as few as 10 or 20 patients. Yet it is well known to statisticians that, in order to avoid the problem that “outliers” will bias statistical findings, it is usually necessary to have hundreds of observations.
2. Before the findings from a survey can reliably be projected to the population in general, it is crucial to ensure that the survey group is chosen in such a way as to be representative of the “population.” Many medical studies survey the patients from a single hospital or clinic, for example. But the findings of such a survey cannot reliably be projected to the general population if that hospital or clinic draws only from a sub-set of the population – for example, only from a relatively wealthy district or only from an urban population. And studies that attempt to contact patients many years after treatment may be biased in the sense that it may be easier to locate certain sub-sets of the group than others. For example, those paraplegics who have had the greatest success adjusting to their condition may be the ones who are most likely to have moved from the addresses they had at the time of admission to hospital and, therefore, may be the most difficult to reach at the time of the survey.
3. Although there are multiple factors that influence the effect of disability on employment and earnings, studies often collect information on only a small sample of these factors. For example, if older individuals are more likely to suffer from a particular disability (like arthritis) than are younger individuals, a data set that did not provide information about the ages of the individuals surveyed might appear to suggest that individuals with that disability earn higher average incomes than those who are not disabled (because individuals’ incomes tend to rise with age). Similarly, studies may overestimate the impact of a disability if more low-income than high-income individuals suffer from that disability.
4. Statistical studies can only show that variables – for example, disability and earnings – are correlated: they cannot show that one “causes” the other. That is, it is not clear whether disability causes low earnings, or whether occupations with low earnings have high accident rates. Labourers, for example, are more likely to experience on-the-job accidents than are office workers. If office workers earn more than labourers, data may appear to suggest that job-related accidents “cause” a significant reduction in earnings “because” those who have been injured earn less than the average person in the population. The more appropriate interpretation may have been that it was low income that had “caused” the accidents – that is, that it was the occupations with low earnings that had high accident rates.
One of the most common problems with medical studies is that they often do not report the age at which the disability became apparent. Yet we would expect that loss of a leg or an eye would affect individuals’ earnings differently if they were injured before they had completed their educations than if they were injured after they had established their careers.
What Can Counsel Do?
Medical experts often go unchallenged by the courts, even when they use unreliable statistical methods. How can the courts circumvent this problem, given that most lawyers and judges lack the expertise to question the bases of statistical testimony? I recommend three approaches:
1. If the opposing expert appears to be relying on his/her own experience as the basis for his/her predictions, at the “qualification” stage counsel should question the expert’s training in statistics. Do this not (necessarily) with the intention of convincing the court to reject his/her credentials, but to prepare the expert to agree with counsel that certain standards of data collection are important. Then use that agreement later to induce the expert to concede that his/her experience is inadequate for drawing reliable inferences.
2. Hire an expert in statistical analysis. Of those professions most commonly seen in court, actuaries and economists generally have the best training in advanced statistical techniques. Actuaries will have greater experience with life expectancy and fringe benefit data; while economists will have greater experience with data concerning education and incomes.
Other professions that also receive advanced statistical training are epidemiologists (often employed by medical schools) and statisticians (found in university departments of mathematics).
3. One option that is used occasionally in Canada, but less often than is justified, is to hire an expert in statistics to conduct original statistical analyses. As such an analysis is likely to cost at least $10,000-$20,000, it can not be justified for small, or “one-off” cases; but in a major injury case, in which the damages approach a million dollars, or with respect to the types of injury that counsel is likely to encounter many times, such an expenditure may well be justified. For example, my firm was hired by the defendants in one of the residential school sexual abuse cases. We used census data to estimate the earnings of individuals similar to the residents of those schools, but who had not been abused, and compared those earnings to the earnings of the plaintiffs. Similarly, in the second part of this paper (to be published in the next issue of the Expert Witness), I report the findings of a set of statistical analyses I conducted using Statistics Canada data, to determine the impact of various types of disabilities on educational attainment and income. These analyses could have been conducted by virtually any Ph.D.-trained economist in Canada.
