The Orphan Area for Meta-Analysis: Personnel Selection
Frank Schmidt
University of Iowa
Meta-analysis (Hunter & Schmidt, 2004) has been applied to over 100 different research literatures in I-O psychology alone. Relationships examined using these methods include job satisfaction and absenteeism, job satisfaction and job performance, job performance and turnover, role conflict and role ambiguity, goal setting and goal attainment, goal difficulty and goal commitment, work–family conflict and life satisfaction, corporate social responsibility and corporate financial outcomes, just to name a few. Such applications are ubiquitous in I-O research journals. These methods are also used in areas outside of I-O psychology, including social, educational, differential, and developmental psychology and in areas outside psychology, such as finance, medicine, economics, and political science (Hunter & Schmidt, 2004, ch. 1).
In all these areas except one, these methods have been accepted, welcomed, and embraced. The one exception is personnel selection, the sole area where these methods have been and continue to be controversial. (In this area these methods are referred to as validity generalization [VG] methods.)
Actually, this is too broad a statement because some applications of these methods in personnel selection are not controversial: applications to personality tests and integrity, for example. Another example is the application to the GATB (General Aptitude Test Battery) of the U.S. Department of Labor, which was used for years in a large nationwide VG-based testing program that was endorsed by three major civil rights organizations. What areas of this sort have in common is the absence of racial or ethnic mean differences in scores. Personality and integrity tests don’t show such differences, and in the case of the GATB program, race norming eliminated all group differences. The one area in which these methods are controversial is selection methods that show group differences—mostly cognitive ability tests, such as verbal, quantitative, and spatial ability and (especially) measures of general mental ability (GMA), which have been shown to be the generally most valid predictor of job performance.
What does this mean? It means that the controversy is not really about VG methods or conclusions. That is just a smokescreen for the real issue: minority hiring. The real issue is the use of racial preferences to attain workforce diversity not the scientific soundness of VG methods. VG methods are strongly endorsed by the 1999 APA-AERA-NCME Standards, the 2003 SIOP Principles, and by two National Academy of Science reports. It is hard to imagine stronger scientific and professional endorsement for any procedure or set of research findings. In the controversy over VG, we are pretending that something is wrong with the research methods and conclusions when the real issue is something entirely different. Workforce diversity may be a laudable goal, but it cannot be attained through pretense and intellectual slight of hand.
With this kind of foundation of scientific and profession support, you would think that I-O psychologists would have done a great job of educating the legal profession, the courts, and the federal enforcement agencies about VG methods and their associated research findings. But they have not. For example, SIOP has for years published the Frontier book series on important research findings in I-O psychology. Despite the fact that it has more research support than practically any other area, there has been no Frontier series book on VG and its findings. Other sciences and professions—medicine, biology, engineering—have done a much better job on this. When lawyers, courts, other organizations, or the media appear to endorse false ideas, these groups launch vigorous public educational campaigns. They are on TV talk shows and the Internet and in magazines and newspapers very quickly. A good example of this is the vigorous way that biologists from top universities fought back publicly against the doctrine advanced in the media by creationists and intelligent design people that evolution was not a fact, only a theory, and should not be taught as a scientific fact in the schools. I-O psychologists have produced no such response. They have been timid and reluctant to publicly defend their well-established research findings. When I presented this talk at the 2006 SIOP conference, someone in the audience stated that this failure was because I-O psychologists did not want to be called racists, even if unfairly. I pointed out that the biologists who defended the theory of evolution were attacked as atheists—but this did not stop them. I could also have noted that the research finding that might stimulate false charges of racism—the well-established finding of predictive fairness of mental ability tests despite the presence of group differences—is not part of VG methods and was not established using VG methods. In addition, this finding has been endorsed by two National Academy of Sciences reports, providing a strong defense against any loose charge of racism.
(Some I-Os have even stated they are opposed to VG because its acceptance would mean far fewer local validation studies would have to be conducted, reducing their work and income. How did dentists react to fluoridation of drinking water? Did they say, “Don’t fluoridate the water because we will have fewer cavities to fill”? No, they met their professional responsibilities and embraced fluoridation. We should do the same.)
Indeed, instead of meeting professional and scientific obligations to educate lawyers, courts, the media, and the public, many I-O psychologists actually look to the courts to educate them on the meaning of and value of VG and VG research conclusions. Some I-O psychologists constantly pore over court opinions looking for nonexistent guidance on what is professionally acceptable and what is not. There are many articles and talks of this sort. This is exactly backward from what should be happening. Even considered only from a legal point of view, this practice is based on a general failure to recognize that the case law does not build up in any cumulative or systematic way in any area related to personnel selection. The decisions of individual judges are highly idiosyncratic and in fact are essentially random. Judges’ decisions depend on accidents of personality and attitudes of individual judges, accidents of which research evidence happened to be presented or not, how well it was presented, and whether the judge was intelligent enough to understand it—all essentially random factors from case to case. Jerome Frank, one of the founders of the legal realist movement, even stated that judicial decisions were sometimes based on nothing more than “what the judge had for breakfast that morning.”
Yet, this is the “database” that some I-O psychologists prefer to consult to find guidance for professional practice. Some I-O psychologists appear to view every judge as some kind of Solomon and to then probe for deep meanings and insights that are just not there. Instead of building on the strong scientific foundation of our field and using this to educate judges, the media, and the public, they are consulting the Delphic Oracle and seeking knowledge in the reading of chicken entrails. You are not likely to find the scientific truths of personnel selection in chicken entrails!
My reaction to all this has been a partial withdrawal of interest from personnel selection. Some years back I realized that personnel selection had become a churning arena of constant irrationality. There is a huge disconnect between what we know to be true from research and what people pretend to be true. There is a serious corruption of scientific truth caused by legal and ideological intrusions into the field of selection and the failure of the profession to respond appropriately to these intrusions. I have found this frustrating. But at the same time I became aware that even within I-O psychology alone there are over 100 other areas of research in which VG-meta-analysis methods are not only accepted and noncontroversial, but welcomed, embraced, and praised. And there are many other such areas outside I-O psychology. So I found it was more satisfying and fulfilling to devote my time to the development and improvement of general meta-analysis methods. Examples include the revision and updating of meta-analysis methods seen in the 2004 edition of the Hunter and Schmidt meta-analysis book and several recent journal articles on general meta-analysis methods. In these areas, contributions are evaluated rationally and logically; there is no ideology, irrationality, or hidden emotional agendas. There is a willingness to credit scientific evidence that seems to have been lost in personnel selection. In this sense, it would be fair to view much of the practice of personnel selection today in I-O psychology as sort of an intellectual backwater. It does not seem to be where the intellectual action and excitement is at present.
But my hope is that we can change this. This is why I am writing this article and why I gave the SIOP talk on which this article is based. This talk was given at a symposium entitled “Validity Generalization at Work: Is it Legal to be Scientific?” (John Weiner, Chair). Excellent presentations were given by John Wiener, Jim Sharf, David Copus, and Keith Pyburn, all stimulated by the same sense of frustration expressed in this article. This is a very hopeful sign. It means we may yet be able to turn this thing around and regain our professional and scientific respect. I certainly hope we can.
Selected Bibliography for Additional Reading
American Educational Research Association, American Psychological Association, and the National Council on Measurement in Education (1999). Standards for educational and psychological testing. (5th ed.). Washington, D.C.: American Educational Research Association.
Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks, CA: Sage.
Hartigan, J. A., & Wigdor, A. K. (Eds.). (1989). Fairness in employment testing: Validity generalization, minority issues, and the General Aptitude Test Battery. Washington, D.C.: National Academy of Sciences Press.
Landy, F. (2003). Validity generalization: Then and now. In K. R. Murphy (Ed.), Validity generalization: A critical review. Mahwah, NJ: Erlbaum (pp. 155–196).
Schmidt, F. L., & Hunter, J. E. (2003). History, development, evolution, and impact of validity generalization and meta-analysis methods, 1975–2001. In K.R. Murphy (Ed.), Validity generalization: A critical review. Mahwah, NJ: Erlbaum (pp. 31–66).
Society for Industrial and Organizational Psychology (2003). Principles for the validation and use of personnel selection procedures. (4th Ed.). Bowling Green, OH.