SIOP 2007 Keynote Address: Evidence-Based Management in Industrial and Organizational Psychology: A Celebration of Accomplishments and Some Aspirations for the Future¹

Jeffrey Pfeffer
Stanford University

¹ This article is based on the keynote address delivered at the Society of Industry and Organizational Psychology Annual Conference in New York on April 27, 2007.

SIOP, in both its research and practice dimensions, aspires to be and is evidence-based (Pfeffer and Sutton, 2006).  Here I want to provide some perspective on what evidence-based management is and also to both acknowledge the amazing accomplishments of industrial and organizational psychology and outline some aspirations for the future.

We wrote Hard Facts for a number of reasons.  First, after we published The Knowing–Doing Gap (Pfeffer & Sutton, 2000), a book about the barriers and obstacles to turning knowledge into action and how to overcome them, people would tell us that they were actually putting knowledge into action.  They would then describe some practice that was completely inconsistent with almost all the evidence in psychology and organizational behavior.  So, we recognized that not only was there a knowing–doing problem, there was also a doing–knowing problem, in which organizations were doing things that were not based on the best available evidence and knowledge.

Second, we recognized that there were evidence-based movements in medicine, education, criminology, and the policy sciences, but that management had somehow been largely untouched by this trend.  We see evidence-based management as a community of practice and as an evolving body of knowledge and experience. To that end, we have launched a Web site, www.evidence-basedmanagement.com, and we invite SIOP members to contribute case studies, course outlines, and other material to help bring the evidence-based practice of management to fruition.

And third, we wrote the book because organizations have profound effects on people, and such effects are not always positive.  The existing evidence on declining levels of job satisfaction, high levels of distrust, disengagement, and workplace bullying and psychological stress is shocking (for a partial review, see Pfeffer, in press).  We have a responsibility to try and change management practice for the better.

What is Evidence-Based Management?

Evidence-based management is not so much about statistical techniques, although the knowledge of statistics is important and useful, and it is not even about building enormous data bases because having data is quite different from thinking in a diagnostic fashion using data.

EBM is first of all a way of thinking—being committed to doing things on the basis of evidence and data and not acting on the basis of “ideology” and belief.  It is important to contrast the objectives and ethos of industrial and organizational psychology, in both its science and practice dimensions, with much contemporary managerial practice that is driven in some substantial measure by vendors with things to sell. I would also contrast industrial and organizational psychology with some other social sciences such as economics, which are almost like religions in their adherence to orthodoxy and where ideas that depart from orthodoxy receive much more scrutiny and skepticism than those that conform to the dominant belief system. 

Second, EBM entails a commitment to learning about human behavior and using that knowledge in the design of organizations and management practices—a commitment to both acquire knowledge and to turn that knowledge into action.  This is what I-O psychology is about, and I honor and commend the SIOP membership for being serious about the study of human behavior and the efforts to implement that knowledge in practice.

An implication is seeing one’s organization as an “unfinished prototype,” where experimentation and learning by doing are encouraged.  For instance, Yahoo runs multiple experiments each day with its Web site design.  Similarly, Gary Loveman, the CEO of Harrah’s Entertainment, has built an experimenting culture in Harrah’s in which trying things out to see what works—be that some marketing promotion or changing the “hold” on slot machines—is the norm.

The Celebration: How I-O Psychology Is Consistent With
EBM Helps Knowledge and Practice

The first way in which I-O psychology is consistent with the ideas of evidence-based management is the willingness to do research on what doesn’t work and is ineffective and to publish “negative” findings about those things that don’t work.  So, for instance, there is a large literature on the causes of unreliability of interviewing as a selection technique (e.g., McDaniel, Whetzel, Schmidt, &. Maurer, 1994), biases of various kinds in performance appraisal (e.g., Banks & Murphy, 1985; Fletcher, 2001), critical assessments of constructs such as emotional intelligence (e.g., Davies, Stankov, & Roberts, 1998), and so forth.  Publishing negative findings and ideas that don’t work is rare because most fields emphasize only positive results and confirmation of hypotheses.  However, learning what doesn’t work is as important as learning what does.  This is certainly true in medicine.  For example, recent research on whether drug-eluting stents are more effective than bare metal has fundamentally caused a rethinking of how heart disease is managed. 

If we are to build an evidence-based science that informs practice, we need to learn about reasonable ideas that don’t work as well as those that do.  In fact, there is an argument that science progresses most effectively if it employs a logic of strong inference, seeking to find conditions where theory fails so that from those failures of theory more refined and accurate insights can be developed (e.g., Mackenzie & House, 1978).  In considering the importance of learning what works and what doesn’t, meta-analyses are valuable.  But they are even more valuable to the extent that we are able to uncover and incorporate in the meta-analyses scientifically sound but unpublished work.  At a minimum, we should at least attempt to correct for this sampling error (e.g., McDaniel, Rothstein, & Whetzel, 2006). 

A second dimension of evidence-based management consistent with I-O psychology is the long-standing interest in implementation.  To take just one example, more than a decade ago Johns (1993) documented the low level of adoption of I-O science into practice, outlined a number of possible explanations for this implementation failure, and then recommended some possible actions.  I suspect the article could be rewritten today with surprisingly little change, except that because the knowledge base of the field has increased, the list of unimplemented knowledge would be longer.

The third evidence-based dimension worthy of note is the commitment to measurement and methods—to scientific rigor as opposed to hype, fads, and fashion.  Here, the emphasis on reliability and validity, on not only identifying constructs but making sure they are accurately and adequately measured, is crucial.  As I consistently remind doctoral students, even if a theory or hypothesis is correct, the likelihood of effectively testing that theory is going to be reduced without good measures of the underlying constructs.

The Aspirations:  What I-O Psychology Might Do

Much is needed from the members of SIOP in a world in which evidence-based action and pragmatism seem to be on the decline.  The U.S. was once a nation of pragmatists and tinkerers, for instance, Thomas Edison and his workshop in Menlo Park, New Jersey.  What has happened to that pragmatism?  In Singapore, the government experimented with a number of different plausible approaches for teaching mathematics in school.  They picked the program that worked the best.  In the U.S., everything, including mathematics instruction, seems to have become a “religious” issue—and I mean that almost literally.  It is critical that in the domain of science and public policy, ideas and data are not twisted by fear and ideology and that we use the best evidence to make judgments about how to take action.  Criminology may be a field that is in even worse shape than management in this respect.  Although there is, for instance, data showing that curfews don’t reduce juvenile crime (and in many instances are imposed without there being data suggesting that juvenile crime is increasing), belief and values have overridden these data, to the harm of both individuals and society.

It is essential that we as psychologists and management professionals engage with the world of public policy and public discourse.  It is great that research results and their implication for practice appear on SIOP’s Web site.  But SIOP needs to also ensure that the research findings of its members don’t go unrecognized.  The Academy of Management has hired a public relations firm to get more attention for its scholarship in public debates, and I hope SIOP can be part of that effort in some way.  We need to ensure that our findings are not just reported in journals but also discussed in the media and become part of public discourse.  My own efforts in this regard have entailed writing a monthly column for Business 2.0 for more than 4 years. 

We live in an era in which the findings of both physical and social science have been “edited” to conform with prevailing political views.  This anti-science, know-nothing, anti-intellectual movement is not only inconsistent with the fundamental values of academia and this organization, these are attitudes which, when implemented in practice, will lead inexorably to the decline of our knowledge-based economy.  Scientists operate in a global labor market and will move to where there are the best opportunities and where there is more interest and investment in knowledge and technology. The growing movement of stem cell research to the UK and Singapore is an excellent example. 

It would also be useful to broaden our studies to include not only what works and why but also the implementation of our findings.  This topic formed one portion, albeit a not necessarily “empirically tested” part of Bob’s and my book—the barriers to using EBM.  The occasional special issue and article on implementation are nice, but turning science into practice might become a more central theme in both our empirical and theoretical work.

In this research, it would be helpful to acknowledge multiple methods, that qualitative data are also data, and that not everything that matters can be counted, to be open to multiple methods.  This is not to in any way deprecate the importance of measurement, rigor, and statistical testing.  But it is simply recognition that there are multiple ways of knowing and that we should not foreclose any that promise to advance our science and understanding.

Finally, I have an aspiration concerned with building integrated theoretical structures, which I  believe is the only way we are going to compete with economics and also the best way to have our results “stick.” One example of such an effort would be our Organizational Science paper on self-enhancement as a way of understanding many power and influence processes (Pfeffer & Fong, 2005).  But neither that particular theoretical perspective nor the specific focus to which it is applied is privileged. We need to get beyond the “list-like” nature of our field to find a few true, enduring insights and observations that help, in a parsimonious fashion, to capture important phenomena.

SIOP and the people who are its members share a commitment to evidence-based practice—to bringing evidence and science to the world of organizational management.  Much has been accomplished and there is much to both acknowledge and celebrate.  But as I read the occasional article on what proportion of our knowledge is being implemented in practice, and when I see the harm that organizations are doing to the mental and even the physical health of many of their employees on a daily basis, I recognize that so much remains to be done.

References

     Banks, C. G., and Murphy, K. R. (1985).  Toward narrowing the research-practice gap in performance appraisal.  Personnel Psychology, 38, 335–345.
     Davies, M., Stankov L., and Roberts, R. D. (1998).  Emotional intelligence:  In search of an elusive construct.  Journal of Personality and Social Psychology, 75, 989–1015.
     Fletcher, C. (2001).  Performance appraisal and management:  The developing research agenda.  Journal of Occupational and Organizational Psychology, 74, 473–487.
     Johns, G. (1993).  Constraints on the adoption of psychology-based personnel practices:  Lessons from organizational innovation.  Personnel Psychology, 46, 569–592.
     Mackenzie, K. D., and House, R. (1978).  Paradigm development in the social sciences:  A proposed research strategy.  Academy of Management Review, 3, 7–23.
     McDaniel, M. A., Rothstein H. R., and Whetzel D. L. (2006).  Publication bias:  A case study of four test vendors.  Personnel Psychology, 59, 927–953.
     McDaniel, M. A., Whetzel D. L., Schmidt F. L., and Maurer S. D. (1994).  The validity of employment interviews:  A comprehensive review and meta-analysis.  Journal of Applied Psychology, 79, 599–616.
     Pfeffer, J. (in press).  Human resources from an organizational behavior perspective:  Some paradoxes explained.  Journal of Economic Perspectives (in press).
     Pfeffer, J., and Fong, C. T. (2005).  Building organization theory from first principles:  The self-enhancement motive and understanding power and influence.  Organization Science, 16, 372–388.
     Pfeffer, J., and Sutton, R. I. (2000). The knowing–doing gap.  Boston:  Harvard Business School Press.
     Pfeffer, J., and Sutton, R. I. (2006).  Hard facts, dangerous half-truths, and total nonsense:  Profiting from evidence-based management.  Boston:  Harvard Business School Press.