Good Science - Good Practice

Marcus W. Dickson
Wayne State University

Jamie Madigan
Ameren Services

For those of you who weren’t in attendance or who don’t pay much attention to all things officially SIOP, our annual conference was held in San Francisco just this April. Like we’ve done in the past, this issue’s column will be dedicated to what we saw at the conference this year. A lot of our favorite topics like personality research, leadership development, staffing, and legal issues were represented. There were also some topics that seemed to be enjoying a wave of newer popularity as well, such as health and safety, employee engagement, and working with international companies.

And cutting across all these subjects was our favorite: good science melded with good practice. This was evidenced not only by the number of separate sessions covering the same topic from the points of view of both academics and practitioners but also the number of sessions that tried to deal with both sets of concerns simultaneously. A special theme track on how well I-O education is serving tomorrow’s scientist/practitioners also cut a wide swath through an entire day. So let’s take a look at what sessions we were able to attend. (Our citations at the end of this article refer to the sessions by their number in the SIOP conference program.)

One of the earlier sessions was entitled “Bridging the Scientist–Practitioner Gap: Senior Executives Identify Critical Research Needs” and seemed to fit the bill pretty well. The panel was headed by members of the Society for Human Resources Management (SHRM) Foundation, which is a nonprofit affiliate of the larger SHRM organization dedicated to examining issues such as these. The entire report can be downloaded on their Web site at http://www.shrm.org/foundation/, but the session gave a pretty good overview focusing mainly on a survey of 526 executives (both from HR and other business areas across many industries and company sizes). The survey questioned those executives about what they saw as the most pressing business challenges facing them in the future. In order of decreasing average concern for those surveyed, the topics were:

• Succession planning (actually by far the most anxiety-producing challenge)
• Recruiting and selection
• Engaging and retaining talent
• Providing leaders the skills they need to be successful
• Dealing with rising health care costs
• Creating and maintaining a performance-based culture

What are these executives looking for from people like us (i.e., I-O and HR professionals) to deal with these challenges? Three things: best practices, tools, and information about return on investment.

When the SHRM Foundation panel and members of the audience were asked to comment on these findings, nobody expressed particular surprise at the findings. In fact, a common theme running through everyone’s responses was that what actually surprises them is the lack of surprise. We’ve been studying these topics for a long time. Besides the fact that systems like succession planning, wellness programs, and leadership development require a long-term effort rather than a flash in the pan to be most effective, several people were a bit confused by the fact that so many executives felt they had no plan in place to meet any of these challenges. “How can that be?” asked one member of the audience. “We have studied these things for decades, and we know how to do them. Why do these executives think they have no plan?”

Indeed, this problem harkens back to issues that we’ve discussed at length in this very column: How do you get the research and the science into the hands of practitioners like these executives? There are plenty of potential vehicles for such a knowledge transfer: executive education, conferences, journals, magazines, consulting, skywriting, singing telegrams, and more. But if the SHRM Foundation’s report is to be believed, there’s still a disconnect. One panel member acknowledged the issue and offered two possible causes.

First is that we, the I-O professionals, have not done an adequate job of communicating the research, models, theories, and best practices. “Research is not complete,” he said, “until it has been communicated.”  I-O scientists and research-oriented practitioners need to bring utility in as part of their models and part of their approach for evaluating the merits of research. How will this idea work? How will it be put into practice? Failure to do this results in overreliance on limited heuristics like benchmarking.

The second reason given for the science/practice disconnect places blame closer to the feet of executives like those who completed the survey. There seems to be a prevailing demand for quick fixes and solutions that can simply be plugged in and turned on. These fixes do not exist for the most part, especially with regards to those slow-cooked systems like succession planning, leadership development, and culture change. These executives need to invite HR and I-O experts to that big, round, strategy table. We also need to examine the interconnectedness of all of these issues, as well as the high-level business concerns that make them challenges to begin with. One audience member astutely asked “What problem is succession planning the solution to? Business continuity? Knowledge transfer? Customer retention? We need to know that kind of thing.”

Many good questions were raised, even if a lot of them have been asked for a while now. The SHRM Foundation members left us with the comment that their research is ongoing and will address many of these exact issues.

“Validation Research Strategies: Ensuring Situational Sufficiency and ‘Appropriate’ Professional Rigor” was the title of another one of the panels we attended. This one took a frank look at a problem that a lot of researchers have working in the realm of test validation: How do you strike the balance between the demand of scientific rigor and the realities of working in an organization? Binning and Barrett’s (1989) model of multiple different kinds of inferential validity is nice, but anyone making an honest assessment of typical organizational research would have to quickly (and maybe a bit sheepishly) admit that we never get anywhere close to that level of comprehensive research. Organizational constraints tend to sweep those kinds of plans aside and force us to make hard decisions about what will do given the situation and stakes involved.

The panel members, all of them accomplished researchers, set out to tackle this issue by answering five basic questions:

• Do operational standards of validity vary as a function of a study’s purpose?
• What factors legitimately limit rigor?
• How does the type of organizational practice being validated affect the research?
• How do existing publications affect the research?
• What are the appropriate role of subject matter experts?

Close examination of these questions reveal a number of subtexts and thinly disguised implications related to many of the common dilemmas researchers face in this area: when to rely on “easier” validation techniques, when to trade off rigor for legal exposure, when to consider alternatives to doing your own research, and so forth. And in fact, the panel did not disappoint in their willingness to step up and grab hold of these issues.

One common theme that ran through their responses was that one’s definition of “rigor” varies from context to context and even person to person within the same context. Doing research for publication in a refereed journal generally demands more rigor (and bookkeeping) than organizational interventions, but this is a somewhat limited point given that there are few, if any, test-validation studies published in journals. Other panel members noted that work done for development and promotional purposes (as opposed to external hiring decisions) might be less rigorous, though they may demand additional work. All of the panel members generally agreed however that “less rigorous” does not translate to “not rigorous” or “lazy.” Professional standards and ethics still demand doing the best we can, even if working within constraints.

What are these constraints to scientific rigor? Just the usual suspects: low sample sizes, inability to sell decision makers on procedures, limited access to data, budgets, uncooperative groups (especially unions or advocacy groups), fear of litigation, unreliability and restriction of range on criteria, limited time, and being saddled with lousy performance appraisal systems for purposes of data collection. We often encounter limited success in explaining technical terms to key decision makers. Maybe you can get the idea of confidence intervals across, but conveying something more esoteric like the inferences needed to establish construct validity is going to be a bit more dicey.

The panelists offered a few specific solutions to these issues, such as developing your own tool to gather criteria data, using multiple raters to combat unreliability, and using forced distributions and correction formulas to combat restriction or range. In the end, though, much of the remaining advice seemed to center around the idea of “do the best you can.”

And then there was the Saturday theme track, focusing on the state of I-O education. We wanted to spend some time talking about this event (formally titled “Preparing for the Future: A Critical and Constructive Look at I-O Education”) because it hits so close to home for this column; much of it was about how our training can be better linked to the demands and requirements of practice. The committee that put together the full-day track did a fantastic job (full disclosure: Marcus was part of that committee and was a presenter and so may be slightly biased), and the line-up of speakers and formats created a compelling event for people concerned about linking good science and good practice.

The opening keynote address by Ben Schneider was full of good insights, including the reminder that our field has been focused on joining science with practice since its inception (Ben noted that Morris Viteles was both an academic [University of Pennsylvania] and a practitioner [director of personnel research, Philadelphia Electric]). He also highlighted some different distinctions than most of us usually consider, including the tension between those doing primarily personnel psychology-type work (where most of our practice has historically been) and those doing more organizational psychology-oriented work (where many newer practice avenues have emerged). Schneider reminded the audience that this tension is also not new, arguing that “The future requires simultaneous entertainment of both I and O in science and practice—and therefore in education” (Lyman Porter said this in 1966!). He also highlighted the suspicions academics have of ideas that originate in practice and that practitioners have of things that appear in the Journal of Applied Psychology. Although Good Science–Good Practice isn’t explicitly about the academic–practitioner schism, much of what Ben had to say rang true for us as we think about the issues that comprise this column. One of the points that particularly struck us was his argument that it is the “young’uns” who are forcing the field to address new topics in new ways, and it is often these people who push for more integration across the I and O branches of our field and across the academic–practitioner divide, as well. Ben’s point? “Let ‘em!”

Rather than go session by session through the rest of the theme track day, we’ll hit a few key points that were emphasized repeatedly and that directly relate to this column’s focus. One of those themes was the extent to which graduate students in I-O psychology receive applied experiences, and the extent to which those experiences were well-designed and useful for professional development.

The “preparation gap” between what I-O graduate programs teach and what I-O practitioners need to know was a recurring theme as well. Rob Silzer made a presentation of the results of the Professional Practice Committee’s recent Practitioner Needs Survey, and this issue arose several times. For example, practitioners reported that the vast majority of the skills and knowledge areas identified in the survey were learned on the job rather than in graduate school. Practitioners were also quoted in the report as saying that graduate training programs should “help graduate programs better prepare their students for the practitioner environment;” though as always, it isn’t entirely clear how best to do that, which of the skills identified as being learned on the job would be most suited to incorporation into graduate training programs, for example. Marcus spoke on behalf of SIOP’s Education and Training Committee, presenting pilot results of a survey on business and consulting skills, and identified disagreement in the results as to whether training in practitioner-oriented skills is even part of what graduate school should be about. Everyone seems to agree that we need good science and good practice, but we continue to debate and discuss how our graduate training programs can contribute to the creation of scientist–practitioners. The afternoon session on “Connecting Education to Practice” took an innovative approach of having four major topic areas (consulting operations in graduate training programs, internships, the skill gap between what is taught and what is needed in practice, and the societal values that I-O professionals have an obligation to uphold), having experts in those specific areas make brief presentations to the whole group and then breaking into smaller groups to focus on each topic in greater depth. This session particularly left us feeling optimistic about the attention being paid to merging science and practice, and to identifying common goals and shared challenges.

Issues of standards were frequently raised as well with focus on both the accreditation of graduate training programs and the licensure of I-O psychologists garnering attention. In an innovative debate format featuring several well-known SIOP members, the first question was whether I-O graduate training programs should be accredited. Jim Outtz argued that they should be, to ensure commonality of training and qualification, with Ann Marie Ryan making a spirited defense against accreditation, pointing out the historic differences in training models and the strength that can come from those differences. There were also questions around the need for licensure as a means of assuring quality control in practice (i.e., ensuring that qualified I-O people do I-O work) and calls for graduate training programs to do more to prepare students for state licensure. As part of this topic, Rob Silzer’s report on the Practitioner Needs Survey suggested that the largest group of full-time practitioners are not sure whether they are eligible to be licensed, and similarly, they don’t know whether their graduate training programs prepared them for licensure or not. So the extent to which licensure is an issue that affects Good Science–Good Practice remains somewhat fuzzy—but it is also not going away.

There were several other worthwhile sessions, including an invited poster session on I-O classroom innovations and an integrative session to wrap up the day, in which an “integration team” had put together questions emerging from throughout the day and posed those questions to a distinguished panel. The theme track planning group, headed up by John C. Scott of APT, did a great job of putting together a full-day exploration of issues about I-O education, many of which have direct impact on our interests in this column. They will be doing a write-up of their conclusions from the day’s presentations for the next issue of TIP, so keep an eye out for that.

One of the respondents to the Professional Practice Practitioner Needs Survey said that “SIOP as a society continues to pay only lip service to bridging the gap between science and practice.” We don’t know the experiences or perceptions behind that comment, but based on what we saw at SIOP, we’re not sure we’d agree. The range of sessions, and the dedication of the many people involved in efforts to bridge that gap lead us to continue having plenty to write about in Good Science–Good Practice. Feel free to send us suggestions, though—Jamie is at HMadigan@ameren.com, and Marcus is at marcus.dickson@wayne.edu.

References

     Binning, J. F., & Barrett, G. V. (1989). Validity of personnel decisions: A conceptual analysis of the inferential and evidential bases. Journal of Applied Psychology, 74, 478–494.
     60. Bridging the Scientist–Practitioner Gap: Senior Executives Identify Critical Research Needs. Fredrick Morgeson, Chair.
     68. Validation Research Strategies: Ensuring Situational Sufficiency and “Appropriate” Professional Rigor. Levi R. Nieminen and John Arnold, Chairs.
     208. Keynote Address: Preparing for the Future: A Critical-and-Constructive Look at I-O Education. Steven G. Rogelberg and John C. Scott, Chairs. Ben Schneider, Presenter.
     217. A Special Debate on the State of I-O Training. John C. Scott, Chair. Frank J. Landy, James L. Outtz, Nancy T. Tippins, Frank L. Schmidt, Angelo S. DeNisi, and Ann Marie Ryan, Presenters.
     238. Meeting Stakeholder Needs: Views From Industry, Consulting, and Academia. Dan Sachau, Chair. Marcus W. Dickson, Paul R. Sackett, Jeffrey J. McHenry, Irwin L. Goldstein, Robert F. Silzer, and Derek R. Avery, Presenters.
     265. Innovations in I-O Teaching and Curricula. Linda R. Shanock, Chair. Elise L. Amel, James M. Conway, Roseanne J. Foti, Tomas R. Giberson, Peter D. Bachiochi, Meridith E. Selden, Zinta S. Byrne, Kurt Kraiger, Bill Attenweiler, Stefanie K. Johnson, and Eden B. King, Presenters.
     277.  Connecting Education to Practice. Sara P. Weiner, Chair. John D. Arnold, Bruce M. Fisher, Richard Moffett, Allan H. Church, Angela K. Pratt, Janet L. Barnes-Farrell, Joe Colihan, Richard A. Guzzo, Jennifer Z. Gillespie, James L. Outtz, and Walter C. Borman, Presenters.
     292. The Future of I-O Education: Theme Track Integration and Open Forum. Michelle (Mikki) Hebl, Moderator. Stephen D. Steinhaus, Chair. Benjamin Schneider, Steven G. Rogelberg, Gary P. Latham, Laura L. Koppes, Kevin R. Murphy, Richard J. Klimoski, William H. Macey, and Sandra O. Davis, Presenters.