Most Important I-O Articles for Research/Practice in the Last 5 (or so) Years

Scott Highhouse, Bowling Green State University

Industrial-organizational (I-O) psychology academics and practitioners (N = 145) from around the world were asked to nominate what they believed to be, in the last 5 years, the most important published article for research and/or practice. The article provides a framework for crowdsourcing important advances in the field and can be directed toward specific types of advances.

I recently shared, on the BGSU industrial-organizational (I-O) psychology program listserv, a meta-analysis by Andrew Speer and colleagues (Speer et al., 2021) showing that the validity of biodata has been underestimated. I shared with my colleagues my belief that, along with a recent meta-analysis by Sackett and colleagues (2017) showing that assessment centers are better head-to-head predictors of job performance than GMA, the biodata meta-analysis shows that our field has been overreliant on the classic 1998 article by Schmidt and Hunter.

Sharing articles that may have been overlooked by busy colleagues and students made me think that it might be possible to crowdsource papers that prominent academics and practitioners in my network believe are the most important publications in recent years. Such a list might have a myriad of uses. It could provide the foundation for seminars on current advances. It could be used by students preparing for their preliminary (comprehensive) exams. And, it could alert busy practitioners, students, science writers, and other stakeholders to the important advances in the field.

I emailed 91 members of the Personnel and Human Resources Research Group (PHRRG or “frog”), a by-invitation group of I-O and related scholars founded over 35 years ago. I also emailed the editorial board of Personnel Assessment and Decisions (PAD; https://scholarworks.bgsu.edu/pad/), consisting of an additional 54 I-O scholars and practitioners from all over the world. This resulted in (N =145) invitations to accomplished I-O scholars, in both academic and applied settings, tasked with nominating a recent article of importance to the field. The invitation was as follows:

I am writing to get nominations for the most important I-O article in the last 5 years for research and/or practice. My goal is to bring to light important advances that may be overlooked by scholars, and that may have never reached practitioners. Please send me your pick for the most important contribution (in any journal), along with a very brief explanation of why you chose it. 

I then provided, as an example, my contribution which appears below. I received 18 nominations for articles. Some contributors nominated two articles. Many of these were edited for clarity and efficiency, so any mistakes are my own.

Nominated Articles

You will note that the articles for which I received nominations are presented in general categories: selection, personality/individual differences, leadership, and work/family/gender issues. A final section includes self-nominations from two late-career scholars. I gave these a separate section because I found it inspirational that both of these self-nominees indicated that their recently published article was the most important work of their careers. It was most certainly encouraging to me personally that my best work may still lie ahead!

Selection

1. Sackett, P. R., Shewach, O. R., & Keiser, H. N. (2017). Assessment centers versus cognitive ability tests: Challenging the conventional wisdom on criterion-related validity. Journal of Applied Psychology, 102(10), 1435.

This article shows that comparing validities is more complicated than pulling effect sizes from Schmidt and Hunter (1998). When samples and criteria are held constant, assessment centers (corrected r = .44) outperform cognitive ability tests (corrected r = .22) for predicting on-the-job performance. 

- Scott Highhouse, Bowling Green State University

2. Tippins, N. T., Oswald, F., & McPhail, S. M. (2021). Scientific, legal, and ethical concerns about AI-based personnel selection tools: A call to action. Personnel Assessment and Decisions, 7 (https://scholarworks.bgsu.edu/pad/vol7/iss2/1/)

This article was already available via PsyArXiv before being published in PAD, and it’s one of the few articles that even before it was printed made an impact on practice (at least here in the Netherlands).

- Janneke Oostrom, VU University Amsterdam

To me, there is “too much technology and too little science” in the application by various firms, and I worry about the effects on non-white males, females, and those who have immigrated.

- James T. Austin, Ohio State University

3. Dahlke, J. A. & Sackett, P. R. (2022). On the assessment of predictive bias in selection systems with multiple predictors. Journal of Applied Psychology. Advance online publication. http://dx.doi.org.ezproxy.bgsu.edu/10.1037/apl0000996

This article examines differential prediction in multipredictor selection systems and explores generalizability across race/ethnic and gender subgroups. This represents an important extension to the literature and addresses both a practically and scientifically important topic.

- Deborah Whetzel, HumRRO

4. Cucina, J. M., Caputo, P. M., Thibodeaux, H. F., & Maclane, C. N. (2012). Unlocking the key to biodata scoring: A comparison of empirical, rational, and hybrid approaches at different sample sizes. Personnel Psychology, 65(2), 385–428.

The study shows empirically how much sample size will dictate the benefit of more refined keying for improving biodata validity. The research is important in its own right but also sets an example for other domains.

- Fred Oswald, Rice University

5. Lievens, F., & Sackett, P.R. (2017). The effects of predictor method factors on selection outcomes: A modular approach to personnel selection procedures. Journal of Applied Psychology, 102(1), 43–66.

This article presents a modular framework that identifies the important measurement components of selection procedures and reviews how each of these components affects validity, subgroup differences, and applicant reactions. I think this article has offered a new perspective in terms of how we should look at and study selection procedures.

 - Janneke Oostrom, VU University Amsterdam

The major contribution of this article is it provides a roadmap for how we can generate more theoretically driven selection research and understand the implications of choices in the design in selection procedures.

- Charles Scherbaum, Baruch College, City University of New York

6. Sackett, P. R., Dahlke, J. A., Shewach, O. R., & Kuncel, N. R. (2017). Effects of predictor weighting methods on incremental validity. Journal of Applied Psychology, 102(10), 1421–1434.

It emphasizes how adding new predictors to a selection battery can actually decrease the overall validity of a selection process depending on the weighting approach used (unit vs. regression weight), intercorrelations between predictors, and the validity of the additional predictors included. Practically, it’s important because lots of practitioners still believe that “adding something else cannot hurt,” which the paper shows is clearly wrong. 

- Nicolas Roulin, Saint Mary’s University

Personality/Individual Differences

7. Credé, M., Tynan, M. C., & Harms, P. D. (2017). Much ado about grit: A meta-analytic synthesis of the grit literature. Journal of Personality and Social Psychology, 113(3), 492–511.

Grit has garnered a lot of attention in the media and is purported to be one of the most important individual differences for predicting performance and success. The authors found significant overlap between grit and conscientiousness, questioning the construct validity of grit. Additionally, they found that grit had very little incremental validity over conscientiousness.

- Jeff Cucina, U.S. Customs and Border Protection

8. Walmsley, P. T., Sackett, P. R., & Nichols, S. B. (2018). A large sample investigation of the presence of nonlinear personality–job performance relationships. International Journal of Selection and Assessment, 26(2–4), 145–163.

Rather than conducting a primary study, the authors sought out a large multijob database (consisting of 123 validation studies) to test the hypothesis that personality has a nonlinear relation with job performance. They found minimal evidence of curvilinearity and provided insightful guidance for practitioners.

- Jeff Cucina, U.S. Customs and Border Protection

9. Roberts, B. W., Luo, J., Briley, D. A., Chow, P. I., Su, R., & Hill, P. L. (2017). A systematic review of personality trait change through intervention. Psychological Bulletin, 143(2), 117–141.

This should be required reading for I-Os who want to work with personality data, coach leaders, or avoid saying things such as “personality is all about faking!” that are not true.

- Ted Hayes, U.S. Department of Justice

10. Van Iddekinge, C. H., Aguinis, H., Mackey, J. D., & DeOrtentiis, P. S. (2018). A meta-analysis of the interactive, additive, and relative effects of cognitive ability and motivation on performance. Journal of Management, 44(1), 249–279.

This is a great piece because it shows the unimportance of interactions, even the most straightforward interaction ability x motivation. Given how attractive interactive explanations of phenomena are for laypeople (and scientists), this is sobering...

- Cornelius König, Saarland University

Leadership

11. Heimann, A. L., Ingold, P. V., & Kleinmann, M. (2020). Tell us about your leadership style: A structured interview approach for assessing leadership behavior constructs.  Leadership Quarterly, 31(4), 101364.

They basically develop a structured interview with behavioral questions designed to capture theory-driven leadership styles/constructs and demonstrate that interview ratings predict a host of relevant outcomes (from subordinates’ organizational commitment to leader income) above and beyond many other predictors (e.g., self-ratings of leadership effectiveness, EI, etc.). Not only their study design/method is exemplary, but it has lots of practical implications to select or promote leaders in organizations.

- Nicolas Roulin, Saint Mary’s University

12. Badura, K. L., Grijalva, E., Galvin, B. M., Owens, B. P., & Joseph, D. L. (2020). Motivation to lead: A meta-analysis and distal-proximal model of motivation and leadership. Journal of Applied Psychology, 105(4), 331–354.

This paper is one of the most downloaded from JAP, and that’s because it addresses a fundamental question: Do different leadership motivations matter? These findings have significant implications for leadership development programs and leadership.

- Ted Hayes, U.S. Department of Justice

13. Finkelstein, L. M., Costanza, D. P., & Goodwin, G. F. (2018). Do your high potentials have potential? The impact of individual differences and designation on leader success. Personnel Psychology, 71(1), 3–22.

The contribution of this article is that it provides a framework that can be used by academics and practitioners for considering potential in organizations and the programs designed to realize that potential.

- Charles Scherbaum, Baruch College, City University of New York

Work/Family/Gender

14. Gabriel, A. S., Volpone, S. D., MacGowan, R. L., Butts, M. M., & Moran, C. M. (2020). When work and family blend together: Examining the daily experiences of breastfeeding mothers at work. Academy of Management Journal, 63, 1337–1369.

The reason why this research is deserving of a nomination is because it is normalizing conversations that were previously taboo: breastfeeding at work. It highlights the challenges that women experience, but also illustrates the benefits of supporting women as they blur the boundaries between work and family.

- Traci Sitzmann, University of Colorado Denver 

15. Sitzmann, T., & Campbell, E.M. (2021). The hidden cost of prayer: Religiosity and the gender wage gap. Academy of Management Journal, 64, 1016–1048.

The theorized religiosity factors explained 37% of the variability in the global gender wage gap, an effect size of especially high magnitude in management and social science disciplines.

-Kristen Shockley, University of Georgia

16. Shockley, K. M., Shen, W., Denunzio, M. M., Arvan, M. L., & Knudsen, E. A. (2017). Disentangling the relationship between gender and work–family conflict: An integration of theoretical perspectives using meta-analytic methods. Journal of Applied Psychology, 102(12), 1601–1635.

This paper addresses an issue often discussed in the popular press of gender differences in work–family conflict. Contrary to popular belief, based on a meta-analysis of over 350,000 participants, they found minimal gender differences in the amount of conflict experienced across men and women.

- Allison Gabriel, University of Arizona

Late Career Self-Nominations

17. Campion, M. C., Campion, M. A., Campion, E. D., & Reider, M. H. (2016). Initial investigation into computer scoring of candidate essays for personnel selection. Journal of Applied Psychology, 101, 958–975.

This has changed my professional life more than any other paper I have ever written. 

We explain how to “train” a computer program to emulate a human rater when scoring accomplishment records. We then examine the reliability of the computer’s scores, provide preliminary evidence of their construct validity, demonstrate that this practice does not produce scores that disadvantage minority groups, and illustrate the positive financial impact of adopting this practice in an organization (N = 46,000 candidates).

- Mike Campion, Purdue University

18. Sackett, P. R., Zhang, C., Berry, C. M., Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology. Advance online publication. http://dx.doi.org/10.1037/apl0000994

I view [this] as the most important paper of my career. It offers a substantial course correction to our cumulative knowledge about the validity of personnel selection procedures. We show commonly used correction procedures result in substantial overcorrection and that the validity of many selection procedures for predicting job performance has thus been substantially overestimated. Most of the selection procedures that ranked high in prior summaries remain high in rank, but with mean validity estimates reduced by .10–.20 points. Structured interviews emerged as the top-ranked selection procedure.

- Paul Sackett, University of Minnesota

Concluding Observations and Limitations

Before examining the nature and content of the nominations, I should emphasize that this is one of many possible lists, depending on the network one is relying on for the crowdsourcing. The scholars asked to make the nominations certainly skew “I” side, even though many of those sampled were OB scholars in schools of management. It seems likely also that the emphasis on research and practice in the instructions slanted nominations toward the “I” side of I-O. The most common topic of the 18 articles nominated (44%) was, not surprisingly therefore, employee selection. Selection is the oldest area of research in I-O (Vinchur, 2018) and is the dominant area of I-O practice (see Aguinis et al., 2014). More nominated articles (44%) appeared in Journal of Applied Psychology (JAP) than any other journal. In fact, no other journal appeared more than twice. This supports the idea that JAP is the flagship journal in I-O psychology (Highhouse et al., 2020). Finally, one-third of the nominated articles included Paul Sackett as an author. This suggests that Sackett is arguably one of the most influential I-O scholars in the last several years.

It is notable that the articles considered most important in the last 5 years did not include more contemporary workplace concerns such as job vacancies, remote work, and work meaning. This may be because these are less mature areas of research where we might expect fewer breakthroughs. We do see, however, multiple nominations for studies on artificial intelligence (AI), nonlinear or interactive relations among constructs, work/family conflict, subgroup differences, and challenging the rank ordering and magnitude of effect sizes for predictors.

It is important to reiterate that this is one list that may or may not be representative of the entire field. With that caveat in mind, this list provides important insight into the kinds of studies that are viewed as advancing the field forward. It also provides a general framework for crowdsourcing important advances in the future, and the approach can be tailored toward one’s specific interest, whether it be theoretical advances, methodological advances, practical advances, or creative advances.   

Note

I am grateful to Maggie Brooks for providing feedback on previous versions of this project.

References

Aguinis, H., Bradley, K. J., & Brodersen, A. (2014). Industrial–organizational psychologists in business schools: Brain drain or eye opener? Industrial and Organizational Psychology, 7(3), 284–303.

Highhouse, S., Zickar, M., & Melick, S. (2020). Prestige and relevance of the scholarly journals: Impressions of SIOP members. Industrial and Organizational Psychology, 13(3), 273–290. https://www.doi.org/10.1017/iop.2020.2

Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124(2), 262.

Speer, A. B., Tenbrink, A. P., Wegmeyer, L. J., Sendra, C. C., Shihadeh, M., & Kaur, S. (2021). Meta-analysis of biodata in employment settings: Providing clarity to criterion and construct-related validity estimates. Journal of Applied Psychology. Advance online publication. https://doi.org/10.1037/apl0000964

Vinchur, A. J. (2018). The early years of industrial and organizational psychology. Cambridge University Press.