The Diversity Report: Assessing Our Impact: In Search of a Metric for Multicultural Research January 2008

Since becoming aware of the multitude of methods for appraising employee performance during my tenure as an I-O graduate student, I’ve wondered about the process of evaluating scholarly productivity. Sure, we all know top-tier publications and high citation counts are important, but how do we really assess the impact of our work? Over the course of my relatively brief academic career, I’ve had the opportunity to observe how faculty at four very different universities, and in two different types of colleges within them (Arts & Sciences and Schools of Business), define and critique scholarship. I also recently read an interesting article discussing the process of ranking publication outlets for faculty evaluation for another school (Marsh & Hunt, 2006). What I’ve come to realize is that (a) there is as much inconsistency as consistency in this process, and (b) the only instances of absolute agreement typically involve extreme cases, either favorable or unfavorable.

Though this evaluative uncertainty certainly affects all of us with tenure-track appointments, its impact is not evenly felt. Garnering acceptance letters is a difficult, often tedious process, irrespective of the focal topic. This is especially true among the more rigorous and prestigious outlets where acceptance rates are commonly less than 10%. Nonetheless, Cox (1990) argued that research on racioethnicity in the workplace often elicits additional hurdles to overcome in the review process. Though slightly dated, his article was acknowledged recently as one of the most important published in the Journal of Applied Behavioral Sciences and was reprinted in the journal in 2004. Assuming Cox’s arguments (and those of his sample) remain valid, this could help to explain why there is relatively less research on this topic than comparable subjects and why most of the existent literature does not appear in our field’s most prestigious outlets (Avery, 2007; Cox & Nkomo, 1990; Cox, Nkomo, & Welch, 2001).

The purpose of this column is to examine how using solely the most prevalent indicators of scholarly productivity could lead to erroneous conclusions about the impact of researchers studying racioethnicity in organizations. My point is not simply to criticize the status quo but rather to provide insight and stimulate thought, dialogue, and action concerning this situation. Ultimately, I hope that we, as a field, begin to consider or reconsider whether we are using the most appropriate metrics to gauge the impact of our research. In the sections to follow, I define impact, review how we currently measure it, and discuss prospective limitations of the current method.

What Is Impact?

It is impossible to begin this discussion without first defining the term impact. Depending on one’s own biases and beliefs, impact can be defined in any number of ways. Because we are a field that values science and practice, impact must involve both of these components. Despite what we have been taught to believe, we cannot profess to have impact if our work fails to inform both science and practice. This is not to undermine the independent value of either basic or applied research. However, in isolation, neither fully allows us to enhance our understanding of human behavior in the workplace. True progress lies at the nexus of science and practice. Interestingly enough, the most recent issue of Academy of Management Journal (vol. 50, number 5) contains a series of provocative editorials, mostly in support of this point.

Beyond the need to be pertinent to scientists and practitioners alike, impactful scholarship must identify and answer previously unidentified and unanswered significant questions. Of course, the most subjective and debatable component of this description is the word significant. What is significant to one person is often of little consequence to another. This is not to say that there should be a universal standard for evaluation. Rather, appraisers should be sensitive to this inherent subjectivity and remain open to the possibility that colleagues’ scholarship may not conform to their own or traditional definitions. In essence, research makes an impact when it extends the existing knowledge base in theoretically interpretable and practically meaningful ways.

Assessing Scholarly Impact

Traditional Approaches
Publication outlet. One of the most common means of assessing scholarly productivity and impact is to examine publications. As Gibby, Reeve, Grauer, Mohr, and Zickar (2002) noted, faculty and departments are often rated according to their publication records. In particular, assessors count the number of publications appearing in outlets considered to comprise the top tier. Although there is considerable dispute about the size of the top tier, there is a fair amount of consistency in the top journals that commonly appear on journal lists (e.g., Journal of Applied Psychology, Personnel Psychology, Academy of Management Journal). Regardless of the level of absolute agreement regarding the occupants of the top tier, there is no disputing the impact of publishing in these journals. For instance, a journal’s tier standing is significantly correlated to its readership size, Starbuck journal impact factor, and rejection rate (Zickar & Highhouse, 2001). Furthermore, Gomez-Mejia and Balkin (1992) found that the number of top-tier publications was, along with institutional changes, the principal determinant of faculty pay.

Citation counts. Another factor typically considered an indicator of impact is the number of times an article has been cited by subsequent research (Judge, Cable, Colbert, & Rynes, 2007). In short, “a citation may be perceived as a signal in the knowledge market that indicates a faculty member’s research has influenced the work of other scholars” (Gomez-Mejia & Balkin, 1992; p. 925). Thus, extensively cited articles are likely to be seen as having exerted greater influence on the field than those referenced less often. Although some research suggests citation counts only tend to benefit those with records containing top-tier publications (Gomez-Mejia & Balkin, 1992), recent scholars have speculated that the importance of citations as an indicator of scholastic excellence will continue to increase (Judge et al., 2007).

Two Exceptional Exceptions

Despite the widespread acceptance of the indicators above, I believe there is a need to consider the use of different metrics. Perhaps the best way to illustrate the importance of considering alternatives is to examine a couple of exceptional examples. Admittedly, these involve individuals whom I know. Nevertheless, I’m quite certain there are a number of other examples that might serve this purpose equally well. The first involves a graduate student named Matthew Harrison in the applied psychology doctoral program at the University of Georgia. The second involves a senior research fellow at Dartmouth College named Quintus Jett.

A study on “Colorism.” With the assistance of his advisor, Kecia Thomas, Matthew Harrison recently conducted a research project to examine the impact of skin color on workplace discrimination. According to Mr. Harrison, they found:

That a light-skinned black male can have only a bachelor’s degree and typical work experience and still be preferred over a dark-skinned black male with an MBA and past managerial positions, simply because expectations of the light-skinned black male are much higher, and he doesn’t appear as “menacing” as the darker-skinned male applicant.

After presenting the paper at the 66th annual conference of the Academy of Management, the paper is now forthcoming in the Journal of Applied Social Psychology (JASP).

Taking nothing away from JASP (a reputable journal in which I’ve published multiple papers), the general consensus regarding its standing is that it is a second-tier journal in our discipline (Zickar & Highhouse, 2001). As Gomez-Mejia and Balkin (1992) noted, “administrators will treat secondary publications as ‘frosting on the cake,’ as an extension of (but not a substitute for) scholarly productivity” (p. 924). Moreover, because the article has not appeared in print, it has yet to be cited. Using the existing metrics would lead one to surmise that this research has had little impact. However, I recently conducted a cursory Internet search using Google.com and discovered more than 500 Web sites discussing the study. Among those covering the project were highly influential sources, such as the Washington Times, Diversity Inc., and Black Entertainment Television. Clearly, this study has had (and likely will continue to have) a considerable impact on the field of I-O psychology (Kudos, Matthew!).

Hurricane recovery efforts. Though the preceding story involves a relatively small deviation from traditional ideas about scholarly impact, the second example is a larger departure. Areas in southern Louisiana and Mississippi suffered catastrophic devastation during Hurricane Katrina in 2005. Since that time, countless volunteers have made invaluable contributions to the rebuilding efforts, which continue to this day. One such volunteer is Dr. Quintus Jett.

Dr. Jett’s specialty is operations management. He opted to apply his expertise to the recovery efforts and has provided the infrastructure and leadership to map recovery efforts in Gentilly (a New Orleans neighborhood in the lower ninth ward) and surrounding areas. His efforts, to date, have had a substantial impact on the lives and prospects of residents in this area. They also illustrate the prospective benefit of applying the relevant aspects of our science. Although his contributions may not translate directly into top-tiered publications and, thus, will not be highly cited in the academic literature, I challenge anyone to make a compelling case that they have not had as much or more scholarly impact than a highly cited top-tier journal article.

Assimilation or Accommodation?

I’m sure the purists amongst us will object soundly to my contention that we should consider additional indices of scholarly impact beyond publication outlets and citation counts. Indeed, even I concede that my argument creates somewhat of a slippery slope. If we should consider media attention or community impact as indicators of scholarly productivity, what else might colleagues argue should be considered? Although I can certainly appreciate this position, it does not negate my belief that we should be considering nontraditional indicators. To further illustrate why I feel this way, let’s consider the current appraisal system in the context of an ongoing debate within the diversity literature, namely assimilation versus accommodation.

In developing an approach to manage organizational diversity, decision makers, inevitably, will be faced with the challenge of balancing organizational pressures for employees to assimilate in conformance with company norms and employee desires for organizations to accommodate their uniqueness. For example, a company may desire to implement a strict dress code for uniformity and safety reasons, but employees may wish to relax the dress code to allow for cultural or religious expression. Those mired in the traditional organizational approach are unlikely to see a need for accommodation because they have become accustomed to the organizational norms and do not see them as unfair, unnecessary, or counterproductive. Conversely, those feeling forced to assimilate often feel constricted and unable to fully invest themselves in their work roles, often resulting in psychological disengagement and withdrawal (Kahn, 1990; 1992).

Applied to the present discussion, those conducting research on racioethnicity in the workplace commonly find themselves at a distinct disadvantage relative to those studying more “mainstream” topics when traditional scholastic impact measures are applied. For instance, topic-related bias in the publication process makes it disproportionately less likely that their research will find its way into top-tier outlets (Cox, 1990). This problem is compounded further by the fact that citations are influenced by (a) the number of top-tier articles previously published by the authors, (b) the impact rating of the journal in which the article appears, and (c) the prevailing subjective prestige associated with the journal (Judge et al., 2007). Consequently, the research and researchers become less accessible and familiar to prospective readers (creating a perpetual cycle), and a powerful signal is sent and reinforced regarding topics the field values versus those it does not. Moreover, because there is less research being conducted on this topic, relative to more “mainstream” topics, citation rates are unlikely to be as high as for those doing other types of research (Judge et al., 2007).

The resulting situation leaves racioethnicity researchers facing a difficult predicament: Do we (a) try to publish research in the top-tier journals despite the odds, (b) seek to publish our research in outlets perceived to be more topically inclusive, or (c) alter our research agendas to make them more palatable to the power brokers in our field? In other words, should we seek accommodation or assimilate? My hope is that this column will help to stimulate a dual-pronged solution to this conundrum. First, I strongly implore those working on racioethnic research to commit to overcoming the obstacles, legitimate or otherwise, to having our work included in top-tier outlets. The work we do is important and deserves the dedication and perseverance it takes to publish in such journals. Yes, it is difficult, but we are certainly making progress on this front! Second, I urge decision makers (e.g., editors, reviewers, department chairs, deans, promotion and tenure committees) to consider these issues prior to developing evaluative criteria for judging the merits of research and researchers. Please do not interpret this as a request to lower our scholastic standards, as nothing could be further from my intention. Instead, view it as a call to ensure that our standards reward scholastic excellence in its multiple and diverse forms.

References

     Avery, D. R. (2007). Does underrepresented = understudied? Recent research on racial and ethnic minorities in I-O psychology. The Industrial-Organizational Psychologist, 44(3), 75–80.
     Cox, T. H., Jr. (1990). Problems with research by organizational scholars on issues of race and ethnicity. Journal of Applied Behavioral Science, 26, 5–23.
     Cox, T. H., Jr., & Nkomo, S. M. (1990). Invisible men and women: A status report on race as a variable in organization behavior research. Journal of Organizational Behavior, 11, 419–431.
     Cox, T. H., Jr., Nkomo, S. M., & Welch, J. (2001). Research on race and ethnicity: An update and analysis. In R. Golembiewski (Ed.), Handbook of organizational behavior (2nd ed., pp. 255–286). New York: Marcel Dekker.
     Gibby, R. E., Reeve, C. L., Grauer, E., Mohr, D., & Zickar, M. J. (2002). The top I-O psychology doctoral programs of North America. The Industrial-Organizational Psychologist, 39(4), 17–25.
     Gomez-Mejia, L. R., & Balkin, D. B. (1992). Determinants of faculty pay: An agency theory perspective. Academy of Management Journal, 35, 921–955.
     Judge, T. A., Cable, D. M., Colbert, A. E., & Rynes, S. L. (2007). What causes a management article to be cited—Article, author, or journal? Academy of Management Journal, 50, 491–506.
     Kahn, W. A. (1990). Psychological conditions of personal engagement and disengagement at work. Academy of Management Journal, 33, 692–724.
     Kahn, W. A. (1992). To be fully there: Psychological presence at work. Human Relations, 45, 321–349.
     Marsh, S. J., & Hunt, C. S. (2006). Not quite as simple as a-b-c: Reflections on one department’s experiences with publication ranking. Journal of Management Inquiry, 15, 301–315.
     Zickar, M. J., & Highhouse, S. (2001). Measuring prestige of journals in industrial-organizational psychology. The Industrial-Organizational Psychologist, 38(4), 29–36.