Michael J. Zickar and Scott Highhouse¹
Bowling Green State University

¹ Author Note: We are grateful to Margaret Brooks-Laber and Steve Russell for their assistance on this project.

The quality or prestige of journals is often used as a surrogate for evaluating the quality of individual publications. It has been our experience that faculty search committees and tenure evaluation committees often make judgments about the research productivity of candidates without reading the individual articles listed in the research vita. Instead, the quality of articles is judged by the prestige of the journal in which the article has been published. This reliance on journal prestige has been noted in management (e.g., Extejt & Smith, 1990; Johnson & Podsakoff, 1994; Kirkpatrick & Locke, 1992) as well as other fields such as sociology (Teevan, 1980). This over-reliance on journal prestige as a measure of journal article quality mandates that alternative measures of journal prestige be explored.

There have been several different indicators of journal quality proposed. One of the most popular indicators of journal prestige has been the Social Science Citation Index (SSCI) impact factor. The logic behind the impact measure is that journals with high prestige will contain articles that are cited more frequently by subsequent articles. One criticism of impact factors is that articles on topics with mass appeal (e.g., training and development) will tend to be cited more frequently than articles in areas of more limited appeal (e.g., psychometrics). Similarly, journals with larger audiences, such as those catering to both business school and psychology faculty, will necessarily have higher impact factors than journals with smaller audiences such as those catering only to psychologists. Other measures such as rejection rates, library circulation, or years in press are also confounded by factors such as audience size and topic popularity.

Another method of assessing journal prestige is to ask members of the field to judge the quality of particular journals. Although surveys of faculty impressions of management journals exist (e.g., Extejt & Smith, 1990; Gomez-Mejia & Balkin, 1992), it is unclear whether rankings of business journals can serve as rankings of I-O journals. One issue that has received attention over the years is the identity of I-O psychology relative to related disciplines, such as human resource management, organizational behavior, or organization development (e.g., Highhouse & Zickar, 1997; Naylor, 1966). Indeed, a considerable number of leading academics in I-O reside in business schools, raising the questions of whether I-O psychologists in psychology departments view journals differently than I-O psychologists in business schools.

To begin to address these questions, as well as to provide a specific measure of journal prestige in I-O psychology, we surveyed a sample of academic members of SIOP concerning their impressions of journal quality in I-O. One advantage of this method is that it focuses on the judgments of constituents of one area thereby avoiding the problem of area size dictating journal impact. We were also able to identify the academic home of these members, in order to examine differences in impressions of journal prestige for psychologists in business schools versus departments of psychology. 

Methods 

We selected 23 journals that, in our judgment, are common outlets for research by I-O psychologists. Journals were selected from previous lists of management journals (e.g., Extejt & Smith, 1990), excluding journals that are exclusively associated with business research (e.g., Sloan Management Review, Journal of Labor Economics) or with psychological research (e.g., Journal of Personality and Social Psychology, Psychological Bulletin). Although we recognized that some of these excluded journals can and do serve as occasional outlets for research by I-O psychologists, we were ultimately concerned with having judges compare I-O journals directly against one another. We also added journals that have not been included in previous lists, either because they are newer (e.g., Human Performance; Human Resource Management Review; International Journal of Selection and Assessment; Organizational Research Methods) or because they have been previously overlooked (e.g., Applied Psychology: An International Review; Journal of Business and Psychology; Journal of Organizational Behavior). We also included space for judges to add journals that they felt should be included on the list.

We sampled from the membership roster of SIOP. We received mailing labels from the SIOP database of all members who reported an academic setting as their primary place of employment. This list included 1,003 members in alphabetical order. We used a spaced sampling technique to arbitrarily sample SIOP members to reach our target sample of 500 members. We excluded people who were sent surveys in an earlier small-scale pilot study. Our sample represented approximately 50% of the academic members of SIOP as of 1999.

A questionnaire was sent to each member of the sample, and responses were made anonymously. Part I of the questionnaire instructed judges to place the 23 journals into one of three tiers and to indicate (a) whether they have published in the journal, and (b) whether they read the journal on a regular basis. The tiers were defined as follows:

First Tier: This ranking should be reserved for journals that present uniformly high quality research and/or review articles. Articles in top tier journals should be both methodologically sound and important in advancing our knowledge base.

Second Tier: Journals in this tier should routinely have high quality articles. However, the quality of content is uneven.

Third Tier: Journals in this tier routinely publish articles with suspect methodology.

The judges were instructed to circle the option labeled ? if they were unfamiliar with the journal.

Part II of the questionnaire instructed judges to choose and rank the top 10 journals (from the list) that in their opinion publish the most important research for I-O psychologists. The purpose of this section was to force judges to make relative distinctions among the higher quality journals. We also recognized that although a journal could be considered first-tier, it may not be seen as publishing research of direct relevance to I-O.

The final part of the questionnaire asked judges to indicate their primary place of employment (i.e., business school, labor/industrial relations department, psychology department, other) and years since they received their PhD. Judges were also asked to indicate areas in which they have conducted research, using the competency areas identified in the SIOP guidelines for education and training.

Results

Two-hundred seventeen usable responses were returned. Nine questionnaires were returned uncompleted or because of a change of address. This represented an effective response rate of 44.2%. The sample was evenly split between those employed in psychology departments (46.9%) and those employed in business schools (43.1%); a smaller number of respondents were employed in labor/industrial relations (LIR) departments (3.8%) and other departments (6.2%). For analyses based on department we combined business school and LIR faculty. The average respondent had completed their PhD 13.9 years prior to completing the questionnaire.

Table 1 reports the average tier ratings for each of the 23 journals. These ratings range from 1 to 3 with lower ratings corresponding to higher prestige. These ratings are presented for the overall sample as well as separated by business school and psychology department respondents. The rank ordering of journals on this index is also presented in the table.

Table 1. Tier Ratings of I-O Journals

^{___________________________________________________________________}