Home Home | About Us | Sitemap | Contact  
  • Info For
  • Professionals
  • Students
  • Educators
  • Media
  • Search
    Powered By Google

Measuring Prestige of Journals in
Industrial-Organizational Psychology 

Michael J. Zickar and Scott Highhouse1
Bowling Green State University

1 Author Note: We are grateful to Margaret Brooks-Laber and Steve Russell for their assistance on this project.

The quality or prestige of journals is often used as a surrogate for evaluating the quality of individual publications. It has been our experience that faculty search committees and tenure evaluation committees often make judgments about the research productivity of candidates without reading the individual articles listed in the research vita. Instead, the quality of articles is judged by the prestige of the journal in which the article has been published. This reliance on journal prestige has been noted in management (e.g., Extejt & Smith, 1990; Johnson & Podsakoff, 1994; Kirkpatrick & Locke, 1992) as well as other fields such as sociology (Teevan, 1980). This over-reliance on journal prestige as a measure of journal article quality mandates that alternative measures of journal prestige be explored.

There have been several different indicators of journal quality proposed. One of the most popular indicators of journal prestige has been the Social Science Citation Index (SSCI) impact factor. The logic behind the impact measure is that journals with high prestige will contain articles that are cited more frequently by subsequent articles. One criticism of impact factors is that articles on topics with mass appeal (e.g., training and development) will tend to be cited more frequently than articles in areas of more limited appeal (e.g., psychometrics). Similarly, journals with larger audiences, such as those catering to both business school and psychology faculty, will necessarily have higher impact factors than journals with smaller audiences such as those catering only to psychologists. Other measures such as rejection rates, library circulation, or years in press are also confounded by factors such as audience size and topic popularity.

Another method of assessing journal prestige is to ask members of the field to judge the quality of particular journals. Although surveys of faculty impressions of management journals exist (e.g., Extejt & Smith, 1990; Gomez-Mejia & Balkin, 1992), it is unclear whether rankings of business journals can serve as rankings of I-O journals. One issue that has received attention over the years is the identity of I-O psychology relative to related disciplines, such as human resource management, organizational behavior, or organization development (e.g., Highhouse & Zickar, 1997; Naylor, 1966). Indeed, a considerable number of leading academics in I-O reside in business schools, raising the questions of whether I-O psychologists in psychology departments view journals differently than I-O psychologists in business schools.

To begin to address these questions, as well as to provide a specific measure of journal prestige in I-O psychology, we surveyed a sample of academic members of SIOP concerning their impressions of journal quality in I-O. One advantage of this method is that it focuses on the judgments of constituents of one area thereby avoiding the problem of area size dictating journal impact. We were also able to identify the academic home of these members, in order to examine differences in impressions of journal prestige for psychologists in business schools versus departments of psychology. 

Methods 

We selected 23 journals that, in our judgment, are common outlets for research by I-O psychologists. Journals were selected from previous lists of management journals (e.g., Extejt & Smith, 1990), excluding journals that are exclusively associated with business research (e.g., Sloan Management Review, Journal of Labor Economics) or with psychological research (e.g., Journal of Personality and Social Psychology, Psychological Bulletin). Although we recognized that some of these excluded journals can and do serve as occasional outlets for research by I-O psychologists, we were ultimately concerned with having judges compare I-O journals directly against one another. We also added journals that have not been included in previous lists, either because they are newer (e.g., Human Performance; Human Resource Management Review; International Journal of Selection and Assessment; Organizational Research Methods) or because they have been previously overlooked (e.g., Applied Psychology: An International Review; Journal of Business and Psychology; Journal of Organizational Behavior). We also included space for judges to add journals that they felt should be included on the list.

We sampled from the membership roster of SIOP. We received mailing labels from the SIOP database of all members who reported an academic setting as their primary place of employment. This list included 1,003 members in alphabetical order. We used a spaced sampling technique to arbitrarily sample SIOP members to reach our target sample of 500 members. We excluded people who were sent surveys in an earlier small-scale pilot study. Our sample represented approximately 50% of the academic members of SIOP as of 1999.

A questionnaire was sent to each member of the sample, and responses were made anonymously. Part I of the questionnaire instructed judges to place the 23 journals into one of three tiers and to indicate (a) whether they have published in the journal, and (b) whether they read the journal on a regular basis. The tiers were defined as follows:

First Tier: This ranking should be reserved for journals that present uniformly high quality research and/or review articles. Articles in top tier journals should be both methodologically sound and important in advancing our knowledge base.

Second Tier: Journals in this tier should routinely have high quality articles. However, the quality of content is uneven.

Third Tier: Journals in this tier routinely publish articles with suspect methodology.

The judges were instructed to circle the option labeled ? if they were unfamiliar with the journal.

Part II of the questionnaire instructed judges to choose and rank the top 10 journals (from the list) that in their opinion publish the most important research for I-O psychologists. The purpose of this section was to force judges to make relative distinctions among the higher quality journals. We also recognized that although a journal could be considered first-tier, it may not be seen as publishing research of direct relevance to I-O.

The final part of the questionnaire asked judges to indicate their primary place of employment (i.e., business school, labor/industrial relations department, psychology department, other) and years since they received their PhD. Judges were also asked to indicate areas in which they have conducted research, using the competency areas identified in the SIOP guidelines for education and training.

Results

Two-hundred seventeen usable responses were returned. Nine questionnaires were returned uncompleted or because of a change of address. This represented an effective response rate of 44.2%. The sample was evenly split between those employed in psychology departments (46.9%) and those employed in business schools (43.1%); a smaller number of respondents were employed in labor/industrial relations (LIR) departments (3.8%) and other departments (6.2%). For analyses based on department we combined business school and LIR faculty. The average respondent had completed their PhD 13.9 years prior to completing the questionnaire.

Table 1 reports the average tier ratings for each of the 23 journals. These ratings range from 1 to 3 with lower ratings corresponding to higher prestige. These ratings are presented for the overall sample as well as separated by business school and psychology department respondents. The rank ordering of journals on this index is also presented in the table.

Table 1. Tier Ratings of I-O Journals

___________________________________________________________________

 

Overall Tier

Psych Tier

Business Tier

AMJ

1.15 (2)

1.17 (2)

1.06 (2)

AMR

1.21 (3)

1.28 (5)

1.10 (4)

ASQ

1.27 (5)

1.482(6)

1.082 (3)

APM

1.82 (9)

1.69 (7)

2.00 (13)

AP:IR

2.33 (19)

2.37 (18)

2.24 (19)

BASP

2.24 (18)

2.22 (17)

2.29 (21)

EPM

1.91 (12)

1.88 (11)

1.92 (11)

GOM

2.33 (20)

2.40 (19)

2.26 (20)

HP

2.08 (15)

2.01 (14)

2.19 (17)

HR

2.21 (17)

2.442 (21)

2.032 (14)

HRMR

2.33 (21)

2.53 (23)

2.23 (18)

IJSA

2.48 (22)

2.43 (20)

2.59 (23)

JAP

1.07 (1)

1.07 (1)

1.05 (1)

JASP

1.87 (10)

1.83 (9)

1.89 (91)

JBP

2.52 (23)

2.49 (22)

2.57 (22)

JM

1.76 (8)

1.85 (10)

1.65 (7)

JOOP

2.07 (14)

2.04 (15)

2.09 (15)

JOB

1.90 (11)

1.91 (12)

1.89 (91)

JVB

1.94 (13)

1.94 (13)

1.93 (12)

LQ

2.18 (16)

2.21 (16)

2.16 (16)

OBHDP

1.28 (6)

1.23 (4)

1.29 (6)

ORM

1.75 (7)

1.82 (8)

1.69 (8)

PP

1.22 (4)

1.20 (3)

1.20 (5)

Note. 1 Indicates a tie; 2 indicates that a significant difference exists between business school and psychology department faculty on this index.

___________________________________________________________________

 

Table 2 presents the number of points that the 23 journals received in the top 10 rankings. Journals were given 10 points if they were ranked in the first position, 9 if they were in the second, and so forth. As before, rankings and breakdowns by psychology departments and business schools are presented in this table.

 

Table 2. Rankings of I-O Journals

 

Overall

Top Ten Points

Psychology

Dept.

Business Dept.

AMJ

1337 (3)

551 (3)

693 (2)

AMR

911 (5)

440 (5)

436 (4)

ASQ

546 (6)

189 (7)

321 (6)

APM

70 (18)

42 (16)

28 (18)

AP:IR

46 (22)

24 (20)

21 (20

BASP

59 (21)

36 (19)

14 (21)

EPM

188 (11)

106 (12)

62 (13)

GOM

69 (19)

14 (22)

51 (16)

HP

238 (10)

146 (9)

59 (131)

HR

106 (15)

39 (18)

52 (15)

HRMR

63 (20)

4 (23)

59 (131)

IJSA

38 (23)

23 (21)

11 (23)

JAP

1884 (1)

898 (1)

860 (1)

JASP

178 (13)

109 (11)

48 (17)

JBP

93 (16)

69 (14)

13 (22)

JM

438 (7)

180 (8)

234 (7)

JOOP

187 (12)

193 (6)

212 (8)

JOB

425 (8)

100 (13)

65 (11)

JVB

284 (9)

126 (10)

123 (9)

LQ

72 (17)

41 (17)

26 (19)

OBHDP

980 (4)

516 (4)

409 (5)

ORM

161 (14)

66 (15)

86 (10)

PP

1484 (2)

718 (2)

668 (3)

_________________

Note. 1Indicates a tie; this index was formed by weighting inversely each mention of the journal in the respondents top ten most influential journals list (e.g., if a journal was ranked first it received 10 points and if it was ranked in the 10th spot, it received 1 point).

___________________________________________________________________________

 

These rankings were highly correlated (r = --.87), so we decided to combine these two indexes of journal quality (after standardizing and reverse-coding prestige) to come up with an overall index. Table 3 presents the overall top 10 index based on this combination.

 

Table 3. Top Ten Overall Journals

___________________________________________________________________________

 

1. Journal of Applied Psychology

2. Personnel Psychology

3. Academy of Management Journal

4. Academy of Management Review

5. Organizational Behavior and Human Decision Processes

6. Administrative Science Quarterly

7. Journal of Management

8. Journal of Organizational Behavior

9. Organizational Research Methods

10. Journal of Vocational Behavior

____________________________________________________________

 

As a final index of interest, Table 4 presents the percentages of respondents who reported that they read the 23 journals in the study. The majority of I-O psychologists read AMJ, AMR, JAP, and PP, whereas most journals included in this study were read by less than 20% of the sample.

 

Table 4. Reading Frequencies of I-O Journals

 

Overall % who Read

Psychology

Dept.

Business Dept.

AMJ

79.2

77.8

83.9

AMR

72.3

66.7

80.9

ASQ

37.3

26.31

49.51

APM

12.9

19.2

8.1

AP:IR

13.3

19.2

9.1

BASP

10.6

14.1

7.0

EPM

20.8

24.3

17.2

GOM

16.6

12.1

23.3

HP

17.1

24.2

11.1

HR

21.2

18.2

26.2

HRMR

14.3

6.11

22.21

IJSA

7.8

10.1

4.0

JAP

82.5

85.9

79.8

JASP

24.0

28.3

21.2

JBP

14.8

19.2

11.2

JM

40.5

27.31

55.61

JOOP

13.3

13.1

12.1

JOB

31.8

26.3

37.4

JVB

20.7

21.3

20.2

LQ

15.7

14.2

16.2

OBHDP

44.3

45.4

43.4

ORM

15.7

13.1

19.2

PP

64.9

68.7

64.6

Finally, Table 5 presents the correlation matrix between the indexes in this study as well as Starbucks impact rating corrected for area norms (Starbuck, 2000) and self-reported rejection rates. Rejection rates were collected from a published compendium (American Psychological Association, 1997) and writing to editors for journals not listed in that publication. As can be seen from the correlation matrix, all of the indexes are highly correlated with each other. However, our three self-reported measures were more highly correlated with each other than the correlations between those measures and Starbucks impact factor and rejection rates.

 

Table 5. Correlation Matrix of Indexes

 

Tier

Ranking

Read

Starbuck's Impact

Tier

 

 

 

 

Ranking

-.87

 

 

 

% Who Read

-.88

.95

 

 

Starbuck's Impact

-.49

.51

.62

 

Rejection Rate

-.72

.44

.56

.72

Note. All correlations are significant at p < .05.

____________________________________________________________ 

Discussion 

The indexes presented here can be considered additional evidence that can be used to judge the quality and prestige of various I-O psychology journals. These indexes, like others, have their own limitations. Our sample excluded nonacademics; the ranking of journals would surely differ if nonacademics were sampled. In addition, responses to our measure were likely influenced by raters exposure to other indexes, such as impact factors or rejection rates. Finally, our ratings might also have differed if we would have included more general psychology journals among the list of 23 journals that we selected. A small number of respondents chose to list and rate Psychological Bulletin and Journal of Personality and Social Psychologytwo journals that publish basic research that is often directly related to I-O psychology. We suspect if those journals were included in our list, they would have received high prestige ratings.

Our data further confirm that there is a core group of journals (AMJ, JAP, PP, AMR, and OBHDP) that are rated as uniformly high on all possible indexes. ASQ would fit into this group if considering only ratings by business school and LIR faculty. It is clear that JAP is the flagship journal for I-O psychologists. Across all indexes, JAP scored highest. We did analyses based on the type of research interests expressed by respondents, and across all areas of research JAP scored highest. We suspect that most readers of TIP already knew that JAP and those other journals were prestigious, and so the more useful aspects of our data may relate to the other 17 journals included in our analyses.

That our self-report prestige measures correlated with other nonself-report measures adds some validity to these ratings. Starbucks (2000) impact factor ratings are based on citation rates for the articles published in each of the journals and, as such, are an objective measure of journal quality. The rejection rates are relatively objective, though there is the possibility that some editors inflate those estimates. Regardless, the correlations among the different measures suggest that our prestige ratings have some validity.

Differences between ratings by business school/LIR faculty and psychology department faculty were relatively few. ASQ and HR had lower (better) tier ratings for business faculty than psychology faculty; ASQ, HRMR, and JM were read more frequently by business department faculty compared to psychology faculty. More journals would have had significant results, but we used an experiment-wise error correction so it appears that these journals were the ones that had the largest differences. There was an interesting asymmetry in that more psychologically focused journals (e.g., JAP, OBHDP) were ranked and read at similar rates across business and psychology faculty. However, some of the more management-focused journals are read more infrequently by psychology faculty.

In an ideal world, hiring, tenure, and award committees would evaluate the quality of publications by actually reading the articles. However, it is our experience that most committees do not read the articles but focus simply on the quantity of publications and the prestige of the journals in which those articles are published. We hope the results from our questionnaire will, at the least, help to better inform judgments about the journals in I-O.

References 

American Psychological Association, (1997). Journals in Psychology: A resource listing for authors. Washington, DC: APA Books.

Extejt, M. M., & Smith, J. E. (1990). The behavioral sciences and management: An evaluation of relevant journals. Journal of Management, 16, 539--551.

Gomez-Mejia, L. R., & Balkin, D. B. (1992). Determinants of faculty pay: An agency theory perspective. Academy of Management Journal, 35, 921--955.

Highhouse, S., & Zickar, M. (1997, October). Where has all the psychology gone? The Industrial-Organizational Psychologist, 35, 82--88.

Johnson, J. L., & Podsakoff, P. M. (1994). Journal influence in the field of management: An analysis using Salanciks index in a dependency network. Academy of Management Journal, 37, 1392--1407.

Kirkpatrick, S. A., & Locke, E. A. (1992). The development of measures of faculty scholarship. Group and Organization Management, 17, 5--23.

Naylor, J. C. (1966). The psychologist in the business school: Some further comments. The Industrial Psychologist, 3, 20--22.

Starbuck, W. (2000). [Online]. Available: http://www.stern.nyu.edu/~wstarbuc/cites.html.

Teevan, J. J. (1980). Journal prestige and quality of sociological articles. American Sociologist, 15, 109--122.

Appendix
List of journals included in study
 

AMJ Academy of Management Journal

AMR Academy of Management Review

ASQ Administrative Science Quarterly

APM Applied Psychological Measurement

AP:IR Applied Psychology: An International Review

BASP Basic and Applied Social Psychology

EPM Educational and Psychological Measurement

GOM Group and Organization Management

HP Human Performance

HR Human Relations

HRMR Human Resource and Management Review

IJSA International Journal of Selection and Assessment

JAP Journal of Applied Psychology

JASP Journal of Applied Social Psychology

JBP Journal of Business and Psychology

JM Journal of Management

JOOP Journal of Occupational and Organizational Psychology

JOB Journal of Organizational Behavior

JVB Journal of Vocational Behavior

LQ Leadership Quarterly

OBHDP Organizational Behavior and Human Decision Processes

ORM Organizational Research Methods

PP Personnel Psychology


April 2001 Table of Contents | TIP Home | SIOP Home