The Academics' Forum: Data Sources and Publication in Top Journals: The Hunt for Participants

Sylvia G. Roch
University at Albany


An important decision when designing a primary research project is what type of participants to use in the research. As is the case in most major research universities, I have a large and convenient undergraduate subject pool available for my research purposes. For some of my research questions, participants from the undergraduate subject pool are an ideal population. I can control confounding factors in my laboratory to a degree not possible in most other settings and, thus, gain a deeper understanding of the phenomena of interest. Obviously, research using undergraduates is most useful in theory building and is often not meant to directly generalize. Mook (1983) does an excellent job of articulating the differences between research investigating if something “can” happen (research for theory building) versus research designed to answer the question of whether something “does” happen (research for generalization). Nonetheless, there are some questions best answered using a working population. Thus far, I have been fortunate. For my research requiring a working participant population, I have been able to find suitable populations. My two most frequent sources of working populations have been alumni from the University at Albany and employees from organizations where my students are either interning or working. However, I can see a scenario where a suitable population may not be easily available, and based on my conversations with colleagues, finding a suitable subject population is a common concern. Thus, I have decided to devote this column to providing both a brief overview of the types of participants used in research published in top I-O journals and a description of some of the more unusual participant populations that I found in my search.

My first step was to conduct a search of literature looking for reviews of the I-O literature addressing this issue. I found a few studies that have looked at research trends in I-O but had difficulty finding recent information. Authors who have examined participant trends in I-O psychology include Dipboye (1990) and Podsakoff and Dalton (1987). Podsakoff and Dalton (1987) examined all the articles published in five organizational journals in 1985 and found that only 29.5% of articles included lab research. Dipboye concentrated on the Journal of Applied Psychology (JAP) and examined all articles published in JAP between the years of 1970 and 1988 and found that, similar to Podsakoff and Dalton, only 33% of the articles consisted of lab research and 67% consisted of field research, including surveys.

Thus, with the help of my research assistants, I decided to conduct an admittedly rough investigation of what type of research participants were reported in articles published between 2002 and 2007 in four journals in I-O psychology. I chose to examine the Journal of Applied Psychology, Personnel Psychology, Academy of Management Journal, and Organizational Behavior and Human Decision Processes because these journals were ranked as the top four by I-O psychologists in Zickar and Highhouse’s (2001) review of the top journals. All of the studies reported in these journals between 2002 and 2007 were classified into one of six categories: experiment with students, experiment with employees, survey of students, survey of employees, survey of employees with outcomes (i.e., survey data and another data type such as supervisor ratings), and other. If an article had multiple studies, each study was independently categorized because, in many cases, articles that reported multiple studies contained studies that fit into different categories. Please see Figure 1 for the results. This table reflects the average number of studies per journal by type. The “other” category is not reflected in this table. The “other” category varied greatly from roughly an average of 74% of the articles per year in Personnel Psychology (PP) to 10% of the articles per year in Organizational Behavior and Human Decision Processes (OBHDP), with the Journal of Applied Psychology (JAP) at 24% and Academy of Management Journal (AMJ) at 48%.

 
Figure 1.  Average number of published studies per year by type from 2002 to 2007

Note:  JAP refers to the Journal of Applied Psychology, PP refers to Personnel Psychology, OBHDP refers to Organizational Behavior and Human Decision Processes, and AMJ refers to Academy of Management Journal.

It is not surprising that a large percentage of the journal articles published in Personnel Psychology did not fall into one of the categories that we used. Personnel Psychology routinely includes review articles, meta-analyses, and book reviews. The book reviews especially contributed to a large proportion of the 74% of “other” articles in Personnel Psychology. It should be noted that in general the “other” category contained a wide variety of articles ranging from theoretical pieces to book reviews to meta-analyses to archival data and interviews; anything that could not easily be fit into one of the first five categories was included in the “other” category.

Based on the results, it appears that if one is conducting experiments with undergraduates, either OBHDP or JAP would be the primary outlets in the journals that we examined. Journal of Applied Psychology also appears to be a good outlet for survey research, as is AMJ, and also to a certain extent PP. What is interesting in this figure is how few articles include surveys with other types of supporting data, such as supervisor evaluations or peer ratings. There is a push to publish articles not exclusively relying on survey data, and this push is not reflected in the averages presented in the table but is still evident when looking at the progression over the years. For example, in 2002 only 9 articles published in JAP included surveys with outcomes, but this number rose to 18 in 2006 and 22 in 2007. Thus, there does appear to be a trend to publish survey data supported by another data source. Lastly, it appears that it is still rare to have experiments with employees as participants. This is not surprising, given the difficulty in gaining access to organizations. Often the best we can do is collect survey data. It is difficult to perform even a quasi-experiment with employees, yet some researchers do manage to employ an experimental design using employees as participants. It should be noted that we did not distinguish experiments from quasi-experiments.

Lastly, I wish to highlight some creative ways of finding research populations that we found while categorizing the studies. Flipping through the articles in the four journals, one sees many articles based on responses from company employees, managers, and members of the military. However, there are also some populations represented that are unusual but are still valuable sources of information. For example, Simonson, Kramer, and Young (2004) recruited 731 travelers at the domestic terminal of the San Francisco International Airport who filled out a “Consumer Preferences” questionnaire. Kumar (2004) intercepted individuals at a public vending area. Dysart, Lindsay, MacDonald, and Wicke (2002) had two female confederates approach patrons in two local bars.  In a study investigating the role of obesity in customer service, King, Shapiro, Hebl, Singletary, and Turner (2006) observed women confederates interact with salespeople. Tan, Foo, and Kwek (2004) observed and coded cashier interactions with customers in a chain of fast food restaurants and followed up the interactions with a survey. Johnson and Raab (2003) recruited participants from local handball clubs in Germany and Brazil. Wiener et al. (2002) placed an advertisement in local newspapers. These are only a few of the creative methods of obtaining participants that we noticed; there were also many more authors who deserve to be mentioned for their creative participant recruitment efforts.

Finally, a few years ago, I heard a symposium at SIOP in which the researchers gave a survey focusing on organizational attitudes to individuals who were waiting for jury duty. I apologize that I cannot properly cite the authors; I just remember that I thought that it was a very creative way of finding willing participants (and I suspect also potentially bored individuals with time on their hands and nothing better to do than fill out surveys). All of these authors remind us that it is worthwhile to think beyond students and employees at their place of employment when designing research projects. A little creativity can be useful. Lastly, I would like to thank Gene Trombini, my research assistant, for his assistance in every aspect of this project. Also, I would like to thank Jenni Higgins, Tracey Drobbin, Ashleigh Flick, Ryan Armstrong, Joshua Rutter, David Sevits, Amber Pease, Tereva Bundy, Kamilah McShine, and Chris Pucheu for their help with this project.

References

     Dipboye, R. L. (1990). Laboratory vs. field research in industrial and organizational psychology.  International Review of Industrial and Organizational Psychology, 5, 1–34.
     Dysart, J. E., Lindsay, R. C. L., MacDonald, T. K., & Wicke, C. (2002). The intoxicated witness: Effects of alcohol on identification accuracy from showups. Journal of Applied Psychology, 87, 170–175.
     Johnson, J. G., & Raab, M. (2003). Take the first: Option-generation and resulting choices. Organizational Behavior and Human Decision Processes, 91, 215–229.
     King, E. B., Shapiro, J. R., Hebl, M. R., Singletary, S. L., & Turner, S. (2006). The stigma of obesity in customer service: A mechanism for remediation and bottom-line consequences of interpersonal discrimination. Journal of Applied Psychology, 91, 579–593.
     Kumar, P. (2004). The effects of social comparison on inaction inertia. Organizational Behavior and Human Decision Processes, 95, 175–185.
Mook, D. G. (1983). In defense of external invalidity. American Psychologist, 38, 379–387.
     Podsakoff, P. M., & Dalton, D. R. (1987). Research methodology in organizational studies.  Journal of Management, 13, 419–441.
     Simonson, I., Kramer, T., & Young, M. J. (2004). Effect propensity. Organizational Behavior and Human Decision Processes, 95, 156–174.
     Tan, H. H., Foo, M. D., & Kwek, M. H. (2004). The effects of customer personality traits on the display of positive emotions. Academy of Management Journal, 47, 287–296.
     Wiener, R. L, Hackney, A., Kadela, K., Rauch, S. Seib, H., Warren, L., et al. (2002). The fit and implementation of sexual harassment law to workplace evaluations. Journal of Applied Psychology, 87, 747–164.
     Zickar, M. J. & Highhouse, B. (2001).  Measuring prestige of journals in Industrial-Organizational Psychology. The Industrial-Organizational Psychologist, 38, 29–36.