What is I-O?

Paul Spector, discussing his experiences with qualitative research in the field of I-O psychology

An informal review of major I-O journals (e.g., Journal of Applied Psychology, Personnel Psychology) over the last 2 years shows that there are relatively few articles based on qualitative research. At first glance, it would appear that qualitative techniques are not a particularly important or valued type of research method in I-O psychology. However, in other areas of psychology and the social sciences, qualitative research is an accepted and well-developed method for examining a wide range of research topics. Are we missing out on a valuable set of research tools that could help us to answer important questions in I-O?

In an effort to explore the issue in some depth, I interviewed (by e-mail) editors from a wide variety of journals read by I-O psychologists to find out why so little qualitative research is generally published. Editors from the following 13 journals participated:

• Applied Psychological Measurement
• Group and Organization Management
• Human Factors
• Human Resource Development Quarterly
• Industrial Relations
• Journal of Applied Psychology
• Journal of Management
• Journal of Organizational Behavior
• Journal of Applied Behavioral Science
• Journal of International Business Studies
• Journal of Occupational and Organizational Psychology
• Journal of Vocational Behavior
• Organization Development Journal

These data were supplemented by responses to additional questions from 11 subscribers to RMNet, a Web site for members of the Academy of Managements Research Methods Division. This group was made up of four people in I-O or applied social psychology, three people in organizational behavior, three people in management, and one in public administration. Most were affiliated with a business school. RMNet subscribers were targeted because they tend to publish in some of the I-O journals, include many I-O psychologists, and may have a wider experience with qualitative research techniques.

Each person was asked a series of open-ended questions about qualitative research. The questions and a summary of responses are shown below (Note: not all questions were asked of all groups).

1. How do you define qualitative research? (All respondents.) The simple answer was Anything thats not measured or analyzed quantitatively, or doesnt have a p value. Participants reported that qualitative research could run the gamut from purely subjective speculation (which wasnt considered to be true research), to descriptive analysis that can be very objective, systematic, and capable of replication, to methods where data collection is qualitative but the material is content analyzed and quantified (or, conversely, where quantitative data is analyzed by qualitative/judgment means). Definitions included research that is recorded and handled in a narrative fashion, uses generally accepted qualitative methods, emphasizes the participants perspective, seeks some kind of meaning behind an event or condition, uses non-standardized methods for data collection and/or interpretation, or doesnt address causal relationships or test hypotheses.

Several editors pointed out that there are a host of useful methods for collecting and interpreting data and that distinguishing between quantitative and qualitative methods isnt a particularly helpful way of looking at things. Instead, the research design and question should drive the choice of research method/techniques. The most important point is that the research, whether qualitative or quantitative, should be rigorous.

2. What are the specific techniques/methods that you would classify as being qualitative? (All respondents.) Responses to this question ranged from simply anything that is not numbers or doesnt involve counting and anything that needs to be analyzed by a person rather than a computer to a detailed list of techniques. The journal editors mentioned familiar methods such as interviews, participant observation, content analysis (of written or spoken materials), and case studies. Other techniques cited included ethnography, profile interpretation, thematic analysis, and grounded theory. Some editors replied that so little qualitative research is submitted to their journal that they were not able to identify any particular qualitative techniques.

In contrast, the RMNet respondents listed a whole host of techniques some of which may be less familiar to I-O psychologists. Their list included the following:

3. What are the specific topics or research questions that you think could best be addressed by using qualitative techniques? (RMNet only.) Respondents felt that a broad range of questions can fruitfully be addressed by qualitative techniques, although it is most useful in the early stages of research to develop theoretical propositions and derive hypotheses. Research that requires extensive contextual background or description to understand the phenomena in question, as well as studies where causation and theory testing are not the primary objective, are also good candidates for using qualitative techniques. Generally, respondents said that the nature of the data and the type of research question being asked would determine the usefulness of qualitative versus quantitative methods.

Topics that were specifically mentioned include: mechanisms, black boxes, and other poorly understood phenomena, group dynamics, creativity, cross-cultural research, topics which require thick description, organizational processes (including organizational change), and leadership.

4. Do you think that there is enough research using primarily qualitative techniques/methods published in the major journals? If you think that there is not enough, what factors do you think contribute to this shortage of published qualitative research? (Editors only.) The general consensus among journal editors was that there was not enough qualitative research published in the major journals. Some felt that this was due to I-O psychologys prejudice against qualitative research. Too often, doctoral students (and junior faculty) are told not to use these techniques if they want to get published. The perception (supported by these interviews) was that reviewers and editors hold a bias against qualitative work and so make it difficult to publish qualitative studies.

Another reason identified by editors was a lack of educationdoctoral students are simply not trained to use qualitative techniques. As a result, many researchers are ill prepared to use qualitative methods and turn out less than perfect qualitative studies that do not get published. The blame is then placed on the methods used rather than the training in the method itself. To do qualitative research well takes a great deal of skill and practice but few people take the time to learn how.

Editors also raised concerns about the reliability and validity of qualitative information, as well as the low replicability and subjectiveness inherent in some of this work. In some cases, the qualitative research submitted to journals was seen as poorly done and too subjective to qualify as scientific research.

A number of editors pointed out that we need to publish good research and not have quotas or goals for publishing a certain amount of research that uses a particular methodology. If researchers are using qualitative methods in an appropriate fashion and their work is being published, then this is enough. If it isnt getting published because a journal doesnt endorse this style of research, then perhaps were not seeing enough qualitative research.

5. Approximately what proportion of articles submitted to your journal use primarily qualitative techniques or methods? (Editors only.) Submission rates ranged from virtually none (Journal of Applied Psychology, Human Factors (HF), Journal of Vocational Behavior, Industrial Relations, Applied Psychological Measurement (APM), Journal of Organizational Behavior, and Journal of Management) to 2030% in Journal of International Business Studies, Human Resource Development Quarterly, The Organization Development Journal and Group and Organization Management and about 50% in Journal of Applied Behavioral Science. In the case of APM and HF, this lack of qualitative research submitted to the journal is likely a function of the journals focusAPM is solely concerned with measurement methods and procedures which, by definition, are quantitative, whereas HF has a policy against publishing case studies (case studies are encouraged at Ergonomics in Design so authors tend to submit qualitative studies there instead of to HF).

Several journals specifically encourage submission of qualitative articles. For example, Journal of Vocational Behavior (JVB), Journal of Occupational and Organizational Psychology (JOOP), and Human Resource Development Quarterly each have designated qualitative reviewers who are experts in qualitative research methods. Journals like JVB have commissioned invited articles dealing with this subject and the editors have written articles promoting qualitative research in their journals. JOOP has published a set of criteria for evaluating papers using qualitative research methods so that reviewers, authors and readers of the journal can judge the quality of research papers that use these methods. These helpful guidelines are found on the JOOP Web site at www.bps.org.uk/publicat/ejournals/op/qual.html.

6. As an editor, what differences (if any) do you see in the quality and/or content of research studies submitted to your journal for publication that use primarily qualitative versus quantitative techniques? (Editors only.) Editors seemed to be divided into different camps on this question.

(1) I see a few lousy case studies a year and zero to two good ones. Some editors felt that what little qualitative work they saw was generally of a poorer quality than the quantitative submissions. Qualitative studies were often highly subjective and had an ill-defined scope, claiming to incorporate far more than could reasonably be supported. In addition, one editor felt that qualitative studies required a higher writing quality and that this was often the most critical failing of the qualitative studies submitted to that journal.

(2) I dont see any systematic difference between the level of quality of the two general approaches. Some editors saw no difference, while others suggested that most quantitative submissions were as weak as, or of an even lower quality than, qualitative submissions.

(3) Qualitative articles typically address more important and interesting questions. Many quantitative articles were seen as focusing on only numbers and providing no context, making the results not particularly meaningful. In contrast, qualitative studies, or quantitative studies supplemented with qualitative data, were seen as providing a richer, more complete picture of the issue under investigation and allow researchers to study more exciting topics.

7. Do you consider research that uses primarily qualitative methods to be less rigorous than research that uses quantitative methods, and if so, why? (RMNet only.) The answer here was a resounding NO! Respondents were quick to point out that lots of bad journalism gets packaged as social science under the banner of qualitative methods . We should not allow this to tarnish the image of good qualitative research. The general consensus was that good qualitative research is hard to do, but that if done correctly, it can actually be more rigorous than quantitative research. Many forms of qualitative research begin with strong frameworks and involve intensive, systematic analysis of data that have been collected in an objective manner.

Respondents also raised the point that rigor does not necessarily equal quantification. Quantitative research is not synonymous with objectivity and qualitative research with subjectivity. Both research approaches (quantitative and qualitative) have a degree of subjectivity because both are influenced by human decisions. Each method must be judged against the standard of what it claims to accomplish. As one respondent said, The dumbest things get said when researchers try to criticize one set of techniques by the standards of the other. Each technique has its own form of rigor. The same technique may stand up well against an appropriate standard and look shoddy in relation to an inappropriate standard. For example, IRT models can make for great psychometrics but lousy ethnography.

8. Do you think that qualitative techniques are a useful/valuable addition to the field of I-O psychology? Why/why not? (All respondents.) The answer from journal editors seemed to be a qualified yes while the RMNet subscribers generally gave a more enthusiastic Absolutely! Most editors saw qualitative techniques as being useful in the early stages of scientific investigation (e.g., obtaining a general understanding of a topic, identifying variables or constructs, developing theory, and generating testable hypotheses) rather than the hypothesis-testing phase. Editors generally felt that more objective and precise methods (typically quantitative) of investigation should take over after the initial qualitative work has been done.

Many respondents saw qualitative techniques as a valuable addition or supplement to quantitative research or mentioned the need to use multiple methods to create and explore theories. Research using one tool from the kit can be provocative, but not necessarily convincing. Anything that adds to the diversity of the field and that challenges conventional thinking was seen as valuable. Combining quantitative and qualitative techniques in the same area can help to broaden the perspective on the topic by approaching the same issue from different directions. Quantitative research methods cannot always provide the needed information, help us to understand and explain phenomena, or provide viable analysis techniquesqualitative techniques can take us places that traditional quantitative techniques cannot. And, as one respondent said, quantitative techniques take the life out of the phenomenon under study.

Finally, a few respondents felt that qualitative research doesnt make many unique contributions to the research literature. According to this view, Purely qualitative research makes the work anthropological, rather than psychological, and doesnt belong in the psychology literature.

9. Do you conduct/have you conducted research that uses primarily qualitative techniques or methods? Why/why not? (RMNet only.) Most respondents had conducted qualitative research, although typically only in limited amounts and as a supplement to quantitative work. Qualitative techniques were used when they provided the best method for addressing a specific question or topic of interest. Those who hadnt conducted this type of research were generally concerned about the riskiness (with respect to getting it published or having it taken seriously) or found that it wasnt appropriate for the research question under investigation.

10. Have you published any research that uses primarily qualitative techniques/methods? If so, what were your experiences in trying to get it published? (RMNet only.) Most respondents had not submitted a piece of qualitative research for review, partly because the I-O journals are seen as having a strong empirical bias. Those who did submit qualitative research faced an uphill battle, with frequent rejections and editors remarks that this type of research does not belong in a traditional psychology journal. Even those who used content analysis to quantify the data and used inferential statistics to test hypotheses found it difficult to publish in the I-O journals.

One respondent summed it up by saying I have become increasingly inclined to take the research to journals in other disciplines where it will be better appreciated rather than attempting to press it on an unwilling audience. It seems to make more sense for all parties concerned to follow the path of least resistance in this case. There is little point in publishing something in a journal whose readership will not find it useful. 

Summary and Conclusions

In sum, several key points can be made regarding the state of published quantitative research in I-O psychology.

First, many I-O psychologists are not very familiar with the wide variety of qualitative techniques that could potentially be applied to our research efforts. Both editors and researchers agreed that the appropriate method or technique should be chosen based on the nature of the research question itself. Unfortunately, if we dont know what qualitative techniques are available, how can we choose the most appropriate approach to our research questions?

Second, good qualitative research is hard to do, but with training and experience, its possible to conduct rigorous, objective and significant qualitative studies. Unfortunately, we dont tend to do much qualitative research and, if we do, we generally dont do it very well.

Third, qualitative research is often seen as a supplement to quantitative work, rather than as a stand-alone method. There is a perception that Good qualitative research involves quantification-in which case the research is no longer qualitative.

And finally, while editors may recognize the value of rigorous qualitative research and express a willingness to publish this type of work, researchers report that they are discouraged from submitting qualitative articles to I-O journals because they get rejected. Why does this discrepancy exist?