TipTopics April 2007

“The end has come!” Greetings friends, TIP-TOPics readers, and fellow graduate students. Sadly, our editorial journey is coming to a close. During our 2-year tenure, we have been very fortunate to work with a number of well-known experts and I-O psychologists from various areas with different research interests. Our experiences as TIP-TOPics editors have been educational, and we have some advice for the next editorial group and graduate students in general: (a) Stick to your plan—we had a relatively smooth time putting together each of our columns because our blueprint for success was all laid out. We followed exactly what we set out to do 2 years ago and, as a result, had few complications; (b) always have a Plan “B”—each of our columns included information from experts in the field. Sometimes, we did not get this information from the people we intended to get it from. We made sure to have multiple outlets and backup plans so we had enough time to get the information that we needed; and (c) don’t procrastinate—we set goals for each column so that there was never any last minute rushing a week or two before the submission deadline. Goal setting worked for us and hopefully it will work for you!

In our final column (tear drops), we will turn our attention to research methodology and statistics. For this issue, we had the distinct pleasure of speaking with Frank L. Schmidt. We also have another remarkable panel of experts who shared their experiences and knowledge about statistics, research methodology, and its application to I-O psychology. Finally, we will end with our last assessment center, where we highlight the topics and interviews from the past 2 years.

I-O 101

What issues make research in methods challenging and rewarding for researchers?

Our experts felt that the challenges and rewards of methods-related research come hand in hand. Vish Viswesvaran suggested that research methods are interesting when they address a substantive problem and when researchers demonstrate that the use of a “new” method substantially improves one’s results (not just with simulation data but also with typical real-world data). José Cortina pointed out that methods research is rewarding because it can be more broadly applicable than empirical research on a specific topic. For example, various obstacles in the climate work of Larry James led him to study interrater agreement. Although the climate–related findings are very influential, it is of interest only to those who work in that area. However, work related to rater agreement is applicable to all researchers studying in the organizational sciences.

Alfred Dansereau felt that the main challenge for methodological research is that any theoretical inquiry raises questions as to how to test the idea. For example, his interest in leadership led him to question how leaders treat subordinates equally in some instances and at the same time treat them differently in others. It took years to figure out, but it turned out to be a question about levels of analysis that he and others were able to address using within-and between-entities analysis (WABA). As this example illustrates, our experts agreed that methodological research is a key component to the advancement of our field.

Within academia, there are researchers who use methods to analyze their data and there are researchers who research methods. How do these groups inform each other?

According to Alfred Dansereau, some questions that researchers want to ask go way beyond the capability of contemporary methods. He also stated that the link between the two groups typically occurs when a researcher asks a question of a methodologist and they work together to find an answer. Vish Viswesvaran stated that another way to foster interaction is through joint workshops, tutorials, and panel discussions at SIOP conferences. José Cortina suggested that few people in I-O actually research methods; many researchers generate ideas for methods papers from problems they confront in their own program of research. An example of this is a paper by Cortina and colleagues (Cortina, Chen, & Dunlap, 2001), in which they needed to include an interaction in a structural equation model and discovered that no one in I-O knew how to do it! They decided to explore different methods of including an interaction variable and then disseminated their results.

There are a number of “hot” methods being used by academicians and practitioners, such as item response theory (IRT), structural equations modeling (SEM), and hierarchical linear modeling (HLM). If you had to select five methods that every I-O psychologist should have in their repertoire, what would they be?

All of our experts agreed that a fundamental understanding of the general linear model, scale development, reliability analysis, validation, and factor analysis is what one should aim to achieve in graduate school. As for “hot” methods, they selected meta-analysis, SEM, HLM, and IRT. However, they stressed that it is more important to let the questions you want to answer dictate the methods that you use for your research.

How should students develop and manage a repertoire of statistical methods?

Alfred Dansereau suggested learning as much as you can about “when” to use different methods to answer different questions. He mentioned that a particular method might be useful for a different theoretical question of interest in the future. Vish Viswesvaran challenges his graduate students to read one book that is not in their area of research and one methodology text every semester (Wow!). José Cortina also suggested that people should not be afraid of learning how to analyze their data the “right way” and that there is nothing magical about statistics.

For example, let’s consider SEM. On the surface it seems terribly complicated with its exogenous and endogenous variables, manifest and latent variables, phi matrices, gamma matrices, beta matrices, theta-delta, and theta-epsilon matrices. The reality, however, is that SEM is nothing more than factor analysis layered onto regression and that most of these terms are just fancy names for things that we already understand (predictors, criteria, indicators, factors, predictor intercorrelations, regression weights, more regression weights, and measurement error variance for those who wanted a translation). The bottom line is that if your data are worth collecting, then they are worth analyzing correctly. The correct way might be the simplest, but if it is not, then teach yourself the correct way.

BI-O

Frank L. Schmidt is the Ralph L. Sheets Professor of Management and Organizations in the Tippie College of Business at the University of Iowa. Dr. Schmidt has published more than 150 research articles and book chapters on personnel selection, selection utility, meta-analyses of validity of selection methods, research methods, and meta-analysis methods and has coauthored six books, including the second edition of Methods of Meta-Analysis: Correcting Error and Bias in Research Findings, which he coauthored with John Hunter. Dr. Schmidt earned his bachelor’s degree in psychology from Bellarmine University and his master’s and doctorate degrees in I-O psychology from Purdue University, and he is currently on the editorial board of the Quantitative Series in the Social Sciences, International Journal of Selection and Assessment, and Psychological Methods. In addition, Dr. Schmidt is a past president of the Division of Measurement, Statistics, and Evaluation (Division 5) of APA, past chair of the Defense Advisory Committee on Military Personnel Testing, and a former member of the Liaison Advisory Group to the Committee on the General Aptitude Test Battery of the National Research Council, National Academy of Sciences. He has received a Distinguished Scientific Contributions Award from both the APA and SIOP as well as the Lifetime Research Methods Contribution Award from AoM and the Heneman Distinguished Career Award for Research Contributions to Human Resources from the AoM Human Resources Division.

Did your graduate school experiences prepare you for working within the field?

Great preparation for practice and in methodology—but the training in theory construction and in thinking theoretically was weak. These were the days of dust bowl empiricism! I learned to think theoretically after I graduated.

How did you go about getting your first job once you had attained your degree? How long were you at your first job?

My first job was at Michigan State University in I-O. It was a great job and a great way to start a career. But after 4 years I wanted to try something new—something less academic. So I took a job heading a research program in personnel selection at what is now the Federal Office of Personnel Management (OPM) in DC. It was in this position that I did my work (with Jack Hunter) on validity generalization (VG), meta-analysis, and utility analysis. This work was the basis for the later APA and SIOP Distinguished Scientific Contributions awards. In addition to the research at OPM, the involvement in court cases was very interesting. (OPM was involved in many selection-related court cases at that time.) I also did quite a bit of consulting with private industry, which I really enjoyed.

What things would you have done differently if you knew then what you know now?

In my case, things worked out wonderfully. I would not do anything differently. Of course, if I’d known how long it takes to change the thinking and beliefs of a field when research clearly shows the need for this, I might not have had the motivation to persist. I am referring to the fact that it took about 20 years to convince the I-O field of the truth of VG. I had always thought that if you can empirically prove something, a science-based field would have to accept it. They did, but it took longer than I thought it would.

How did you go about developing your current research interests?

I have the same basic research interests now that motivated my dissertation: What is the meaning of data? How can data be interpreted properly? My dissertation showed that regression weights—considered statistically sophisticated—often produced prediction inferior to simple unit weights. My subsequent research showed that single group validity was an illusion caused by data artifacts—an illusion that was accepted as fact at that time. The later work on VG and meta-analysis showed that acceptance of data in individual validity studies at face value leads to very erroneous conclusions. Other research of mine showed how failure to correct for measurement error leads to erroneous interpretations of data, in particular, underestimates of the size of relations between constructs, an error that distorts the construction of theories. The common theme in all this is the deceptiveness of data if taken at face value and the need to look deeper to determine the real meaning of data. The proposition that “the data speak for themselves” is false. Data often lie to researchers. If we cannot solve this problem, we cannot have cumulative knowledge—or any reliable knowledge.

Is the work that you do now related to the work you did early in your career?

Yes, I am still working on the general question of how best to extract the truth from data. In recent years I have developed more accurate methods of meta-analysis for doing this. Many of these are discussed in the 2004 Hunter-Schmidt meta-analysis book (2nd Ed.).

What obstacles in graduate school and in your career did you experience that you were not anticipating and what advice would you give to students and young professionals to help overcome these challenges?

I did not anticipate that it would take 20 years to change the thinking of a field after you present overwhelming empirical evidence showing the need for such change. I do not think this is unique to the field of I-O. I have learned through my reading that this is true of pretty much all fields. For example, consider the researchers in Australia who showed that stomach ulcers are caused by a bacterium, not by stress as was the dominant belief. Convincing the medical profession of this took nearly 15 years, even though the evidence was clear. So young researchers should realize that there will be this resistance to new findings and should be prepared psychologically to deal with it. In our case, the 1985 Q&A article in Personnel Psychology turned out to be the most effective way to deal with this resistance. It led to an amazing turn around in the field.

What is your typical day at work like?

There is no typical day—some days I teach PhD students, some days I work on research all day, some days I consult. This variety is a great thing about I-O. Every day is different.

What were your greatest doubts in graduate school and how did you overcome them?

I could not decide whether I wanted to be a practitioner or an academic. I had both kinds of job offers. I took the MSU faculty offer—the lowest paying one! But it was the best one for my career and it allowed me to discover the fact that research was my true calling.

What were the most appealing characteristics/qualities of the career you selected and why did you choose this over the other side (i.e., applied or academic)?

The appealing thing about being an academic is that you get to do everything: teach, do research, and do consulting. And you have a great deal of autonomy in deciding how to distribute your time across these. However, the most satisfying thing of all is seeing your research have a major impact on the field, seeing it change the beliefs and practices of the field. I have had this experience in the areas of VG, meta-analysis, test fairness/differential validity, and selection utility. On the other hand, training and developing PhD students who go on to be top players in your field is just about as satisfying, so it is hard to decide. I think these two things sort of go together. I have had some really outstanding PhD students (and still do), and I am grateful for that.

What are the most satisfying and dissatisfying aspects of our field to you? How has this related to your career?

The most satisfying thing is the fact that we have a field that is truly science based, a field that bases its theories and practice on solid research findings. The most dissatisfying thing is how long it takes for beliefs and practices to catch up with research—the time lag is greater than it should be.

Assessment Center

Those of you who actually make it this far into the article will notice that we have once again departed from our modus operandi. Because this is our last issue, we felt that it would be appropriate to recap our purpose as well as outline the topics we’ve covered and the people we’ve chatted with over the past 2 years (see table). As we boldly declared in our first article, our objective was not to advise but to present information we have collected about how others have succeeded and what others are currently doing to succeed. To this end, we laid out three objectives and developed a three-part structure for our articles:

To examine research areas that are important and that all I-O graduate students should possess a general knowledge about. To accomplish this goal, we conducted a literature review of three of the top I-O journals (Journal of Applied Psychology, Academy of Management Journal, and Personnel Psychology) to see the most studied topics in the last 3 years. In addition, we surveyed I-O psychologists about the topics they felt will be important in the future. Based on these efforts, we generated a list of topics for our articles. We then contacted experts on each topic and had them share their thoughts in a section we called I-O 101.
To provide students with information on how successful I-O psychologists arrived at where they are today. For this objective, we surveyed graduate students about the questions that they would like to see answered by someone who has achieved success in our field. Based on this information, we developed a structured interview and posed these questions to some of the big names in our field. Their answers were presented in a section we entitled BI-O.
Provide a snapshot of what I-O graduate students around the country are doing to succeed as well as provide an open venue for alumni and students to share survival TIPs with other students. This was an ambition that was addressed in many ways. During our tenure, we have conducted surveys, presented stories, as well as provide a directory in a segment named Assessment Center. Although varied, our aim was always to provide some space for graduate students to have a voice and see what other graduate students were thinking and doing.

Table 1
Summary of Columns
__________________________________________________________________________

Vol.
_____

I-O 101 Topic and Contributors
______________________________

BI-O
Interviewee
_______________

Assessment Center
____________________

43(2) Occupational Health Psychology (OHP): Peter Chen, Leslie B. Hammer, Steve Jex, James A. McCubbin, and Paul E. Spector James Campbell Quick Survey of and anecdotes about stress in graduate school

43(3) Emotions in the workplace:
Neal Ashkanasy, Vanessa U. Druskat, Frank Landy, Brian S. O’Leary, David Van Rooy, and Howard Weiss Richard E. Boyatzis Survey of emotions in graduate school

43(4) Leadership in the workplace: Bruce J. Avolio, David V. Day, Cynthia D. McCauley, and Ronald E. Riggio W. Warner Burke Survey of leadership in graduate school

44(1) Counterproductive work behaviors (CWB): Rebecca Bennett, Lilia Cortina, Jerald Greenberg, Joel Neumann, and Sandra Robinson Paul E. Spector Survey of “graduate CWB”

44(2) Teams in the workplace: Michael Beyerlein, Micheal Brannick, John R. Hollenbeck, and Susan Mohammad Eduardo Salas Anecdotes related to teams

44(3) Cross-cultural psychology: Ronald Fisher, Michelle Gelfand, Paul Hanges, Cong Lui, Kong Knok Yee, and Ng Kok Yee Harry C. Triandis Directory of
international I-O programs

__________________________________________________________________________

Conclusion

Special thanks to all our experts for this column and from all our previous entries. Our column would not have been what it turned out to be if it wasn’t for your input and assistance. In particular, our research methodology experts included: Alfred Dansereau (State University of New York), José Cortina (George Mason University), Frank Schmidt (University of Iowa), and Vish Viswesvaran (Florida International University). If you would like additional commentary from our panel, please feel free to e-mail us at tipsontopics@yahoo.com. We hope we have been able to help you on your graduate educational journey over the last 2 years. Best of luck to the new editor(s), and we’ll see you at SIOP in New York!