Informed Decisions: Research-Based Practice Notes
Steven G. Rogelberg
Bowling Green State University
This month's column is substantively different from past "Informed
Decisions" columns. This month, instead of discussing ways and approaches
to hopefully improve the practice of I-O psychology, I have asked Jim Smither
and colleagues to discuss how sound practice can be translated to research
publications. Their valuable publication insights are helpful to practitioner
and academic alike. If you have any comments/questions concerning this column
please contact Jim Smither at Amysean@aol.com.
If you have any ideas for future columns or would like to propose authoring a
column, please contact me at rogelbe@bgnet.bgsu.edu.
Publishing Applied Work
Anthony T. Dalessio
Bell Atlantic
Calvin Hoffman
Southern California Gas Company
Richard R. Reilly
Assessment Alternatives Inc.
John Hinrichs
JR Hinrichs, Inc.
Ken Pearlman
Lucent Technologies
Nancy Tippins
GTE
James W. Smither
La Salle University
Steven Rogelberg recently asked me (Jim Smither) if I could put together
some thoughts on publishing applied work. This is a topic that is of
considerable interest to me. Many of my own publications have dealt with field
data (for example, growing out of my previous role as a full-time practitioner
at AT&T and my current role as a consultant). Also, I currently serve as
Associate Editor of Personnel Psychology where I am handling submissions
for the Scientist-Practitioner Forum (SPF). The goal of SPF is to offer
solutions, insights, lessons learned, guidelines, tools, and methods for
addressing the problems and issues that confront practitioners. Unlike magazines
or newsletters (that may sometimes address similar concerns), all decisions are
made via a rigorous peer review process to ensure that all material reflects
state-of-the-art thinking and practice.
After thinking about Steven's request, I thought it would be best to seek
advice from some experts, so I asked members of our Editorial Board to offer
their responses to these questions.
What advice would you offer to practitioners who are interested in
publishing their applied work?
Think about your experiences as a reviewer of applied work that has been
submitted to peer-reviewed journals. What are the most important features you
look for in such a submission? What captures your attention and heightens your
interest? What makes you lose interest in an article? What are some of the
characteristics that distinguish promising from less interesting (or less
promising) submissions? What kinds of articles (topics, formats, etc.) would you
like to see more of? Less of? What insights can you offer to practitioners about
the "mind of a reviewer"?
I received responses from Skip Dalessio (Bell Atlantic), John
Hinrichs (JR Hinrichs, Inc.), Cal Hoffman (Southern California Gas
Company), Ken Pearlman (Lucent Technologies), Dick Reilly
(Assessment Alternatives Inc.), and Nancy Tippins (GTE). Each of these
psychologists exemplifies the scientist-practitioner model. Each is an excellent
practitioner and each has published applied work in top-quality peer-reviewed
journals. Following is a summary of their thoughts and suggestions.
Advice to Practitioners Interested in Publishing Applied Work
Our expert panel emphasized that practitioners are often able to create
research opportunities and capitalize on unique data (of which many academics
would be envious). Everyone noted the value of collaborating with others.
Others' advice dealt with setting goals and making public commitments, obtaining
preliminary feedback, managing your time, dealing with the all too common
corporate barriers, adopting the role of an "archaeologist," and
staying current with the literature and other trends.
Create Research Opportunities and Capitalize on Potentially Unique Data
Dick Reilly: Opportunities to collect interesting data often occur in
applied settings but these opportunities are frequently missed because the I-O
psychologist is focussed on getting the project implemented. My advice would be
to periodically consider the projects that you are planning or working on, and
whether some of them might represent an opportunity to shed some light on an
issue or problem of interest to the field. An example would be the introduction
of a new program. With a bit of planning, you might be able to either find or
collect relevant data that could be used to assess the impact of the
intervention on employee behavior, attitudes, and so forth.
Cal Hoffman: What kind of creative spin can you put on the data you
have available? Have you come up with some novel solution which you think could
be useful for others in practice? This is to some extent the luck of the draw in
terms of the types of projects you get to do at work, but it also has a lot to
do with how you look at things, especially having a knack for finding secondary
or tertiary uses for data and solutions.
Ken Pearlman: Create research opportunities within existing projects
and look for angles or twists that might exist (or could be created) within an
apparently mundane project.
Jim Smither: While I was at AT&T, we were conducting a validation
study with recent college graduates we had hired for entry-level professional or
management positions. I had been told by campus recruiters that applicants often
complain about selection procedures such as written tests. But I only had
anecdotal stories. I realized that we could use our validation study as an
opportunity to gather reactions to selection procedures (different types of
tests, interviews, biodata, etc.). It only took a little extra time to capture
these data from our new college hires. This very modest "twist" on a
validation project ultimately led to a publication in Personnel Psychology
("Applicant Reactions to Selection Procedures," 1993).
Ken Pearlman: Think about research and practice gaps you have
discovered in your own practice. What did you do to bridge these gaps? For
example, instances where there was no guidance from the research literature, and
no tool available to do X, soyou had to invent one. Develop new twists on
established practices to meet new needsor situations, that is, long-standing
practice applied to a novel or interesting setting (e.g., cross-cultural testing
or assessment). Don't assume that a particular practice, problem, or solution is
too trivial or not of wide enough interest or frequent enough occurrence to be
publishable. Often many others have had the same problem and could benefit from
your practice.
John Hinrichs: If you're chartered to conduct some study, try to
piggy-back issues fitting some of the themes you are interested in. In working
with managers, don't be shy to suggest data collection or even full-blown
research designs to deal with topics of interest or concern. Don't be shy, be
creative.
Skip Dalessio: Many practitioners and consultants often have very
interesting data that can only be gathered as part of an applied research
project. Such data often are not available to academics, and provide a basis for
a potentially interesting article.
Collaborate with Others
Dick Reilly: Practitioners are busy and often do not have extra resources
for the tasks involved in developing a publishable paper. On the other hand,
many academic I-O psychologists would welcome the opportunity to work with field
data. One approach is to collaborate with an academic researcher who may have
the time and resources to enter and analyze data, and do some of the other work
that is necessary to produce a publication. This kind of collaboration can be a
"win-win."
Cal Hoffman: My advice for practitioners who want to publish, even in
a venue like SPF, is to seek out someone you know who has published themselves,
and who has expertise in the topic you want to share. There are tricks of the
trade you can pick up from such people, especially in areas like not overstating
the importance of your study, and they usually have a better knowledge of the
current literature in an area. My own experience in getting involved in
publishing was that a friend (Barry Nathan) who was in academia at the
time commented on a poster on criterion predictability I had taken to SIOP. He
felt it was bringing in valuable, real world data that was often in short
supply, and encouraged me to write up an article for publication. If Barry had
not given me that nudge, I probably would not have become involved in publishing
(and reviewing for SPF) like I did. I collaborated with Barry in revising my
SIOP paper, and shared an authorship (Hoffman, Nathan, & Holden, Personnel
Psychology, 1991) in exchange for a lot of valuable guidance, advice, and
work in repackaging the article to be in a form more suitable for publication. I
have since partnered with two other I-O psychologists outside my company (one
academic, one consultant) to publish other articles in addition to authoring
several by myself.
Ken Pearlman: Find co-authors with more time than you, or additional
incentives, to work with you on either conducting research or writing (for
example, junior or younger staff, interns, colleagues in academia, etc.)
Nancy Tippins: Enlist the help of others. Publishing good research is
a formidable task. Finding a cohort in the endeavor facilitates the effort.
Maybe misery loves company but perhaps your colleagues will keep you motivated
and vaguely on schedule. There seems to be something to public goal setting and
a sense of responsibility to other people.
Skip Dalessio: Work with other committed colleagues on the manuscript.
If all the authors are committed, each is likely to encourage the others to
complete the product. However, if the authors are not committed to finishing the
manuscript, a diffusion of responsibility may result, and it may actually be
more difficult to bring the product to closure.
John Hinrichs: See if you can engage colleagues in other settings for
cooperative work.
Obtain Some Preliminary Feedback
Cal Hoffman: Ask someone other than co-authors to read the paper. When
you are engrossed in writing a manuscript, it is very easy to overlook little
things, or to think you have done a great job explaining something which is
already very clear to you. Manuscripts frequently need work in areas like
operational definitions, or in introducing intermediate concepts that form the
foundation for understanding what you are trying to describe. Giving the
manuscript to a "naive" reader is a good way of learning if someone
who has not worked directly on your project can understand what you are saying.
Skip Dalessio: Before submitting to a journal, submit the paper as a
poster or as of part of a practitioner forum at the SIOP conference. By doing
this, the author(s) will be able to write a first draft and obtain some initial
feedback. If the manuscript is accepted to the conference, the author(s) will
have an opportunity to revise it for presentation and then submit a more
polished draft to the journal.
Cal Hoffman: If SIOP submissions do not get accepted, that doesn't
mean you should give up on revising and submitting to a journal. My last paper
(Hoffman & McPhail, 1998) was rejected as a SIOP poster session, but I was
still able to get it accepted elsewhere. The key is listening to what the
reviewers say and trying to be responsive.
John Hinrichs: Test the waters with a draft write-uppreferably one
intended for a non-refereed publication. Get colleague reaction (and input from
friendly management as well, if feasible) about the reasonableness of your
thesis, presentation, and its value. Revisit, revise, and submit. Perhaps two
versions: one for the popular press and one for an I-O journal.
Set Goals and Make Public Commitments
Nancy Tippins: If sharing research in a public forum is something you
want to do, then make it a goal. I've found it useful to share my goals with
other professionals. Sometimes it's helpful to put such a goal in a performance
agreement at work. If nothing else, I will be embarrassed that I didn't fulfill
my commitments.
Make it a Priority and Manage Your Time
Nancy Tippins: For me, the big barrier to writing is not the lack of
appropriate data; it's not coming up with a good idea; it's not the dim view
some managers take of this kind of activity; it's not the lack of corporate
reward for this activity; it's not getting approval from the legal department;
it is time and priority. From my perspective, many jobs in private industry have
become extremely time consuming. Many of the people I know work 60+ hours a week
on a regular basis. Then there are the bad weeks! Decide if writing for
publishing is a priority. Many of us finish graduate school with a sense of
obligation regarding research. The first step is to decide if this is an
important aspect of our career that we want to pursue. I believe it's something
we ought to be doing. Our profession is stronger because of input from
scientist-practitioners in academic and organizational settings. In addition, I
have found my professional activities to be extremely rewarding (intrinsically).
Assuming publishing research is a priority and you have set your goals
accordingly, the next step is to make the time. That's much easier said than
done. I've tried lots of things: 6:00a.m._8:00 a.m., 9:00_11:00 p.m.; any hour a
day, Saturday mornings, Sunday afternoons, and so forth. All strategies have
worked at times, and all have failed. Be flexible; be systematic; set goals; be
flexible; and keep trying.
Ken Pearlman: Push yourself to take the extra step to write up your
worksometimes that's all that's missing. Find the motivation.
Skip Dalessio: Like most people, practitioners, as well as academics,
are very busy. Completing professional work is actually valued as part of an
academician's job, but not often as part of a practitioner's job. Time needs to
be set aside to work on professional papers. Potential authors need to think
about times when they would otherwise be reading the newspaper or watching TV
that they could work on professional papers; perhaps on a plane or a train.
These are good opportunities to spend part of the time working on the paper.
Several 30-, 45-, or 60-minute time periods working on a paper will result in
substantial progress. It may be easier to find this kind of time, rather than 3-
to 5-hour blocks.
John Hinrichs: Start files of notes and relevant background materials
(reports, letters, citations, etc.) of any themes that you see. Noodle ideas
about publishing something from one of your files. At least once a month,
attempt to sketch out a rough outline or research design that would address the
ideas in one of your folders. Re-file your rough notes and set a tickler for
yourself to look at them again in a month or so. Ask yourself: "How can I
shape my work priorities so that at least part of the time I'm dealing with one
of my items of interest?"
Manage Corporate Barriers
John Hinrichs: Don't ask permission; just do it! In some organizations
what you are doing may be considered subversive; if caught, plead ignorance and
go underground. Don't worry, if it's ever published in a refereed journal,
nobody in power will ever know.
Dick Reilly: There are lots of obstacles to publishing from applied
settings. Publishing is typically not rewarded and may even be seen as a risk by
legal, since it makes data public and explicit. I am not sure what you do about
legal. Legal tends to be highly risk aversive, so if you ask them, they will say
no. My advice would be to use your own judgment and if you think that what you
are doing is not placing the company at risk, don't ask.
Nancy Tippins: Ignore the corporate barriers. Many of us allow
corporate rules to keep us from pursuing our research or writing an article for
publication. Corporations (sometimes with good reason) are fearful of what we
might say or do that subjects them to unnecessary legal scrutiny. I've worked
for several companies that had their fair share of legal problems.
Interestingly, not one of them ever said "no" once they saw the
article or presentation. I'm sure it happens: The legal department balks. My
personal experience is that it's rarer than I thought. (I once had a public
affairs type tell me that an article was the most boring thing he ever read and
he only finished the abstract!)
Ken Pearlman: Educate or persuade your management as to the advantages
to the organization of published research (prestige, credibility to both you and
them, etc.) or offer compromises (e.g., if you can do the research on company
time, you'll do the writing on your own time).
Be an Archaeologist
Ken Pearlman: Become an I-O "archaeologist"dig up old data
that you (or your predecessors) collected but never had time to do anything
with. If you don't have the time, get help (interns/junior staff) analyzing it
and writing it up.
Dick Reilly: Make yourself aware of archival data that might be used
in longitudinal studies or quasi-experiments. Most companies keep lots of
records and today most of these are computerized, so knowing what is available
could help you in thinking about how you might be able to put what you are
working on into a more rigorous research framework.
Stay Current with the Literature and Other Trends
Ken Pearlman: Stay current with the literature, so you are more likely to
see opportunities.
Skip Dalessio: Address a practical question, but tie the question back
to relevant literature.
John Hinrichs: Be continually aware of what are the
needs/interests/hot buttons of the three differing parties to the
dilemmaprofessionals (I-O) in organizations, their employers/managers who
control their organizational life and careers, and the academic community.
Recognize that you are but one point in a triangle of competing priorities and
your task is the practically impossible one of satisfying all three, but first
of all, satisfying an itch to publish. (Without the itch, it probably won't
happen, and I don't have any magic itching potion.) Assess where each of these
constituencies are spending their time, what challenges they face, what's
causing concern, and which, if any, converge with your job setting. Think about
possible linkages of these issues to the literature and to the work that you are
aware of in other organizations.
Advice for Academics
Cal Hoffman: My advice for academics who are looking to get access to
organizational data and maybe another publication is to browse the poster
sessions at SIOP and see if anyone has preliminary results in an area of
interest to you. This type of collaboration is an effective way of bridging the
science/practice gap!
What Do You Look for When Reviewing a Journal Submission?
In response to this question, our expert panel (all of whom review for Personnel
Psychology and probably other journals as well) discussed innovation,
practical implications, quality, clarity, organizational context,
generalizability, interdisciplinary work, not being buried by the data, and
responding to reviewers' suggestions.
The "So What?" Test
Cal Hoffman: When I am reviewing a manuscript, one of the first things I
look for is the answer to the "so what" question. What is new or
innovative about the solution offered? Does it shed new light on an old problem?
How easily can it be generalized to other settings and/or help other
psychologists?
John Hinrichs: The "so what?" test. What does it mean to
organizational practice and to academic theory? What's the value added?
Dick Reilly: The description of an innovative approach, or a topic
that has not been addressed in the I-O and related literature will have a higher
probability of success. You may have developed an innovative approach to an
organizational problem that could be useful to other practitioners. Just as
important may be topics that have not been addressed in the literature. As I-O
psychologists we often find ourselves "behind the curve" on business
trends. Often, organizations implement new programs or practices that have no
research support. Practitioners can make a major contribution by providing data
on the impact of such programs or practices. A combination of careful planning
and persuasion (to get the data) can result in important field research
findings.
Ken Pearlman: Application of I-O principles to new contexts, that is,
where I-O research findings or practices logically have applicability but
haven't been used (e.g., the application of task analysis to validation of
medical requirements). Particularly valuable nowadays are practices that reflect
new solutions and new ways of doing things because of the changing workplace;
that is, situations where traditional practice no longer "works"
because of how work or the workplace has changed, there is little or no research
to draw from, and new tools, practices, or techniques need to be developed or
invented (a recent example was the publication in Innovations in Research-Based
Practicethe predecessor to Personnel Psychology's
Scientist-Practitioner Forumof a piece on the development of a job analysis
tool for capturing the personality-related dimensions of work).
Practical Implications
Nancy Tippins: I like to read an article and have an idea of how I could
apply the results in my own company. If I can't figure out what to do with the
information, then the value is limited. I really like the article that shows me
how to do something in an efficient way without compromising the resulting
product. Several of the articles I've reviewed recently dealt with shortcuts in
the development of biodata forms and structured interviews. I also like articles
that take a topic that has been discussed academically and is now being applied
in a real world situation. One of my favorite articles recently was a corporate
application of IRT.
Skip Dalessio: Authors need to develop and discuss practical
implications of their work. If the manuscript does not clearly address practical
implications, it will not have much of an interest outside the academic
community.
Cal Hoffman: Articles for SPF should include a reasonable emphasis on
how to use the output or results (practice implications or advice), and not
focus solely on how to do something.
Quality
Nancy Tippins: The first thing I look for in an article is how well the
research is done. Is the research designed so that the conclusions logically
flow from the evidence provided? Every now and then, I review an article written
by someone who seems to think that poor research is OK because it's written by
and for practitioners. In my mind, a poor research project serves no one. There
is a thin line between research that adapts to organizational restraints and
research that compromises too much. A good article will share how good research
was conducted in an efficient way.
Dick Reilly: Field studies, by nature, tend to be low-constraint
research; however, scientific methods should be applied regardless of the level
of constraint. Quasi-experiments, use of SEM or other techniques that can lend
some rigor to the research will enhance the probability of publication. Also,
recognition of the limitations of the research should be made explicit.
John Hinrichs: Are the data credible to any real issues in industry?
Are they representative of the population to which they are being generalized,
or are they suffering from unacceptable non-response, bias, or distortion?
Clarity and Persuasiveness
Cal Hoffman: I look for readability, editing, and support. Readability,
to me, means the manuscript is easy to read, uses active voice (where possible
within the limits of APA format), and makes a fairly direct case for what is
being described. Editing refers to good old-fashioned quality control: Did the
authors leave in run-on sentences; do they use proper punctuation; and make sure
all the references in the paper are listed in the bibliography? While these
items sound little, when a manuscript has these kinds of bugs, it begins to cast
doubt on other areas. By support, I mean things like the order in which concepts
are presented, how references are used to support the author's ideas, and how
statistical analyses (and interpretations of same) are used to buttress
arguments.
Nancy Tippins: Of course other aspects of the article are important,
too. I'm big on clarity. Every now and then I don't understand what is being
written. The grammar is too convoluted and the vocabulary is arcane. I'm too
lazy to work very hard to figure it out and too busy to be your editor.
Ken Pearlman: What makes me turn off or lose interest as a reviewer:
poor writingthat is, writing that's disorganized, unclear, not concise, has
many grammatical or spelling errors, or is written in a pedantic, academic, or
"jargony" manner.
John Hinrichs: The verbiage. Even professionals don't have time to
read 20-page articles on 2-page topics. A picture is worth 1,000 words. Big
words and techno-babble obfuscate (whoops, sorry for the big word!).
Organizational Context
Ken Pearlman: Another turnoff is papers that ignore or do not report on
the organizational context for the application, that is, what drove the need for
the application, what obstacles had to be addressed or overcome and how was this
managed, and so forth.
Generalizability
Dick Reilly: Some submissions describe work that has broad and useful
application for other practitioners. It might represent an interesting method
that could be used to address a particular organizational problem in almost any
organization. On the other hand, a description of work that does not have any
generalizability beyond a specific situation or a specific context is of less
interest.
Interesting Topics (At Least to Reviewers)
Nancy Tippins: At this point in my career, I'm fairly egalitarian in
content areas. I always like articles on selection and would like to see more. I
would also like to see more good "O" research on topics where many
practitioners are working: leadership development, employment development,
organizational development.
Ken Pearlman: Interdisciplinary workwork that brings in practice
from other disciplines to enhance an I-O issue or practice (e.g., the use of
economic concepts in utility analysis, or the application of principles of
social psychology in addressing applicant reactions to selection procedures).
Skip Dalessio: Applied work that I find most relevant are validity
studies of various types of tests for particular positions. Papers that address
questions about both test validity and adverse impact are useful. Papers that
discuss what standards courts accept for test validation are useful. Papers that
address practical approaches for solving business problems are useful, such as
how to establish a process for developing a competency model, or 360 feedback,
and so forth. Papers that present points and counter-points on topics of
practical concern are useful, such as surveys, 360 feedback, competency
models, test development, interview development.
Don't get Buried by the Data
Ken Pearlman: Other common pitfalls include papers that get so buried in
the data analysis that they fail to provide readers with enough information and
context to understand the substantive meaning and implications of the work.
Also, significance-test-based interpretations and conclusions.
John Hinrichs: Is the author using a sledge hammer to swat a fly? Is
she or he merely demonstrating that advanced statistics and computer literacy
have been mastered? Is all that really necessary, or would a simpler analysis
have sufficed? If necessary, does the author explain why, and what it means in
terms understandable to a reasonably intelligent layman manager. A word of
advice I've used with interns and new I-O hires: "Never present something
to managers that requires taking a square root." That's a bit extreme for a
professional publication, but not too far off target.
Responding to Reviewers
Dick Reilly: A final bit of advice has to do with what happens after you
actually submit your work for publication. Submissions almost never get accepted
after the first set of reviews. Criticism, even when it is intended to be
constructive, can be difficult to accept. You should take your reviews seriously
and try to learn from them. Don't adopt a defensive or argumentative posture. It
will be much more productive to think carefully about what the editor and
reviewers have said and decide whether you want to try to revise your
submission. If you decide to revise, try to address the reviewer's concerns.
Also, tell the editors exactly what you have done to address their specific
points where you could. If you cannot address a specific concern, explain why
not. This will improve your chances of success.
Skip Dalessio: Once a manuscript is sent to a refereed journal, and is
accepted contingent upon revision, be willing to accept the input, and complete
the extra work to revise the article. I believe that this is a place, quite
often, where particularly valuable articles fall off track and never see the
light of day. Be willing to follow through and perhaps complete some additional
analyses, read some additional literature and integrate it into the paper, and
re-think conclusions and implications of the research.
There's a lot of good advice here from people who have managed to succeed for
many years as scientist-practitioners. On behalf of Personnel Psychology
(and, I suspect, editors at many other journals), we really encourage
practitioners to submit their work. You all have so much to offer. Please let us
know if we can help. For example, at the Scientist-Practitioner Forum of Personnel
Psychology, we welcome calls from potential authors to discuss their ideas.
In some instances, potential authors may want to submit an outline of a proposed
article. We (e.g., the editorial board) will be pleased to review the outline
and provide preliminary feedback. Potential authors may contact Jim Smither
(215-951-1797, amysean@aol.com) to discuss
ideas or submit an outline for preliminary review and feedback.
January 2000 Table of Contents | TIP Home | SIOP Home