What is I-O?

Industrial-organizational (I-O) psychology is the scientific study of the workplace. Rigor and methods of psychology are applied to issues of critical relevance to business, including talent management, coaching, assessment, selection, training, organizational development, performance, and work-life balance.

The Real World: The Political Polling Process:
Surveys Run Amok?

Janine Waclawski
W. Warner Burke Associates, Inc.

Happy New Year! Welcome to the anti-penultimate year of the 20th century. Before I launch into my rant-du-jour, I just want to wish the readers a happy holiday season. Hopefully, you are relaxing somewhere, enjoying some time off and leisurely reading through TIP. For any of you who still engage in the age-old, self-delusional habit of making a New Year’s resolution, good luck! Personally, I gave ’em up years ago. I decided I don’t need any more self-imposed failures in my life, thank you very much.

It’s the End of the World as We Know It. But I Feel Fine

Not really, but I wanted to use that line from an R.E.M. song somewhere in this column. So sue me! But enough of my feckless ramblings and misguided holiday cheer, let’s get down to business. These days the nation’s interest in the nature of the relationship between our president Bill Clinton and former White House intern Monica Lewinsky has really put survey techniques (in the form of public opinion polls) in the limelight. It seems that it is nearly impossible to turn on the TV news without being shown the results of the latest poll regarding the public’s opinion of the president. For example, is he trustworthy, did he lie during his grand jury testimony, did he lie to the American people, is he more or less trustworthy than Ken Starr, should Hillary leave him, is he competent to hold office, and should we impeach him—just to name a few. In fact, the shear volume of questions being put to the public on this subject is mind-boggling (at least it boggles my feeble mind).

This, like a lot of things in life, puzzles me. Why is it that most of the time no one seems to care about my individual opinion on anything, but, as part of the collective, my opinion on Bill and Hillary’s marriage is a newsworthy item? Maybe it has something to do with me personally, but I tend to doubt it. The newsworthiness of such items is even more puzzling when you consider the way polls are conducted and reported. In fact, exceedingly small numbers of individuals are chosen to represent the entire populace of the United States, which is currently hovering somewhere around 250 million people and counting.

For example, according to many of the public opinion polls I’ve read recently, 1,006 seems to be the magic number. Specifically, many nationwide polls (which are discussed below) sample approximately 1,000 people to represent 250 million! This represents a sample size of about .0004%. This is astoundingly small. Nevertheless, in general, political polls (save the infamous predicted triumph of Tom Dewey over Harry Truman in the 1948 presidential election) are uncannily accurate. Can you imagine going to a client organization that consists of 250,000 employees and telling them you can accurately survey their company based on a sample of 1 person? Unless, unlike Bill Clinton, you recently did inhale, you probably can’t imagine it. Of course, as always I am exaggerating a bit to make my point. As we all know, there are established parameters for determining the appropriate sample size for a given population, but the fact remains that the samples used for opinion polls are much smaller than anything we use in I-O.

So by now you have probably sussed out my rant for this issue of TIP. Yes, you guessed it—surveys (of course this wouldn’t be too difficult as surveys are mentioned in the title)! Not that I have anything against surveys, in fact some of my best friends are survey practitioners (including myself). However, the recent rash of public opinion polls seems excessive even to me. Which, by the way, to my way of thinking is just another indication of our nation’s propensity to do everything to death. It seems that whenever we find something we like (e.g., TV, fast food, celebrities, exercise, etc.) we overindulge in it. How else could our country have one of the highest rates of obesity and anorexia at the same time? Let’s face it, we can be a bit extreme. It seems to me that as a group, we will beat a horse to death, beat the dead horse, and then take a poll about it! But as always, I digress.

Back to the topic at hand, recently while poking around on the internet I located an interesting site called "www.PollingReport.com important trends in public opinion." This site gives the user frequent updates on a myriad of polls conducted by the media to assess public opinion. News services and pollsters such as ABC News, CBS News, CNN, Gallup, NBC, Pew, and Zogby are all canvassed and posted on this site on a semi-daily basis. The topics on this site range from the political, with ratings of the Clinton scandals and investigations and political figures like Hillary Clinton, Monica Lewinsky, and Ken Starr, to economic issues such as ratings of consumer confidence and our country’s economic outlook, to opinions on pop culture with ratings on sporting events, celebrities, and Viagra. Some of these polls are intriguing; others are simply funny. On the intriguing side, this widespread solicitation of public opinion makes me wonder whether excessive polling like this trivializes important information. For example, are these polls sending messages that equate the importance of public opinion about the use of Viagra to that of impeaching the president? Probably not, but it’s a little disconcerting nevertheless. This also makes me wonder about all of the organizational surveys being conducted. Do the end users think they are being surveyed too much? Do the questions asked on organizational surveys aggrandize certain issues and trivialize others? For example, do they send messages that equate the importance of, let’s say, leadership practices and job security to that of office cleanliness? Maybe, maybe not. It’s something to think about anyway.

The Age of Paradox

I guess I have always had a love/hate relationship with surveys. One the one hand, they are informative, a great way to collect a lot of data in a short time period, fun and challenging to create and analyze, and allow me to help my clients change for the better. One the other hand, working on a survey requires that I use my brain. This can be difficult. Like most people, I often don’t like to use my brain. It’s hard work. Seriously, though, while surveys are relatively user-friendly for the recipient, they are often fraught with complexities for the survey practitioner. I have a lot of time to think about things like this. In my current job, I spend about one-third of my time on survey work (developing, analyzing, and presenting survey findings). I also teach a course that is focused in large part on organizational surveys and just recently co-authored my first book, Designing and Using Organizational Surveys, with Allan H. Church (1998). So, I guess you could say I’m a bit of a masochist (as well as a shameless self-promoter). Yes, life truly is an enigma to me. It’s so full of extremes. Although I would consider myself to be generally even-keeled in nature, certain things just get under my skin and drive me crazy. Surveys fall into this category.

This reminds me of one of my early I-O related job experiences. I was working as a personnel research intern analyzing survey data for a large high-tech corporation. I can clearly remember one of the first meetings I attended on the job. I was sitting in on an internal presentation of the company’s annual survey results. It was pretty standard fare except for one item in particular that I thought was a scream. The survey question was "Are we over surveying"? Although the question certainly is a valid one, it seemed very ironic to me at the time. Looking back on it, it seems less funny. What it highlights to me now is the tremendous amount of response burden that most people in organizations have to deal with. In addition to just doing their work, they have to participate in all sorts of activities based on survey methodology, such as performance appraisals, customer satisfaction surveys, upward appraisals, employee opinion surveys, and on and on. Of course the most recent addition to this list is multi-rater (a.k.a. 360-degree) feedback, which I affectionately call a "survey for one." So not only does the survey practitioner need to concern himself or herself with the content of the survey and its psychometric properties, but he or she also needs to be sensitive to other organizational initiatives that compete for the attention of the potential survey respondent.

So given these complexities, why are surveys so popular? Many factors have helped to secure and increase the acceptance and use of employee surveys. One example is the emergence of consortiums like the Mayflower Group, a cadre of blue-chip companies that exchange selected items from their annual employee survey data with one another. Just to give you a feeling for how much growth has occurred in organizational survey use in the past 20-odd years, the Mayflower Group was founded in 1971 with only 15 charter members. By 1996, membership was up to 42 companies (Johnson, 1996). For obvious reasons, elite groups such as these by their very existence serve to validate the importance of the survey effort. The logic being that if IBM, Xerox, and Merck are doing it, so should we. Another reason surveys are so popular is that they are relatively easy to administer and allow the surveyor to reach a large number of people quickly. Surveys can be conducted in person, by mail, by phone, by fax, and by e-mail, just to name a few. This makes them among other things one of the more user-friendly and pervasive methods for collecting data. Yet another reason surveys are so popular (and this is only my opinion) is because in general they are nonthreatening to the lay person and make the individual participating feel important. So while surveys may be tricky for the practitioner, when done right they are relatively simple and unobtrusive for the recipient. Yes, indeed, surveys are a very important and well-used instrument in the I-O psychologist’s toolkit.

Therefore, aside from my own existential angst about all of this, why should we as a field be concerned about the future of surveys? That fact is that surveys are a mainstay of I-O psychology and their use is on the rise. Recent studies indicate that more than half of U.S. companies are using employee surveys (Kraut, 1996). Moreover, some researchers contend this number is higher than 70% (Paul & Bracken, 1995). However, despite or perhaps because of their popularity, research also indicates that survey nonresponse may be on the rise as well (Rogelberg & Luong, 1998; Schwarz, Groves, & Schuman, 1998). So while surveys are being used more and more, people may be responding less and less. If this is indeed the case, this is not good. Therefore, I thought it might be important to think about how surveys are perceived by their end users (i.e., subjects and/or clients). Although public opinion polls differ in many ways from organizational surveys, they do employ the same method of inquiry. This makes me wonder if there is any crossover between the two. Specifically, do these polls and their seemingly incessant intrusion into our everyday lives impact the perceived validity and usefulness of organizational surveys? With this in mind I asked several survey practitioners the following questions:

1. As an I-O practitioner, what is your reaction to this application of survey methods?
2. Do you think the extensive use of public opinion polls by the media (vis--vis recent political events) downgrades, enhances, or has no impact on the general public’s view of the usefulness and credibility of survey methods? Why?
3. To what extent do public opinion polls shape rather than measure opinions (from deciding political futures to keeping issues that might otherwise fade in the forefront)?
4. Likewise, to what extent do organizational surveys shape rather than measure opinions?
5. What, if anything, can we learn about the conduct of surveys from these types of polling processes?

***

Subj:

Re: comments for tip column on organizational surveys

Date: 98–10–31 08:22:25 EST

From: edwards2@erols.com   (Jack Edwards)

To: j9151@aol.com

At their best, findings from political and consumer polls offer a productive means for gauging and documenting events and concerns. Such findings tell us about others and ourselves (e.g., how much we are similar or different). At their worst, findings from polls may be used to mislead us. Among other things, our reactions to poll findings are related to the salience of the information addressed in a poll. What is important for some may be trivial to others.

Your question about the relative importance of possible impeachment and Viagra can be used to illustrate this point. Apolitical people could view possible impeachment as unimportant because they don’t perceive that it affects their everyday lives. Others could consider this to be one of the most serious concerns in decades. The amount of Viagra sold suggests that its effectiveness is exceedingly important to a large number of individuals. Others might see Viagra as very important for a different reason—a question of whether or not insurance companies should pay for contraceptives if they pay for Viagra. Conversely, Viagra may be seen as a trivial issue for those who have not experienced the problems addressed by this drug or do not perceive a gender-based inequity in the products and services covered by medical insurance policies.

In sum, the expense of conducting a well-designed poll suggests that the issues covered in it are important to someone. Although the findings from particular polls may not be salient to us, others may view the findings as being of paramount importance. Our job as polling and survey professionals is to create even-handed questions, procedures, and analyses to address societal and/or organizational concerns fairly.

In general, I think the extensive use of polling is having a negative effect on the usefulness of such polls. One manifestation of this negative impact might be found in survey response rates. Kalton (1988) noted that "there has been considerable concern that the rate of total nonresponse has been increasing in recent years" (p. 116). More recently, achieving a representative sample has been made all the more difficult by the widespread use of answering machines to screen calls from pollsters. Increasing growth in the polling industry may result in killing the goose that laid the golden survey egg. Without representative samples, the usefulness and the credibility of survey findings come into question.

In conducting focus groups to pretest organizational surveys, very few of the hundreds of participants that I have interviewed have mentioned anything regarding public opinion polls. At the same time, some focus group participants have complained about getting too many surveys from the organization. It is probably a good time for organizations to step back and re-evaluate what information really needs to be gathered with surveys.

Public opinion polls are used to both measure and shape opinions. The number of polls being conducted, the groups commissioning the polls, the reasons for conducting polls, and other factors make it is impossible to estimate the percentage of polls conducted for one or the other purpose. I might conclude that a poll provides an objective measure of opinions, whereas others might label it as an obvious attempt to shape attitudes or behaviors. Often, the perceived purpose of a poll and its accuracy are in the eye of the beholder.

A minor change in question wording can shape a finding and consciously or unconsciously give an issue a positive or negative spin. Kagay and Elder (1992) described a survey that asked if the U.S. was spending too much, too little, or about the right amount of money on "assistance to the poor." About two thirds said that too little was being spent. When "welfare" was used in place of "assistance to the poor," about half the respondents said that too much was being spent on welfare.

I’d like to think that the primary purpose of organizational surveys is to provide accurate measures of members’ opinions, values, needs, and so forth. If this goal is accomplished and members can see positive actions tied to survey findings, the behaviors and opinions of organization members can be shaped in hopefully good ways. For example, members might come to believe that their opinions are important to the organization. In turn, this could result in more open communications. Other people may cite less benign uses of organizational surveys. An organization may use biased questions or present only those findings that support its views in an attempt to slant or shape members’ views of the workplace.

Public polling researchers (e.g., the American Association of Public Opinion Researchers—AAPOR) have produced a wealth of high-quality research. Some (e.g., sampling and question wording) offer I-O practitioners and researchers valuable insights for improving organizational surveys. At the same time, other polling practices might be very disruptive to an organization. For example, the polling practice of providing monetary rewards to increase survey response rates might establish unrealistic expectations if such rewards were used in an organization.

Jack Edwards

Defense Manpower Data Center

***

Subj: Re: Comments for TIP Column on Organizational Surveys

Date: 98–11–03 20:52:06 EST

From: sweiner@us.ibm.com   (Sara Weiner)

To: J9151@aol.com

Hi Janine, and thanks for inviting me to comment. I shared your questions with my colleagues and we had a good discussion at one of our I-O staff meetings!

Both public opinion polling and organizational surveying have become excessive in recent years. We’ve become very information hungry and numbers needy! However, surveys often lead to discussions in the workplace as well as in the public forum, which is a powerful and positive result.

Poor-quality polling certainly affects the perceived credibility of all polls and surveys. Sometimes questions are contrived and leading; sometimes sampling methods are not explained and therefore are open to skepticism (for example, without extensive explanations it certainly is difficult to conceive that 500 people could be representative of all voting-age adults in the U.S.). If more information were shared about sound methods, the credibility could be enhanced.

Many politicians have stated that they learn about the needs of their constituencies by reviewing polls. In this way, polls certainly contribute to shaping policy.

I am going to answer a slightly different question than you asked! There is often a debate in our field about whether to use surveys as communication vehicles. I don’t think there is any question that surveys CAN be used to communicate; however, they are certainly not as effective as other mediums (e.g., memos, policy statements) for communicating things like policy or cultural changes. The primary purpose of an organizational survey is to measure opinion, and due to the possibility of misinterpretation of intended messages they are NOT a reliable means for shaping opinion.

There are several things poor polling procedures have taught us in organizations. First, we need to distinguish surveys grounded in sound design from those that have questionable quality. Then we need to communicate clear and simple information about the reliability of the sample (credibility). Reducing the overall number of surveys so that important ones get the attention they deserve (higher response rates) is also required. We also need to monitor the quality of surveys going to employees so poorly designed questions and surveys do not taint the perception of all surveys (integrity). Finally, we must let employees know how the results are used (investment of employees’ time is then viewed as worth the effort).

Sara P. Weiner

Global Employee Research, IBM

Subj: Re: Comments for TIP Column on Organizational Surveys

Date: 98–10–30 15:52:54 EST

From: kkuhnert@arches.uga.edu   (Dr. Karl Kuhnert)

To: J9151@aol.com

Janine, thanks. Your questions were really provocative.

Although political opinion and organizational surveys share a common methodology, I see them as quite different in major respects. First, organizational surveys offer employees a chance to react to matters important to their daily work life. Issues of pay fairness, working conditions, and job satisfaction directly affect the lives of those completing an organizational survey. Opinion surveys, on the other hand, do not always ask questions that are relevant to the daily life of the public. In addition, an organizational survey has a defined purpose and specified population of respondents. It is rarely made clear in opinion surveys who completed the survey, the sampling technique used, the purpose, or how opinion data will be used.

The potential loss of credibility for organizational surveys is real. I am not sure when this happened, but political polling is now very politicized. A few years ago, while watching C-SPAN, I saw Republican and Democratic pollsters tearing into each other with results from their own polling services. We have known that the way a question is phrased determines how people will respond. What is new, however, is the fine art—if you can call it that—of phrasing questions and getting polling data to justify a political position. I guess if we can have advocate journalism, we can have advocate pollsters, too. The real danger to I-O psychologists is if employees and managers believe we can "spin" survey questions to get the answer we want. Such beliefs threaten our credibility along with confidence in the survey/feedback process.

The fact is public opinion polls are used to shape opinions. Organizations from the Broccoli Growers of North America to the NRA are actively engaged in influencing public perceptions. Opinion polls shape opinions because they are, for the most part, scientifically conducted and appear objective. In addition, poll results are graphically represented, which make them easily accessible and interpretable. For example, listen to the talk around the office water-cooler and see how many people use polling data to construct their arguments. This is all you need to know about how polling data shapes opinion.

Organizational surveys shape opinion in two fundamental ways. First, the survey dimensions and questions inform people what is important about their organization. The old maxim of "what gets measured is what gets done" applies here. Second, survey results shape opinions by confirming what organizational members think they already know about their company. When opinions are validated, organizations gain the confidence and commitment to try out new ideas and behaviors.

Without a plan to feedback and use data, organizational surveys run the risk of becoming a less meaningful, less relevant, less important organizational development tool. The biggest risk to organizational surveys is that they become opinion polls where people do not invest in their responses and do not know, or care much, how their input is used.

Karl Kuhnert

University of Georgia

***

So, what is the moral of this story? I guess I would have to quote that well-known philosopher Forest Gump (or rather his mother) and say "Stupid is as stupid does." If well-constructed and parsimoniously used, polls and surveys can provide us with a wealth of valuable information (as Jack said). However, if they are ill-conceived and overused, it is not likely that they will provide us with anything worth having. So much for trying to reduce my survey-related angst.

Off the record, on the QT and very hush, hush.

Well, not exactly, but I just wanted to use that line from L.A. Confidential somewhere in this column. So sue me! Anyway, following are some "on the record" survey results I want to share with you from www.PollingReport.com. As always, I’ll surprise and delight you with my running commentary (see some things never change, not even for New Year’s—not even if they should)!

Our first entry comes from a survey on customer loyalty. I figured I would start with the straightforward, I-O relevant example first and then work my way into the weird and poorly constructed ones later.

EXAMPLE 1 Source: Shell Oil Company Shell Poll conducted by Peter D. Hart Research Associates July 17–20 1998 N=1,123 adults nationwide

"In general, how loyal do you think companies are to their employees today? And, in general, how loyal do you think employees are to their companies today?"

References