What is I-O?

Industrial-organizational (I-O) psychology is the scientific study of the workplace. Rigor and methods of psychology are applied to issues of critical relevance to business, including talent management, coaching, assessment, selection, training, organizational development, performance, and work-life balance.

Informed Decisions: Research Based Practice Notes 360 Degree Feedback: Methodological Advice from Multiple Sources

Matthew E. Sederburg and Steven G. Rogelberg

Bowling Green State University

Implementing an effective 360 feedback program can be a complicated process. Many methodological issues (e.g., how many raters are needed from each source, how to interpret discrepant data between sources, etc.) must be considered. Further, the manner in which you deal with such issues can have a profound impact on the quality of your intervention. Much basic research has been done on 360 feedback (such as self–other comparisons), but unfortunately little or none has examined the optimal means of implementing and carrying out a successful 360 program. From a methodological standpoint, practice has been far ahead of research in this area.

Given the current state of affairs, a review of the existing literature seems ill suited to finding answers to the types of questions we wish to address in this column. The "state of the art" in 360 feedback lies with those who practice it regularly and in the knowledge they have gained from years of experience. In order to meet our goal of providing the best advice possible, we have interviewed four seasoned practitioners in the area, and present their responses to our questions in this paper. We’d like to thank Allan H. Church, Manny London, Walt Tornow, and Fran Yammarino for their participation in this project, and hope that their words of wisdom enable you to design better 360 programs in the future.

London: "One is getting people involved in the process right from the start. Then you focus on the survey instrument itself—getting them involved in determining what the items should be, writing the items, making sure they understand what is being measured. 360 is a vehicle for communicating to people what is important, and so you want to be sure that the performance dimensions being measured are ones that you buy into as a corporation or as an executive. Facilitation of the results is important as well, and that might take the form of workshops where people get the results and you help them work through it individually. Remember though, it is not just a matter of them getting the results, but encouraging people to share the results with others to make it a more commonplace experience to talk about performance management issues within the organization. Finally, there is follow-up. You look at whether people have improved, have they accomplished the goals they have set for themselves, has their behavior changed, and has their performance improved over time."

Tornow: (a) Consider the power of purpose—why the organization wants to use a 360 feedback program. For example, is the feedback going to be used for performance appraisal or development? Treat development as a system and look at 360 as one part of that overall system and as a process, not an event. (b) Be longitudinal in perspective. It takes time to develop. Emphasizing follow-up can also create impact. (c) Link individual with organization-level change—how organizational change gets translated at the individual level, and how individual-level change plans get integrated into organizational business plans. (d) Focus on both raters and ratees—involve both in the design and use of the 360 system.

Church: "One key is to make sure that you’ve determined the desired outcome of the 360 process, paying particular attention to how the feedback system is going to be linked to existing and ongoing objectives in the organization. If an organization decides to do 360 because it’s a fad—without knowing what’s going to happen to the data, who is going to have access, and how it’s going to be implemented—it’s probably going to flop. Another key is ensuring the confidentiality of the process. The third is making sure the instrumentation is solid. It doesn’t necessarily have to be developed based on a job analysis or even similarly rigorous research methods, but it should at least makes sense to the user. At a bare minimum the instrument should not break the cardinal rules of item writing (e.g., no quadruple barreled items, no mismatched scales, and no jargony or culturally biased items). Also, the items need to be linked to some organizational concept that is meaningful to people. Finally, the delivery of the feedback itself has to be in some kind of system that is effective and makes an impact. It should not be just a ‘deskdrop.’ It is critical to provide the recipient with opportunities for focused attention on the results. Specifically, feedback can be most effective when it is delivered in the context of a developmental program with professionals on hand who can provide interpretation and coaching. Ideally, these coaches should be people who can make on ongoing commitment to work with the feedback recipients to help them work on the things they are trying to change, and to provide them with resources to do so. Sometimes, the relationship can be almost like therapy. You need to have a more extended connection to the recipient than just a single discussion."

Yammarino: "The number one key is you’ve really got to have top management’s commitment to do this. It can’t be just window dressing; they really have to be behind it. The second key is that the people who are going to be involved in the program, both raters as well as the people receiving the feedback, have to understand that this is being done for developmental purposes rather than performance appraisal. The answers can get kind of weird from some sources when people know that it’s linked to performance evaluations or selection. So, I think if you get top management commitment to the process, and use it for developmental purposes, these can really go a long way to making the initiative successful."

Church: "When the data are not the same, you have to get the individual to think about his/her current situation, how he/she relates to different groups of people, and what might be driving the differential set of ratings. It’s really an issue of looking at the specific items, and asking the individual to explore possibilities as to why the ratings differ among groups. But you have to pull it out of the person—you can’t second guess the whole thing. It can be useful to get the individual to think of recent, specific behaviors or interactions that may have contributed to these ratings. Sometimes as a coach you can observe behaviors that your client cannot see in him or herself. Getting your client to discover these tendencies through his or her own feedback and experiences will be more effective and valid than making pronouncements about what you have observed. Again, it can be a little like therapy; you have to get that person to tell you what the problem is."

Yammarino: "The rating sources don’t always see the focal individual in the same way. And my own personal view is that that’s ok. These are different people, they have different perspectives on that individual and the role he or she plays, and that’s all valid information. So the last thing you want to do is combine rating sources, or average across them. What you need to do is feed back to the focal individual what the different sources say. And you present them as independent, unique pieces of information."

Tornow: "One of the key learnings about 360 feedback is that there are multiple realities, or perspectives, that different people have. Therefore, differences are to be valued and understood in order to do a better job in relating to the different constituencies."

London: "This is part of the process; it is not necessarily the case that these differences are an indication of unreliability. It could indeed be that there are distinctly different perspectives. You need to go through the data and see what it means. It may depend on the questions asked. It might depend on the situation that the manager is in. The manager may be behaving differently with different constituent groups. So those are the things that need to be processed as the person gets feedback, and the recipient needs to be encouraged to think about the meaning of those differences and try to understand them. He/she needs to determine if they are a valid indication of actual performance, and then incorporate that into the meaning of the results and the goals that are set."

Yammarino: "That’s a tougher issue. The standard practice has been to average the ratings. The assumption that you’re making in this case is either one of two things: the raters are all seeing the individual the same, or the variability in their ratings is really error. The reality is that neither one of those assumptions may be correct. What we do is report frequency counts, or category responses on the feedback forms. In this way individuals can see the dispersion of their scores. One of the interesting things we’ve found is that you get these splits. You might have a couple of people clustered on the high end of the response format, and a couple on the low end. People can then start to get the sense that even within this group, people see me somewhat differently. And that’s useful information."

London: "There is probably some general thinking that within sources at least there ought to be agreement, but that is not necessarily the case. A person can behave differently with different subordinates, and the subordinates will have different views. What all this says is that processing of 360 is a difficult cognitive and emotional process, and one that needs to be gone through very carefully. You really need to have a broad understanding as a recipient as to how the results were derived and to be able to think about what they mean. So you have to look at the items within each dimension, and understand that there may be disagreement there too because the internal consistency in the dimension might not be that strong. People need to be educated recipients of feedback. In particular they need to be educated as to how to process it cognitively."

Church: "Most of our feedback systems don’t focus on this aspect. It’s our belief that we shouldn’t provide dispersion information on direct reports or peers as this will likely jeopardize the confidentiality of the data. It’s particularly dangerous with direct report data because we don’t want somebody assuming that the lowest ratings are all from a single particular person they can identify. Having said that, we have provided this information for external client ratings, as the nature of the power in this relationship is different. In a client scenario, if you see discrepancies, whether it’s within or between, you have to ask why this is happening."

Tornow: "Data should be displayed so that it is easy for the ratee to quickly see how different rater sources see him or her. These should also be compared with self-ratings. Norms should be presented as well."

Church: "We use a combination of graphics and text. Typically this consists of bar charts and tables. And it’s basically displayed on a summary level, across the key competencies of the organization. Usually, we provide an overall summary score followed by individual item means. The feedback is always displayed separately for each constituency (i.e., self- and each of the various rating perspectives). Alongside that, we typically provide importance ratings and some type of norm, which is usually the average for everybody in that organization."

London: "Reports vary across the board. I’ve seen some very complex ones and then I have seen some very simple ones. Generally, something in between is best. But it is certainly something to be tested within the organization, and it depends on how you are going to provide the feedback, and how much info and help you are going to be giving to people in using it. If you are not going to be giving them much help, it had better be self-explanatory and easily useable. I also think it is good to have norms presented so they can see how other managers at their level in the organization did, so they can compare themselves with others. I also like to present some measure of dispersion, such as a range. Usually the mean is presented, by source. It is important not to aggregate it across sources, but to report each source separately. The norm might be presented as the average of all other ratings across sources, or it might be presented for each source separately; it probably should be for each source separately. It is a difficult issue and there is no one right way to do it, but presenting something that is understandable and where people can get help in understanding it is really necessary."

Yammarino: "We’ll give the self scores, the average of the other scores by source, then we’ll give a breakout or frequency count of the other scores, also by source. What we don’t do is report things in terms of variances or standard deviations, or some of these other indicators, because the reality is that the managers don’t understand it or don’t care about it. You want to compute those things for yourself, but you don’t necessarily want to report that information to the focal individuals who are getting the feedback.

London: "Often we have asked for open-ended measures or responses at the end of the survey. That information is generally transcribed so that identifying material is taken out and those verbatim responses are presented as part of the feedback report."

Yammarino: "What we are doing is asking a general, open-ended question at the end. We ask a question such as "What are the three best things that this person does relevant to the survey items, what are the three worst things, and is there anything else that you want to tell us about this individual?" When we report back this information it is verbatim, unedited. We literally pipe from the surveys onto the feedback form. And all we do is identify it by source. We scramble the order, so that they can’t tell that good comment #2 comes from the same person as bad comment #2."

Tornow: "Yes, open-ended comments can be very useful in providing some ‘flesh’ to the quantitative data."

Church: "Yes, but they are more difficult to keep clean. What I mean is, you have to make sure they’re carefully content coded and analyzed, and cleaned of all identifying information. As an aside, we prefer not to use verbatim comments (without content coding) even if cleaned for the same reasons. Actually, we’ve found that managers and executives love write-ins. It gives them some additional context that they wouldn’t otherwise have. We’ve done various 360 programs with and without write-in comments, and it seems that participants get much more out of the standard numeric feedback when write-ins are included in their reports. It also seems easier for them to create developmental action plans for the future. But not everyone offers their comments on the rating forms, so this type of data can be more skewed in one direction or another."

Church: "Typically, we let the participants choose who will rate them because the participants are the ones who will either use or not use the data they receive. While they may not get everything back from everyone using this method, and they won’t know which individuals responded, at least participants will know the pool of people they’ve picked. We recommend strongly that they choose people of different types in order to get wider representation. If you do it the other way (i.e., someone else selects the raters), participants won’t necessarily know who is in the pool at all—it could be any of a number of people. This can cause credibility and validity issues with the data. They don’t know why there might be discrepancies among the sources because they didn’t pick the raters. You’re left with data without a tag. Of course, you certainly do run the risk of individuals picking out raters who like them the best, but if you can establish the credibility of the process and its confidentiality in the larger system, we’ve found that you tend to get pretty honest ratings anyway."

Tornow: "Raters should be selected on the basis of representing the key constituencies defining job success, and being knowledgeable about the ratee’s performance."

Yammarino: "We’ve done it a variety of ways—we’ve had the focal person pick, we’ve had HR departments pick, we’ve done random selection kinds of things, and we’ve also done this thing where we get everybody that we can who’s in that source group. My experience is that it’s the latter two that work best, either some kind of random selection of people or everybody. With the other methods, people pick raters for a variety of reasons, and not necessarily to give the most accurate and comprehensive feedback. And so we’ve not put a lot of faith in data coming from those types of selection systems."

London: "I think that having the focal individuals choose the raters makes most sense, particularly initially when you are introducing the process. Because once again, it reduces defensiveness and gives them a chance to give it to people who they value, people whose input they care about. As a result they will pay more attention to the results than they would otherwise. So I think if you want to enhance that credibility and attention to the results, you can have the ratees give it out to people they have confidence in."

Yammarino: "My feeling is that you want to get all the people, or a reasonable random sample of the people, in that source. The other rule of thumb we use is that if we get fewer than 3 or 4 responses from any particular source, we won’t present the breakout data. We might even be reluctant to report mean data, because at that point the data can be skewed or colored in some way. The anonymity of the rater also goes out the window. If you’re going to do a 360, and you can’t identify a reasonable customer pool, you might want to consider not presenting data from that source. Likewise, if a person only has one direct report, you don’t want to report that data because you can no longer protect anonymity. In the end we use reasonable judgment. You want to be able to get accurate information and protect the anonymity of sources, but you don’t want to force the issue just to have a complete set of 360 ratings."

London: "I think I have seen three, four, and five used in various companies. Four is probably a good number. Three is too small, I suppose, particularly if measures of disbursement are used, if you’re presenting the frequency of responses per item. Four is probably a reasonable number to protect the anonymity of the rater, as well as getting a fairly reliable data point."

Church: "At least three—three peers, three direct reports, and three clients. This can often be a problem. The worst case is you don’t give them the data from that group. We do lots of follow-up, and try to get as much data as possible, but if you don’t have enough responses, you simply can’t give them the data. One client of ours has actually forced us to collapse ratings together on peers and direct reports when one of them is incomplete, but we typically don’t advocate that because then you’re mixing up perspectives, making interpretation even more difficult. On the flip side we have had clients request as many as 15 direct reports and peers each to assess them. This can also be problematic though because of central tendency effects."

Tornow: "A minimum of three, preferably five, to assure anonymity."

London: "Response rates have been pretty good in the programs that I am familiar with, 40–50–60%, some as high as 90%. So you do the normal things by explaining what this process is for, encouraging people to participate, and follow up if necessary. Increasingly, 360 is being administered electronically, and that makes it easier for people to respond."

Church: "In our work, we have found that 360 rater response rates are not nearly as problematic as survey response rates. In fact, it’s not really a problem at all. Nevertheless, we do lots of follow-up, stressing confidentiality. Basically it’s by telephone, fax and/or e-mail. Some organizations are culturally very good at responding, and some are consistently poor. The general level of response depends on a few factors, such as (a) the developmental orientation of the organization, (b) how often the organization does other types of data collection efforts (i.e., people can get simply sick of responding), and (c) whether or not raters are comfortable with the process."

Yammarino: "We’ve had tremendous luck with response rates, even with customers when they have been included in the process. If you get top management’s commitment, and people know that the feedback is for developmental purposes, people seem much more willing to participate and respond. Even before the 360 process begins it is important to get the word out that top management values and endorses the feedback process. Then, you need to let people know that the 360 process is coming. When the feedback materials arrive, they should include a cover letter that describes the feedback process in detail. This letter should lay out the purpose and importance of the 360 process, as well as specific instructions for completing the questionnaires. Moreover, the letter should firmly state top management’s commitment to the process and reconfirm that the feedback will be used solely for development. Finally, appeals, sponsorship and support are also important to include in the cover letter. When it comes time to follow up, postcards, memos, and phone calls are all effective techniques."

Though our experts did not agree on every issue discussed here, several common themes seemed to emerge. Commitment and buy-in from top management and the participants in the 360 program is crucial. The decision to make use of the feedback ultimately lies with the recipient, so gaining his/her confidence and establishing credibility is vital to the success of the project. The rating form itself should be designed to reflect organizational goals and relevant performance dimensions in order to communicate what the organization holds important. Furthermore, the use of open-ended items in the form can be useful, as they provide valuable information that would not otherwise be available. When selecting raters, choosing people the feedback recipient views as credible and who can accurately report on the relevant performance dimensions are the most important qualities to look for. Concerning interpretation, each source, as well as each rater, has a story to tell. Effective coaches, in turn, facilitate the process whereby the participant uncovers and understands the underlying themes and messages from each data source. Although aggregating data within a particular source is necessary to preserve anonymity, between-source information should be kept separate to reflect the different perspectives accurately. Also, remember that when you have fewer than 3 or 4 raters within a given source (excluding supervisor and self-ratings), it is probably best not to present that data to the recipient. Finally, 360 programs should promote a process orientation to development rather than an event orientation. Specifically, a long term developmental perspective should be encouraged, with participants working interactively with their coaches during and after the 360 program occurs.

In conclusion, 360 feedback is a potentially powerful intervention that can positively impact individual and organizational effectiveness if educated and careful methodological decisions are made and implemented. We hope that the information contained in this article will help you in your quest to provide high quality services to organizations. Although each organization is in a sense unique, and operates in its own contextual environment, these important issues identified by our experts will certainly present some relevance to your efforts. Again, we would like to thank Allan Church, Manny London, Walt Tornow, and Fran Yammarino for their time and great insights. Feel free to e-mail us at sederbu@bgnet.bgsu.edu if you have any questions or comments about this article, or about 360 feedback in general.

TIP

Vol. 36/No. 2  October, 1998

Return to Table of Contents