Reactions to Ricci
by Clif Boutelle and Stephany Schings
SIOP Members Discuss How the Recent Supreme Court Decision Will Affect Testing, Selection, and the I-O Profession
With its recent narrow 5-4 decision in the Ricci v. DeStefano case, the Supreme Court ruled that results of a valid, job-related test cannot be thrown out simply because the test may result in adverse impact. The decision has underscored the importance of valid, job-related tests, and for many I-O psychologists, it has reaffirmed their status as central and vital players in the testing process.
The case revolved around employment testing and selection practices at a municipal fire department in New Haven, Connecticut. Seventeen incumbent White firefighters and one Hispanic firefighter who scored high enough on a test battery to be promoted filed a lawsuit against the city of New Haven when the city refused to certify the overall results after finding that no incumbent African-American firefighters scored high enough on the exam to be promoted. The city argued that it was within its rights to protect itself from an adverse impact lawsuit while the plaintiffs argued the city was guilty of disparate treatment of the firefighters based on race. The city provided no evidence that the test was invalid or unrelated to the firefighter position. The Court sided with the plaintiffs, stating a city needs a “strong basis in evidence” that it would lose an adverse impact case to reject a promotional examination on these grounds.
First and foremost, said SIOP Member Arthur Gutman, we must recognize that this was a disparate treatment case, not an adverse impact case (despite claims by the City). The Court ruled that New Haven had an illegal motive for not certifying the test; they didn't like the results, said Gutman, a psychology professor at Florida Institute of Technology and an authority of evaluating personnel selection systems.
SIOP members have differed in reactions to the Court’s decision, but they all agreed that the ruling would have a strong impact on employee selection, highlighting the need for I-O psychologists to be involved in every stage of the selection process.
SIOP members give their opinions on the case below, but you can share your own opinion right now on the SIOP Exchange!
“In general, this is a very good decision based on the facts of the case,” said SIOP Fellow Gerald Barrett, thepresident of consulting firm Barrett & Associates, Inc. who has designed selection, testing, and promotional processes for more than 30 years for individual companies, the civil service, and the safety forces. “It sets the boundaries for when you can and cannot certify an exam. In the past, it wasn’t clear what a city should do.”
Catherine Cline, a New York City-based independent consultant who has designed and administered numerous testing and selection systems, mostly in the public sector, recently designed a promotional test for New Haven for fire investigators. She said she has been following the case very closely because of her interest in the city’s testing as well as for what the decision means to I-Os.
“I have been following this decision because it is a problem cities do face: What do they do with a test when they believe they have adverse impact?” she said. “I’ve seen tests thrown out before when I know they have been done with a good job analysis, when the city officials don’t like the results, or they don’t think it is politically good for them.”
David Arnold, general counsel at Wonderlic, Inc., which specializes in employee recruiting and selection solutions, said the decision now tells employers that they can’t simply throw the results of a test or any other hiring procedure out merely because of the existence of disparate impact.
“You can’t abandon the results, as New Haven attempted to do, because the numbers don’t come out the way you want them to,” he said.
In addition to the decision, the message delivered by the Court goes further, Gutman said. He said the decision stresses that employers, especially municipalities, need to ensure they are using tests, starting with a job analysis and covering the entire job domain, that can be verified as to the critical skills needed to perform the job.
“From now on it will be difficult for an organization to discard test scores, as the city of New Haven did in this case, in absence of solid evidence supporting the test’s outcome,” said SIOP Fellow Herman Aguinis, professor of organizational behavior and human resources in the Kelley School of Business at Indiana University. “The Court’s decision reinforces the view that test validation is important, both practically and legally speaking.”
New Haven was put in the unusual position of arguing that after the results of the test it administered to fire fighters were shown to be racially disparate, the selection process itself was invalid and that therefore the results of that process should not be used to make promotion decisions, said SIOP Fellow Wayne Cascio, the Robert H. Reynolds Chair in Global Leadership at the University of Colorado Denver, who has testified extensively on firefighter promotion issues. “It’s unusual because the city did not defend its own test,” he said.
Cascio and Aguinis, along with SIOP Fellows Irwin Goldstein, James Outtz, and Sheldon Zedeck, submitted an amicus curiae brief in support of the city of New Haven in the case, identifying flaws in the tests that undermined their validity. However, the Court ruled that no evidence was presented to show the test was invalid or that there were alternatives that could have been used, Cascio pointed out.
“Employers who use hiring and promotional tests, and many of them do, need to be concerned with disparate impact,” Arnold said. “They need to administer a test that is valid and minimizes adverse impact. A prudent employer will ensure the test is valid and focuses on job skills and business necessities.”
Barrett said employers also need to understand that minimizing bias and creating valid tests doesn’t ensure there won’t be adverse impact and that you need to develop the most valid test possible.
“Most tests do have adverse impact,” he said. “You can’t predict ahead of time what will have adverse impact and what will not. What you do is first, it’s a very simple process, you develop for the city the best selection procedure you can, one that’s valid under professional standards, and determine what you can do to reduce any adverse impact before you give the test. But until you give the test you don’t know if there is going to be any adverse impact because you can’t control how hard people study.”
Barrett said cities always have to expect litigation. “I’ve been in the safety forces field since 1973, and some cities have said no matter what you do there is going to be a lawsuit,” he said.
Arnold said employers can defend themselves by following established practices.
“Ultimately good business and employment practices are defensible and can avoid litigation or prevail in litigation,” he said.
How does an employer know if a test is going to meet validity requirements?
“You have to look at past cases and rely on professional judgment,” Gutman said. “That’s where industrial and organizational psychologists can be of immense help to employers because of their vast knowledge in selection procedures that will stand up to challenges.”
Cascio noted that I-O psychologists adhere to rigid standards in developing tests and follow the Society for Industrial and Organizational Psychology’s Principles for the Validation and Use of Personnel Selection Procedures, which demands accountability by test developers and administrators to provide strong evidence supporting the claims they make about a test.
“I-O psychology is the only profession that specializes in the creation and validation of employment and professional tests,” he said, adding that the 1978 Uniform Guidelines on Employee Testing Procedures, issued by the federal government, encourages organizations to utilize the abilities and experience of professionals like I-O psychologists to develop and administer tests. “Organizations need the knowledge and credibility of I-O practitioners to defend their hiring and promotion procedures.”
The Supreme Court acknowledged the need for qualified professionals for this purpose, Barrett added.
“Right now there is going to be a strong push to ask, ‘Is your test valid and up to professional standards?’ and those people are I-O psychologists who do these types of things,” he said. “I think the Supreme Court understood that there is a process we go through in using testing. This decision reaffirms that I-Os should be the ones to design and administer tests.”
This is especially true for municipalities, which are “lightning rods for litigation,” Arnold said.
Thurston Cosner, a Cleveland-based police and fire safety psychologist who has worked with small to midsized municipalities and villages for more than 30 years, noted that many of the tests administered for hiring and promotion purposes stem from civil service exams, which, he says, need updating.
“It’s a process that needs revising. Most are multiple choice written tests and physical agility evaluations and have very little predictive validity for predicting job performance,” he said. But there’s more to basing a promotional decision than on a test or assessment center.
“Past performance in the public service departments must count for something, otherwise individuals who perform at high levels will see their past behavior eliminated from the promotional process, which could have a demoralizing effect upon them,” Cosner said.
He added that departments should take a more active and useful role in developing, with consultation from testing professionals, more valid and effective performance evaluation systems that will become part of the promotional process.
Cosner also pointed out it is important for departments to receive feedback on how well they are doing with eliminating disparate and biased treatment of individuals in their development programs.
“When we find evidence of adverse or disparate impact, it is likely that the results accurately mirror the factors that are operating in the department to create such an effect,” he said. He feels SIOP could perform a valuable service to municipalities by introducing testing models into their selection systems.
Despite the Supreme Court decision, it is likely the issue will be revisited, Aguinis said.
“This was a highly divisive vote (5-4), and in her dissenting comments Justice Ruth Bader Ginsberg said the majority ruling will not have staying power,” he said. “We haven’t heard the last of this issue.”
In the dissent, Ginsberg said context must be taken into account in promotion and hiring tests reflecting the stakeholders, which in the New Haven case, would include city residents, most of whom are African-American.
“This implies a dual role which says that the sole emphasis should not be only on the validity of the test but should also include the broader societal impact,” said Aguinis, whose research was cited in this dissentas well as that of Cascio, SIOP Member Deborah Rupp, and SIOP Fellow George Thornton. “The Court majority focused on validity of the New Haven test while the dissenting opinion said that tests need to take the broader context and implications of hiring and promotion decisions into account, which clearly leaves the door open for further discussion.”
Cascio agrees that the Court’s ruling leaves some unsettled areas of testing.
“There’s a gap between what many I-O psychologists believe to be sound testing standards and what the Court has ruled,” he said. For example, Cascio said there were other aspects of a candidate’s job readiness that should be measured, which the New Haven test failed to do, such as measuring how a candidate handles an emergency and whether the candidate possesses command presence.
Barrett agreed that while the current best evidence is that the written job test had the highest validity in job performance, other selection procedures that mimic actual job situations could be implemented. One such method is a situational judgment tests where an incident is outlined and the candidate is asked how she or he would respond. An assessment center is another method where the candidate engages in a simulation that requires on-the-spot judgments about how he or she would handle emergency circumstances, though these centers can cost ten times more than written tests, Barrett said.
Also, a procedure called “banding” could be used where instead of giving the promotion to the top scorer on a test, several candidates whose scores are in a close range are put into a band or group, Cascio said. Other business-related selection criteria are used to select the successful candidate, including experience, past job performance, training, and licensure to break the ties. Thus, in settings like New Haven, the use of banding procedures as developed by I-O psychologists could have resulted in valid testing called for by the Court majority but also taking into account the “context” as preferred by the Court minority.
However, Barrett said he believes the Court decision would prohibit banding.
“It sounds like to me that the Supreme Court, in their decision, is extending their reasoning to the banding process so that will not be an alternative selection procedure for cities,” he said.
Arnold said the court’s decision left banding as an option but not after the fact when it is motivated by a racial disparate impact, as in this particular case.
“Here, the court said banding was not a valid alternative for this reason,” he said, quoting the court’s decision: “Had the City reviewed the exam results and then adopted banding to make the minority test scores appear higher, it would have violated Title VII’s prohibition of adjusting test results on the basis of race.”
Whether a written test, interviews, assessment centers, or numerous other tests are used, SIOP members involved in testing say the test needs to measure necessities for the job.
“We (I-O psychologists) should be advocating tests that meet the standards of validity and that tap into all of the mission-critical aspects of the job,” Cascio said. “We need to measure more about these candidates.”
At any rate, SIOP members say, the Court decision in this case confirms that employers should strive to use valid assessment procedures that measure candidates’ fitness for a position or promotion.
“This decision puts more emphasis on good testing,” Cline said. “Which means, I guess, this will bring in more work for I-Os.”