Expression of Concern
Michael A. McDaniel
Virginia Commonwealth University
On December 8, 2005, the New England Journal of Medicine issued an editorial documenting the editors expression of concern over the deletion of data from a Vioxx study that resulted in an understatement of the risks of Vioxx for heart attacks.
Although inaccurate data reporting by I-O psychologists is unlikely to have life-altering consequences, there are serious repercussions for both employers and employees. For example, incomplete or inaccurate representation of findings by I-O psychologists can result in employers rejecting a good test and accepting a poor test for use in personnel selection. Such inappropriate reporting practices by I-O psychologists can result in some applicants being inappropriately rejected and others inappropriately accepted for employment. The effects of poor selection have been well documented to have financial consequences for organizations and psychological effects on applicants and employees.
How pervasive are inappropriate reporting practices among employment test vendors? I recently received an e-mail advertisement from a well-known employment test vendor. The e-mail indicated that the vendor had helped thousands of organizations to reduce turnover and increase productivity by effectively identifying quality candidates. The president told me (personal communication, November 1, 2005) that the company has not conducted a single criterion-related study on its testing products. In the absence of data, how does this organization know that that their tests reduce turnover and increase productivity? In the absence of their own data, I would be happy to see the validity assertions supported by validity generalization or validity transportability studies. However, the vendor does not make these arguments.
Another well-known employment test vendor states on their Web site that they have built their companys reputation on a scientific instrument for personality assessment that has been validated by more than 4 decades of research. I have written the company and asked for a technical manual on the test and they refuse to release it (personal communication, October 31, 2005). I have written the president of the company, a member of the American Psychological Association, and asked for the technical manual and he has not responded. If I had 4 decades of research on a test that I was selling, I would distribute the validity information.
Although many test vendors provide accurate representation of their validity data, some employment test vendors make claims unsupported by data, refuse to release alleged data, and inappropriately report the data that they have.
What is SIOPs role in enforcing its own Principles? The Principles for the Validation and Use of Selection Procedures
(SIOP, 1987, 2003) provides clear guidance on reporting validity data. In a discussion of the presentation of statistics, both the 1987 and 2003 editions of the
Principles states Tables should present complete data, not just significant or positive results (p. 52, 2003; p. 30, 1987). It is hoped that this expression of concern will encourage discussion among SIOP members and the SIOP leadership concerning the inappropriate behavior of some test vendors.
References
Bombardier, C., Laine, L., Reicin, A., Shapiro, D., Burgos-Vargas, R., Davis, B., et al. (2000, November 23). Comparison of upper gastrointestinal toxicity of Rofecoxib and Naproxen in patients with rheumatoid arthritis. New England Journal of Medicine,
343(21), 15201528.
Society for Industrial and Organizational Psychology. (1987).
Principles for the Validation and Use of Personnel Selection Procedures (3rd Ed.). College Park, MD: Author.
Society for Industrial and Organizational Psychology. (2003).
Principles for the Validation and Use of Personnel Selection Procedures (4th Ed.). Bowling Green, OH: Author.
April 2006 Table
of Contents | TIP Home
| SIOP Home