Fax-Back Scoring: A Cautionary Note1
1 This study was sponsored by the Valspar Corporation Human Resources Department. Valspar is a Minnesota-based global manufacturer of paints, resins, and coatings.
Vernon A. Peterson
Consulting Psychologists, Inc.
Introduction
Test publishers have tried a variety of methods to distribute and administer tests in order to protect their intellectual property and to preserve the revenue streams that testing can produce. The oldest and most common method of test distribution and administration relies on a hard copy of the test booklet, scoring template, and answer sheet that clients purchase on an as-needed basis. This method is relatively inexpensive and is well received by clients, in that it allows testing to occur in a variety of environments. It does little, however, to protect intellectual property and the revenue streams to which test publishers are entitled. The Production Personnel Test (PPT) developed by Consulting Psychologists, Inc. is a good example.
The PPT is distributed throughout the United States, Canada, and some European countries in hard copy form. The test booklets and scoring keys are reusable and one-time-use answer sheets are purchased by the client on an as-needed basis. While the sale of test booklets and scoring keys generates some revenue, the cost of development and its maintenance are primarily covered by the on-going sale of one-time-use answer sheets, which if photocopied deny the test publisher revenue.
The wholesale disregard of U.S. and International Copyright laws has led some test publishers to distribute tests electronically with encrypted scoring algorithms, which are not easily duplicated. This method protects the intellectual property rights of the test publisher as well as the revenue streams to which the test publisher is entitled. Unfortunately many clients do not find such tests convenient.
While software-based or Web-based test distribution and administration of tests is commonplace, the client organization needs to have a suitable number of PCs available to test multiple applicants simultaneously, or it must be willing to pay someone to manually enter applicant responses after the fact. So what is the alternative?
Psychological Services Inc. (PSI), a premier test publisher, has refined fax-back scoring to an art form. While the client organization purchases test booklets and answer sheets on an as-needed basis, the already inexpensive purchase price includes the cost of scoring. In the case of the Viewpoint, a personality test, applicants are given an inexpensive but reusable test booklet as well as an answer sheet onto which they record their responses. Once complete, the HR professional enters a client code and fax number and then faxes the answer sheet to a predetermined test-scoring center that is carefully managed by the test publisher. The HR professional receives a score report from the test-scoring center via fax. The beauty of this approach is substantial.
First, because a hard copy of the test is used, testing can be completed in environments that do not require PCs. Second, HR professionals do not need to use a scoring template nor do they need to manually enter applicant responses onto a scoring disk. Best of all, the client organization does not need proprietary software nor must it carefully monitor site and/or scoring codes. In sum, fax-back scoring could be among the most convenient and cost effective methods of distributing and administering testing. However, it does rely on some simple but important assumptions.
First, the client organization must have at hand a fax machine to transmit answer sheets and/or receive score reports. Second, test administrators must carefully review each answer sheet before transmitting to make sure that there are no stray marks, unclean erasures, or improperly darkened responses. Third, test administrators must assume that the score reports that they receive are produced reliably. That is, they must assume that they would receive the same score report upon rescoring. But is this the case?
Research Design
To answer this and related questions, Consulting Psychologists, Inc. (CPI) conducted a study in which 42 HR professionals were randomly assigned to two treatment conditions. Each member of the control group was asked to fax two Viewpoint answer sheets (prepared by applicants) to the test publisher for scoring, after having been instructed to review each answer sheet prior to transmission for stray marks, unclean erasures, and improperly darkened responses, using only a #2 lead pencil. Each member of the experimental group was given two Viewpoint answer sheets completed by CPI. One of the answer sheets was prepared using a #2 lead pencil; the other was completed using a Sense-a-Mark pencilpurported to increase machine-scoring accuracy. All of the HR professionals who participated in this study were provided with a detailed description of the steps that they needed to take to ensure accurate scoring. In addition, all participants had prior experience using the fax-back scoring process. Finally, all participants had received feedback concerning the accuracy of the score reports and the factors at play, prior to participating in this study.
The score reports received at each location were subsequently compared to the score reports that CPI received when it faxed the two Viewpoint answer sheets to the test publisher for scoring.
Results
When CPI completed the Viewpoint answer sheets, in a manner that was fully consistent with the guidelines provided by Psychological Services, Inc., 100% of the score reports faxed back to participating locations matched in every detail the score reports faxed back to CPI
(r = 1.00). When applicants completed the answer sheets, which were subsequently reviewed by HR professionals prior to transmission, the score reports faxed back to participating locations matched the score reports faxed back to CPI only 32% of the time.
While in many cases, score-report differences were quite small and unlikely to negatively impact hiring decisions, more than half of the discrepancies were significant enough to be of concern. In one case the score report prepared by the applicant and reviewed by an HR professional produced a total score of 2 when in fact the actual score should have been 64, a 62-point difference that precluded the applicant from further consideration.
While this study did not assess the error rate when hand-scoring templates were used, prior studies conducted by CPI have found hand-scoring errors to be almost as prevalent as the fax-back scoring errors reported in this study. Hand-scoring errors, however, have almost without exception been limited to 1 or 2 pointsdifferences that in most cases have little or no impact on the hiring decision.
Conclusions
This simple but important study demonstrated that the fax-back scoring process can be extremely reliable when either a #2 lead pencil or a Sense-a-Mark pencil is used AND when the answer sheets are carefully prepared to ensure that:
- There are no stray marks
- All erasureswhen they occurare clean
- Selected response options are carefully and fully darkened as suggested by the test publisher
This study also demonstrated that the fax-back scoring process is robust even when old and in some cases poorly maintained fax machines are used. As a practical matter, however, fax-back scoring error rates tend to be relatively high and their impact is potentially damaging, given that scoring errors are often large enough to negatively
impact hiring decisions.
While it is true that the results of this study could argue more for the failure of the process used to train the HR professionals who participated in this study than the fax-back scoring process itself, the conditions under which this study was conducted are, at the very least, representative of the conditions under which fax-back scoring is commonly applied.
Recommendations
We strongly suggest that test administrators carefully coach applicants to avoid making stray marks and to ensure that they make clean erasures and fully darken responses in light of the fact that their answer sheets will be machine scored. Further, we recommend that test administrators take the time to carefully review answer sheets before transmission and to clean erasures and/or redarken applicant-selected response options, when needed. We also recommend that HR professionals periodically conduct score report and calibration audits to identify and take the steps needed to ensure that answer sheets are being properly reviewed and corrected by the test administrator and that fax machines are properly calibrated.
October 2003 Table
of Contents | TIP Home
| SIOP Home