Information to Consider When Creating or Purchasing an Employment Test
Employers have two choices when implementing an employment test. They can either purchase a test or create their own test. There are many employee testing products and services on the market today. When deciding to purchase a test, managers may experience information overload when reviewing information on testing products and services. Most test publishers provide a technical summary or manual that describes the most important qualities and characteristics for any given test. The technical manual should provide information on most, if not all, of the factors to consider before purchasing a test. Obtaining professional help in interpreting testing information is often necessary. While most managers will not have the time, resources, or background to engage in test development, this material is provided to assist the manager in planning and reviewing work done by professionals.
Creating a test is a complex and time-consuming process, so experts either inside or outside the organization should be involved if a decision is made to create rather than purchase a test. Employers develop their own tests for a variety of reasons such as cost effectiveness, test security concerns, company culture, position uniqueness, and other factors. The need to develop a test is often based on a cost/benefit analysis. This analysis addresses the cost to develop and use ones own test compared to buying someone elses test and paying to use it over the life of the instrument.
If purchasing a test, obtaining information on these factors is important. If creating a test, developing documentation related to these factors is important. These factors apply regardless of what you plan to measure with the test, what type of test you are considering, or what mode of administration (computer vs. paper-and-pencil vs. performance) you are planning.
- Test development information. The research and development that went into creating the instrument should be documented. What was the theory or experience on which the test was based? Was the test developed on people that are similar to this organizations applicants or employees? What was the process used to develop the test? At the very least, this background information on the test is important because it provides information on the logic, care, and thoroughness by which the test was developed.
- Reliability. Reliability refers the consistency of test results. There are several ways to assess the reliability of a test, and some are more appropriate for certain situations (e.g., when multiple raters or evaluators are involved; if one wishes to know about stability of results over time). Experienced and knowledgeable test publishers have information on the reliability of their testing products.
- Validity. Validity refers to the accuracy of the inferences made based on test results (e.g., how accurate is it to say that a higher test score indicates that a person is more likely to be a better performer). There are many forms of validity evidence. For example, evidence might consist of showing a relationship between test scores and some outcome of interest (e.g., supervisory ratings of job performance, average monthly sales, turnover). Evidence might consist of documentation of links between the content of the test and the requirements of the job. Evidence might include showing that the test relates to other measures of the same thing. Experienced and knowledgeable test publishers have (and are happy to provide) information on the validity of their testing products. Judgments regarding what types of validity evidence are appropriate for a given test depend on a number of factors, and these are outlined in The Standards for Educational and Psychological Testing. Trained professionals can help interpret whether the evidence supporting the particular inferences an employer wishes to make with a test is sufficient.
- Test bias. Test developers should provide evidence that the test does not contain bias on the basis of race or sex, that is, that the test is related to outcomes in a similar manner for all individuals. This does not mean that the test will have similar results for different groups of people, but that it is not a biased indicator of an outcome of interest. For example, in a typical employment decision context, more women than men will score low on a test of upper body strength, but the test would not be considered biased if women and men with similar scores achieved similar performance on the job.
- Information on administration. The documentation should include a description of all materials required for administration (e.g., test booklets, answer sheets, scoring keys, etc.) and administration instructions. Instructions should discuss issues such as standardization of testing conditions (e.g., noise, lighting, time) and how to avoid nonstandardized administrations. Qualifications for administering should also be clearly stated.
- Data for test interpretation. Test scores cannot be interpreted in isolation, but are interpreted in light of other information. For example, whether a test score is considered good or poor may depend on the distribution of scores of a comparison group. This comparison group is typically referred to as a norm group. The test publisher should provide information about the different norm groups that are available for the test being considered. Ideally, one uses a norm group that is similar to the group of people that are in the position for which testing is being used. There are other ways to interpret test results including expectancy charts and cut scores, which are developed based on information about how the score relates to outcomes of interest. Information should be made available on data that can aid in appropriate test score interpretation.
- Scoring options. Determine what scoring and test reporting needs are to determine whether the test has appropriate options. Some tests can be scored on-site, either by hand or by machine. Other tests require that an employer call, mail, or fax the test results to the test publisher for scoring. Qualifications for scoring should be explicit.
- Ongoing research/refinement of the test. The test publisher should indicate when the test was developed and when the test was last updated. Test publishers often update their tests to comply with new legal requirements or to reflect changes in vocabulary or terminology.
- Time requirements. Some tests have time limits while others provide the test taker with unlimited time to take the test. How time limits were determined and why they are necessary should be documented. For tests that have time limits, greater administrator training may be needed.
- Credentials and experience. The educational background and work experience of the persons who developed the test should be documented, as well as references that can speak to the capabilities and experience of the test developer or vendor. Some tests require the test administrator or individuals interpreting test scores to have certain credentials (e.g., MA, PhD) that reflect coursework in statistics, test interpretation, or test development and validation.
- Cost. Direct costs of the test usually include test booklets, answer sheets, and a test administrators manual. Hand-scored tests usually include a scoring template as well. Computer-based testing generally includes software that is valid for a prepaid number of uses. Testing fees usually need to be considered as an ongoing expense, since few test publishers will license a test for unlimited usage. This situation often leads organizations to create their own tests. However, remember that part of the cost of the test is for the substantial investment that the publisher made in researching and developing a high quality measure. Creating a high-quality and effective test requires time, money, and people to research and develop, revise, and validate the tests. In many instances, the more cost effective approach is to purchase a test from a test publisher rather than to create a test.
Choosing or developing a test is a challenge, and this list just mentions some of the information you should obtain. A comprehensive source of the information you should consider in testing is provided by the Standards for Educational and Psychological Testing (www.apa.org/science/standards.html). You might also want to check out, www.onetcenter.org, for Testing and Assessment: An Employers Guide to Good Practices. This guide helps managers and others understand and use employment testing and assessment practices.