Information to Consider When Creating or Purchasing an Employment Test
Employers have two choices when implementing an
employment test. They can either purchase a test or create their own test.
There are many employee testing products and services on the market today.
When deciding to purchase a test, managers may experience information overload
when reviewing information on testing products and services. Most test
publishers provide a technical summary or manual that describes the most
important qualities and characteristics for any given test. The technical
manual should provide information on most, if not all, of the factors to
consider before purchasing a test. Obtaining professional help in
interpreting testing information is often necessary. While most
managers will not have the time, resources, or background to engage in test
development, this material is provided to assist the manager in planning and
reviewing work done by professionals.
Creating a test is a complex and time-consuming process,
so experts either inside or outside the organization should be involved if a
decision is made to create rather than purchase a test. Employers develop
their own tests for a variety of reasons such as cost effectiveness, test
security concerns, company culture, position uniqueness, and other factors.
The need to develop a test is often based on a cost/benefit analysis. This
analysis addresses the cost to develop and use ones own test compared to
buying someone elses test and paying to use it over the life of the
If purchasing a test, obtaining information on these
factors is important. If creating a test, developing documentation related
to these factors is important. These factors apply regardless of what you
plan to measure with the test, what type of test you are considering, or what
mode of administration (computer vs. paper-and-pencil vs. performance) you are
1. Test development
information. The research and development that went into creating
the instrument should be documented. What was the theory or experience on which
the test was based? Was the test developed on people that are similar to
this organizations applicants or employees? What was the process used
to develop the test? At the very least, this background information on the
test is important because it provides information on the logic, care, and
thoroughness by which the test was developed.
2. Reliability. Reliability
refers the consistency of test results. There are several ways to assess
the reliability of a test, and some are more appropriate for certain situations
(e.g., when multiple raters or evaluators are involved; if one wishes to know
about stability of results over time). Experienced and knowledgeable
test publishers have information on the reliability of their testing products.
3. Validity. Validity refers to
the accuracy of the inferences made based on test results (e.g., how accurate is
it to say that a higher test score indicates that a person is more likely to be
a better performer). There are many forms of validity evidence.
For example, evidence might consist of showing a relationship between test
scores and some outcome of interest (e.g., supervisory ratings of job
performance, average monthly sales, turnover). Evidence might consist of
documentation of links between the content of the test and the requirements of
the job. Evidence might include showing that the test relates to other
measures of the same thing. Experienced and knowledgeable test publishers
have (and are happy to provide) information on the validity of their testing
products. Judgments regarding what types of validity evidence are appropriate
for a given test depend on a number of factors, and these are outlined in The
Standards for Educational and Psychological Testing (www.apa.org/science/standards.html)
. Trained professionals can help interpret whether the evidence supporting
the particular inferences an employer wishes to make with a test is sufficient.
4. Test bias. Test
developers should provide evidence that the test does not contain bias on the
basis of race or sex, that is, that the test is related to outcomes in a similar
manner for all individuals. This does not mean that the test will have
similar results for different groups of people, but that it is not a biased
indicator of an outcome of interest. For example, in a typical employment
decision context, more women than men will score low on a test of upper body
strength, but the test would not be considered biased if women and men with
similar scores achieved similar performance on the job.
5. Information on administration.
The documentation should include a description of all materials required for
administration (e.g., test booklets, answer sheets, scoring keys, etc.) and
administration instructions. Instructions should discuss issues such as
standardization of testing conditions (e.g., noise, lighting, time) and how to
avoid nonstandardized administrations. Qualifications for administering
should also be clearly stated.
6. Data for test interpretation.
Test scores cannot be interpreted in isolation, but are interpreted in light of
other information. For example, whether a test score is considered good or
poor may depend on the distribution of scores of a comparison group. This
comparison group is typically referred to as a norm group. The test
publisher should provide information about the different norm groups that are
available for the test being considered. Ideally, one uses a norm group
that is similar to the group of people that are in the position for which
testing is being used. There are other ways to interpret test results
including expectancy charts and cut scores, which are developed based on
information about how the score relates to outcomes of interest.
Information should be made available on data that can aid in appropriate test
7. Scoring options.
Determine what scoring and test reporting needs are to determine whether the
test has appropriate options. Some tests can be scored on-site, either by
hand or by machine. Other tests require that an employer call, mail, or
fax the test results to the test publisher for scoring. Qualifications for
scoring should be explicit.
8. Ongoing research/refinement of the
test. The test publisher should indicate when the test was
developed and when the test was last updated. Test publishers often update
their tests to comply with new legal requirements or to reflect changes in
vocabulary or terminology.
9. Time requirements. Some
tests have time limits while others provide the test taker with unlimited time
to take the test. How time limits were determined and why they are
necessary should be documented. For tests that have time limits, greater
administrator training may be needed.
10. Credentials and experience.
The educational background and work experience of the persons who developed the
test should be documented, as well as references that can speak to the
capabilities and experience of the test developer or vendor. Some
tests require the test administrator or individuals interpreting test scores to
have certain credentials (e.g., MA, PhD) that reflect coursework in
statistics, test interpretation, or test development and validation.
11. Cost. Direct costs of the test
usually include test booklets, answer sheets, and a test administrators
manual. Hand-scored tests usually include a scoring template as well.
Computer-based testing generally includes software that is valid for a prepaid
number of uses. Testing fees usually need to be considered as an ongoing
expense, since few test publishers will license a test for unlimited usage.
This situation often leads organizations to create their own tests.
However, remember that part of the cost of the test is for the substantial
investment that the publisher made in researching and developing a high quality
measure. Creating a high-quality and effective test requires time, money,
and people to research and develop, revise, and validate the tests. In
many instances, the more cost effective approach is to purchase a test from a
test publisher rather than to create a test.
Choosing or developing a test
is a challenge, and this list just mentions some of the information you should
obtain. A comprehensive source of the information you should
consider in testing is provided by the Standards for Educational and
Psychological Testing (www.apa.org/science/standards.html).
You might also want to check out, www.onetcenter.org,
for Testing and Assessment: An Employers Guide to Good Practices.
This guide helps managers and others understand and use employment testing and
Employment Testing Table