Seven Questions to Ask a Vendor Before Purchasing a Test
Youve decided that testing is a viable and useful option
for your organization. You also know what knowledge, skills, abilities, or
other characteristics you want to assess. Now what? How do you find
a good quality test or tests that measures what you want to measure?
Thousands of employment testing products and services exist
in the marketplace. They vary in many ways including cost, time to
administer, and overall quality. When deciding to purchase a test,
managers typically experience information overload. Obtaining professional
help in interpreting testing information is often necessary. While
most managers will not have the time, resources, or background to engage in test
development, this material is provided to assist the manager in planning and
reviewing work done by professionals.
To help you make a good decision related to employment
testing, ask the following seven questions before purchasing a test.
If a test publisher or vendor cannot or will not answer the following questions,
you should be wary. Knowledgeable and experienced test publishers
typically welcome a clients detailed interest in their product. Test
publishers and vendors also want to provide whatever information is necessary to
help an organization make a good employment testing related decision.
1. What does the test measure?
Look for a clear and concise answer to this question. It is
difficult to develop a good quality test without a clear definition of what is
to be measured. Also, consider how the answer compares with what needs to
be assessed in your organization. Be wary of a vendor who describes their
test as measuring many seemingly unrelated factors. Also be wary of tests
that consist of a limited number of itemsfor example a 25-item test that
purports to measure 10 different factors. Measuring a characteristic or
quality consistently and accurately typically requires more than just a couple
2. What research and process was used to develop the test?
What was the theory or experience on which the test was
based? Was the test developed on people that are similar to your
organizations applicants or employees? What was the process used to
develop the test? At the very least, this background information on the
test is important because it provides information on the logic, care, and
thoroughness by which the test was developed.
3. What experience and/or education do you have that qualifies you to
develop and/or sell this test?
The educational background and work experience of the
persons who developed the test is important, as well as references that can
speak to the capabilities and experience of the test developer or vendor.
To have confidence in the test and in the event of a legal challenge, you want
test developers or vendors who have education and/or experience related to the
specific content of the test and related to test development and validation.
Also, some tests require the test administrator or individuals interpreting test
scores to have certain credentials (e.g., MA, PhD) that reflect coursework
in statistics, test interpretation, or test development and validation.
4. What evidence do you have related to the
reliability of this test?
Reliability refers the consistency of test results.
There are several ways to assess the reliability of a test, and some are more
appropriate for certain situations (e.g., when multiple raters or evaluators are
involved; if one wishes to know about stability of results over time).
Experienced and knowledgeable test publishers and vendors have information on
the reliability of their testing products. For more detailed information
on how to assess the reliability of a test, check out Testing and Assessment:
An Employers Guide to Good Practices at www.onetcenter.org.
5. What evidence do you have related to the validity of this test?
Validity refers to the accuracy of the inferences made
based on test results (e.g., how accurate is it to say that a higher test score
indicates that a person is more likely to be a better performer).
Knowledgeable and experienced test publishers typically have many forms of
validity evidence. For example, they may have evidence that shows a
relationship between test scores and some outcome of interest (e.g., supervisory
ratings of job performance, average monthly sales, turnover). They might
also have evidence that documents a link between the content of the test and the
requirements of the job. Other evidence might include showing how the test
relates to other measures of the same thing. Experienced and knowledgeable
test publishers have (and are happy to provide) information on the validity of
their testing products. Judgments regarding what types of validity evidence are
appropriate for a given test depend on a number of factors, and these are
outlined in The Standards for Educational and Psychological Testing (www.apa.org/science/standards.html)
. For more detailed information on how to assess the validity evidence
associated with a particular test, check out Testing and Assessment: An
Employers Guide to Good Practices at www.onetcenter.org.
6. What evidence do you have that demonstrates the lack of bias or
discrimination of your test?
Look for evidence that the test does not contain bias on
the basis of race or sex, that is, that the test is related to outcomes in a
similar manner for all individuals. This statement does not necessarily
mean that the test will have similar results for different groups of people.
This statement does mean that the test is not a biased indicator of an outcome
of interest. For example, in a typical employment decision context, more
women than men will score low on a test of upper body strength. The test,
however, would not be considered biased if women and men with similar scores
achieved similar performance on the job.
7. What data do you have that will help me interpret test scores in my
You cannot interpret test scores by themselves. Whether a test score is
considered good or poor may depend on the distribution of scores of a comparison
group. This comparison group is typically referred to as a norm group.
The test publisher should provide information about the different norm groups
that are available for the test being considered. Ideally, you want to use
a norm group that is similar to the group of people that are in the position for
which testing is being used. There are other ways to interpret test
results including expectancy charts and cut scores, which are developed based on
information about how the score relates to outcomes of interest.
Information should be made available on data that can aid in appropriate test
Employment Testing Table