Home Home | About Us | Sitemap | Contact  
  • Info For
  • Professionals
  • Students
  • Educators
  • Media
  • Search
    Powered By Google

Making Global Assessments Work1

Nancy T. Tippins
Valtera Corporation

1 This paper is an outgrowth of my work in researching and developing global selection and assessment programs.  I gratefully acknowledge the contributions of my colleagues, the clients with whom I’ve worked, and Ann Marie Ryan with whom I coauthored Designing and Implementing Global Selection Systems (2009).

The globalization of business and industry has placed new demands on industrial and organizational psychologists to create assessment programs that work across international boundaries. Some of these assessment programs are selection systems that are designed to evaluate candidates for jobs that are found across geographic and cultural boundaries within one company as well as determine the ability of an individual to work cross culturally. Other types of assessment programs may be used with people from multiple cultures, including certification of skills, evaluation of potential for leadership roles, and identification of personal strengths and weaknesses for development purposes. These assessments must work in many different environments and be acceptable to the people in all the geographic areas in which they are used.

The benefits of global assessments are numerous. A single, common assessment program can significantly reduce the costs of development, deployment, and implementation when compared to multiple local programs. A well-done global selection program will ensure a consistent quality of candidates for a job in terms of knowledge, skills, abilities, or other characteristics (KSAOs), as well as present a common image to candidates around the world. In addition, data from the same assessment program allow comparisons across geographical boundaries and serve as a basis for strategic talent management, which ultimately contributes to the organization’s long-term success.

Although the benefits of global assessments are substantial, the challenges are not insignificant. Simply finding a test item format that works globally can sometimes be a challenge. In addition, changing attitudes of candidates and test administrators about new selection and assessment procedures and complying with local employment laws can be daunting tasks. Some candidates may not be familiar with certain types of items, and others may not believe some types of items are appropriate for the job to which they are applying or acceptable to individuals with their background. Test administrators may have different perceptions of fair treatment, confidentiality, and test security, and these differing attitudes may lead to divergent practices and unanticipated behaviors that compromise the test materials or results. Access to technology to support test administration, scoring, and data storage cannot be assumed. Implementing the same procedures across cultures is often viewed as a cost savings. However, the cost of translation and the research to establish equivalency are not trivial in large-scale testing programs. Moreover, global assessment requires considerable coordination among many stakeholders everywhere the assessment may be used.

The processes for developing and validating global assessment programs can be complex and surpass the limits of a TIP article. The following is a list of nine tips distilled from a number of projects for ensuring the success of a global assessment program. The list is intended to remind the practitioner who is working globally of things to do before and during the process of developing, validating, and implementing a global assessment program.

1.Determine if there is a common job or job family.

Before a global selection system can be created, the psychologist must first determine if there is a single job with similar task requirements that require a common set of KSAOs. Frequently, jobs have identical titles yet very different responsibilities. For example, a sales representative in one country may engage in consultative selling and be expected to have a deep understanding of the product or service, as well as one or more of its applications. In another country, the same sales representative may fulfill an order-taker role. The responsibilities of technicians in manufacturing facilities may differ according to the type of equipment used. Even when the assessment program is developed for a broad job family, the assumption of a common set of core work behaviors may not be warranted. For example, some leaders may be expected to develop strategy and long-term visions, but others must execute the plans of others.

2. Verify that organizational support will be forthcoming.

Unless the I-O psychologist is in a very powerful role within the company, he or she will often lack the authority to execute a large project. Most I-O psychologists will need an advocate who supports the idea of global assessment and can convince others to support the project as well as help acquire the resources (e.g., people, technology) needed for development, validation, implementation, and maintenance. In addition, this advocate must possess the patience and tolerance to sustain interest in a global project that can have a lengthy timeline.

In addition to having a powerful advocate who can make things happen, the I-O psychologist must also win the support of local human resource (HR) personnel. Without local support, implementation of a global assessment system will be difficult and continually fraught with problems. The local HR must assume many responsibilities including identifying the local constraints on assessment practices, communicating attitudes about the assessment program, providing information on what test formats are acceptable in the culture, ensuring proper training of administrators and protection of testing materials, explaining to clients the purposes and intended use of the assessment program, arranging for subject matter experts to participate in the development and validation phases, and so on.

3. Understand the expectations about the assessment process and the context in which it will be used.

Once the I-O psychologist has determined a common job exists, identified an advocate, and established a relationship with local HR personnel, he or she must learn who the primary stakeholders are and understand their requirements and expectations of the assessment process. For example, some organizations will emphasize efficiency of the assessment process as a goal, but others will focus on a high-touch experience that stimulates development. The goal of some assessments is to predict immediate job performance; for others, the goal is to assess potential for leadership in the long term. A thorough understanding of the environment for assessment administration (e.g., locations, technology, personnel capabilities, time frames, etc.) is a prerequisite for designing a system that will work. Similarly, a clear idea of how the data are to be used will help the I-O psychologist fulfill the company’s goals. An assessment program designed for selection may be different in content and administration protocol than one designed for development. Similarly, knowledge of how results will be stored and communicated to the participant and organization, how data will be protected, and what level of confidentiality will be promised will shape the assessment design.

4. Take culture into account in the design, development, validation, implementation, and use of assessment programs.

“Culture” refers to the stable characteristics of a group and can apply to a country, a geographical section of a country, an ethnic group within a country, a corporation, an organization within the company, and so on. The I-O psychologist must identify the salient aspects of culture relative to the assessment program and take steps to eliminate the effects of culture on test and assessment performance that are not job related. For example, if members of a cultural group tend not to respond accurately on self-rating instruments such as personality inventories, another approach to evaluating job-related personality traits may be required. Or, if some groups tend to inflate responses on such scales, the use of local norms may be warranted. Although cultural differences do not usually change the requirements of the job, they may change how the relevant skills and abilities are measured.

5. Consider the other characteristics of the local environment: labor market, legal environment, labor unions, and economic conditions.

In addition to culture, there are a host of other variables that will affect how a test is received by assessment participants and by stakeholders in the organization, as well as how useful the results will be to those making decisions. Prior to designing the assessment program, the I-O psychologist must learn about the characteristics of the labor market, including education and training. An assessment program that is designed for a college graduate may not be particularly effective for an applicant population consisting of people who have not finished high school. (The I-O psychologist may also want to rethink the idea of a common job!) The depth of the applicant pool can also have implications for the length of the assessment process. If the labor supply is limited, applicants may not be willing to invest a significant amount of time in an assessment process. Employment laws and labor unions can dictate the conditions under which assessments can be used. Therefore, careful attention must be paid to who is protected, how data privacy must be protected, what technical requirements must be met, and who may use assessments. In addition, local laws or customs may define who may design and administer certain kinds of assessments or require activities associated with assessment such as face-to-face feedback.

6. Develop assessment tools for universal use.

A significant challenge in most global assessment programs is developing content that works internationally. From the very beginning of content development, the psychologist must identify content, techniques, and formats that are familiar to all of the intended population. At the same time, the I-O psychologist should avoid material that includes jargon, metaphors, and other language that is difficult to translate, that is culturally offensive or confusing, that relies on systems of measurement that are not universal (e.g., currency), that includes country-specific topics, that does not translate well or depends on the nuances of words or their difficulty, or that incorporates concepts that are less familiar or simply don’t exist in other cultures.

7.  Adapt assessments for universal use.

Once content is developed, it is usually translated into multiple languages. The goal of the translation process is not only to provide a faithful replication of verbal material but also to retain the same level of difficulty and clarity. Typically, a series of translations and reviews result in the best adaptation of verbal material. Simple back translations without review often do not result in equivalent assessment materials.

8. Create the tools necessary for effective administration and scoring.

A common, yet fatal, mistake is to pay close attention to the assessment instruments and neglect the materials that are necessary for consistent administration and scoring across cultures. The problem of consistency is exacerbated when the capabilities of the local assessment staff vary widely across countries. Participant instructions, as well as scoring instructions must be as carefully written, adapted, and tested for cross-cultural understanding.

In addition to providing effective instructions, other tools can enhance the value of the assessment results. Building flexibility into the order of assessment components may allow HR personnel to adapt the process to fit the local staffing model. For example, if most candidates for a job are not local and must travel to a company office, a structured interview conducted over the telephone may precede proctored testing. In contrast, if most candidates are local, then the proctored testing preceding the interview may be a more cost-effective approach because fewer candidates are interviewed.

9. Determine if the global assessment meets technical criteria.

Although cross-cultural research is difficult to do and sample sizes are often not large enough for statistical analyses, the I-O psychologist should strive to evaluate the extent to which the global assessment program is valid, reliable, free from bias, and job relevant. Failure to do so may leave questions about the extent to which a business need is met or compliance with local discrimination laws is achieved. Regardless of the purpose of the assessment, the assessment program is useful only if it meets these criteria.

Many American I-O psychologists rely on the Uniform Guidelines on Employee Selection Procedures (EEOC, 1978), the Principles for the Use and Validation of Employee Selection Standards (SIOP, 2003) or the Standards for Educational and Psychological Testing (AERA, APA, & NCME, 1999) to define technical standards; however, there are others guidelines and standards that are relevant in an international setting. Documents produced by the International Test Commission such as International Guidelines for Test Use (ITC, 1999) and International Test Commission Guidelines for Test Adaptation (ITC, 2000) are important as are the Guidelines and Ethical Considerations for Assessment Center Operations (2000) from the International Task Force on Assessment Center Guidelines. In addition, ISO 9000 standards for assessments are in progress.

10. Collect evidence of equivalence across cultures.

In addition to demonstrating validity and reliability, the equivalence of assessment tools that are translated or used cross culturally should be demonstrated. For an assessment to be effective internationally, the I-O psychologist must ask if all versions of the assessment measure the same construct at the same level in each culture and determine if the observed differences are a result of true differences between the groups or a result of the test materials themselves. Statistical approaches, including differential item functioning, metric or scalar equivalence, or confirmatory factor analysis, can help the psychologist understand the extent of equivalence; however, many of the statistical approaches are difficult to implement when the sample in any one country or culture is small. Nevertheless, “conceptual equivalence” can usually be established through careful translations and cultural reviews.

References

      American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
     Equal Employment Opportunity Commission, Civil Service Commission, Department of Labor, & Department of Justice. (1978). Uniform guidelines on employee selection procedures.  Federal Register, 43, 38290-38315.  

     International Task Force on Assessment Center Guidelines. (2000). Guidelines and ethical considerations for assessment center operations. Retrieved April 23, 2010 from
www.assessmentcenters.org/pdf/00guidelines.pdf.
     International Test Commission. (1999).  International guidelines for test use. Retrieved April 23, 2010 from http://www.intestcom.org/Downloads/ITC%20Guidelines%20Download%20Version%204.doc.
     International Test Commission. (2000). International Test Commission guidelines for test adaptation. Retrieved April 23, 2010 from http://www.intestcom.org/test_adaptation.htm.
     Ryan, A.M. & Tippins, N.T. (2009).  Designing and implementing global selection systems. New York: Blackwell Press.
     Society for Industrial and Organizational Psychology. (2003). Principles for the validation and use of personnel selection procedures. Bowling Green, OH: Author.