Home Home | About Us | Sitemap | Contact  
  • Info For
  • Professionals
  • Students
  • Educators
  • Media
  • Search
    Powered By Google

Practicing What We Preach: Competency-Based Assessment of Industrial/Organizational Psychology Graduate Students

Herman Aguinis and Kurt Kraiger

University of Colorado at Denver

Ask any former Industrial/Organizational (I/O) psychology graduate student to reflect on their comprehensive examinations and the response will likely involve sighs, groans, or expletives. Graduate programs in I/O psychology typically implement some type of written or oral comprehensive examination (comps or prelims; Society for Industrial and Organizational Psychology, 1995) to ensure that future graduates have the necessary knowledge and expertise before graduation. However, for most students, these comprehensive exams are perceived as much as a measure of test anxiety as an assessment of true capability. Moreover, comprehensive examination procedures typically assess knowledge of content, rather than the strategic and competent implementation of knowledge. While traditional testing strategies may validly assess these knowledge requirements, they may do little to distinguish future good and poor actual performers.

So, why do we torment our students by putting them through the grueling experience of taking comps? Good question. Although most of us would probably never admit it, it seems that in this instance, our methods may justify their madness. We have become creatures of habit with regard to testing and evaluating graduate students. As I/O psychologists, we have the knowledge and expertise to do better.

In this article, we advocate the implementation of a competency-based approach for evaluating I/O psychology graduate students. First, we briefly define and illustrate the concept of competency. Then, we describe a newly implemented competency-based assessment system at the University of Colorado at Denver.

Competencies: Definition and Benefits

As I/O psychologists, we often recommend that organizations identify competencies necessary for successful job performance (e.g., Kesler, 1995). A competency refers to an individual's demonstrated knowledge, skills, or abilities (KSAs; Ulrich, Brockbank, Yueng, & Lake, 1995). Note, however, that competencies go beyond the more traditional KSAs; they are KSAs that are demonstrated in a job context influenced by the organizational culture and business environment. The circumstances of the business environment directly influence what and how specific KSAs are demonstrated (Boyatzis, 1982). Moreover, competencies are combinations of KSAs. Traditionally, I/O psychologists have been concerned with distinguishing and separating the KSAs required for effective job performance. At present, however, it is recognized that it is a cluster of demonstrated KSAs that defines a competency and makes a real difference for success in each organizational environment (Wisher, 1994). For example, planning can be a competency. This competency would be composed of such skills and activities as setting goals, assessing risks, and developing a sequence of actions to reach the goal (Boyatzis, 1982).

Lawler (1994) eloquently described how organizations need to change their structure, work design, and human resources management practices in order to become more adaptable and to add more value to products and services. I/O psychologists working in organizations have realigned their roles as practitioners to support these changes, as they promote a shift from job-based to competency-based organizations (Lawler, 1994). Accordingly, our comprehensive examination policies in I/O psychology graduate programs must also be realigned. Graduate programs in I/O psychology adopting a competency-based approach to evaluate their students will maximize the chances that graduates will not only have the necessary KSAs, but that they will also be able to implement them in specific work environments (i.e., academic as well as business organizations). In short, real competence involves proper application and demonstration of KSAs within a dynamic environment. Because it is the role of a graduate program to train its participants for scholarly activity in a subsequent job environment, it is appropriate to define graduate student competencies in terms of that environment.

Changing our graduate program assessment practices to parallel what we preach to business organizations regarding the adoption of a competency-based approach will demonstrate to organizations that our students possess the competencies necessary for attaining an increasingly elusive competitive advantage. This, too, will add value to our academic programs by narrowing the gap between I/O psychology research and practice (cf. Aguinis & Kraiger, 1996).

In addition, the competency-based approach mirrors a growing trend within educational psychology to link assessment, instruction, and application. Educational researchers have begun to emphasize methods of authentic assessment or portfolio assessment to contextualize evaluation. Authentic assessment refers to the evaluation that requires achievement to be determined by performance or products that interpret, apply, or operationalize knowledge in meaningful situations (Wiggins, 1989). Assessment becomes simultaneous with learning; rather than being a retrospective documentation of learning, evaluation is itself a tool for learning. In contrast to traditional forms of evaluation, methods of authentic or portfolio assessment require that learners apply new concepts to real-world problems, display performance publicly, work in social contexts to solve problems, and recognize success criteria that mirror real-world complexities.

Competency-based Assessment of I/O Psychology Graduate Students at the University of Colorado at Denver

The University of Colorado at Denver has an M.A. program in I/O Psychology. The objective of the program is to train individuals to perform psychological research, evaluation, and services in public or private sector organizations. Students also receive state-of-the-art training in theories and methods in I/O psychology, which in turn can prepare them for further (i.e., doctoral level) graduate training. The I/O psychology program at the University of Colorado at Denver has recently designed and implemented a competency-based comprehensive examination (Aguinis & Kraiger, 1997). Nine competencies were defined representing the professional topic areas emphasized in our program. These competencies were drawn from the Guidelines for the Education and Training at the Master's Level in Industrial/Organi-zational Psychology published by the Society for Industrial and Organizational Psychology (1994). Table 1 shows the competencies included in our system as well as the competencies included in the Guidelines (SIOP, 1994). As is shown in Table 1, our competency-based assessment does not include all the competencies listed in the Guidelines (e.g., small group theory and process, organization theory). The competencies chosen to be included in our competency-based assessment system were defined in terms of knowledge and skills necessary for individual accomplishment as an M.A.-level practitioner at the University of Colorado at Denver. Other programs may choose to include a different (i.e., more or less inclusive) list of competencies depending on the goals and areas of emphasis of each program.

Successful demonstration of the competencies may be accomplished through a (a) traditional paper-and-pencil examination, (b) work sample, or (c) project. Students are responsible for selecting their evaluation format for each competency. Thus, through consultation with the program coordinators, students are able to individually determine the process by which they will demonstrate each of the nine competencies. In addition, each student is required to have two committee members evaluate their proposal for demonstration of a competency. The value of this lies in having an internship or project supervisor from the business domain take a more active role in the student's development-adding value to the program as well as their organization. The requirement of choosing evaluation formats for each competency forces students and faculty to create what amounts to an individualized development plan for each student. Each student has an individual evaluation plan which is a living document, modified as they complete competency requirements or choose new evaluation formats.

Table 1

Competencies Included in the Competency-based Assessment System at the University of Colorado at Denver and Competencies Included in the Master's Level Guidelines for the Training of Industrial/Organizational Psychologists (SIOP, 1994)


University of Colorado at Denver Master's Level Guidelines (SIOP, 1994)

Ethical, Legal, and Professional Issues Ethical, Legal, and Professional Issues

Univariate Statistics Statistical Methods/Data Analysis

Measurement Measurement of Individual Differences

Criterion Development Criterion Theory and Development

Job Analysis Job and Task Analysis

Performance Appraisal Performance Appraisal and Feedback

Selection Employee Selection, Placement,

Classification

Training Training: Theory, Program Design,

Evaluation

Multivariate Statistics Work Motivation

Attitude Theory

Small Group Theory and Process

Organization Theory

Organizational Development

Research Methods

Optional (desirable but not essential):

Career Development Theory

Human Performance/Human Factors

Consumer Behavior

Compensation and Benefits

Industrial and Labor Relations


Written Examination

The written examination is offered twice a year, at the beginning of fall and spring semesters. Students have one hour to complete one essay question corresponding to each competency. The questions are focused on specific problems and situations that students need to resolve. Thus, although a student may choose this option to demonstrate one or more of the nine competencies, questions do not emphasize encyclopedic knowledge. Rather, questions focus on the application of knowledge to resolve specific situations.

Work Sample

A work sample consists of hands-on experience that demonstrates that students possess knowledge relevant to a specific competency. For example, a thesis or internship project may involve extensive application of univariate and multivariate statistics or the development of a performance appraisal system. Thus, students may fulfill these competency requirements. It is the responsibility of the student to present his/her case that the completion of a work sample meets the stated requirements for demonstrating a specific competency.

Project

A project is similar to a work sample because it requires that students demonstrate that they possess knowledge relevant to a specific competency and the application of this knowledge. However, a project does not require hands-on work in a "real" project. For example, a project may consist of a written proposal regarding a fictitious organizational intervention, or the statistical analyses and interpretation of a data set already collected and analyzed by a faculty member.

Summary and Conclusion

As I/O psychologists, we often recommend that organizations make a transition from job-based to competency-based organizations. Likewise, our own educational programs must modify their curricula to reflect this paradigm shift if we wish to produce students prepared to succeed in a highly competitive job market. Ensuring our graduates have a fighting chance in the extremely competitive academic and business job markets requires innovative methods of demonstrating their professional competence. Unfortunately, our academic practices have been lagging the market. We believe that I/O psychology graduate programs should be among the leaders implementing competency-based academic testing and evaluation practices. This format offers several advantages over the traditional methods. First, it is consistent with the current theoretical developments in our profession. Second, it forces students to take a more active role in their own development. Third, the evaluation process occurs within the actual learning context. This focus on experiential learning especially benefits adult learners. Finally, students seem much more receptive to the format. Although we have not yet conducted a quantitative evaluation of the new competency-based system, qualitative illustrations of typical testimonies of students who were asked to judge the competency-based system and compare it to the old written examination format include the following:

"I can honestly say that being able to fulfill comps by competency-based procedures will have a large effect on my stress level. I like being able to make the attempt at the comp in class or other applied things we are doing. There's nothing to lose-only to gain in completing a comp. Besides, some things are really just beyond paper-and-pencil tests. I learn much more from 'doing' than 'saying'."

"I do not view the new system with nearly as much trepidation as if it

were a series of tests to be taken on one day that would determine whether I passed or failed my graduate career. I do, also, feel that the applied version tests students more thoroughly on the knowledge that they acquired during the graduate program."

"I definitely think that the new competency-based system is less anxiety producing than the old one. One reason is that we can do it on our own time, just as long as it is completed by graduation. A conscientious student could theoretically try to plan out the fulfillment of the individual competencies over the 2-year span of the program instead of having a 3 day period of high stress right before graduation."

In short, the new format lessens the anxiety associated with lengthy do-or-die exams and provides students with an added sense of control over the outcome of the evaluation and increases face validity.

In closing, it is our hope that the present article will stimulate further discussions regarding the implementation of competency-based assessment systems in our very own ateliers.

References

Aguinis, H., & Kraiger, K. (1997). Industrial and organizational psychology program at the University of Colorado at Denver. International Journal of Selection and Assessment, 5, 69-79.

Aguinis, H., & Kraiger, K. (1996). Intervening in emerging markets: Back to the basics or go home empty-handed. The Industrial-Organizational Psychologist, 33(3), 84-87.

Boyatzis, R. E. (1982). The competent manager: A model for effective performance. New York: John Wiley & Sons.

Kesler, G. C. (1995). A model and process for redesigning the HRM role, competencies, and work in a major multi-national corporation. Human Resource Management, 34, 229-252.

Lawler, E. E., III. (1994). From job-based to competency-based organizations. Journal of Organizational Behavior, 15, 3-15.

Society for Industrial and Organizational Psychology, Inc. (1994). Guidelines for education and training at the master's level in industrial/ organizational psychology. Arlington Heights, IL: Author.

Society for Industrial and Organizational Psychology, Inc. (1995). Graduate training programs in industrial/organizational psychology and related fields. Bowling Green, OH: Author.

Ulrich, D., Brockbank, W., Yueng, A. K., & Lake, D. G. (1995). Human resource competencies: An empirical assessment. Human Resource Management, 34, 473-495.

Wiggins, G. (1989). A true test: Toward more authentic and equitable assessment. Phi Delta Kappan, 75, 703-713.

Wisher, V. (1994). Competencies: The precious seeds of growth. Personnel Management, 26, 36-39.