Practicing What We Preach: Competency-Based Assessment of
Industrial/Organizational Psychology Graduate Students
Herman Aguinis and Kurt Kraiger
University of Colorado at Denver
Ask any former Industrial/Organizational (I/O) psychology graduate
student to reflect on their comprehensive examinations and the response will likely
involve sighs, groans, or expletives. Graduate programs in I/O psychology typically
implement some type of written or oral comprehensive examination (comps or prelims;
Society for Industrial and Organizational Psychology, 1995) to ensure that future
graduates have the necessary knowledge and expertise before graduation. However, for most
students, these comprehensive exams are perceived as much as a measure of test anxiety as
an assessment of true capability. Moreover, comprehensive examination procedures typically
assess knowledge of content, rather than the strategic and competent implementation
of knowledge. While traditional testing strategies may validly assess these knowledge
requirements, they may do little to distinguish future good and poor actual performers.
So, why do we torment our students by putting them through the grueling
experience of taking comps? Good question. Although most of us would probably never
admit it, it seems that in this instance, our methods may justify their madness. We have
become creatures of habit with regard to testing and evaluating graduate students. As I/O
psychologists, we have the knowledge and expertise to do better.
In this article, we advocate the implementation of a competency-based
approach for evaluating I/O psychology graduate students. First, we briefly define and
illustrate the concept of competency. Then, we describe a newly implemented
competency-based assessment system at the University of Colorado at Denver.
Competencies: Definition and Benefits
As I/O psychologists, we often recommend that organizations identify
competencies necessary for successful job performance (e.g., Kesler, 1995). A competency
refers to an individual's demonstrated knowledge, skills, or abilities (KSAs;
Ulrich, Brockbank, Yueng, & Lake, 1995). Note, however, that competencies go beyond
the more traditional KSAs; they are KSAs that are demonstrated in a job context
influenced by the organizational culture and business environment. The circumstances of
the business environment directly influence what and how specific KSAs are demonstrated
(Boyatzis, 1982). Moreover, competencies are combinations of KSAs. Traditionally, I/O
psychologists have been concerned with distinguishing and separating the KSAs required for
effective job performance. At present, however, it is recognized that it is a cluster
of demonstrated KSAs that defines a competency and makes a real difference for
success in each organizational environment (Wisher, 1994). For example, planning can be a
competency. This competency would be composed of such skills and activities as setting
goals, assessing risks, and developing a sequence of actions to reach the goal (Boyatzis,
1982).
Lawler (1994) eloquently described how organizations need to change
their structure, work design, and human resources management practices in order to become
more adaptable and to add more value to products and services. I/O psychologists working
in organizations have realigned their roles as practitioners to support these changes, as
they promote a shift from job-based to competency-based organizations (Lawler, 1994).
Accordingly, our comprehensive examination policies in I/O psychology graduate programs
must also be realigned. Graduate programs in I/O psychology adopting a competency-based
approach to evaluate their students will maximize the chances that graduates will not only
have the necessary KSAs, but that they will also be able to implement them in specific
work environments (i.e., academic as well as business organizations). In short, real
competence involves proper application and demonstration of KSAs within a dynamic
environment. Because it is the role of a graduate program to train its participants for
scholarly activity in a subsequent job environment, it is appropriate to define graduate
student competencies in terms of that environment.
Changing our graduate program assessment practices to parallel what we
preach to business organizations regarding the adoption of a competency-based approach
will demonstrate to organizations that our students possess the competencies necessary for
attaining an increasingly elusive competitive advantage. This, too, will add value to our
academic programs by narrowing the gap between I/O psychology research and practice (cf.
Aguinis & Kraiger, 1996).
In addition, the competency-based approach mirrors a growing trend
within educational psychology to link assessment, instruction, and application.
Educational researchers have begun to emphasize methods of authentic assessment or
portfolio assessment to contextualize evaluation. Authentic assessment refers to the
evaluation that requires achievement to be determined by performance or products that
interpret, apply, or operationalize knowledge in meaningful situations (Wiggins, 1989).
Assessment becomes simultaneous with learning; rather than being a retrospective
documentation of learning, evaluation is itself a tool for learning. In contrast to
traditional forms of evaluation, methods of authentic or portfolio assessment require that
learners apply new concepts to real-world problems, display performance publicly, work in
social contexts to solve problems, and recognize success criteria that mirror real-world
complexities.
Competency-based Assessment of I/O Psychology Graduate
Students at the University of Colorado at Denver
The University of Colorado at Denver has an M.A. program in I/O
Psychology. The objective of the program is to train individuals to perform psychological
research, evaluation, and services in public or private sector organizations. Students
also receive state-of-the-art training in theories and methods in I/O psychology, which in
turn can prepare them for further (i.e., doctoral level) graduate training. The I/O
psychology program at the University of Colorado at Denver has recently designed and
implemented a competency-based comprehensive examination (Aguinis & Kraiger, 1997).
Nine competencies were defined representing the professional topic areas emphasized in our
program. These competencies were drawn from the Guidelines for the Education and
Training at the Master's Level in Industrial/Organi-zational Psychology published by
the Society for Industrial and Organizational Psychology (1994). Table 1 shows the
competencies included in our system as well as the competencies included in the Guidelines
(SIOP, 1994). As is shown in Table 1, our competency-based assessment does not include all
the competencies listed in the Guidelines (e.g., small group theory and process,
organization theory). The competencies chosen to be included in our competency-based
assessment system were defined in terms of knowledge and skills necessary for individual
accomplishment as an M.A.-level practitioner at the University of Colorado at Denver.
Other programs may choose to include a different (i.e., more or less inclusive) list of
competencies depending on the goals and areas of emphasis of each program.
Successful demonstration of the competencies may be accomplished through
a (a) traditional paper-and-pencil examination, (b) work sample, or (c) project. Students
are responsible for selecting their evaluation format for each competency. Thus, through
consultation with the program coordinators, students are able to individually determine
the process by which they will demonstrate each of the nine competencies. In addition,
each student is required to have two committee members evaluate their proposal for
demonstration of a competency. The value of this lies in having an internship or project
supervisor from the business domain take a more active role in the student's
development-adding value to the program as well as their organization. The requirement of
choosing evaluation formats for each competency forces students and faculty to create what
amounts to an individualized development plan for each student. Each student has an
individual evaluation plan which is a living document, modified as they complete
competency requirements or choose new evaluation formats.
Table 1
Competencies Included in the Competency-based Assessment System at
the University of Colorado at Denver and Competencies Included in the Master's Level
Guidelines for the Training of Industrial/Organizational Psychologists (SIOP, 1994)
University of Colorado at Denver Master's Level Guidelines (SIOP,
1994)
Ethical, Legal, and Professional Issues Ethical, Legal, and Professional
Issues
Univariate Statistics Statistical Methods/Data Analysis
Measurement Measurement of Individual Differences
Criterion Development Criterion Theory and Development
Job Analysis Job and Task Analysis
Performance Appraisal Performance Appraisal and Feedback
Selection Employee Selection, Placement,
Classification
Training Training: Theory, Program Design,
Evaluation
Multivariate Statistics Work Motivation
Attitude Theory
Small Group Theory and Process
Organization Theory
Organizational Development
Research Methods
Optional (desirable but not essential):
Career Development Theory
Human Performance/Human Factors
Consumer Behavior
Compensation and Benefits
Industrial and Labor Relations
Written Examination
The written examination is offered twice a year, at the beginning of
fall and spring semesters. Students have one hour to complete one essay question
corresponding to each competency. The questions are focused on specific problems and
situations that students need to resolve. Thus, although a student may choose this option
to demonstrate one or more of the nine competencies, questions do not emphasize
encyclopedic knowledge. Rather, questions focus on the application of knowledge to resolve
specific situations.
Work Sample
A work sample consists of hands-on experience that demonstrates that
students possess knowledge relevant to a specific competency. For example, a thesis or
internship project may involve extensive application of univariate and multivariate
statistics or the development of a performance appraisal system. Thus, students may
fulfill these competency requirements. It is the responsibility of the student to present
his/her case that the completion of a work sample meets the stated requirements for
demonstrating a specific competency.
Project
A project is similar to a work sample because it requires that students
demonstrate that they possess knowledge relevant to a specific competency and the
application of this knowledge. However, a project does not require hands-on work in a
"real" project. For example, a project may consist of a written proposal
regarding a fictitious organizational intervention, or the statistical analyses and
interpretation of a data set already collected and analyzed by a faculty member.
Summary and Conclusion
As I/O psychologists, we often recommend that organizations make a
transition from job-based to competency-based organizations. Likewise, our own educational
programs must modify their curricula to reflect this paradigm shift if we wish to produce
students prepared to succeed in a highly competitive job market. Ensuring our graduates
have a fighting chance in the extremely competitive academic and business job markets
requires innovative methods of demonstrating their professional competence. Unfortunately,
our academic practices have been lagging the market. We believe that I/O psychology
graduate programs should be among the leaders implementing competency-based academic
testing and evaluation practices. This format offers several advantages over the
traditional methods. First, it is consistent with the current theoretical developments in
our profession. Second, it forces students to take a more active role in their own
development. Third, the evaluation process occurs within the actual learning context. This
focus on experiential learning especially benefits adult learners. Finally, students seem
much more receptive to the format. Although we have not yet conducted a quantitative
evaluation of the new competency-based system, qualitative illustrations of typical
testimonies of students who were asked to judge the competency-based system and compare it
to the old written examination format include the following:
"I can honestly say that being able to fulfill comps by
competency-based procedures will have a large effect on my stress level. I like being able
to make the attempt at the comp in class or other applied things we are doing. There's
nothing to lose-only to gain in completing a comp. Besides, some things are really just
beyond paper-and-pencil tests. I learn much more from 'doing' than 'saying'."
"I do not view the new system with nearly as much trepidation as if
it
were a series of tests to be taken on one day that would determine
whether I passed or failed my graduate career. I do, also, feel that the applied version
tests students more thoroughly on the knowledge that they acquired during the graduate
program."
"I definitely think that the new competency-based system is less
anxiety producing than the old one. One reason is that we can do it on our own time, just
as long as it is completed by graduation. A conscientious student could theoretically try
to plan out the fulfillment of the individual competencies over the 2-year span of the
program instead of having a 3 day period of high stress right before graduation."
In short, the new format lessens the anxiety associated with lengthy
do-or-die exams and provides students with an added sense of control over the outcome of
the evaluation and increases face validity.
In closing, it is our hope that the present article will stimulate
further discussions regarding the implementation of competency-based assessment systems in
our very own ateliers.
References
Aguinis, H., & Kraiger, K. (1997). Industrial and organizational
psychology program at the University of Colorado at Denver. International Journal of
Selection and Assessment, 5, 69-79.
Aguinis, H., & Kraiger, K. (1996). Intervening in emerging markets:
Back to the basics or go home empty-handed. The Industrial-Organizational Psychologist,
33(3), 84-87.
Boyatzis, R. E. (1982). The competent manager: A model for effective
performance. New York: John Wiley & Sons.
Kesler, G. C. (1995). A model and process for redesigning the HRM role,
competencies, and work in a major multi-national corporation. Human Resource Management,
34, 229-252.
Lawler, E. E., III. (1994). From job-based to competency-based
organizations. Journal of Organizational Behavior, 15, 3-15.
Society for Industrial and Organizational Psychology, Inc. (1994). Guidelines
for education and training at the master's level in industrial/ organizational psychology.
Arlington Heights, IL: Author.
Society for Industrial and Organizational Psychology, Inc. (1995). Graduate
training programs in industrial/organizational psychology and related fields. Bowling
Green, OH: Author.
Ulrich, D., Brockbank, W., Yueng, A. K., & Lake, D. G. (1995). Human
resource competencies: An empirical assessment. Human Resource Management, 34,
473-495.
Wiggins, G. (1989). A true test: Toward more authentic and equitable
assessment. Phi Delta Kappan, 75, 703-713.
Wisher, V. (1994). Competencies: The precious seeds of growth. Personnel
Management, 26, 36-39.