What is I-O?

Industrial-organizational (I-O) psychology is the scientific study of the workplace. Rigor and methods of psychology are applied to issues of critical relevance to business, including talent management, coaching, assessment, selection, training, organizational development, performance, and work-life balance.

Outcomes Assessment for I-O Psychology
Graduate Programs

Stefani L. Yorges
West Chester University

Many of you have probably heard of the term outcomes assessment by now. Recent legislative mandates now require most academic institutions to provide evidence of their effectiveness and their value to students. Faculty are expected to justify high tuition with proof of the knowledge and skills that students have gained. In some cases, it can determine future funding provided to that institution. In other cases, it is used simply to provide feedback as the university strives to improve its educational programs.

While many universities now have undergraduate general education assessment programs in place, little attention has been paid to outcomes assessment at the graduate level. Despite its importance, graduate education remains relatively unexamined and less defined in terms of goals and outcomes than undergraduate education programs. Having recently created an outcomes assessment plan for the I-O master's program at West Chester University, I hope to provide some suggestions to those of you that may be required to create similar plans in the near future.

Graduate Level Assessment

Graduate level assessment will differ from undergraduate assessment plans in several respects. The essence of graduate training lies in the breadth of research and critical thinking skills that are developed in the course of the graduate degree program. Strong graduate programs can be critical to the national welfare because they may lead to significant advances in basic and applied research as well as provide the technical and professional skills necessary for advancing quality of life (Commission on Higher Education, CHE, 1996).

Two main types of graduate programs exist and are distinguished by their primary objectives. Research-oriented programs tend to focus on preparing graduate students for scholarly or research activities that are directed toward the acquisition of new knowledge. The objective of practice-oriented programs is to prepare graduate students for professional practice where they will apply existing knowledge.

I-O Graduate Program Objectives

The first step in creating an assessment plan for your I-O graduate program is to identify several primary objectives. Similar to procedures employed in job analysis, you will identify what knowledge, skills, and abilities are essential for achieving success in your program. You will also want to consider the expected impact of your educational program on students. You will be clarifying what goals you have for students that complete your program. Identifying your program as research-oriented, practice-oriented, or a combination of both will be essential at this point. Specific goals and objectives may be identified in a mission statement, if you already have one developed for your program.

You will also be expected to demonstrate specifically how your educational program meets those objectives. One way to do this is to identify which courses (or aspects of those courses) contribute to your goals and objectives. You may also look at other program requirements (e.g., thesis completion, internship completion) and how they meet your educational goals for students. At this point, direct links between your educational program and student learning must be established. If weaknesses in your program are later identified through assessment measures, these links will provide a guide for where improvements are required.

In our assessment plan at WCU, we identified three primary objectives for our master's program in I-O psychology. First, students who graduate from our program should have basic knowledge of I-O psychological theory and research. Second, we expect that students will gain research and statistical skills that will enable them to think critically and independently about research, analyze research data, and communicate results and conclusions through the use of statistics. Third, students should obtain professional skills that will allow them to practice and apply I-O theory and principles in field settings. You may wish to identify alternative or additional objectives for your individual I-O graduate program.

These three competencies became the core of our assessment plan. In each of these three categories, more specific objectives were then identified (See Figure 1).

In gaining a basic foundation in I-O theory and research, we expect students to be able to: (1) assimilate new psychological knowledge, (2) recognize ethical considerations, and (3) understand key theories and principles in I-O psychology.

By gaining research and statistical skills, we expect students to be able to: (1) understand published research, (2) use a wide variety of research methods and statistics, and (3) exhibit critical thinking.

By obtaining professional skills, we expect students to demonstrate: (1) an awareness of career options, (2) an ability to apply their basic knowledge to real-world situations, and (3) effective oral and written communication skills.

Assessment Options

Once you have identified your specific objectives and goals for students who complete your I-O program, the next step is to determine how those objectives will be assessed. At this point, you will have a number of options to consider.

Figure 1. Sample Competency Model for I-O Psychology Graduate Programs

First, you must choose whether your plan will include measures of "growth" or "achievement," or a combination of both. The distinction between the two is largely of function of the timing of the observation. Growth and development indicators require periodic observation while the student is progressing through the program. These can provide some opportunity for corrective feedback. Some examples of growth indicators might include internship supervisor evaluations, thesis or dissertation preparation, individual course grades, and research mentor evaluations. On the other hand, achievement indicators are appropriate for summarizing all of the experiences that reflect mastery of the required knowledge, skills, and abilities (CHE, 1996). Examples of achievement indicators might include comprehensive examination scores, successful thesis or dissertation defense, secure employment, and cumulative grade point average.

Second, your plan may also include a combination of direct and indirect measures of student achievement. Direct indicators may be derived from evaluation practices already embedded in your program. These might include completion of a research thesis or comprehensive examination performance, both of which are common to many I-O graduate programs. Less direct indicators may be derived from evaluation of components of the educational environment which are believed to contribute to successful completion of the graduate degree (CHE, 1996). Such measures might include peer interactions, seminar attendance, library use, membership in professional organizations, or clarity of career goals.

A similar consideration is the distinction between objective and subjective measures of achievement. Objective indicators will be derived from more formal testing and evaluation practices in your program, where student performance is largely a function of their own preparation and ability. In this case, quantifiable outcomes are easily obtained (e.g., exam scores, number of presentations at conferences) and comparison with local or national norms may be possible. More subjective measures will rely on the opinion of others in the educational environment. Because these types of indicators are subject to bias and inaccuracy, they should be regarded with caution.

Table 1. Examples of Assessment Measures for Required Competencies

Assessment Measures

As you begin to choose assessment measures, remember that the purpose is to identify whether your specific objectives have been achieved. Keep in mind that it is better to successfully measure a concise set of outcomes than to fail with a more ambitious set. Outcomes that can be assessed with reliable and valid measures are the basis of successful assessment plans. As with all research methods, it is best to measure each outcome in more than one way.

Table 1 provides examples of assessment measures for three competencies identified as critical for success in an I-O graduate program. It is recommended that you select two or three indicators from each category. The list of assessment options includes both growth and achievement measures, direct and indirect measures, as well as objective and subjective measurement options.

You should be prepared to defend your selection of measurement tools and to demonstrate the link between the required competencies and your chosen methods of measurement. For example, you should be able to demonstrate, particularly for observers outside of your specialization, that the internship supervisor's evaluation is an appropriate indicator of professional skill development.

Criterion Development and Implementation

The last step in the assessment plan is to establish specific criteria and determine how the measurement phase will be implemented. The criteria developed for your chosen measures will depend heavily upon personal judgment, but should reflect achievable goals and realistic objectives. If you choose to use comprehensive examination scores as your measure of achievement in basic foundations of I-O theory, for example, you might expect that 80% of students taking that exam will receive a passing grade on their first attempt. Other specific criteria might be developed that incorporate time to completion (e.g., students will complete their thesis within 3 years), ratings on Likert scales (80% of internship supervisor ratings will be at or above a 7 on a 9-point scale), or number of activities (e.g., students will present at least two papers at regional or national conferences).

You will also be asked to identify how these measures will be implemented. Again, specific details about your individual I-O program will need to be provided. For example, if you choose "number of successful job placements" as your measure of achievement in professional skills, you will be expected to identify how you will gather such data. In this case, you might wish to use an exit interview where students provide information about their employment status upon graduation from your program. On the other hand, if you choose to use internship supervisor ratings, you will be expected to demonstrate that a standardized form is available with which that information can be gathered and returned to your department.

The purpose of gathering these data is to draw conclusions about the effectiveness of an I-O graduate program. Following these procedures, you will be able to identify the strengths and weaknesses of your program. The process of using these results to change and improve your educational practice is the critical phase of the assessment plan. Your assessment program should become a cyclical process of measurement, analysis, refinement, and measurement. Ideally, the result will be a dynamic program that is responsive to feedback and changing in order to enhance student learning.

References

Commission on Higher Education (1996). Framework for outcomes assessment. Philadelphia, PA: Middle States Association of Colleges and Schools.

January 2000 Table of Contents | TIP Home | SIOP Home