A Potpourri of Cursory Thoughts on Evaluation
Robert Perloff
University of Pittsburgh
E. Jane Davidsons article, The Discipline of Evaluation: A Helicopter Tour for I-O Psychologists, in
TIP, October 2002, is very useful and generally on target. But as one who has been around the block once or twice in the field of program evaluation and evaluation research, I would like to add a few footnotes to Davidsons thoughtful article.
First, she refers to evaluation as a discipline. I hope she will forgive me for demurring that a discipline evaluation is not. Disciplines are systematic, coherent, founded more often than not on sound theory, and offered as programs in accredited colleges, universities, and professional schools. Evaluation, without detracting in the least from its multitude of contributions and creative authors and practitioners, is not systematic, coherent, theory-driven, and offeredoh perhaps with an exception here and thereas a program of study at institutions of higher learning. Evaluation is a helter-skelter mishmash, a stew of hit-or-miss procedures, notwithstanding the fact that it is a stew that has produced useful studies and results in a variety of fields, including education, mental health, and community development enterprises.
Theory
I have long maintained that program evaluation, which is hoisted on the shoulders of evaluation research, desperately needs theory. At the heart of this theoryand by theory I mean to showcase concepts and ideas and not
numbers, designs, or statisticsshould be the nobility of experimentation, of trying things out, of the daring that should insinuate itself into the maelstrom of personal problems, education, community development, and social welfare thrusts into the idealism of trying to make the world and its institutions better. Don Campbell wrote about this in his experimenting society.
Any theory of evaluation must take into account the natural inclination of individuals or undertakings that understandably resist efforts at having themselves evaluated. What works, for example, in persuading people to risk being exposed for failing ideas or enterprises? How do we encourage people to welcome and not resist feedback? What are the ways that are efficacious in reducing perceived threats to ones ideas and ones initiatives lest they be viewed as vulnerable houses of cards? To be evaluated is to be subjected to the oversight of others; others who by definition are on a higher plane and who can say that what you are doing is flawed and vulnerable. This is at the heart of evaluation and any theory of evaluation needs to address this issue.
Evaluation Research Versus Program Evaluation
As suggested above, what should come first is evaluation research and then, only then, program evaluation. How can a program be fairly and enduringly and replicably evaluated absent the foundation that research provides? Evaluation research is an activity which explores and creates and modifies the methodology and designs that must be the hallmarks of decent, sophisticated program evaluation projects. On the other hand, program evaluation is, as the phrase says, the evaluation of a particular program. Is this method of intervention better than that? Is this treatment more enduring and lasting than that? Does this means of educating minority kids represent an improvement over what is now being done? Program evaluation, in a word, is the evaluation of an operational entity, an entity which will not be funded anew unless data attesting to its efficacy are forthcoming. Programs that are the targets of evaluation are very often multimillion dollar programs funded by laws of Congress, and are subject to review by the mother of all program evaluators, the General Accounting Office of the United States.
Formative Versus Summative Evaluation
Formative evaluation is the process whereby an activity that is under development is shaped, molded, and transformed into an entity that was originally envisioned as a social, educational, personal, or educational program. The evaluator uses his bag of evaluation tricks to determine how the program is shaping up. Does this nascent program look like what its creator intended it to be and to look like? If not, slice it here, glue it differently there, twist it into the shape it was intended to assume in order to benefit its intended beneficiaries. There is a strong place for formative evaluation in the panoply of evaluation enterprises and strategies. It might well turn out to illustrate that an otherwise promising program will go down the toilet because it was not built to its architects precise specifications, and formative evaluation can therefore prevent the occurrence of Type II error. On the other hand, summative evaluation is bottom-line oriented. Does it work? Does it save money? Does it prolong life? Does it increase safety? Does it reduce delinquency? Yes or no. Up or down. In or out.
Knowledge Utilization
An outgrowth of program evaluation is a field that has come to be known as knowledge utilization. The question arises as to how to implement the core essentials of a program that has weathered the storm of program evaluation and has shown itself to be a winner. What do you do to make sure that further forays into a particular domain avail themselves of the benefits of a particular program whose evaluation has been demonstrated via a rigorous program evaluation test of the programs efficacy? For example, if certain procedures for encouraging gays to pursue safe sex are shown, via control groups and all that jazz, to in fact reduce HIV infections because the men are pursuing safer sex, then the question arises as to how those protocols can be widely and pervasively disseminated in order to have a broadly salubrious effect. How can the knowledge shown by the evaluation of this program that this procedure workshow can that knowledge be utilized?
Program Evaluation and I-O Psychology
The above ruminations have been presented, I know, with no reference to their applicability to I-O psychology. My aim here was to shed a little light on the basics of evaluation research and program evaluation, leaving it to the intelligence, wisdom, and experience of savvy members of SIOP to divine how program evaluation and evaluation research can be positioned into their particular programs, whether they be programs designed to determine the outcomes of casual dress on Fridays, the effects of paintings on the walls of employees in businesses and industries, the usefulness of mentors from large corporations who venture forth into ghetto areas to help deprived boys and girls raise their vocational sights in order to increase the likelihood that they will be able to cavort amid the blessings of the American Dream, or to see what the outcomes of flextime or of daycare facilities on worksites have been.
January 2003 Table
of Contents | TIP Home
| SIOP Home