Harold A Edgerton

This story of a career in "Industrial Psychology" is organized around sets of events rather than a chronology of career events. This should let the reader understand more clearly the patterns of thinking, activities and opportunities which were the basis of that career.

To help to understand some of these related but separated events, a resume is included as Appendix A. This shows the various positions, employers and dates which describe the framework of my career. I have been a pragmatist or empiricist rather than a theorist. Both Dr. Herbert A. Toops, under whose guidance I obtained the doctoral degree and Dr. Albert P. Weiss, a gentle but hard-nosed behaviorist, had influences in my adopting the essence of the behavioristic approach to problems. Even as an undergraduate I accepted McCalls dictum, that anything which exists, exists in quantity, and that quantity is capable of measurement. This includes direct and indirect measurement and also use of surrogates. My activity was more in the direction of organizing a job according to the ends sought rather than as a means of developing or validating a theory.

Most of my career has been built around a core of "measurement"-- measurement of achievement, measurement of aptitudes, measurement of skills, measurement of attitudes, and analysis of the interrelations of such measures. It has involved teaching of undergraduates and graduate students. It has included vocational and career counseling of people from high school age up to 65 or older. Training of counselors was part of this program. Research, which has always been a substantial component, has included development of research tools and experimental design as well. There has been consulting work with different industries and government departments, as well as industrial research per se.

My work led me into a variety of industries. Chemical manufacturing, public utilities, electrical manufacturing, mining, paper and pulp, trucking, and oil well equipment, as well as many U.S. government departments such as Army, Navy, Marine Corps, Air Force, Labor, General Services, and Health, Education and Welfare. I've been asked if it took special knowledge of the techniques of the various industries, but as far as I could see the questions I had to deal with were "people problems."

I have served as an expert witness on the use of tests for selection and promotion for a government agency and for a labor union.

I have been unusually fortunate in my associations, either as student or as co-worker, with so many giants of the period especially in my earlier years in psychology. These experiences were invaluable, both in terms of what I learned from them and in terms of the support, motivation and opportunities they provided.

My career choice was the culmination of factors such as family, finances, training, experiences, opportunities and persons who influenced my attitudes and opened doors for me.

My original choice of career was medicine. Four of my father's brothers practiced medicine and I saw it as a good kind of career-offering service to others with social approval based on science. I do not recall that I thought of it except as general practice. The only medical specialty I knew about at that time was in the treatment of eye, nose and ear problems. At one time, when working in a music and jewelry store, I thought seriously about becoming a watchmaker. When I commented on this to my high school chemistry teacher she seemed horrified at the idea, but gave me no reason for her feeling on the matter.

Immediately after graduation from high school I went to Kansas State Normal, later Kansas State Teachers College and recently Emporia State University. This choice had about three elements in it. It was the school attended by both of my parents; it was probably more economical to attend than any other school in the state, and my father was enrolled there for summer school, thus starting me off with free room rent.

I was responsible for earning almost all of my college expenses, so I had to have some sort of job. I started as a dishwasher in a restaurant, and soon was able to carry the title of waiter. Later I obtained a job in a local jewelry store, keeping it clean and doing some of the simpler repair work on watches and clocks.

At the end of my first college year I failed the final examination in French, and this proved to be a turning point in career choice and development. My teacher, Mme Dudley, decided I needed a better job so I could study more. Through her I was hired as shipping clerk in the school's Bureau of Educational Measurements at the rate of 35 cents an hour. In addition I was given a re-test for the failed final--the construction of every word in four pages of "Le Voyage De Monsieur Perrichon," which I passed with flying colors and received a B in the course. My job duties included keeping track of test supplies on hand, filling orders for tests, packaging them for shipment and making scoring stencils for the Army Alpha Test.

Within a few months, another position became available with some such title as research clerk, which paid five cents per hour more than the job of shipping clerk. This job required that I score tests, make scoring keys, and compute means, standard deviations, correlation coefficients and other needed statistical measures. I could do the statistical work before I knew the formulas, and before I had any clear idea of the uses and limitations of such measures. This brought some knowledge of how tests were made, how used, and what the related costs might be. It was in fact an introduction to measurement. The most important skill learned from this job, and maybe from my undergraduate training, was the operation of a Monroe calculator. This skill opened so many opportunities to me that it has seemed to be my most effective "entering wedge" into psychology.

Another set of factors which had an impact both on vocational choice and on early development was the influence of my "honorary foster parents." This term I applied to individuals who went out of their way to give me encouragement and suggestions, to push me ahead and to open doors of opportunity for me. Dr. Dean A. Worcester, then Director of the Bureau of Educational Measurements, was one. A month or so before I had hoped to graduate, he suggested that since he was going to Ohio State University to complete work for his PhD degree, why shouldn't I go too and get my graduate studies started. At that time I'd never thought seriously about graduate study. I had to beg off because I lacked one hour of credit for graduation and also was shortof money. Dr. Worcester got E. R. Wood to arrange for the needed additional credit, and then obtained all the part time work at Ohio State that I could handle. As time went on, he would recommend my services to others, really stretching my skills to live up to his advertising.

A second "honorary foster parent" was Dr. Ernest R. Wood, then an instructor at Kansas State Teachers College. He was also working on his dissertation (University of Chicago) which included a graphic method for computing partial correlations. He hired me to operate the calculator for him, helping him calculate the necessary values for drawing his graphic devices. This not only improved my facility with the machine, but it also opened my eyes to the possibilities of graphs and monographs for computing. Both of these men did much to push me toward a career in psychology and measurement.

These two men developed the first statewide educational achievement contest. It was held at Emporia. Kansas in May of 1923. High school students came from all over the state to take the tests to see who could score highest in various school subjects and which schools had entered the most effective teams for the intellectual competition. Students took their tests in the morning and at three in the afternoon assembled to hear the announcement of the winners. I was chief clerk for the operation. That meant that my staff and I had to have all scoring done and checked in time for the awards ceremony.

A third "honorary foster parent" was Dr. Herbert A. Toops of Ohio State University. He opened the door to my first full time "professional" job as statistical assistant for the study of mechanical aptitudes at the University of Minnesota. This contributed to my concept of the nature of aptitudes, and to the idea of objective behavioral criteria for measurement of performance and for the validation of tests. In addition to this job, I had the opportunity to take seminars with Dr. Karl Lashley and Professor Donald G. Paterson. Each of these, too, left a substantial imprint on my thinking.

A second large door which Dr. Toops opened for me was an opportunity to work for Dr. Lewis M. Terman at Stanford University in the summer of 1926. Since Dr. Terman was teaching elsewhere that summer, he left me one page of single spaced typewritten instructions telling me what he wanted me to do on his study of the masculinity and femininity of personality. In the summer of 1927, I went back to Stanford to work on this project while taking a course in statistics with Dr. T. L. Kelley and another course with Dr. Harold Hotelling. These experiences increased my self confidence and added well known names to my list of references. In addition there were several very capable graduate students who made an impression on me. Among them were Robert Bernreuter, Jack Dunlap, Floyd Ruch, Albert Kurtz, Philip Rulon, Harry Harlow and Barbara Burks.

A third door which Dr. Toops opened for me was that of willingness to try new ideas and to look for new approaches. I shall never forget, for example, the consternation in his statistics class when the assignment was "Develop a new statistical formula. Assignment due in two weeks." Completion of this assignment became one of my first published articles, a derivation of a formula for the computation of the average of a set of intercorrelations without computing any one of them. Then there was the afternoon when I argued that a concept presented by Dr. Toops was wrong. I could not put a finger on why I believed it wrong and finally gave up. The next morning I found the flaw in his presentation and so reported to him. That was the only time that he severely criticized me. His criticism was that I had not stayed with my argument the previous day.

III. COUNSELOR TRAINING PRACTICUM

In 1937, soon after returning to Ohio State University from the Occupational Research Program (ORP) of the U. S. Employment Service, I met with a small group concerned with the vocational guidance of young people, especially those seeking jobs and not in school. One result was to organize a vocational counseling practicum for graduate students in cooperation with the Junior Division of the Ohio State Employment Service Office In Columbus, Ohio. This was one of the earliest practicum training programs for counselors dealing with young people trying to find jobs. It must be remembered that this was a depression program. It had urgent reality and as such offered qualified graduate students an unusual opportunity to participate in supervised on‑the‑job training in vocational counseling. Graduate students in psychology, education. sociology and social administration were included. This program existed for several years, until war conditions changed the employment picture to one in which employers were actively recruiting workers.

The program included counseling interviews with clients, use of test data, conferences on job hunting, motivation and encouragement. It also included a case board each Saturday morning in which each counseling trainee reviewed his case for discussion and advic6 from the staff and his fellow trainees. The courses received credit via a minor problem course number in psychology. In addition, under the same course number an Occupational Information course was offered. This included occupational classification, descriptions of the more common opportunities open to young persons, a look at wage scales, employer requirements and the use of the new Dictionary of Occupational Titles. Later, in order to regularize the situation, I requested that the College of Education approve three core courses to be included in any program for vocational/career counseling. These included the counseling practicum, and the occupational information courses. The third proposed course was designed to cover the kinds of community Information needed for such counseling, how to obtain it, organize it and keep it up to date. The college curriculum committee approved the counseling practicum only, and recommended the inclusion of two courses in John Dewey's educational philosophy instead of the other courses. I do not know of any students who followed the latter recommendation. Nonetheless, several students in the practicum program later became directors of University counseling centers in large midwestern universities.

About this same time I was appointed Counselor in the College of Arts and Sciences, with special attention to students who were enrolled but had little idea of their career goals. These were designated "Exploratory Students," and were permitted to try various subject matter areas to explore their content and meaning to the student rather than following all of the junior division requirements. Aptitude tests and interviews were used extensively to aid this exploratory process.

Thus the University was becoming more aware of the need for more career information and making it available to counselors and students. In 1941, the Occupational Opportunities Services (OOS) was organized and I became its director. Its primary purpose was to aid the advisors in the various colleges on questions of occupational information. At first no counseling was done by the organization, since that was the function of junior division counselors in each of the colleges; but we did administer special aptitude tests at the request of the college counselors and on request of individual students. Soon we were "interpreting" the test results. One early product was the publication "Ohio State and Occupations," a book reporting for each department of the University the kinds of jobs and careers its graduates entered.

Soon after this service center was organized World War II was upon us and the University was asked to organize a Star Unit to assign enlisted military personnel to appropriate academic training programs on the basis of tests, previous academic history, and Individual preference. I was asked to take over the program and have it ready to operate in about ten days. The program was organized, personnel employed, tests purchased and space obtained. Then the War Department told us to dismantle it. One week later they asked the University to put together their form of such a program and have it ready to operate within one week. The only difference between the4version of the program and ours was that military personnel made the academic assignments; we used the same space, same personnel, same tests as we had planned earlier.

As the war came to a close the Occupational Opportunities Service was given responsibility for a vocational counseling contract with the Veteran's Administration. The counselors were mature, experienced personnel who wanted to continue their graduate studies. Better counselors couldn't be had, they were excellent.

IV. THE SCIENCE TALENT SEARCH

The Science Talent Search for the Westinghouse Science Scholarships was an important part of my career. Early in 1941, Dr. Stuart H. Britt, Office of the American Psychological Association, called me and explained that there was a matter in Washington, D.C. which could be of interest to me. My expenses would be covered if I would come. On my arrival, the two of us had breakfast together where I learned that the Westinghouse Educational Foundation was offering support to Science Service In taking over Science Clubs of America, and that the program to give new life to Science Clubs of America would be a search for science talent among the high school seniors of the nation.

At 9:30 that morning, we met with representatives of the organizations concerned, and outlined a program for selection. We proposed the following selectors:

A "Science Aptitude Examination," a college aptitude test in science clothing.
A "Personal Data Blank," an anecdotal recommendation asking what the student had done to show his status in such areas as creativity, work habits, scientific attitude, ability to work with others, leadership, etc.
A transcript of the student's high school academic record.

We were asked to prepare the selection program, construction of the Science Aptitude Examination and the Personal Data blank. The plan was to operate on a successive hurdles arrangement. The first hurdle was to complete the entrance requirements. All data were required for each student before he became an entrant in the Search. The second hurdle was the examination and those who came successfully over that were evaluated on their high school transcripts. These "hurdles" have been used in the searches since 1941. The third hurdle was the score on the Personal Data bank. Following this, at least two scientists read and rated the project reports of the surviving contestants.

Each year about 2,000 to 4,000 high school seniors entered the program with 300 of these being listed as "Honorable Mentions" in the Search; from this number 40 became the "Winners" whose expenses were paid to come to Washington, D.C. There, they were interviewed by each member of the Board of Judges before the final awards were decided. Interviewing these contestants was a stimulating experience. It also afforded one a view of the growth in high school curricula and community opportunities available to such talented young people. One factor adding greatly to the validity of selection was the selection ratio, in this case choosing 300 from among the 2,000 to 4,000 entries. With such data available for 2,000 to 4,000 outstanding high school seniors each year, follow up studies were needed to tell how effective the selection procedure had been, to describe the educational and career development of such talented young persons and to offer information which might make better counsel and guidance possible for them. The few studies completed showed that the winners (top 40 each year) made on the average greater academic and professional advancement than did the rest of the Honorable Mention group and the latter in turn had greater success than the other contestants. Selection procedures were modified both in form and in use over the years. About the second year the project requirement was changed to "My Scientific Project" and required the contestant to report on work that he had done. It was also found that it made no essential difference whether a teacher or the contestant filled out the Personal Data blank, since it was a factual behavioral document rather than a set of evaluative statements. The 29th Annual Search (1969) was the last to use the aptitude examination since our data showed that it no longer seemed to be refining the selection. The areas referred to in the Personal Data blank have been modified from time to time in the light of the judges' opinions about which furnished the more useful Information and what information about the contestants seemed to be lacking.

The program is continuing and I hope that it goes on for many years, seeking out the enormous pool of scientific talent among the seniors of our high schools and giving credit to many outstanding and dedicated teachers. Probably leading into scientific careers many young people who had not considered such a possibility.

V. OCCUPATIONAL RESEARCH PROGRAM--U.S. EMPLOYMENT SERVICE

One direct outcome of my experience in the study of mechanical aptitude at the University of Minnesota was an invitation, ten years later, to join the staff of the Occupational Research Program of the U.S. Employment Service. There my duties included research design and statistical analysis in a psychological measurement operation.

One program was the development and validation of trade questions as an aid to employment interviews. The questions were designed and validated in terms of the extent to which they could differentiate those craftsmen who were highly skilled from those of less skill and experience who were trying to "muscle in." Questions were obtained by field interviews with craftsmen. To become USES trade questions substantially more craftsmen had to answer the questions correctly than did helpers, trainees and others. To select valid items, we used a simple statistical procedure based on the overlap of the score distributions from the two samples.

A second area was in the design and treatment of data for the "Department Store Study." This was undertaken to develop valid measures of on-the-job performance of sales personnel in department stores as a basis for validation of selection tests. The primary study used a large store in Baltimore. Later the findings were cross validated in department stores in two other large cities. First of all we wanted to measure sales performance in terms of recorded evidence of on-the-job performance rather than in terms of ratings by supervisors. These included such data as total dollar value of sales, value of goods returned, the selling of "specials" and errors in making out sales slips. Then there was the question of how to compare sales performance of persons in one department, as in notions, with those in another department, as in ladies' ready-to-wear. The study assumed that the quality of sales from department to department was about equal, and that variability of sales performance from department to department were about equal. On such bases the score for each salesperson was the ratio of his own performance record to the mean of his own department.

Then we faced the question of how the several evidences of quality of sales performance could beat be combined into a single index which would be a fair measure of overall sales performance. One could average the scores for each salesperson (expressing the scores as standard scores). Another concept, and the one we finally used, was that the best overall index of sales performance should be a weighted sum of the several variables for which the weights were those obtained by minimizing the sum of squares of the differences among all pairs of performance measures.*

*Edgerton, Harold A. and Kolbe, Laverne. The Method of Minimum Variation for the Combination of Criteria, Psychometrica 1, Sept. 19369 183-187.

A very interesting phenomenon which we observed was that of the changes in performance scores of individual sales personnel over the years. The index rose rapidly during the first five or six years on the job, then started a very slow decline over a period of many years. Such a decline was not enough to suggest "coasting in" to retirement but rather that job performance had become routinized. Since that study I have not seen so careful and objective a measurement of job performance; although I have in many cases sought such objective behavioral data.

VI. ON APTITUDES

From the time when I was first introduced to objective testing (in 1921, as a high school senior), I have been much interested in the nature of aptitudes, how they were acquired, and how they could be used to help people gain greater satisfaction in living. My participation in the Minnesota study of mechanical abilities* had contributed several concepts. The central problem had been one of obtaining a sound, acceptable and objective measurement of mechanical ability and then exploring the validity of the available mechanical aptitude tests, especially their validity relative to a performance criterion.

*Paterson. D. Go . Rlliott, F. M. , Anderson, L. D., Toops, H. A. , Heidbreder, E. , Minnesota Mechanical Ability Tests, University of Minnesota, Minneapolis, 1930.

In constructing the performance criterion of mechanical ability we assumed that those who possessed greater amounts or degrees of aptitude would perform more effectively on sample mechanical tasks, and that those who performed less well on those same tasks had less aptitude. The subjects for the study were junior high school boys. Each boy normally took a series of five shop courses: woodworking, sheet metal work, printing, electricity and mechanical drawing. For purposes of the study each shop course introduced a series of projects which all students were required to do, such as making a wooden game board, a metal cookie cutter, a wired electrical circuit, a drawing of one of the projects, and the setting of type for a news item. Mechanical ability was assumed to be reflected by how rapidly and how precisely each boy completed each project. Measures used were those utilized by the shop teachers in their subjective evaluation of performance, as squareness of the game board, accuracy of dimensions, uniformity of chamfers on each edge, alignment of holes and smoothness of finish.

problems of measurement. For example, how to measure the quality of a Western Union splice--a splice required in the projects of the electrical shop. To construct a scale for evaluating the splices made, we obtained a sample of some 200 splices made by the boys included in the study. These splices were rated for quality by a number of raters. Then we selected some ten of them representing degrees of quality, ranging from poorly done splices up to those which had been really excellently done--somewhat analagous to the handwriting scales then in use to furnish standards for rating samples of handwriting. With such a large number of measures there was need to combine them into a single index. For want of some estimate of which characteristics were more important, and because each measurement reflected a facet of the quality of performance, the score for each boy for each project was the sum of his standard scores of all the measurements and ratings given for his project.

The aptitude tests were mostly improvements of the then available tests. For example, the Minnesota Assembly test had 15 items as compared with the Stenquist, an earlier test using 10 items. Other tests as well were lengthened to improve their reliability and revised where needed to give greater currency to their content. One test which attracted my attention was a "Things Done" checklist, a list of things which a boy might have done such as wiring a light socket, planning a board, splicing two electric wires, soldering, setting rubber type, making a stamp pad, etc. The score was the number of items checked. And this "Things Done" score correlated higher with the mechanical ability criterion than any other "aptitude" test! One might conclude that a person who had high mechanical aptitude would have done many more mechanical tasks because he had the aptitude. Another possible conclusion was that aptitudes were learned skills and that score on the aptitude tests reflected a learning curve for that group of skills.

I have found the latter definition very useful in vocational and educational counseling. It also seems to be more in alignment with our concepts of schools as a place in which to learn new skills.

I remember the late Dr. H. H. Goddard stating that, given enough time, anyone can learn anything, and that even a moron could learn calculus; given enough time, but seldom would live that long.

This concept has important implications for career counseling. For many years we have been administering aptitude tests and telling people that they had certain aptitudes and lacked others and should make their vocational /educational choices in line with the test score patterns. This approach has a limiting finality about it. One young man, a freshman, had a low college aptitude test score, so low that I told him that his test score indicated that his chances of completing college were no more than 1 in 13 because he lacked or was weak in those skills which were needed for academic success. This approach did not kick him in the teeth. It did tell him that his present skills were inadequate for the task of academic work. This man accepted my verdict as a dare rather than a daunt, and we spent the rest of our time looking at what he could do to raise his chances for satisfactory college grades from 1 in 13 to some more acceptable level. The ultimate result was that he did graduate from college. In further exploration of the significance and patterns of aptitudes, I asked all of my special counselees in the College of Arts to take the Thurstone Primary Abilities Test. These special counselees were students who had entered college with little or no idea of their vocational goals. At the end of the college year, these test scores were correlated with their marks in various courses.*

*Ellison, Mary Lou and Edgerton, Harold A., The Thurstone Primary Mental Abilities Tests and College Marks. Educational and Psychological Measurement, Vol. I, No. 4, October 1941.

Should Theory Precede or Follow a "How-to-do-it" Phase of Training - ONR, 1956, Edgerton.

The Inductive Reasoning factor showed its strength in marks in Romance Language courses. We learned that these courses were being taught primarily by inductive methods. The Deductive Reasoning factor showed its strength in its correlations with marks in Psychology and Zoology. These courses were taught using a deductive approach.

Later, under a study sponsored by the Office of Naval Research, the relationship of aptitude pattern and teaching method was explored further. The study was made in the beginning course in Meteorology with cooperation of the staff of that school. About half of the sections of the course were organized so that the practical or "how to" parts came as early as possible in the course. In the other sections the theoretical concepts were presented first with the "how to" put as late as possible in the course. All trainees in these courses were given the Thurstone Test of Primary Mental Abilities. The correlations with the memory factor were highest for the courses with "how to" at the early part while the reasoning factor showed the highest correlations with course grades for those with theory first.

There remains much to be done in this area. At present our teaching assumes that teaching method is determined by the subject matter to be taught rather than by the characteristics of the learners. I believe that efforts to teach according to the measured aptitudes of the student would greatly increase our success ratio.

VII. TESTS AND TEST CONSTRUCTION

Many of the positions I have held have entailed some test construction as well as the use of tests.

My first introduction to the large scale production of tests was assisting Dr. Herbert A. Toops in the construction of Forms 6 to 26 of the Ohio State University Psychological Examination. This required using rigid rules in the selection of materials, styles of items, and difficulty indices for test items. It involved trying out proposed test materials on a sampling of college freshmen to determine item difficulty and to note non-functioning answer alternatives. It also required that we determine the equivalence from form to form of the tests, so that test data would be comparable from form to form, year to year and person to person.

A somewhat different approach was used in constructing 29 successive forms of the "Science Aptitude Examination" which was used as one step in selection of winners for the Westinghouse Science Talent Search. Some of the materials tapped science information, and others involved reading paragraphs drawn from science publications with questions requiring that the contestant be able to read and to reason from these materials. Basically it was an academic aptitude test dressed in science clothing. It was not really necessary that the several forms be comparable since each was used for only one year. Each form of the test was made public after it had been used, thus it was necessary to produce entirely new test items each year. One of the most difficult parts of the test construction was to develop materials sufficiently difficult to discriminate among the more able of the contestants. For most of the test forms the frequency distribution was skewed with the upper fifty percent of scores covering a much shorter range than those of the lower fifty percent.

A markedly different kind of test we developed at OSU was a test of Supervisory Judgement. For this we tried a multiple choice format, requiring the identification of a best answer and a worst answer among the alternatives for each question. This seemed to work quite effectively in breaking up the pattern of having a "textbook answer," or using cues inadvertently included in the construction of the questions. This format was used in a training program for Quality Assurance personnel. Since the trainees were experienced in Quality Assurance rules and procedures, the crucial question was how to build an examination which would legitimately measure their knowledge of the course content at the beginning of the training session and again at the end. The best and worst answer scheme seemed to be quite effective. The use of the examination also proved to be a good method of getting these experienced people to pay attention during the class sessions and to do their homework.

Another interesting job we undertook was to adapt some commonly used Mechanical Aptitude tests for use in Venezuela where an oil company was setting up a training program to train Venezuelans in some of the technical crafts of the oil field including diesel mechanics, welding, and pipefitting. Requirements for entry to these training programs called only for graduation from common school, about our fifth grade level. (Interestingly, no one from Caracas was permitted to apply. It was assumed that city dwellers would not like to live and work out in the oil field.) There actually were 150 qualified applicants for 15 training slots--an excellent selection ratio. We used such tests as the Paper Form Board and Test of Mechanical Comprehension. We did visit a number of villages to see what kind of mechanical content we could find boys involved in. We also talked to many of the Venezuelan employees asking about their mechanical experiences. As far as we could see there were no indications that tests used in the USA for similar purposes could not be used here.

However, because the applicants had little or no experience with objective form tests we took steps to insure that they understood what they were to do. Directions were rewritten in simplified English and these were then translated into Spanish. To administer the test the cover page of directions was enlarged to about 3 x 5 feet so that it was viewable by all who were taking the test, As the examiner (a Spanish speaking Italian engineer) read the directions aloud he would point to the words and diagrams in the directions. This procedure seemed to work very well. In fact these Venezuelans with only a common school education seemed to know better how to work on the Paper Form Board test than did North Americans at the 12th grade or higher level.

In a situation like this, the company did not feel that it was necessary to have a tryout and validation of test materials. Knowing of our experience with the same tests in the USA and with such a high selection ratio. (150 applicants for 15 openings) they felt that they should be able to get good quality trainees, and it must be noted that they seemed much pleased with the final selection.

Other test construction programs I have been involved in were construction or critique of various tests for military use and as achievement examinations for college courses involving military personnel.

A technique which has proven very useful in construction of criterion measures and tests could be called "looking through the eyes in position to see." A good example of this was the development of the Ohio State University Flight Inventory. This work was done under a research contract with the National Research Council aimed at developing measures of progress in learning to fly. The time was just prior to World War II, when young college men were given the opportunity to learn to fly and were trained through a Civilian Pilot Training Program (CPT) to the level of earning their private pilot license. There was a lack of uniformity in the instructor ratings of the performance of these students both from time to time and from instructor to instructor. With Dr. Robert Y. Walker, our approach was to systematize the kinds of observations used by flight instructors in grading or rating the performance of their student pilots. We dealt only with flight performance, not with learning in ground school. It was assumed that flight instructors could report on what the student flyer did on any given flight with considerable accuracy and reliability. The discrepancies in instructor ratings seemed to rest in the differences in standards for rating any given item of performance, such as how the student pilot performed in a climbing right turn soon after take-off. These differences in standards seemed to vary some from time to time for the same instructor and also from instructor to instructor.

Our program set out to develop a set of standard tasks and maneuvers for all of those required to obtain a private pilot license. We developed a check list for each required maneuver so that an instructor/observer could report his observation of how well the student performed each of the required maneuvers. The observer's check list was based on the instructor's knowledge of how the maneuvers were to be performed, common errors made and degree of errors made. A series of maneuvers, constituting a check flight, was used as a basis for an over-all performance report or final examination by the flight instructor. The inspector's flight check for a private license was done by the inspector in his own fashion. Dr. Walker learned to fly and obtained both his private license and his single engine commercial license in learning how a flight instructor sees the performance of his student.

The same technique--looking at a job situation through the eyes of those in position and skilled enough to see--was the first step in many studies, especially those relating to criteria for selection of tests or to development of performance reports.

Another area which raises interesting questions lies in the effects of "cramming" by practicing on material similar to the aptitude or selection tests used. Does the Increase in aptitude test score obtained by such means reflect a real increase in aptitude and in later on-the-job performance? In the academic world, training on vocabulary and interpretation of paragraph material commonly used for academic aptitude tests, does enable students to obtain better grades in their courses. If such rapidly learned material increases one's aptitude as well as aptitude test score, we may have the question in counseling, "What aptitudes does your vocational choice call for?" rather than "How well do your aptitudes match those which are considered essential to the chosen field?"

VIII. THE CRITERION

Ever since I entered the field of Psychology, I have heard about the "criterion problem." The problem is that of determining what evidences can properly be used and how they may be combined in construction of a measure of performance--a dependent variable--by which effectiveness of the Independent variables or predictors may be measured or judged. It is usually a question of what evidences or behaviors are available for the purposes of the study. Seldom is it possible to approach the question in any other way. This section is not intended as a review of the "criterion problem" but rather as a look at some experiences in the search for recorded behavioral evidence which might contribute to a clearer, more valid estimate of the quality of on-the-job performance. In my extensive work on the development of criterion measures for validation of selection tests, I have not found an oversupply of really valid criterion measures. In most cases the best criteria available have been ratings of the immediate superior. Reliabilities of such ratings varied considerably. Sometimes reliability was too low for the rating to be an acceptable criterion, while in other situations it was very good. When the reliability has been too low, interview and question procedures have been used to obtain better ratings from supervisors. In the Minnesota study of mechanical aptitudes the criteria used were constructed from measurements and ratings of a standard set of shop products. This Minnesota study did much to shape my attitude toward the nature of aptitudes. These attitudes were further molded by a study, reported in the 1920's, comparing the selection of tests when validated on the basis of supervisory ratings to tests selected, when validated using worker production records as criteria.

Later, when constructing and validating selection tests or performance reports for industry, I tried to search out any behavioral records which might shed more light on performance. In a study of nine foremen, we had quite reliable ratings by supervisors but there seemed to be some behavioral factor trying to crop up. Finally we asked that each first line foreman keep a record of work delays on his shift each day. He was asked to indicate the reason for the delay but did not record the man-hours lost. I was assured that work delays due to the railroad failing to bring in cars to load couldn't be the fault of a foreman, nevertheless the foreman recorded all work delays. At the end of 30 days the number of work delays for each foreman was counted. We demonstrated that the number of delays due to the railroad was unrelated to the rest of the work delays and independent of the supervisor's ratings. These railroad delays were promptly put back into the work delay score of each foreman. Had we not done this, I'm sure I would have had requests/demands that I remove other categories of work delays, leaving possibly no significant variance. The number of work delays for foremen correlated -0.44 with the average of ratings by immediate superiors and nine superintendents !

Another source of information in quality of job performance of foremen might be found in distribution of wage increases. Insofar as wage increases differed among the foremen, they must represent some estimate of differences in quality of performance. So the measure average wage increase over the previous five years" was added as a criterion variable. The final criterion used for the validation of tests to select foremen included all of the above variables.

Another interesting bit of information came to light in a study of on-the-job performance in two samples of chemists in the same plant. Discussion of possible behavioral criterion variables with the Director of Personnel and Training covered such possible variables as number of experiments performed, average salary increases, absenteeism, number of papers published, etc. None of these seemed to offer any useable data since there seemed to be no uniformity in the situation relative to any of them. We did obtain performance rankings by superiors which had acceptable reliability. Then our contract monitor suggested that there were some job analysis, data for all of the chemists which had been obtained a few months earlier by the ratio-delay method. We reclassified the observations for each chemist into two categories. Either he was engaging in some professional or technical work, or else he was doing some non-technical activity such as washing laboratory glassware, smoking, visiting, going to the restroom, etc. The non-technical score for each chemist was tried as a measure. In each of the two samples. this variable showed a substantial negative correlation with ratings by superiors. The superiors had not seen the data which was collected for job description purposes. This kind of measurement would be risky or impossible to use in most plants, since the workers would probably regard it as a "Big Brother" spy situation. Misrepresentation of what each was doing at the moment each observation "What are you doing now?" was called for would be encouraged.

Consultants find that there is a danger of looking at criterion data not acceptable to a client. In starting to evaluate a vocational counseling program for the Department of Defense, I proposed that staying with a vocational training choice might be as good a criterion as we could find. Did the man complete his training course? Did he pursue the same field of training or work after he became a civilian? Was he still in that vocational field six months after discharge? At that point I was assigned a new contract monitor, a clinical psychologist who was horrified at such concepts.

There is room for much exploration of objective behavioral evidences of quality of on-the-job performance. With increased use of such objective criteria, it Is possible that tests for selection, hiring interviews and qualifications for the job might change. Along with this. as there are changes in the social structure of the work situation, one can expect some changes in the qualities thought most desirable in employees.

APPENDIX A

RESUME

HAROLD A. EDGERTON

Birth:

1904, Russell, Kansas

Education:

Humboldt High School. 1921, Humboldt, Kansas
B. Sc., 1924, Kansas State Teachers College, now Emporia State University
M.A.0 1926, Ohio State University
Ph.D., 1928, Ohio State University
Additional graduate study at University of Minnesota and Stanford University

Experience:

Part time student assistant. Bureau of Educational Measurements, Kansas State Teachers College, 1922-1924. Duties involved test construction and evaluation, statistical computation, assisting with state scholarship contests. Under direction of Dr. D. A. Worcester and Dr. E. R. Wood.

Research Assistant, Mechanical Abilities Investigation, University of Minnesota, September. 1924-June, 1925. Assisted in the development and evaluation of tests. compilation of criteria of mechanical ability. direction of statistical work. Under direction of Professor Donald G. Paterson.

Graduate Assistant, Department of Psychology, Ohio State University, June, 1925--August, 1928. Duties included administration and scoring of intelligence tests, statistical consultation and computation. Put together 20 sets of mechanical assembly tests for use in vocational guidance. Assistant to Dr. Herbert A. Toops.

Research Assistant, Department of Psychology, Stanford University, Summers 1926 and 192?. Development of tests of non-intellectual traits. Work done under the supervision of L. M. Terman, Stanford University.

Research Assistant, Department of Psychology, Ohio State University, 1928-1930. Assisting with various researches on vocational guidance, college personnel, intelligence tests, and personality rating; consulting with staff members and graduate students relative to statistical procedures of their researches; teaching courses in statistics, vocational guidance, general and educational psychology.

Assistant Professor of Psychology, Ohio State University, 1930--1941. Duties similar to those listed above but with greater responsibility for their initiation and conduct, and in addition, direction of researches of graduate students, vocational guidance counseling, assisting in statewide testing and guidance program of the Ohio College Association.

General supervisor-Statistical Unit, Occupational Research Program United States Employment Services Washington, D.C., January 1, 1935--October 1, 1936. Statistical Consultant to the Standards and Research Division, United States Employment Service in Winter Quarter 1937. Responsibilities for statistical aspects of studies of worker qualifications and job opportunities. Construction of criteria, designing new statistical techniques, and supervision of statistical staff (on leave from Ohio State University).

Counselor, College of Arts and Sciences, 1937--1941. (Also carried title and duties indicated as Assistant Professor.) Development of Exploratory Program, guidance and testing as part of the junior division services of the College. Established a Placement Bureau for the College.

Director, research project for the National Research Council on Development of Criteria for Use in the Study of Selection and Training of Pilots, 1939--1942. (In addition to regular employment.)

Director, Occupational Opportunities Service, Ohio State University, 1941--1945Direction of program of vocational counseling, aptitude testing, and occupational studies for students of Ohio State University.

Associate Professor of Psychology, Ohio State University, 1941--1945, consulting in regard to statistical aspects of student researches. teaching courses in counseling, personnel, and statistics.

Civilian Education Advisor, Star Unit, 154th S. U.. April, 1943--April. 1944. (Part of duties as Director of the Occupational Opportunities Service.)

Director, Test Construction Project at Ohio State University for the Army Specialized Training Program, August, 1943-April, 1944.

Consultant to Science Service, Inc., Washington, D. C., on the Westinghouse Annual Science Talent Search. Developed and administered selection processes and served as Chairman of the Board of Judges in selection of winners of the Science Talent Search, 1942-1969.

Expert Consultant to Secretary of War, 1945-1946. Consultant with Personnel Research Section, AGO, on research design and procedures.

Professor of Psychology, Ohio State University, 1945-1947. Consulting in regard to statistical aspects of student researches, teaching courses in counseling, personnel, and statistics.

Vice President, and later President, Richardson, Bellows, Henry & Co.. Inc., New York, New York, 1947-1962. Directed studies related to performance, evaluation and personnel selection in various industries; maintained other professional activities, contacted government agencies in seeking contracts Involving research, consulting and training and directing work on those contracts.

Adjunct Professor of Psychology. New York University, 1948-1949, conducting graduate seminar on measurement in Industrial Psychology.

Performance Research, Inc., Washington, D.C., 1962-1970, President. Sought contracts related to research and consultation on personnel selection, training, and performance evaluation, and directed staff in work in those contracts.

Adjunct Professor of Psychology, University of Maryland, 1962-1964. Conducted seminars on Industrial and Personnel Psychology.

Other Professional Activities:

Member of Special Committee to study education/training at Staff and Command School, 1946-19470 Ft. Leavenworth, Kansas.

Deputy Civilian Member, Research and Development Committee, DOD, an Training and Training Devices, 1948-1955Member, Committee on Professional Certification, N. Y. State Psychological Association Committee on Professional Certification. Member. American Psychological Association Committee on Construction of Code of Ethics. Member, American Psychological Association Legislative Committee Chairman .1967-1968.

Member, FFA Committee to Study Facilities of its Academy.

Member, DC Psychological Association Committee on Standards for Psychological Services. American Psychology Association, 1970--1975.

Member, Board of Cooperating Editors, Educational and Psychological Measurement.

Member, 1963-1966, and President, 1966-1969, Committee on Certification of Counseling Agencies, American College Personnel and Guidance Association. Member, American Psychological Association Committee on Standards for the Delivery of Psychological Services, 1970--1975. Professional Societies and Recognitions: Diplomate, 1951, American Board of Examiners in Professional Psychology. Memberships: American Psychological Association President. Division 14, Industrial and Business Division, 1953-1954. President, Division 17, Consulting Psychology. Psychometric Society; President 1947. American Association for the Advancement of Science. Eastern Psychological Association.

Honorary Societies:

Sigma Xi (Science)

Pi Kappa Delta (Forensic)

Phi Delta Kappa (Education)

Alpha Psi Delta (Mychology)

Pi Mu Epsilon (Mathematics)

APPENDIX B

PUBLICATIONS

HAROLD A. EDGERTON

1. Edgerton, Harold A.--What Acquaintances with Statistics is Necessary? Educational Research Bulletin, January 6, 1926: V: No. 1.

2. Edgerton,Harold A. and Paterson, Donald G.--Table of Standard Errors and Probable Errors of Percentages for Varying Numbers of Cases, Journal of Applied Psychology, September 1926: X: 378-391.

3. Edgerton, Harold A.--An Abac for Finding the Standard Error of a Proportion and Standard Error of
the Differences of Proportions. Journal of Educational Psychology, February,1927.

4. Edgerton, Harold A. and Toops, Herbert A.--A Table to Facilitate the Calculation of the Probable Error of the Mean. Educational Research Bulletin, November 9, 1927, VI: No. 16.

5. Toops, Herbert A. and Edgerton, Harold A.--An Abac for Determining the Probable Correlation over a Longer Range Knowing it over a Shorter One. Journal of Educational Research, December, 1927.

6. Edgerton, Harold A. and Toops, Herbert A.--A Formula for Finding the Average Intercorrelation of Unranked Raw Scores without Solving Any of the Individual Intercorrelations. Journal of Educational Psychology. February, 1928, XIX: 131-138.

7. Edgerton, Harold A. and Toops, Herbert A.--A Table for Predicting the Reliability and Validity Coefficients of a Test When Lengthened. Journal of Educational Research, October, 1928, 18: 225-234.

8. Edgerton, Harold A. and Toops, Herbert A.--Academic Progress: A Followup Study of the Freshmen Entering the University in 1923. The Ohio State University Press, 1929, PP. 150-x.

9. Edgerton., Harold A.-Measuring the Validity of Predicted Scores. Journal of Educational Psychology, May, 1930, 388-391.

10. Edgerton, Harold A.-Academic Prognosis in the University (PhD dissertation); Warwick and York, 1930, PP. 83+vii.

11. Edgerton, Harold A.--A Table for Finding the Probable Error of R obtained by the use of Spearman-Brown Prophecy Formula (n = 2). Journal of Applied Psychology, XIV, June, 1930, 296-302.

12. Edgerton, Harold A.--Intelligence Tests of Delinquents, Journal of Higher Education., Vol. 1, 161-162, March 1930.

13. Toops, Herbert A. and Edgerton, Harold A.--The Second Annual Statewide Intelligence Testing Program of Seniors in Ohio High Schools. Ohio High School Bulletin,, No. 4, O.C.A. Committee on Intelligence Tests, March, 1931, 31p.

14. Toops, Herbert A. and Edgerton, Harold A.--Opportunities in Ohio Colleges, Guidance Manual No. 3,, State Department of Education, 1931, pp. 203.

15. Edgerton, Harold A.--A Graphic Method of Finding Standard Errors and Probable Errors of Differences. Journal of Educational Psychology, XXIII, January, 1932, 56-57.

16. Edgerton., Harold A.--Recording and Reporting, Rev. of Educational Research, 3, June, 1933, 205, 208.

17. Edgerton, Harold A.--Characteristics of Pupil Population, Review of Educational Research, June, 1933, 209-213.

18. Edgerton, Harold A.--A Formula for Finding the Average Correlation of Any One Variable with the (n-1) Other Variables Without Solving Any of the Individual Correlations. Journal of Educational Psychology, Mays 1935, 373-376.

19. Edgerton, Harold A.--Psychological Improvements in Statistical Processes -Paper read at meeting of American Statistical Association. December 30, 1935.

20. Kolbe, Laverne, and Edgerton, Harold A.--A Table for Computing Biserial r. Journal of Experimental Education. March, 1936, 245-251.

21. Edgerton, Harold A. and Kolbe, Laverne E.--The Method of Minimum Variation for the Combination of Criteria. Psychometrika 1, September, 1936, 183-187.

22. Edgerton, Harold A.--A Note on the Computation of the Correlation Between Two Lengthened Tests. Journal of Experimental Education, March, 1937, p. 289.

23. Edgerton, Harold A. and Starbuck, Edmund O.--A Table to Aid in the Computation of Fisher's "t" Function for Comparison of Two Means. Psychometrika, 2, Vol. 2, June, 1937.

24. Edgerton, Harold A., Shartle, C. L. and Otis, J. L.--The Department Store Study, ORP, Department of Labor, 1937.

25. Edgerton, Harold A.--Table of Values for Transforming Values of r Into Fishers z Correlation Function. Printed by Irving Lorge.

26. Edgerton., Harold A.--J. P. Guilford, Psychometric Methods, A Review, Journal of Applied Psychology, XXI, No. 6, December 1937, p. 726.

27. Edgerton, Harold A. and Valentine, W. L.--A Factor Analysis of Learning Data; Psych. Pulletin., 1938, 32, 719.

28. Edgerton, Harold A.--The Exploratory Program: Report of the Sixteenth Annual Meeting of the American College Personnel Association, 1939, 133-136.

29. Kurtz, A. K. and Edgerton, Harold A.--Statistical Dictionary, 1939, John Wiley and Sons, 191 + xi.

30. Edgerton, Harold A. et. al., Statistical Studies of a College Freshman Testing Program, Journal of Experimental Educations VIII, No. 3. 247-292.

31. Edgerton, Harold A.--The Exploratory Programs Journal of Higher Education. November, 1938, X No. 8, 421-424.

32. Edgerton, Harold A., Bordin, Edward., and Molish, Herman--Some Statistical Aspects of Profile Records. Journal of Educational Psychology, March, 194I, 185-196.

33. Edgerton, Harold A.--E. F. Lundquist, Statistical Analyses in Educational Research, a Review, Am. Jour. Psych., July 1941, 54, No. 3, 471-472.

34. Ellison, Mary Lou and Edgerton, Harold A.--The Thurstone Primary Mental Abilities Tests and College Marks. Educational and Psychological Measurement, Vol. I, No. 4, October 1941.

35. Edgerton, Harold A.--Women in Science Careers, Journal of The National Association of Women Deans and Counselors, Vol. XXV, No. 4, June 1942, 166-9.

36. Edgerton, Harold A.--How Science Talent Winners Were Chosen, Told by Judge. Science News Letter, July 25, 1942, pp. 54-55.

37. Edgerton, Harold A. and Thomson, Kenneth F.--Test Scores Examined With the Lexis Ratio. Psychometrika, December, 1942. 281-8.

38. Edgerton,Harold A. and Britt, Steuart Henderson--The First Annual Science Talent Search. American
Scientist, January, 1943, Vol. 31, No. 1.

39. Edgerton, Harold A. and Britt, Steuart Henderson--How Science Talent Search Winners are Chosen. Science and the Future, 1943, pp. 112-115.

40. Edgerton, Harold A. and Britt, Steuart Henderson--Further Remarks Regarding The Science Talent Search. American Scientist, 1943, Vol. 31, No. 3.

41. Edgerton, Harold A. and Britt, Steuart Henderson--The Science Talent Search, Occupations, The Vocational Guidance Magazine, December 1943, 177-180.

42. Edgerton, Harold A. and Britt, Steuart Henderson--The Science Talent Search, Reprinted from Occupations in The Education Digest, IX, No. 6, Feb. 1944, 37-39.

43. Edgerton, Harold A. and Britt, Steuart Henderson--The Third Annual Science Talent Search. Science, April 21, 1944, Vol. 99, No. 2573, pp. 319-320.

44. Edgerton, Harold A. and Britt, Steuart Henderson--Sex Differences in The Science Talent Test. Science, September 1, 1944, Vol. 100, No. 2592, pp. 192-193.

45. Edgerton, Harold A., Britt, Steuart Henderson and Davis, Helen M.--Is Your State Discovering Its Science Talent? Science Education, October, 1944, Vol. 28, No.4.

46. Edgerton, Harold A--A Symposium, Ohio State University, The University, The G. I. Bill and the Veteran. School and College Placement, December, 1944.

47. Edgerton, Harold A. and Britt, Steuart Henderson--Science Talent in American Youth. Science, March, 1945, Vol. 101, No. 2619, pp. 247-248.

48. Edgerton, Harold A. and Walker, Robert Y.--History and Development of the Ohio State Flight Inventory, Part I: Early Versions and Basic Research. Civil Aeronautics Administration, Division of Research, Washington, D. C., Report No. 47, July, 1945.

49. Edgerton, Harold A. and Walker, Robert Y.--History and Development of the Ohio State Flight Inventory, Part II: Recent Versions and Current Applications, 1945.

50. Edgerton, Harold A. and Britt, Steuart Henderson--The Science Talent Search in Relation to Educational and Economic Indices. School and Society. March 9, 1945. Vol. 63, No. 1628. Pages 172-175.

51. Edgerton, Harold A.--Ohio State and Occupations. The Ohio State University Press, 1946.

52. Edgerton, Harold A. and Britt, Steuart Henderson-- Technical Aspects of the Fourth Annual Talent Search. Educational & Psychological Measurement, Vol. 7., No. 1, Spring 1947, p. 3-21.

53. Personnel Research and Procedure Branch TAGO - Survey of the Educational Program, The Command and Staff College, 1947.

54. Edgerton, Harold A., Britt, Steuart Henderson and Norman, Ralph D.-Objective Differences Among Various Types of Respondents to a Mailed Questionnaire. American Sociological Review, Vol. XII, No. 4, Aug. 1947, 435-444.

55. Edgerton, Harold A., Britt, Steuart Henderson and Lemmon, William B.-Reliability of Anecdotal Material in the First Annual Science Talent Search. Journal of Applied Psychology, Vol. 31, No. 4. Aug., 1947, 435-444.

56. Edgerton, Harold A.--Counseling & Its Relation to Total Personnel. Ohio Nurses Review, October 1947, p. 142-148.

57. Darley, J. G., Alexander, W. B., Bailey, H. W., Cook, W. W., Edgerton, H.A. and Vaughn, K. W.--The Use of Tests in Colleges. American Council on Education Studies, Series VI, Student Personnel Work, No. 9, XI, Dec. 1947, pp. 86.

58. Edgerton, Harold A., Britt, Steuart Henderson and Norman, Ralph D.--Physical Differences Between Ranking & Non-Ranking Contestants in the First Annual Science Talent Search. American Journal of Physical Anthropology, December 1947, p. 435-52.

59. Edgerton, Harold A., Britt, Steuart Henderson and Norman, Ralph D.--Later Achievements of Male Contestants in the First Annual Science Talent Search. American Scientist, Vol. 36, No. 3, Summer 1948.

60. Berkshire, J. R., Bugental, J. F. T., Cassens, Frank P. and Edgerton, Harold A.-Test Preferences in Guidance Centers. Occupations, The Vocational Guidance Journal, 1948, p. 337-343.

61. Edgerton, Harold A.--The Place of Measuring Instruments in Guidance, pages 51-58 of Measurement of Student Adjustment and Achievement, Donahue, Wilma T., Coombs, Clyde H. and Travis, Robert M. W. (Ed) University of Michigan Press, 1949.

62. Edgerton, Harold A. and Britt, Steuart Henderson--The Annual Science Talent Search for the Westinghouse Scholarships, Transactions of The New York Academy of Sciences, February 1949, p. 118-120.

63. Fryer, Douglas H. and Edgerton, Harold A.--Research Concerning "Off-the-Job Training.", Personnel Psychology, Autumn 1950, Vol. 3, No. 3, p. 261-284.

64. Committee on Ethical Standards for Psychologists, Vol. 1. The Code of Ethics, American Psychological Association, 1952, pp. 16.

65. Committee on Ethical Standards for Psychology, Ethical Standards for Psychologists, Vol. 2, Source Bock of Ethical Problems, Incidents, and Principles, American Psychological Association, 1952.