What is I-O?

Industrial-organizational (I-O) psychology is the scientific study of the workplace. Rigor and methods of psychology are applied to issues of critical relevance to business, including talent management, coaching, assessment, selection, training, organizational development, performance, and work-life balance.

Informed Decisions: Research-Based Practice Notes

Steven G. Rogelberg
Bowling Green State University

The World Wide Web (WWW), in time, may dramatically alter how we as scientists/practitioners survey, train, recruit, select, and disseminate information to employees. To effectively use the WWW for these work-related purposes, a number of methodological concerns and issues must be broached. To that end, I have asked Jeff Stanton to author this quarter’s Informed Decisions column. Jeff very effectively addresses practical considerations and constraints when using the WWW for personnel recruiting and selection. If you have any comments/questions concerning Jeff’s column please contact him at stanton@bgnet.bgsu.edu. If you have any ideas for future columns or would like to propose authoring a column, please contact me at rogelbe@bgnet.bgsu.edu.

Validity and Related Issues in Web-Based Hiring

Jeffrey M. Stanton
Bowling Green State University

An examination of the human resources trade literature indicates that the World Wide Web (WWW) has found burgeoning application to the tasks and problems of personnel recruiting and selection (Bell & Sutton, 1998; Greengard, 1998; Mottl, 1998). Market research projections predict that an overwhelming majority of medium and large U.S. organizations will utilize web-based recruiting by 2002 (Hodges & Rodgers, 1998). Optimistic promotional materials claim that the Internet provides methods to cut costs and increase the efficiency of the entire hiring process. Minimal research has been conducted, however, on the validity and utility of using the Internet for hiring tasks.

The options for online recruitment and selection methods have escalated with every new quarter. Electronic employment bulletin boards provide a medium for widely distributing both job advertisements and resumes. Robotic web crawlers use keyword matching and network search algorithms to track down eligible resumes (Oullette, 1998). Automated resume handling systems collect resume text, sort and categorize applicants, and send out their own acknowledgement letters (Meade, 1998). Turnkey software packages for managing the whole hiring process over the WWW integrate some or all of these functions and add applicant testing to the mix (see Appendix). Each of these techniques performs one or more of the typical functions of recruiting and selection: locating and attracting job seekers, screening job seekers prior to other selection hurdles, and selection testing.

Several important issues arise in using the Internet for selection purposes. Each of these issues impinges upon practitioners’ choices of whether and how to use the Internet for selection. In keeping with the mission of this column, I will discuss the methodological implications of each issue: data collection needs, practical design advice, and avoidable pitfalls. First, because of the demographic skew among Internet users, Internet-based hiring efforts could raise substantial applicant flow problems. Second, the use of web-based interviewing, screening, or testing techniques requires at least the same degree of careful validation that more traditional selection techniques entail. A closely related concern is that the uncontrolled nature of WWW interactions may create substantial opportunities for faking and cheating. Finally, the collection of personnel information such as test scores and biographical data over the Internet raises some security and personal privacy concerns that need to be addressed.

This article provides a brief, I-O driven overview of each issue. In each case, I attempt to reference the available research that can inform intelligent practice. Although the WWW itself is a new development, prior research on computer-based testing, faking, web user demographics, and privacy can provide at least a few insights into the present situation. This review will also reveal that considerable additional research needs to be accomplished.

Unequal Employment Opportunity on the Web

Efforts to ensure equal employment opportunity for protected classes of applicants may become more difficult when using web-based recruiting and selection strategies. The situation has gradually improved, but the demographics of Internet users are still substantially skewed. On-line Internet usage surveys (see Appendix) converge on the findings that WWW users in the U.S. are mainly white, male, and younger than 40. Although opponents of affirmative action have dismantled legislation in some states, many organizations still have affirmative action plans that depend upon attracting sufficient proportions of minority and female candidates. These plans may become more difficult to fulfill using a web-based recruiting strategy because the proportion of protected class job seekers that become applicants may diminish as web-based recruiting enlarges the total pool.

Recent research (Foderaro, 1998) also suggests that applicants in certain job markets, such as K–12 teachers, infrequently consider the WWW as a useful tool for the job search. In all likelihood, this finding crosses over to any job market where computer usage and computer skills are relatively peripheral. My efforts to sample the most frequent categories of job listings from one job search web site uncovered 2,947 job listings over a 10-day period. The same source provided 15,118 categorized resumes posted during a 120-day period. Table 1 shows percentages by category and clearly indicates the predominance of computer programming and engineering jobs for both job listings and resume postings. This evidence suggests that, at the present time, web-based recruiting may be appropriate only for a limited set of positions. Not coincidentally, these positions correspond strongly with the regular use of computers on the job.

Table 1: WWW Job Listing and Resume Percentages by Category

Source: http://www.careershop.com

The demographic skew and the technical/computer bent of the on-line job market interact to worsen the EEO picture. Minorities and women are underrepresented in highly technical fields and underrepresented in the pool of Internet users. If affirmative action plans are an organizational priority, then human resource practitioners must plan carefully for how the use of web-based recruiting will change the applicant mix. The methodological implications are straightforward: The organization’s human resources webmaster should try to ensure that specialty sites for minority and women job seekers have links leading to the organization’s hiring site. In addition, the hiring web site should include a voluntary affirmative action survey, so that HR can track who views the site, who seeks additional information, and who actually applied using on-line techniques. Finally, organizations should avoid using the web as the sole medium for recruiting.

On-Line Validity

Assuming one has solved the applicant flow problem, the next link in the on-line hiring chain becomes screening and/or testing applicants. McBride (1998) provides an excellent and brief introduction to computerized selection testing which, though it does not mention the WWW, maps neatly onto this domain. McBride identifies two major applications of computers to selection testing: computer conversions of printed tests and tests developed specifically for computer administration. In the former category, organizations such as Procter and Gamble have experimented with cognitive tests, personality tests, and biographical inventories implemented on computers. In the latter category, organizations such as Ford Motor Company have developed computerized test batteries that tap psychomotor abilities, situational judgment skills, troubleshooting ability, and numerous other unique domains of human performance. Both categories of testing can be implemented for WWW usage, and the proliferation of software companies offering such products indicates that organizations are beginning to do so.

In general, the problems encountered by creators of computerized tests become exacerbated by implementation on the WWW. I will discuss why this is so in reference to the following three issues. For personality, honesty, and biographical data testing, faking has always been a matter of concern (e.g., Zickar, 1997). Whether driven by social desirability bias, or other motives, faking may become a larger problem when applicants complete an inventory on the WWW. For cognitive testing, the difficulty of converting paper and pencil instruments to computerized formats is compounded by item and test security problems. Finally, while new tests developed specifically for computer delivery have perhaps the greatest promise for web-based usage, one must carefully weigh the costs of development of such tests.

Faking by Browser

My recent research on WWW responses to attitudinal measures provides encouraging results about the factor equivalence of web-based psychological measures (Stanton, 1998). At present, however, this research has not been extended to web-based cognitive or personality tests. Thus, I instead examine prior research on the equivalence of computer-based measures. For example, some research evidence suggests that administration medium does not adversely affect the measurement equivalence of personality measures (King & Miles, 1995). This optimism is tempered, however, by other research that indicates that responses to computer-based assessments can vary depending upon respondents’ beliefs about anonymity (e.g., Kantor, 1991).

Respondents usually control the time and place of (self) administration of web-based materials. This control may provide a psychological sense of anonymity. Most respondents presumably understand, however, that assessments completed for selection purposes are identifiable. This contrast is reflected in available research on administration mode and degree of socially desirable responding. Although some research found a lower degree of socially desirable responding on computer-based assessments (e.g., Kiesler & Sproull, 1986), Lautenschlager and Flaherty (1990) found that computer-based administration of personality assessments resulted in a greater degree of socially desirable responding than paper and pencil assessments. Specifically, their results suggested that respondents increased their efforts at impression management when responding by computer. Research by Allred (1986) also documented higher impression management in computer versus paper and pencil administration of a personality measure.

These results suggest that faking on web-based, noncognitive tests may have an effect on scores. At least four methodological implications for practitioners arise from this supposition. First, when converting a paper and pencil inventory to the web, confirm the validity of the instrument in the new administration mode with additional research. Second, validate such web-based, noncognitive selection tests on motivated respondents (i.e., actual applicants who have a stake in the results and thus strong motivation for impression management). This point argues against a purely concurrent strategy. Third, choose items based on response transparency: for example, choose items that do not have obvious best answers. Empirical keying against good performance criteria may help support this effort (because high faking can suppress the predictive power of an item). Fourth, use such tests for selection of individuals for whom impression management skills may be a desirable attribute (e.g., sales personnel; see Hogan, 1988). Utilizing some or all of these approaches apparently has the potential to provide valid and useful noncognitive tests. Reports from the field suggest that validity coefficients between r = .20 and r = .50 can be achieved with web-based biographical data and noncognitive competency assessments (N. J. Mondragon, personal communication, October 27, 1998, see Appendix for AspenTree URL).

Ability or Computer Ability?

In the area of cognitive testing, Carretta and Ree (1993) reported distributional differences in test scores of pilot candidates who took either a paper and pencil or computerized version of a cognitive selection test. Scores had to be transformed to equate the two tests. Likewise, Kovac (1990) found that computer administration of a cognitive selection test slowed applicants down relative to those who took a paper version. Scoring differences have been found both on computerized test batteries that did not require typing proficiency (Kublinger, Formann, & Farkas, 1991) and on those that did require good typing skills (Cockrell, 1991). McBride (1998) underscores these issues with descriptions of the pitfalls of translating speeded tests into computerized formats.

These results also have methodological implications for practitioners. As previously stated, any existing cognitive test that is translated into a computer format should have its validity freshly documented in the new medium. Implementing speeded tests on the web is ill advised. Besides the technical problems documented by McBride (1998), the web also suffers from transmittal delays that make the delivery and timing of tests extra difficult. Thus, cognitive testing over the web should be limited to power tests, at least until software to carefully control remote test timing becomes both cheap and sophisticated. Finally, even when a significant validity coefficient signals the utility of a web-based test, one should assume that norms and cutoff scores will require equating to paper and pencil versions of the same tests.

A Web of Lies

Concerns for validity and test equivalence, however, fail to capture the full complexity of web based applicant screening and testing. As I argued in Stanton (1998), administering psychological measurement instruments on the web provides one of the least controlled environments for responding. Unless web-based assessments are administered in controlled environments (e.g., walk-in centers, supervised kiosks, etc.), the applicant has a large degree of control over the administration of the test. This control raises the triple threat of applicant identification, cheating, and test security.

Applicant identification is a critical problem in web-based selection procedures. Research on cheating and plagiarism by college students suggests that a disturbingly high percentage of students have represented the work of others as their own (e.g., Hale, 1987). Insofar as many web-based hiring efforts are directed at recruiting and testing college students, this finding warns that the user of a web-based selection system may not be the actual job applicant. In a related vein, an applicant may have a coach nearby, informational materials pertaining to the test, or even a set of items gleaned from previous administrations of the test. With these advantages, a respondent could easily cheat on a cognitive test.

Each of these issues has methodological implications for practitioners. First, from a test security standpoint, cognitively oriented measures of aptitude and achievement simply cannot be freely published on the web. Instead, organizations must use an access control system (e.g., passwords for one time use), to help ensure that applicants do not preview test items. Likewise, specialized software for presentation of items (i.e., Java rather than HTML), can help to prevent printing or easy copying of test content. The use of adaptive testing may also help to maintain item security (Overton, Harms, Taylor, & Zickar, 1997). Finally, two approaches can be used for applicant identity verification (N. J. Mondragon, personal communication, October 27, 1998). First, prior to starting the test, the applicant should provide some pieces of verifiable personal data (perhaps a randomized list of specialized information such as mother’s maiden name). Second, the test content should overlap to a certain extent with later face-to-face selection procedures.

Multimedia on the Web

Coovert (1995) discusses the impact of new technologies on office jobs and, at one point in his chapter, mentions the possibility of transforming everyday computer tasks into selection tools in the form of work sample tests. Coovert’s discussion, along with McBride’s (1998) chapter, highlight one of the areas of great potential for web-based selection: tests of psychomotor skills and other specialized skills and abilities that are difficult to tap in multiple choice format. Some organizations have already put advanced technology to work in the form of video-based assessments (e.g., Dalessio, 1994; Smiderle, Perry, & Cronshaw, 1994). McHenry and Schmitt (1994) provide an excellent overview of some of these efforts along with recommendations. Implementing the same techniques on the WWW has the same pitfalls that McHenry and Schmitt highlight, plus the technology hurdles of reliably delivering suitable quality video and audio over the Internet. Software vendors have recently made substantial progress in overcoming these hurdles such that the incremental validity provided by multimedia selection techniques may soon be worth the programming costs of putting such assessments on the web.

Privacy and Security Considerations

No discussion of the Internet would be complete without a consideration of privacy-related issues. Although APA ethical guidelines provide guidance on the handling of test results, these rules only govern psychologists, and the operation of selection web sites will typically involve the efforts of many nonpsychologists. Selection sites will inevitably start collecting social security numbers, personal and demographic information, responses to biographical data inventories, results of cognitive tests, and numerous other pieces of sensitive, personal data. In addition, in today’s litigious employment environment, selection procedures often include background checks (and these can now be accomplished automatically on-line). Collecting and using these data may include multiple transmissions over the Internet, storage in vendors’ web servers, storage in the organization’s servers, and review by personnel outside of human resources (e.g., information technology personnel). At each stage, the potential for information "leakage" exists and must receive careful consideration by human resources professionals who design and use on-line screening and selection tools.

Numerous legal cases have underscored the importance of communication by organizations concerning privacy expectations for personnel information (e.g., Bohach v. City of Reno et al., 1996; State v. Bonnell, 1993). These cases strongly indicate that organizations must take the initiative by communicating appropriate and accurate expectations for privacy. One clear methodological implication is that selection web sites should contain disclaimers that explicitly tell applicants what happens to their data, who has access to their data, and the degree to which their data and their involvement in the hiring process are kept private and confidential.

Summary

This article has reviewed research on computer-based testing, faking and cheating, the demographics of web users, and other topics relevant to putting the hiring process on the web. This research, though limited, was useful for developing a set of methodological recommendations. Market indications suggest that web-based recruiting and selection will continue to grow rapidly over the next few years. Industrial-organizational psychologists have critical roles to play in assuring the validity and integrity of these efforts to apply technology to these essential human resource management activities.

References

Zickar, M. (1997, April). Computer simulation of faking on a personality test. In G. Alliger, Faking matters. Symposium conducted at the 12^th annual conference of the Society for Industrial-Organizational Psychology, St. Louis, MO.

Appendix: Web Sites Used in Researching This Article