The Psychometrician will be responsible for professional research and validation of the Division’s examinations, and for guiding professional test development, administration, analysis and scoring activities.
The Psychometrician will serve as a special project leader, working with various members of the Division’s test development and test administration staff to plan and carry out validation studies and testing initiatives. Key projects currently planned include Statewide validation of entry-level and promotion examinations for uniformed services (police, fire, corrections, etc.) and development of a competency modeling system and assessment tools for the State’s professional supervisors, managers, and administrators. To be effective, both projects require the guidance of a professional Psychometrician, preferably one with experience in conducting both traditional KSA-based job analysis and competency-based studies.
The Department’s high-volume, high-stakes tests may be subject to legal scrutiny and judicial review, and the Psychometrician’s testimony will be a critical element in defending the testing program.
The Psychometrician will also play a key role in advising Division and Department managers on issues related to assessment, selection, and testing.
Under the general direction of the Director of Testing Services, the incumbent will be responsible for providing guidance and expertise in the design, analysis, and validation of tools and tests created and/or used in personnel assessment and selection by the New York State Department of Civil Service.
The Psychometrician will work with professional staff in the Testing Services Division in carrying out these duties and may personally supervise a small contingent of staff. The Psychometrician will advise the Director of Testing Services and the Department executive team on high-profile issues, such as the validity, reliability, and defensibility of assessment and selection strategies; research methods and techniques; test development and validation techniques; statistical and other analytic tools and processes; and professional and legal standards; and other assessment-related topics.
Incumbent responsibilities may include but are not limited to:
· Lead personnel assessment and accountability-related psychometric research for a large-scale workforce selection and development program
· Design, plan, and conduct a variety of test validation studies and research projects
· Coordinate competency modeling and traditional job analysis studies as a basis for personnel assessment and employee evaluation programs
· Coordinate development of valid and legally defensible personnel assessment and evaluation strategies for selection, retention, employee development, and performance management
· Guide development and administration of job-related tests in a variety of types and formats, including written, multiple-choice, situational judgment, job knowledge, job simulation, constructed response, evaluation of training and experience, oral, as well as structured interview and performance evaluation systems and similar tools.
· Conduct ongoing statistical analysis and validation of existing tests and assessment programs
· Provide analysis of special populations and assist in ensuring that test items are free from statistical, racial, gender, cultural or other bias
· Ensure tests and assessments comply with legal and professional standards (e.g., Uniform Guidelines on Employee Selection, Standards for Educational & Psychological Testing, Principles for the Validation and Use of Personnel Selection Procedures, etc.) and industry best practices
· Prepare technical documentation on test validity sufficient for legal inquires and in accordance with the Uniform Guidelines on Employee Selection Procedures
· Conduct test equating analyses and demonstrate equivalency in test forms
· Develop and train staff in test and measurement concepts, statistical analysis and test validation techniques
· Align tests and assessments with organizational goals and performance expectations
· Advise executive management on psychometric and technical assessment and selection issues, programs, and trends