Description

Position Summary

The Director of Assessment (DOA) will be responsible for overseeing the full cycle of certification exam development, new exam product development, exam and program maintenance, and evaluation of exam performance. This work will involve close cooperation with volunteer committees of subject matter experts. The goal will be to maintain the high level of integrity and reputation of the AWS Certification Examinations, and to improve the development, maintenance, and assessment functions as much as possible.

Essential Duties

• Lead and manage the Assessment Group. Build an effective team. 
• Serve as the Secretary to the Qualification and Certification Exam Bank (QCEB) volunteer subcommittee and subject matter experts (SMEs) to develop and maintain personnel certification programs. 
• Strategically plan the short-term and long-term assessment roadmaps for the Assessment Group and the Certification Department. Align the Assessment team goals with AWS goals.
• Implement the Assessment Industry best practices. The activities include but not limited to:

1.      Lead job task analysis (JTA) and validation process for target personnel certification program. 
2.      Lead and support certification program design. 
3.      Define test specifications, lead exam development, beta test, and establish the psychometrics. 
4.      Evaluate and revalidate existing programs to align with best practices. If deemed necessary, sunset a program in collaboration with the Q&C and subcommittees, and cross-function team members. 
5.     Participate and support in the development of personnel qualification standards and personnel certification standards.

•         Conduct psychometric analyses of certification exams with the goals of:

1.      Perform item and exam analysis.      
2.      Perform equating to ensure exam form equivalence. 
3.      Identifying breaches of exam security.    
4.      Lead key validation. 
5.      Be responsible for designing and maintaining score reports.  
6.      Present updates about certification quality for the stake-holders. 

•         Respond to candidate inquiries pertaining to their exam performance and score interpretation. 
•         Manage the relationship with vendors and collaborate with them for smooth running of computer-based testing. This includes regularly meetings to communicate and solve issues related to online item authoring system, secure share of confidential materials, exam setup, publishing, new features of systems, score report and meet project timeline. 
•         Manage the contract of vendors for psychometric services and translation services. 
•         Have business acumen and keep AWS business needs as a priority when making decisions with assessment work, vendors, and projects. 
•         Support the Managing Director, other groups of the Certification, as well as departmental projects. 

1.      Budgeting; 
2.      Qualification and Certification Main Committee activities; 
3.      Setting SMART Department Goals and achievement them. 
4.      Evaluation and choice of third-party testing agencies;
5.       Resolve or assist in resolving exam administration issues or nonconformities;
6.      Support Certification Operations with their administration and implementation of the certification programs:
7.      Provide advice and guidance to other groups and departments.

•  Actively participate in the assessment industry conferences such as annual meetings by Institute of Credentialing Excellence and Association of Test Publishing. 
•         Collaborate with Information Science and other departments. 

1.      Continuously evaluate systems in order to obtain best psychometric results.
2.       Provide guidance in core system upgrade.

Supervision Exercised

•         Exam Developer I
•         Exam Developer II
•         Data Analyst, Certification

Requirements

Required Knowledge, Skills & Abilities include:

•                     Quantitative, statistical, and psychometric data analysis. 
•                     Critical thinking. 
•                     Proficiency of Classic Test Theory and Item Response Theory
•                     Project management.
•                     Test development. 
•                     Group consensus building. 
•                     Strong communication skills. 
•                     Software: statistical software, Office 365 suite.

 Required Qualifications, Education and Experience

•         Advanced degree in psychometrics, educational measurement, quantitative psychology, statistics, or related field. 
•         Minimum 7 years’ experience in the Assessment Industry, preferably experienced in industry credentialing organization.
•         Minimum of 5 years’ management experience. 
•         Experience with leading subject matter expert groups, designing testing programs, and leading psychometric activities including job task analysis and standard setting.

Preferred Licenses or Certifications

                    Certification and certificates in psychometrics and credentialing

AWS Competencies

Customer Success 
Business Acumen
Uncompromising Integrity
Build effective team
Widespread Respect for People 
Collaboration
Diplomacy 
Communicates Effectively
Managing Ambiguity 
Continuous Improvement Mindset
Results Oriented