Behind The Curtain: ASHRAE Certification Exam Development

From eSociety, October 2017

Is it surprising that the recently updated BEAP and OPMP certification exams don’t have a passing score established yet? It might seem unusual, but even for established certification programs like these, determining a passing score is just one step in a rigorous process designed to ensure a valid and reliable certification exam.

Before exam questions are even written, an Exam Subcommittee, whose members are appointed by the ASHRAE Certification Committee, conducts a survey of certified professionals on the continued importance of the job tasks currently tested on an exam. The Exam Subcommittee then reviews the survey data to determine if a job task should be retained or removed to better fit the requirements of current job practice.

After finalizing what job tasks should be tested, the Exam Subcommittee weights the importance of the information and assigns a cognitive level—Recall, Application or Analysis—to the job tasks.

Once this is completed, the Exam Subcommittee knows what should be tested, how many questions to write and at what cognitive level. For instance, the recently revised OPMP exam section “Costing for Building Components” contains eight questions, of which two are Recall, four are Application and two are Analysis.

Writing certification exam questions, in particular valid and reliable ones, is as challenging as it sounds! Not only must questions test the correct job tasks and be at the right cognitive level, but also the correct answer must be the best answer, and the “distractors,” or incorrect responses, must be plausible enough to distract, yet clearly not be the correct answer. Over time, once there have been enough exam administrations, the Exam Subcommittee evaluates question analysis data to understand how well the exam questions distinguish between passing and failing examinees and if the distractors work.

So, how is the passing score determined? Exam Subcommittee members weight the difficulty a “minimally competent” practitioner would have with selecting the correct answer for each question on a new exam, calculate the average, and that average becomes the preliminary passing score. Once enough exams have been taken, the Exam Subcommittee evaluates the preliminary results to ensure the questions discriminate and that the preliminary passing score in fact identifies “minimally competent” practitioners.

What safeguards are in place to ensure a valid, impartial exam development process? ASHRAE benchmarks management of its certification programs with the ISO/IEC 17024 accreditation standard. Five ASHRAE certification programs—CPMP, BCxP, BEAP, BEMP and HBDP—are ANSI-Accredited Personnel Certification Programs under ISO/IEC 17024–Accreditation #1139. To further ensure content validity, the Certification Committee requires all Exam Subcommittee members to sign a Conflict of Interest agreement, and partners with a third-party exam development company, PSI Services Inc., to facilitate meetings.

For the past 100+ years, ASHRAE has worked hard to earn and maintain a worldwide reputation for being the leader in HVAC&R design. ASHRAE certification programs with their adherence to a rigorous, valid and reliable exam development process serve to reinforce that reputation. With the detailed content outlines of the BEAP, BEMP and OPMP complete and new exams released, the Certification Committee through its HBDP and HFDP Exam Subcommittees is now working hard to ensure these programs remain relevant to current job practice. To this end, it is anticipated new HBDP and HFDP exams will launch at the 2018 Annual Conference in Houston.