JOINT BASE SAN ANTONIO-RANDOLPH, Texas – As the Air Force pivots to electronic administration of the Weighted Airman Promotion System testing process beginning with the CY24 E6 promotion cycle, many inside the service may not know the team responsible for test development.
Based here under Headquarters Air Education and Training Command, the Studies and Analysis Squadron, or SAS, is responsible for development of all enlisted promotion tests with the goal of promoting Airmen with the competencies, knowledge, skills, and abilities needed to prevail in Great Power Competition.
“We are committed to creating fair, credible, and valid tests to provide objective information on the Airmen most qualified to lead the Air Force,” said Lt. Col. Marcus McNabb, SAS commander. “Our senior leaders have made it clear the new electronic testing process substantially improves promotion assessment and is an investment for the Air Force and our Airmen.”
Other critical partners in the promotion testing ecosystem include the Headquarters Air Force Directorate for Manpower, Personnel Services (HAF/A1) and the Air Force Personnel Center (AFPC). The HAF/A1 team sets promotion testing policy, and manages resources related to modern test development and delivery software, while AFPC enforces promotion testing policy, schedules and manages exam administrations, integrating test scores with other inputs in the WAPS.
The SAS team consists of a highly effective blend of military experience as well as psychology and testing expertise to accomplish its mission,
“More than 60% of SAS test developers are retired military and 40% have advanced degrees in industrial-organizational psychology, the main profession involved in setting standards and principles for developing and validating workplace employment tests,” said Dr. Mark Rose, SAS’s chief of test development and Airman advancement. “Several test developers also have extensive experience in the commercial test development and administration industry.”
This blend of experience and expertise allows SAS to apply scientific methods and advanced analytics to develop rigorous tests relevant to Air Force mission requirements, Rose added.
As the Air Force transitions to electronic testing, SAS is advancing promotion testing along three lines of effort: modernization, analytics, and customer focus.
Modernization
One principal benefit of electronic testing is the potential inclusion of technology-enhanced test questions. Currently, the Air Force only allows for multiple choice questions on paper tests; however, SAS recently completed an internal study of 13 types of testing innovations that can be applied to promotion testing.
“SAS found many of these enhancements can be used across multiple career fields to assess more in-depth, job-relevant knowledge while increasing examinee engagement,” Rose said.
For example, one enhancement may involve using interactive images and asking examinees to identify specific locations on a geographical map or select the main areas of the brain involved with memory. A more complex enhancement could involve job-relevant scenarios that provide in-depth background information (e.g., nurse’s notes, lab tests, and physical history) and require processing and integration of the information to (a) make decisions (e.g., patient diagnosis), (b) identify likely causes, and (c) select follow-up actions, according to Rose.
Another area in which SAS is heavily invested is exploring the use of generative artificial intelligence/large language models for creating new test questions.
“An initial study showed that many of the AI-generated questions performed as well as historical human-generated questions,” Rose said. ”However, developing AI questions to specific difficulty levels proved problematic in this initial study. These results indicate the viability and usefulness of this approach and ultimately may save time in test development, allow for development of larger item banks, and reduce impact of security breaches.”
Analytics
As part of the move to electronic promotion testing, SAS also migrated test development and post-test analysis to a modern system, Envision (part of the DAF Data Fabric). This new system provides post-test analytics nearly instantaneously compared to weeks or months in the previous system.
“This system also makes longitudinal analysis across testing groups and years much quicker and simpler,” Rose said. ”In addition to traditional, industry-standard statistics, the system calculates advanced statistics to generate more precise ability estimates, better understand cohort differences in ability from year-to-year and enable partial credit scoring.”
With these advancements, SAS is pursuing numerous efforts to improve test development capabilities and address critical issues. For example, prediction of item difficulty is a widely recognized challenge in the testing industry. Item difficulty is important because variation in difficulty, from easy to hard, is critical to distinguish performance levels amongst testers.
“As such, SAS is creating prediction algorithms to measure item difficulty using commonly studied variables such as question type, reading load, content complexity, and differences in on-the-job opportunities for learning new knowledge,” Rose said. ”Insights gained here will also help guide efforts to identify sources of potential demographic differences in test scores.”
Other analytics initiatives include examining the impact of varying the number of possible responses to multiple choice answer options (e.g., four possible responses vs. three); developing analytic methods to study test performance as a predictor for job performance; examining the impact and mitigation of test anxiety on exam performance; and evaluating the impact of testing on force diversity, Rose said.
Customer Focus
Another major effort revolves around understanding Airmen’ study habits and how to improve both study materials and test questions.
“One focus area we’ve really looked at is examinee’s use of third-party study materials,” Rose said. “We recently completed the third data collection from test examinees and used the results to strengthen Air Force study materials, including AF Handbook 1, Airman, which is also produced, managed, and updated by SAS.”
Recent improvements have included publication of AFH-1 as an audiobook in 2023.
Similarly, SAS pulled together what was previously a dispersed array of study materials into a one-stop website for all study materials: https://www.studyguides.af.mil/.
“The website contains study guides for the Promotion Fitness Exam and Specialty Knowledge Tests, the Enlisted Promotion References and Requirements Catalog, and an information guide on situational judgment test questions,” Rose said. “The website will also soon house over 2300 practice exam questions to further help Airmen study and prepare.”
SAS is also pursuing post-test examinee feedback to further enable and reinforce Airmen’s development including general and career-specific knowledge. Post test feedback will enable Airmen to not only focus on specific areas related to test performance, but also provide valuable feedback to Airmen on areas to focus to improve overall job knowledge and performance in the field, Rose said.
“Early returns are positive, with feedback from Airmen strongly favoring the advancements and innovations currently implemented,” McNabb said. “As SAS continues to execute its vision of being a world-class testing organization and a leader in innovation, we are proud to be part of AETC’s overall Force Development effort and the promotion of Airmen with the competencies, knowledge, skills, and abilities needed to prevail in Great Power Competition."