<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=353110511707231&amp;ev=PageView&amp;noscript=1">

Strategies for Using Automatic Item Generation (AIG) in Test Design

November 11, 2020

There is a never-ending, urgent need for high-quality test items. However, typical item development is both time consuming and cost prohibitive. The needs have increasingly surpassed the ability to generate large quantities of test items with minimal human involvement, especially under the circumstances we’ve seen in 2020 thus far – a travel ban and social distancing due to a pandemic.  

iStock-1143790720Nonetheless, new challenges bring new opportunities to innovate. One of the ways to address the issue is to use Automatic Item Generation (AIG). AIG offers an intuitive way to generate new items efficiently and effectively to produce hundreds, and even thousands, of items from a single item model.  

AIG is an evolving research area where cognitive and psychometric theories are being used to create items via computer technology. This allows the implementation of item templates/models by controlling for cognitive levels and performance characteristics quickly and effectively, thereby minimizing the efforts of human intervention across the development process. 

Why Use Automatic Item Generation (AIG)? 

AIG has several critical benefits throughout the test development process. 

  • Test Security – A large number of items drastically decreases item exposure rate. Item models are constantly manipulated and varied, making it challenging to memorize items or cheat. Read more about methods to improve test security.

  • Cost Effectiveness – Because the model is continuously reused automatically, as opposed to developing each item manually, it presents a cost savings on development that would not be attained if created from scratch.    

  • Training and InstructionSubject-matter experts can be trained to develop item templates or models and come up with lists of variables that can be manipulated. For examinees, personalized and self-directed learning activities can be supported by preparing appropriate and individualized test items easily, automatically, and with little effort. 

  • Multi-Modality – Automatically generated items can be exported and imported into web-based, multi-modal test delivery systems to support administrations where online, remote proctoring, and testing centers are being used. 

How to Create AIG 

 So where do you start? An AIG model can be created based on three approaches: 

  1. Based on a Theoretical Framework, where an item template is created from cognitive and psychometric theory. 

  2. Based on Successful Items, where an item template is created from an existing item. 

  3. Based on a Content Outline from Job Analysis, where a template is created from a consensus skill description, objective, or competency standard.  

There are several key elements of AIG. First, the item templates of models are created to highlight the key features in the task that can be manipulated. These features can then be converted into variables with certain parameters as defined. These then can be manipulated with logic or formulas to capture responses and common errors.  

Data sources can be generated using manual entry, import, or even within a database with a formula, text, images, and media (including audio and video) format. All item types are supported with AIG as well, including multiple choice, matching, drag-and-drop, fill-in-the-blank, true/false, and hotspot. These variations can be reviewed as a batch or separately and documented with corresponding references and rationale.  

Takeaways and Recommendations  

AIG can be used in all the ways you’re traditionally using items. It can be used with a variety of item types and modeling, and items can be exported for usage via text or QTI format in any banking or delivery system. Moreover, AIG data sources and templates can be shared, reused, and uploaded. AIG templates can be used widely for calculation or numerical items and can be extended to graphic, audio, and video items. And lastly, AIG item development and review processes are very similar to those of traditional items.  

When using AIG, you’ll also be working closely with subject-matter experts who have knowledge about the item content to help you draft templates. Some places you can start are previously developed items, and then you can discuss challenges and shortcomings of those items in order to improve them. It’s important to collect relevant data beforehand, such as tables, figures, images, links, etc., and then discuss plausible distractors and the best method to generate them. Once you’ve collected and discussed those areas, the subject-matter expert will be able to help you finalize the template before generating variations of the item to review.  

Overall, AIG  is an exciting, accessible, and expanding area with numerous research studies in progress regarding item model calibration, item property prediction, and item survival rate.   

Multi-Modal Testing Webinar 2020 and Beyond

Xin Li, PhD Xin Li, PhD is the Vice President, Test Development & Psychometrics, Licensure & International at PSI Services. She has over 14 years of assessment experience and leads all test development and psychometric activities related to PSI’s regulatory licensure and international programs. She also provides expert consultancy and innovative solutions with her psychometric background, including job analyses and standard-setting studies. Her recent research interest has been focused on automatic item generation (AIG), which has been presented at multiple conferences. She previously conducted and published research related to Item Parameter Drift (IPD), Differential Item Functioning (DIF), and Multidimensional Item Response Theory (MIRT).