To LOFT or Not to LOFT? That is the question.

To LOFT or Not to LOFT? That is the question.

Advice on the Development of a LOFT Testing Model

Historically, a huge effort has been involved in ensuring that the exams associated with any qualification or course are practical and fit for purpose. This typically includes a detailed review of all questions, to not only ensure that they are clear and correct, but also that they adequately represent all parts of the syllabus or learning outcomes on which the user is to be tested.

Often many people are involved in the exam creation procedure, which can be quite lengthy, and at the end of the process, an official exam paper is produced with all questions set in stone. There are then significant levels of security put in place, to ensure that this highly-valued paper is not compromised. Indeed it is often the case that a reserve or fall-back paper is also created “just in case”.

LOFT or Linear-on-the-fly testing

With the advent of computer-based testing, many organizations are now taking the plunge and moving to a Linear-on-the-Fly Testing model (LOFT). This means that a bank of approved questions is set up at the start of the process, and every time a candidate sits the exam, the computer-based assessment system generates a new exam paper in real-time specifically for them. This is based on the question of “picking” rules that are defined within the system.

Some of the factors used to determine questions picked for an exam in the LOFT model

Learning outcomes:

Questions can be associated with one or more learning outcomes, based on whatever pedagogies or taxonomies have been set up. For example, select 2 questions on topic 1 and 4 questions on topic 2.

Question metrics based on previous performance (can incorporate Item Response Theory – IRT):

In this case, the system might track various metrics on a question such as a facility index (how many people have answered the question correctly/incorrectly in the past). These metrics can then be used to select questions, for example, select 3 questions that are difficult (low facility index) and 1 that are easy (high facility index).

Custom fields:

This is where individual questions are labeled in a certain way. For example, let’s say that an exam for the most part is delivered by computer-based assessment, but occasionally there is a need for certain cohorts of candidates to take the exam via pen and paper. In this case, a custom field could be used to indicate whether or not the question is suitable to be selected for paper-based exams. If the question required a video to be watched, it obviously cannot be selected for a printed exam.

Friend or Enemy questions:

Friend questions are always picked together and enemy questions are never picked together.

Benefits of the LOFT or Linear-on-the-fly Testing Model

A big benefit of the LOFT model is that once good question banks are set up, the ongoing effort is centered on maintenance. This is often a less arduous process than creating completely new papers for every exam sitting.

There is also the benefit of not being reliant on one exam paper, which has significant security risks. Using the LOFT model, every candidate gets a different set of exam questions, so even if somehow one of those exams is leaked, it does not compromise every exam.

There are also various options to reduce the risk of any collusion, such as the use of the variables in question. So for example, one student might see a question stem that starts, “Acme corporation produces high-quality leather goods and is based in France”, but another candidate would see, “Beta corporation produces high-quality cosmetics and is based in Spain”.

In addition, Trojan horse questions can be included to help identify any compromises such as the leaking of questions. This is where an incorrect answer is recorded as correct. For example, if we have the question “What color is crimson?” and the options are (1) red, (2) blue, and (3) green. If (3) is recorded as the correct answer on the system, then anyone answering (3) may potentially have had access to a leaked question.

Creating a good and effective question bank

As mentioned above, there are a lot of benefits to the LOFT model, however, the effort to create a good, effective question bank (or refine an existing question bank so that it can be used with the LOFT model), should not be underestimated. The success of the LOFT model is very much dependent on the quality of the question meta-data. So has each question been associated with the correct learning outcomes? Have friend and enemy questions been correctly flagged? etc.

Our advice is definitely to make sure you allow adequate time to completely sanitize question meta-data as part of the process to move to LOFT. Based on our experience it is often the case that the bulk of the effort is on ensuring that questions properly and effectively test the required objectives. However with the LOFT approach, ensuring the quality of question meta-data should absolutely be given the same level of priority, resources, and time.

As an interim solution, most good assessment systems can generate randomized papers for cohorts of candidates. This is where every candidate sitting on the paper at the same time gets the same set of randomly-generated questions, which are shuffled so everyone sees them in a different order. Once the random paper has been generated by the system, it can be reviewed to ensure the picking is correct and the question meta-data has been performed, and question substitutions can be made if necessary. This allows organizations to effectively test the picking and transition to LOFT in a phased approach.

Whatever your thoughts on LOFT, it offers much more flexibility than the traditional static approach to exam paper generation and mitigates much of the risk.

References

Linear-on-the-fly testing or loft | testreach blog. (n.d.). Retrieved December 19, 2022, from https://www.testreach.com/blog-post/To-LOFT-or-not-to-LOFT-Linear-on-the-Fly-Testing.html 

Test authoring. SwiftAssess. (2021, December 5). Retrieved December 19, 2022, from https://swiftassess.com/features/test-authoring 

Total
0
Shares
Previous Post
Redefining Assessment Security with AI Proctoring

Redefining Assessment Security with AI Proctoring

Next Post
Digital Vocational Assessment

How Digital Assessment Platforms are Beneficial for Vocational Assessments

Related Posts
SwiftAssess Grading Solutions for Educational Institutions

SwiftAssess Grading Solutions for Educational Institutions

SwiftAssess GradeMate is a platform that provides innovative grading solutions to speed up the grading process without sacrificing the quality of data analysis or student feedback. Not only does GradeMate save educators time, but it also unlocks a new world of grading and data sorting features that promote informed instructional delivery.
Read More
The Importance of Open-Ended Questions in Digital Assessments, and How to Present Them Effectively

The Importance of Open-Ended Questions in Digital Assessments, and How to Present Them Effectively

In today's digital age, it is important for educators to recognize the significance of open-ended questions in digital assessments. Open-ended questions promote higher order thinking skills, assess students at a deeper level of knowledge, and help demonstrate content mastery. However, the digital shift in assessment has resulted in a lack of familiarity with answering open-ended questions on a digital device. This article explores why open-ended questions are essential in digital assessments, how educators can present them on digital platforms, and what tools can be used to evaluate open-ended questions. With the right platform and training, educators can effectively implement open-ended questions in their digital assessments, promoting critical thinking skills and digital literacy for 21st century learning.
Read More