Online exams: are the colleges doing it right?

FEBRUARY 19, 2018, was indeed a black day for hundreds of junior doctors taking the make-or-break Written Divisional Exam of the Royal Australasian College of Physicians (RACP). After months of study, while generally holding down demanding full-time jobs, and after stumping up the $1900 for the pleasure of sitting the exam, the candidates found themselves locked out of the computer-based testing system halfway through. The decision was taken to abort the exam, leaving many distraught and in tears.

The RACP has rescheduled the exam for two different dates and will be reimbursing candidates the full amount they paid to sit the exam. The computer system’s provider, Pearson VUE, is yet to report to the RACP on the reasons for the failure, except to say that it was due to human error.

The fiasco has put the spotlight on the rapid computerisation of assessment and examination by medical schools and colleges that has taken place over the past few years. What are the benefits and, more importantly in this case, what are the risks?

Steven Godinho of Genix Ventures, which has implemented online exam platforms for the Royal Australian College of General Practitioners and the Royal Australian and New Zealand College of Radiologists, says it’s not clear what went wrong with the RACP exams, as there are many potential points of failure.

“There are three broad types of failure. The most benign is computer crashes. If you have a big exam with 2000 candidates, you’re likely to have between two and eight machines crashing. In this case you just move the candidate to a new machine.”

More seriously, you can have software failure. To mitigate this possibility, you have to test your system “to death”, Godinho says, typically in the environments where the exams will actually be held.

The third and most important type of failure is one of infrastructure, involving hardware in a data centre and connectivity between the data centre and exam sites. Building in redundancy is the key to mitigating this type of disaster.

“You have an extra data centre sitting there on the off-chance something goes wrong. We replicate the online database and that mitigates to a large extent the sorts of catastrophic failures that do occur. The problem you have is if connectivity goes down at a venue. You can have all the backups in the world and it won’t help.”

In this case, the last resort is to have paper copies of the exam to fall back on. Godinho says it’s what his company does, and not having paper copies at the ready may have been what ultimately cooked the goose for the RACP.

Given the potential for things to go wrong, should we really be putting all our faith in computerised exams? Godinho makes the point that there are also plenty of risks in the paper-based model. Exam papers can and do go missing and they can be incorrectly marked.

“There are risks, certainly, but they’re not greater than in a traditional exam.”

Dr Andrew Fluck (PhD), an IT expert at the University of Tasmania who has been involved in designing computerised exam systems, says that the two main drivers of implementing computerised exams are administrative efficiency on the one hand, and pedagogical opportunity on the other.

“Computerising exams allows for quicker turnaround in terms of setting up the exams, marking them and getting the results out, which makes the process cheaper and more efficient. But they can also change the way we teach and test students as well, leading to new educational opportunities. If we can transform assessment, we can transform learning.”

He points to the massive TeSLA project that is currently underway in Europe, which will allow people to study and do exams anywhere and at any time, once they have had their face, voice and typing style registered at a centre.

“There’s huge pressure to move to the online space because it matches the world of work,” he notes. “There are very few places in the world of work where people sit down and write for 3 hours without consulting anyone else and using a pen and paper.”

Steven Godinho agrees that computerised testing can better match actual conditions in the real world than paper-based exams can.

“For the college of radiologists, we built the first film-reading exam in 2013. Radiologists typically use electronic systems in hospitals to look at images, and for the first time, we were able to replicate that in an exam environment.”

Both Steven Godinho and Andrew Fluck agree that the online environment won’t make cheating any easier, and in fact may make it harder.

“We have a locked down browser and, just as in paper-based exams, there are stringent efforts to make sure people don’t bring mobile phones or other devices into the exam room,” says Godinho. “We can also randomise the order of exam questions, which means it would be more difficult, for example, for a candidate to communicate the answer to Question 12 to another candidate, because their Question 12 won’t be the same.”

In any case, there doesn’t seem to be any stopping the computerisation of exams, and both experts are of the opinion that within a very short space of time, all high-stakes exams, medical or otherwise, will be done on a computer.

“The larger colleges are already onto it, because they have the numbers where the benefits start to be compelling,” Godinho says. “And the universities are realising they have to move because of costs and turnaround times.”

To find a doctor, or a job, to use GP Desktop and Doctors Health, book and track your CPD, and buy textbooks and guidelines, visit doctorportal.