1. Developing high-stake exams according to industry best practices

Developing high-stake exams according to industry best practices

The question for India’s professional test owners is: as students crammed for your exams, how have you been developing them?

By: | Published: May 25, 2015 12:02 AM

The competitive examination season is drawing to a close and we are yet again witnessing the intense pressure placed on those taking these exams. Tens of thousands of students give up on sleep and social activities, and attend expensive cram schools to earn a seat in a handful of premier technical and medical colleges. For example, over half a million students take the IIT entrance exam every year and fewer than 10,000 clear it—statistically harder than getting into America’s Ivy League colleges.

In Agra, psychologist LK Singh reported in February the effects of exam-related stress on some candidates. “There is a 40% rise in the OPD cases in my clinic, which are mostly associated with exam-related issues and student anxiety. Since January, we are getting calls seeking counselling,” he said.

The intense competition to ace exams no longer ends at school or university-level, with more candidates than ever competing to get pass marks in professional and government exams across a range of job sectors including finance, IT, engineering and medicine in order to get certified for a job or gain a promotion. Some of these tests can be booked on-demand throughout the year, yet there is no less pressure on them to attain the crucial pass marks.

However, the question for India’s professional test owners is: as students crammed for your exams, how have you been developing them?

The consequences of perceived flaws in test design—for example, if the test is perceived to be somehow not a fair or accurate assessment of relevant aptitude or skills—can be serious and damaging for test owners.

Recently, it was reported that a student who took the state-conducted CET in Mumbai in mid-March filed a PIL in the Bombay High Court that over 60,000 of his fellow students encountered incorrect questions. The Directorate of Technical Education then decided to eliminate all the questions considered incorrect from the marking scheme.

Last year, aspiring UPSC students staged a protest at Mukherjee Nagar in Delhi. They accused that the test was unfair because those from rural Hindi-speaking states had to answer questions in English without translations.

These examples suggest that test owners should be putting as much effort into exam design as that of the students desperately cramming to pass those exams. This is why test owners must strive to design and build exams according to industry best practices to be fair, reliable and valid.

Validity means that inferences made from test scores are justified because the test reflects critical aspects of relevant ability, knowledge or skills. Doing well in a university admissions test should mean an individual is more likely to do well at university because the admission test assesses the prerequisite knowledge and skills needed for further study in a given field.

Reliability is about the consistency and reproducibility of test scores, which are never perfect measures of the test takers’ ability but rather always contain some amount of error. An exam with high reliability contains relatively less error than an exam with low reliability, and thus scores on reliable exams are more precise estimates of the test takers’ ability.

A fair exam is one that is equitable for all test takers, regardless of their background or social status, and exam scores should be equivalent for test takers having an equivalent ability level.

For exams to be considered high quality, a multi-disciplinary blending of expertise is needed. Of course, subject matter expertise is integral to being able to develop test questions, but understanding best practices for developing good test questions is required.

Educational measurement—the science which underlies the development of educational assessments—is a highly specialised field of work, typically carried out by psychometricians with advanced university degrees.

Psychometricians analyse test takers’ responses to each question to evaluate how well they function as tools for assessing what they are meant to assess.

They analyse the test as a whole to evaluate the extent to which the scores are fair and reliable reflections of test takers’ ability, and can assist test owners with establishing a passing standard. They typically equate and scale scores so test results can be directly and meaningfully compared across different exam versions and different administrations of the exam, meaning that all test-takers are held to the same standard. For example, a doctor should not pass an exam simply because the version of the exam she took was easier than the version taken by others.

An number of test sponsors in India are choosing to administer their exams via computer-based testing (CBT). With an increasingly tech-savvy populace, sitting in a CBT comes naturally and is often more attractive to the candidate, especially because of the convenience it offers in terms of on-demand testing and immediate results. Test owners can also reduce the administrative burden associated with pencil-and-paper testing because exam papers do not have to be physically shipped to and from testing locations.

Regardless of the mode of delivery, high-stake exams should be developed according to industry best practices.

As test owners across India—whether they cover university admissions, professional or government and regulatory exams—have discovered, students cramming for their exams must be rewarded for their efforts with test scores that are fair, reliable and are valid reflections of their ability.

The author is director of Client Development, Pearson VUE

Get live Stock Prices from BSE and NSE and latest NAV, portfolio of Mutual Funds, calculate your tax by Income Tax Calculator, know market’s Top Gainers, Top Losers & Best Equity Funds. Like us on Facebook and follow us on Twitter.

Go to Top