In 2009, CAT, for the first time, decided to do away with the good old paper and pencil and instead embraced an online format. Another victory for technology? Not so much.
What with systems whimsically rejecting passwords, computers spontaneously shutting down, and students gaping in pure horror watching their dreams of getting into the top mba colleges go bye bye. CAT 2009 was, to put it plainly, a flagrant embarrassment-for both the Indian Institutes of Management (IIMs), as well as Prometric, the globally acclaimed testing vendor commissioned by them for conducting the test. They would’ve been quick to dive behind the “technological mess” excuse, but it’s not as if the question paper was entirely error-free either.
Poor Sumitra Roy, Prometric India’s Managing Director, could be seen scuttling hither and yon for months, hushing up ill-advised comments betraying satisfaction at how CAT 2009 was conducted made by Stephen Williams, Prometric’s Vice President of Test Development Services, and then going and making a few ill-advised remarks of her own, daringly discarding the tech-atrocity that was CAT 2009 as a case of “isolated technical issues”.
So what is the new CAT Evaluation Method?
In 2010, to avoid a similar disaster, Prometric has determined that it will use fewer testing centres-only those that adhere to the Prometric norms created for ensuring infrastructure quality. The method for CAT scoring, however, will remain the same. It is a rather sound three step process supported by the Standards for Educational and Psychological Testing and the ETS Standards for Quality and Fairness. The first step includes calculation of the raw score based on the answers given. This is done thus: A +3 is awarded for a correct answer, and there is a -1, i.e. a one mark deduction, for a wrong answer. There is no negative marking for a question left unattempted. The rationale behind the negative marking is that it discourages “guessing”, and better ensures that a student with a greater skill-and-knowledge base procures more marks.
Finally, these equated scores are “scaled” by placing them on a common metric. Each candidate is presented with four sets of scaled scores-the total score, which is scaled to a 0-450 range and three sets of section scores, with a 0-150 range apiece.
CAT is deliberately and matter-of-factly developed as a very difficult exam so as to best identify only top performing candidates of utmost caliber. After all, getting into the best MBA Colleges of India should not be piece of cake right? Therefore, CAT is simply not one of those exams designed to allow a 100% score. Since an exam constructed to be easy would not solve the distinct purpose of an IIM, the theoretical top-score of 450 is made near-impossible to achieve.
However, it is imperative that the test be uniformly difficult for all, to ensure fairness, and to this end, anchor questions and cloned questions are added to each paper.
An anchor question, explains Mr. Williams, is a question that helps adjust the differences in the difficulty-levels of two sets of question papers. Here’s how it’s done-Four anchor questions are added to each question paper, two of which it has in common with a previous paper, and two with a paper to be used in a consequent slot. This helps establish a common standard of difficulty in two question papers and the raw scores can later be adjusted accordingly.
A cloned question is a question essentially similar, but different in ultimate form. For instance, an algebraic equation repeated across slots, with each version having different substitution values.
Whether questions of the cloned or anchor variety increase the risk of “leakage” by way of students memorizing them and relaying them to candidates in subsequent slots, is matter for further frivolous debate. Why a genuine candidate would want to chip away at his/her own chances of cracking CAT by giving away questions to subsequent batches is anyone’s guess. However, examples of “proxy-students”, who are actually teachers at coaching institutes, are now to be seen, and this might be a more reasonable cause for alarm, since they have no motivation whatsoever to keep these questions to themselves.
Predictably, Mr. Williams rubbishes of this possibility because of no statistically noted “strange patterns with the answering of cloned questions”. Given that their statistical analysis is most likely not much better than their technical prowess, this provides little or no comfort.
The IIMs will release the CAT 2010 advertisement near the end of August. With fewer testing centers, a longer testing window, and a hopefully larger pool of distinct questions, CAT 2010 seems poised to undo the many stark wrongs done by its predecessor. With the bitter criticism faced by them in 2009, they seem motivated to rectify their ways. And if error is indeed minimized, then the satisfactory implementation of the novel online format may well be worthy of some applause.
*Prometric is a trademark of Prometric Inc. All brands, company or product names or trademarks belong to their respective holders
©2010 copyright NiA India