CAT 2009 - A Case Study

Finally, CAT 2009-2010 comes to an end. Most of the people who have appeared for CAT this year called it a debacle. I don't think the entire situation was that bad but the process could have been managed more professionally and more systematically.

For all the readers who have no idea what CAT is - CAT stands for Common Admission Test. It's an all India test conducted by the Indian Institutes of Management (IIMs) as an entrance examination for the management programs of its seven business schools. Besides the IIMs, CAT scores are used in several other B-Schools for admissions.

Last year, it was announced that CAT 2009 would be going online, that is CAT 2009 was becoming a computer based test. As time progressed it was decided that the major decisions regarding the examinations would be taken by the IIMs and the actual management of the exam would be outsourced to Prometric.

The IIMs are a few of the finest B-Schools in India. They produce a few of the best managers in the world and are known for their expertise in management. Prometric is a U.S. company in the test administration industry. Prometric is spread across 135 countries and conducts examinations for several big players in the Computer sector like Microsoft, IBM and also several government organizations.

It's really surprising that a test conducted by the best management academy and the best test administration organization in the world could get into so many controversies.

The CAT exam was conducted in a 10 day window beginning on 28th November 2009 to 7th December 2009 and in addition two more days because of technical glitches - January 30th and 31st 2010. Finally the results were declared yesterday, i.e. 28th February 2010. That's three whole months.

CAT 2009 was a big thing for Prometric. In India, Prometric conducts a lot of exams, but most of them are score-based and a majority of these are qualification exams rather than competitive exams. But CAT was completely different for Prometric India. It was percentile based (rank based) and a national level competitive exam.

A major difference between competitive exams and qualification exams is that in qualification exams performance is absolute and students really don't care about other students. Prometric conducts a lot of exams like this in India, examples include certification exams for organizations like Sun, Oracle, Microsoft, etc. and has been successful in their administration. But, competitive exams are different, the performance of students is relative so every student should be given the same opportunity to show their expertise. The major factors which influence this are -

• Questions should be never repeated across slots
• The question paper pattern should be same
• The difficulty level of the questions should be same

There were a lot of speculations that questions repeated across slots and that questions even came from question papers of previous years.

Though the question paper pattern was the same, it wasn't declared before. However, I don't think Prometric can be blamed for it.

Estimating the difficulty of the question paper is indeed difficult, so methods like Computerized adaptive testing should have been used. I am not sure if Prometric used such techniques in CAT, but if they did, a thumps up to Prometric.

Another major problem faced by the students was that there was a lot of technical problems with the computers. A few technical issues can be tolerated but it was found that 8000 students were allowed to reappear for the exam, which was a huge embarrassment for Prometric. The pathetic part of it was that Prometric attributed this problem to common viruses and what was more surprising was that the test wasn't even internet-based but was LAN-based. Maybe Prometric should have invested on a good Anti-Virus program or at-least installed a free one like Avast on the systems ;)

Personally I faced no problems when I took my test and I believe that despite a few lapses, Prometric did do a reasonably good job.

Now coming to the bigger picture, I didn't understand a lot of decisions that were taking by the administration committee of CAT -

• Why wasn't the pattern of the paper announced before the exam? This did give an advantage to the students who wrote the exam in the later part of the window as they knew the pattern as opposed to the students who wrote it in the beginning. I guess IIMs were being over-optimistic when they expected that details of the exam won't go out when the students press an "I Agree" button to the terms and conditions.
• Why were the results declared so late? Initially they said the results would come out in January, then in the third week of February and then finally at the end of February. This did create a lot of frustration in the students.
• How were the marks allocated? I am sure a lot of students are still wondering how a score of 450 was divided between 60 questions. If this was because of normalization, why wasn't the formula published?

I am sure, the administration committee have their reasons for this.

A Comp. Sci.s view on CAT 2009

Being a Computer Science student, I was very happy when CAT was announced as a computer based test as I assumed it would very efficient and well organized. In fact, I expected the results to be out soon, as once the entire data was available, a formula would have given the results within a day; which of course wasn't the case. During the entire CAT process, I found a lot of features in their software interesting.

During slot reservation, the queue system implemented was very good. It made the process of reserving slots more professional and at the same time eased the load on their server. However the venue details on a few hall tickets were wrong, which created a panic among the students. For example, Andhra Pradesh was misprinted as Arunachal Pradesh in the hall tickets of several students. I was also a victim of this error. Prometric should have done a quality check on the software for such issues as well.

The exam software very good and comfortable for the students. Time and session management were implemented flawlessly. But the students did face some trouble with an "Exit" button, despite the Prometric guys warning everybody about it.

But coming to the results, there were a lot of things which I didn't understand. One big question was - Why didn't they just email the results to the students instead of putting up a server. This would have made the result declaration a lot easy and obviously mail servers of Google and Yahoo wouldn't have crashed on heavy loads. This would have even saved the students from F5 blues.

When they decided to go for a result server, it is surprising why they didn't make sufficient arrangements to accommodate the huge number of requests. In fact, they did implement load balancing and had nearly four servers running, but most of the students didn't know this. However even that wasn't sufficient and within a few hours, the entire system came down crashing.

Another big mistake was that they actually displayed the exception messages outside for the world to see. Here's a screen-shot of one of them -

Does the world really need to know that the IIMs/Prometric servers used a MySQL database? A custom error page would have been a lot better.

Maybe even the best managers in the world need efficient computer engineers ;)

Well, that's the story of CAT 2009. I personally felt the movement towards an online CAT was very good and will be really beneficial in the future.

And for all those who would like to know my percentile in CAT this year - it's 88.47. Not a great percentile, but at the same time not even a bad one.

Before I conclude, I would like to congratulate all the students who are satisfied with their percentile and wish the best of luck to all of them for their interviews at various B-Schools. For the others, I have only statement to make - "Failure is the stepping stone to Success". :)