Consistency in moderation

Yesterday, we saw how CBSE English marks were mod­er­ated up­wards and down­wards. Let’s ex­tend our data forensics in­to Karnataka’s state board ex­ams in class 10 and look for pat­terns.

This is the dis­tri­bu­tion stu­dents marks in English as a 2nd lan­guage. The height of each bar shows the num­ber of stu­dents who ob­tained a given mark.English 2nd

A few ob­vi­ous points stand out. Firstly, a huge num­ber of stu­dents get ex­actly 30 marks. This is not sur­pris­ing, as 30 is the pass mark. We also ob­serve that very few stu­dents get between 20 – 29 marks. Presumably, they are be­ing pushed up, or “mod­er­ated” to pass the ex­am.

This happened in the CBSE Class XII English ex­am last year too – where no stu­dent “just failed”, i.e. failed with­in 7 marks of the pass mark.

English 2nd just failedHowever. that is not the case with the Karnataka English ex­ams. A hand­ful of stu­dents (124 in num­ber) were among the ex­tremely few that failed with 20 – 29 marks. Of these stu­dents, 72 could have passed if only they re­ceived the same con­sid­er­a­tion. Here are some of those stu­dents’ marks. (Each row shows the marks of one stu­dent.)

Just failed because of English

It seems a trifle un­for­tu­nate – these stu­dents would have passed if they had been given the same con­sid­er­a­tion that over 1.2 lakh stu­dents had been given, by bump­ing their marks up to the pass mark.

We are not sure how the poli­cy of mod­er­a­tion is ad­min­istered. However, it does have to be a little more con­sist­ent to avoid un­fair­ness.

Note: We have chosen the Karnataka English ex­ams at ran­dom. This is true of most sub­jects in many states, where mod­er­a­tion is im­ple­men­ted with vary­ing levels of con­sist­ency.

Reverse moderation of marks

Forensic sci­ence ap­plies sci­en­ti­fic prin­ciples to evid­ence to dis­cov­er past fraud. Let’s ex­tend that to data, and ini­ti­ate a jour­ney in­to data forensics. We’ll be­gin with school marks.

One power­ful tool at the hand of a forensic data sci­ent­ist is the humble his­to­gram: plot­ting fre­quency dis­tri­bu­tions of val­ues. For ex­ample, birth­days are not ran­dom and some birth­days are sys­tem­at­ic­ally avoided (e.g. April Fool’s day, the 13th of any month.)

It is power­ful for the same reas­on that fin­ger­prints are ef­fect­ive: they are easy to leave be­hind, dif­fi­cult to erase, and high­light who did what and where.

Let’s ap­ply this to the marks scored in English by the CBSE class 12 stu­dents in 2013. English is the single most com­mon sub­ject taken by stu­dents – over 8.5 lakh stu­dents wro­te the English ex­ams out of the 9.4 lakh stu­dents.

Typically, such mark dis­tri­bu­tions are nor­mal dis­tri­bu­tions – smooth, thin near the ends and thick at the center. This is mainly be­cause most ex­ams re­quire a com­bin­a­tion of abil­it­ies (spelling, gram­mar, com­pre­hen­sion, cre­ativ­ity, etc.). Few people ex­cel in all of these. Few suf­fer from the com­plete lack of all of these. Hence we ex­pect to see few­er people at the edges than at the center.

Normal distribution

What we ob­serve, in fact, is the fol­low­ing dis­tri­bu­tion. The height of each bar rep­res­ents the num­ber of stu­dents who got a spe­cific mark between 0 – 100.

CBSE Class 12 English Marks

Several items are note­worthy. Let’s be­gin with the two large spikes.

Pass mark

The spike on the left ap­pears at 33 marks. Further, no stu­dent has marks between 26 – 32.

According to the CBSE:

The qual­i­fy­ing marks in each sub­ject of ex­tern­al ex­am­in­a­tion shall be 33% at Secondary / Senior School Certificate Examinations. However at Senior School Certificate Examination in a sub­ject in­volving prac­tic­al work, a can­did­ate must ob­tain 33% marks in the the­ory and 33% marks in the prac­tic­al sep­ar­ately in ad­di­tion to 33% marks in ag­greg­ate, in or­der to qual­i­fy in that sub­ject.

That gives us a plaus­ible ex­plan­a­tion: the kind souls cor­rect­ing these pa­pers give bor­der­line stu­dents the be­ne­fit of doubt, and en­sure that no one has “just failed”. Either stu­dents fail to reach 25%, or they are un­of­fi­cially bumped up to 33%. This is pop­ularly ter­med mod­er­a­tion. However, it is not doc­u­mented in any guide­book that we know of.

95 percentThe second spike on the right is equally in­ter­est­ing. This time, it is at 95%, and there are very few stu­dents scor­ing above 95%.

Unlike 33%, the 95% score might be more driv­en less by the struc­ture of ex­ams and more by me­dia. In May 2013, sev­er­al me­dia or­gan­isa­tions an­nounced the stead­ily rising num­ber of stu­dents scor­ing 95% and more, and the fact that this trend has con­tin­ued across many years.

Our be­st guess is that this shows a pat­tern of re­verse mod­er­a­tion. To re­duce the num­ber of stu­dents scor­ing above 95%, the marks of sev­er­al such stu­dents was brought down to 95%, lead­ing to the large spike.

Clearly, for the stu­dents that are nat­ur­ally good at English, 2013 was not a good year to have taken the CBSE ex­am – es­pe­cially given that most col­leges have cut-offs at above 95%.

We will be peri­od­ic­ally re-visiting edu­ca­tion data (marks, in­fra­struc­ture, spend­ing, etc.) to see what else we can learn – both about the Indian edu­ca­tion sys­tem as well as data forensics.