Making Public Service BIG with #BigData

3 Months back while the world was watch­ing, amid­st much hy­pe, a new gov­ern­ment an­nounced its suc­cess with a lot of prom­ise to one and all.

One of those prom­ises was that of min­im­um gov­ern­ment, max­im­um gov­ernance. This state­ment is as au­da­cious as much as it’s suc­cinct. The new gov­ern­ment prom­ises to re­in­vent pub­lic ser­vice, mak­ing it more ef­fi­cient, in­tro­du­cing trans­par­ency and stead­i­er and sus­tain­able growth. With a gov­ern­ment more tech-embracing than ever and the ad­vent of fin­ger­tip tech­no­logy to the people, a lot can be hoped. How much is achieved is yet to be seen.

To solve a prob­lem, know­ing the prob­lem clearly is the key. This key is held by the huge loads of Data that we have hid­den in the or­gan­iz­a­tion­al silos of our gov­ern­ment. At Gramener, we at­tempt to solve this prob­lem through rich­er, bet­ter data-driven in­sights, mak­ing it avail­able to the com­mon Joe. The ad­vent of Big Data in today’s world is not un­known. Big Data is a term that every­one is us­ing today. From board rooms to col­lege canteens, it’s now be­come the buzz in the more priv­ileged world.

Making Public Service Big With Big Data

Fraud Detection

Some stats to put the prob­lem at hand in per­spect­ive:

$314 bil­lion is what India loses from tax eva­sion an­nu­ally, de­priving it of funds for in­vest­ment in roads, ports and power.7 With so little rev­en­ue, the gov­ern­ment must bor­row more to fund a planned $1 tril­lion five-year in­fra­struc­ture pro­gram.

$462 Billion is what India lost due to tax eva­sion, crime and cor­rup­tion post-Independence.

Click here to see more on govt. spend­ing.

The more start­ling fact is that this money is not only from the big-scale frauds that we read about in the dailies. Small, un­re­por­ted frauds add up and form such bizar­re num­bers.

What if we could track these num­bers to their last ru­pee like in the fin­an­cial ser­vices in­dustry? What if we could have sys­tems to de­tect ir­reg­u­lar­it­ies in each micro-transaction? Government should in­vest in the in­fra­struc­ture to cap­ture data from all corners of the gov­ern­ment ma­chinery to one place. Read about how the UK govt saved 33bn a year us­ing Big Data Analytics.

How about Internal Security?

CCTV foot­ages, RFIDs and scan­ner ma­chines and oth­er elec­tron­ic data, al­though un­struc­tured but when used with deft­ness, Wirelessly in­ter­cep­ted in­form­a­tion, Internet brows­ing activ­it­ies can really help ex­tract use­ful in­form­a­tion for ana­lyses to de­tect crime, ter­ror­ist activ­it­ies and track­ing wrong­do­ers faster and easi­er and way more ef­fi­cient.

Law en­force­ment agen­cies need to ad­apt to such prac­tices for the great­er good. This re­quires a con­scious ef­fort to­wards skill ac­quis­i­tion, train­ing etc but it’s worth the ef­fort.

Public Services

Ever thought of filling that form for your PAN-Card purely on­line and get­ting it at your door­step without hassles. Ever wondered what it means to get up­dates about that sky­walk in your neigh­bour­hood, its status real-time on that smart­phone like your Facebook no­ti­fic­a­tion. With mul­tiple sources of data and your de­tails in­teg­rated in­to one place, your up­dates, ser­vices can be more and more per­son­al.

Will there be a time when we real­ize the above is not a hy­po­theses alone? Lets hope our gov­ern­ments real­ize it soon. Watch this space out for some more thoughts on how hav­ing the data is the new ne­ces­sity.

Visualising India’s budget history

As we await the Indian Government’s budget to­mor­row, here are some visu­al­isa­tions we cre­ated to ex­am­ine the his­tory of our past budgets.

Working with The Economic Times, we cre­ated a view of how the budget break­up has changed by min­istry over time.

Sectoral trends

The large pink re­gion is the Economic Affairs min­istry, which takes up the bulk of the spend. Though it has been grow­ing in ab­so­lute terms, in re­l­at­ive terms, it has been shrink­ing in im­port­ance. You can see the break-up by plan and non-plan break­ups at our ET Ministry-wise Budget Allocation page.

Sectoral trends - PlanSectoral trends - Non-plan

In ab­so­lute terms, through, the spend on every sec­tor has been grow­ing smoothly and stead­ily, bar­ring a few kinks. Among these, the ag­ri­cul­tur­al spend­ing is not­able. It spur­ted up in 2009 to Rs 77 cr, but no sub­sequent budget has spent as much on ag­ri­cul­ture.

Sectoral trends - Total

One oth­er change that has happened is the re­l­at­ive ra­tion­al­isa­tion of budgets between 2002 and 2014. While in 2002 (left), the budget was re­l­at­ively more con­cen­trated among a few de­part­ments such as Economic Affairs, Defence, etc, by 2014, this dis­par­ity has re­duced mar­gin­ally.

Sectoral breakup 2001-2002Sectoral breakup 2013-2014

But how do mar­kets re­act to the budget?

One way of cap­tur­ing this in­form­a­tion is to look at how the mar­ket cap­it­al­isa­tion of com­pan­ies has moved on the day of the budget. For ex­ample, on the day of the 2007 budget, every single sector’s mar­ket cap­it­al­isa­tion fell, with the sole ex­cep­tion of Tobacco. The ex­act same thing happened on the day of the 2009 budget as well.

Market movement 2007Market movement 2009

But on the day of the 2010 budget, there was a per­fect re­versal of the situ­ation. Every sec­tor ex­cept Tobacco im­proved, while Tobacco (in a rare turn of events), lost con­sid­er­able value.

Market movement 2010

In fact, the Tobacco sec­tor is primar­ily just a single com­pany – ITC, and its fate of­ten moves counter-cyclically on budget days. But budgets are gen­er­ally good for ITC. In the last 11 budgets, ITC (and Tobacco) has grown ex­cept in 2010. On the oth­er hand, the Media and Entertainment in­dustry typ­ic­ally suf­fers set­backs on budget day. Barring 2010 and 2011, the mar­ket cap of this seg­ment has shrunk on every single budget day in the last 11 years.

Market movement history

For more in­sights and to ex­plore the his­tory of the Indian Budgets, please vis­it our site on The Economic Times and on

Comparing school performance

Continuing the design jams, we had one at Akshara’s of­fice last week­end. The data­set we de­cided to pur­sue was the Karnataka SSLC res­ults, which we had for the 5 years.

We ad­dressed two ques­tions:

  1. How do Government schools per­form when com­pared to private schools?
  2. How does the me­di­um of in­struc­tion af­fect marks in dif­fer­ent sub­jects?

When com­par­ing Government and private schools, here’s the res­ult.


Each box is a school. The size of the box rep­res­ents the num­ber of stu­dents from that school who ap­peared in the Class X ex­am. (Only schools with at least 60 stu­dents were con­sidered.) The col­our rep­res­ents the av­er­age mark – red is low, and green is high.

What’s im­me­di­ately ob­vi­ous is that private schools per­form much bet­ter on av­er­age than Government schools, what’s less clear is when this dif­fer­ence starts. The series of graphs be­low show the num­ber of schools at vari­ous mark ranges. The first shows schools with an av­er­age of 0 – 30%. The next, from 0 – 40%, and so on un­til 80%. Then it shows schools with an av­er­age of 30% – 100%. The next, from 40% – 100%, and so on un­til 80% – 100%.


From the first graph, you can see that there are as many poor schools (av­er­age 0 – 30%) among the private and Government schools. But from the last graph, you can see that there are far more good private schools (av­er­age 80 – 100%) than Government schools.

So, there are poor per­form­ing schools among the private schools as well. However, there are very few ex­cel­lent Government schools.

We com­pared the im­pact of me­di­um of in­struc­tion again­st the sub­jects as well. The table be­low shows boxes for each sub­ject taken un­der each me­di­um of in­struc­tion. The size of the box rep­res­ents the num­ber of stu­dents tak­ing that com­bin­a­tion. The col­our in­dic­ates the av­er­age mark (red is low, green is high.)


Clearly, Sanksrit is a high scor­ing lan­guage. (At least one per­son at the design jam chose Sanskrit for this very reas­on.) Kannada scores well too – es­pe­cially as a first or third lan­guage; but not as well as a second lan­guage.

On av­er­age, English me­di­um stu­dents have the highest marks, fol­lowed by Kannada me­di­um stu­dents. Students study­ing oth­er in me­di­ums of in­struc­tion per­form poorly in most sub­jects bar­ring their lan­guage.

There’s clearly a strong cor­rel­a­tion between the me­di­um and the sub­ject. Kannada me­di­um stu­dents score high in Kannada, Urdu me­di­um stu­dents shore high in Urdu, and so on. But while English me­di­um stu­dents do score high in English, they tend to score much bet­ter at Kannada, Urdu and Sanskrit!

You can ex­plore these res­ults at http://gramener/karnatakamarks/