From http://www.thehindu.com/sci-tech/technology/article3696951.ece
Big data that’s worth big bucks
Deepa Kurup
Huge amounts of data are being crunched to create meaningful information
Last week, an official business meet between the chiefs of social media giant Facebook and retail behemoth Walmart raised a few eyebrows.
While officially it was given to understand that Walmart, a retail major which lags behind others such as Amazon in online retail, was looking to enhance its social media presence, tech forums deliberated on the real purpose of the “relationship meet” — data. With over 800 million users, and needless to say, a lot of intricate and often geo-tagged personal data uploaded by them, Facebook presents a data trove like none other, and Walmart, which has been on the ball as far as technology goes, knows that. Just a few months ago, Walmart’s acquisition of ‘Social Calendar’ — a hugely popular Facebook app that people use to track birthdays — was also, obviously, about getting access to and using data, mostly personal, to make better and more customised business decisions.
Today, companies, both at home and globally, are waking up to the value of data. The growing interest in big data has obviously to do with the fact that it is worth big bucks. Driven by the explosion of social media, the all-pervasive use of mobile networks and cloud storage, data has gotten bigger and bigger, so much so that the term ‘big data’ — used in tech parlance to refer to data sets that are large and tough to manage — has come to be known as one that has no prescribed upper limit.
As storage capacity, computing power and parallel processing capabilities expand, the value of data is being realised better. That is, huge amounts of data (this could be data generated within the enterprise or data on it generated online or on social media) is being crunched to create insights or meaningful information. And increasingly, this process, which used to take hours and even days, is now being done in real time. While tools such as Hadoop allowed for real-time analysis of data, Google’s Dremel and other Open Source implementations that are developing in this ecosystem, allows for ad-hoc querying of big data in real time.
Around half a decade ago, when analytics was still much in its infancy, a popular and provocative article in wired.com asked if analytics signalled the ‘end of theory’. In the petabyte age, the article pondered, will scientific analysis based on hypothesis, modelling and testing be rendered obsolete? Is theory not relevant anymore?
Today, big data enthusiasts agree. An ‘analyst’ is more of a “tool expert”, or someone proficient in using various data analytical tools, and there is a lot of demand in the market for someone who can do this well, says Rahul Kulkarni, senior product manager at Google India.
DATA CRUNCHING
More and more businesses, even in India, are looking to crunch their large data sets to see what works and what doesn’t.
“And people are seeing the value in that. Earlier, people were not enthusiastic about storing data, but now they know that data contains insights that can aid crucial decision-making,” he explains. Earlier, taking this data and analysing it was a two to three week cycle, but now most of this is possible in real time, and the benefits of that are immense, he says.
However, several obstacles limit their ability to turn this massive amount of unstructured data into profit, points out Mitesh Agarwal, Chief Technology Officer and Director, System Solution Consulting, Oracle India. The most prominent obstacle among them is a lack of understanding on how to add big data capabilities to the overall information architecture to build an all-pervasive big data architecture. “When big data is distilled and analysed in combination with traditional enterprise data, enterprises can develop a more thorough and insightful understanding of their business, which can lead to enhanced productivity, a stronger competitive position and greater innovation — all of which can have a significant impact on the bottom line.”
Technology-wise, companies are now focussing on ways to make the analytics and query interface as simple as possible. While internally Google uses Dremel to do this for its own processes, for its clients, Google provides analytics as a service. “What we attempt to deliver is analytics interfaces that are so simple that a marketing officer can use it to pose ad-hoc queries to the data set, and be able to extract information that can be used meaningfully,” Mr. Kulkarni explains.
ANALYTICS OUTSOURCED
As an emerging tech field, several Indian companies, big and small, have their eyes set on analytics. The bigger outsourcers, such as Wipro, TCS and Infosys, are into analytics services; several other larger global companies across segments ranging from automobile to pharmaceutical, are getting their analytics done here.
Apart from them, many smaller companies and start-ups are into analytics services, and in some sense it is a natural progression from business process outsourcing to knowledge process outsourcing to analytics, says S. Anand, Chief Data Scientist at Gramener, a data visualisation company. His company is into analytics products and specialises in the emerging tech field of data visualisation.
“During the nineties, the services model did well and the products-model in IT did not pick up. That seems to be changing, and in a field like analytics, it now appears we may have the advantage on both,” he says.
