The language of tweets

This post is part of the output of the Bangalore Fifth Elephant Hacknight.

followers

What you see above are the words most often used on Twitter by Indians. (Click for a larger image). The size of the bubble indicates how often the word is used.

We were looking at whether there are specific words that people with a large number of followers use, that are distinct from people with few followers. The words on the left (also coloured red) are used mainly by people with few followers. The words on the right (also coloured green) are mainly used by people with many followers.

(At this point, it’s worth discussing the dataset. These are 1 week’s worth of geocoded tweets, mainly around India (but including Pakistan, Nepal, etc.) It’s interesting that there were just 80,000 geocoded tweets in this period – and many of them were FourSquare entries.

It’s interesting that people )with low followers often talk about “know”, “high” and ‘”traffic”. People with many followers have significantly more hashtags. Whether this is a cause or an effect of having many followers is, of course, debatable. But the correlation is quite definite.

It also appears that those with more followers are polite. The “good morning”s and “thank you”s are quite to the right. Those with more followers are more likely to say “good” than “bad”, and vice versa. Perhaps there’s something about having Twitter followers that leads to happiness – or is it the other way around?

replies

This picture shows you the words more often used in replies (on the left, in red) when compared to new tweets (on the right, in green).

“haha” and “lol” appear rather prominently in replies. Either folks who reply are an amused bunch, or it’s the funny tweets that get more replies. A lot of replies are also to thank people. The dominance of Mumbai, Maharashtra and Delhi on the right is easiest explained by the presence of the words “@foursquare” and “mayor” – most of these tweets appear to be FourSquare related.

morning

The above shows the words used in the morning (up to 12 noon) vs the evening. Clearly, people mention “morning” in the morning – often, but not always, in the context of “good morning”. The evenings were, at least on this week, were dominated by Euro 2012.

The visualisation used above is a document contrast diagram. Each word is drawn as a bubble, whose size represents its frequency. The horizontal position determines whether the word is closer to one aspect or another – e.g. replies on the left vs new tweets on the right. This is a very quick and easy way of understanding what characterises an aspect (e.g. which words are often used with good vs bad), as well as the context in which words are used.

Gramener at OpenDataCamp

Our COO, Naveen Gattu, spoke about OpenDataCamp Hyderabad to Hybiz.tv.

(OpenDataMeet is where)… all the data enthusiasts come on board talk about the data initiatives [and] what they’re doing… It might be NGOs, it might be the corporates, enterprises, academicians, everyone. The key goal is: how we can liberate data, get it into the environment so that it’s accessible for everyone; for public use, for enterprise use, for corporate use, for the whole benefit of the society.

We at Gramener, right from day one, [have] been part of Datameet. We are supporting this event. We believe it can make a strong impact to humankind.

So we at Gramener, (introducing to you Gramener: it’s an analytics and visualisation company) we convert numbers into pictures. What that means is: you have tons and tons of data, coming from different places. That might be social media, that might be enteprise, that might be corporate, your own spreadsheets — whatever it might be, (even) your mobile phones generate data. We take all those things from heterogenous data sources, process it in a non-traditional analytics way, and present it out in a visually appealing manner, so that the insights can be derived in the span of a second.

Essentially, (1) it enables you from a decision making perspective. (2) If you’re looking at 10 spreadsheets to make a decision, our target is: can that be made one spreadsheet, (and) still you get more.

The Mahabharatha in Pictures

At 1.8 million words, the Mahabharatha is one of the largest epics – roughly 10 times the size of the Iliad and Odyssey combined. At some level, this represents “big data”. Text is generally considered “unstructured” and therefore tough to analyse. But the growing field of text analytics and text visualisation tell us that there’s a lot more structure to plain text than one might think.

To begin with, a word cloud can tell us a lot about the story.

mahabharatha-wordcloud

The story is obviously about a battle between great kings and sons, with the principal characters being Arjuna, Pandu, Bhishma, Bharata, Karna, Duryodhana, Yudhishthira, Vaisampayama, etc. That’s decipherable without having to read the text.

The structure that we gleam out of it arises from a frequency distribution of the words – i.e. a count of which words occur how many times. The word cloud plots the words at a font size proportional to the frequency of occurrence. (Wordle is a good place to create word clouds.)

Now what we know who’re the principal characters, the next questions are: where are they mentioned? Who’re closely related? etc.

Our Mahabharatha browser provides a simple interface to browse the full text of the Mahabharatha and find where the characters appear.

mahabharatha-mentions

The Mahabharatha is made of 18 books, each with several sections. This visualisation shows each section as a block (the length of the block is proportional to the size of the section.) When you click on a character’s name, the positions in each section where they are mentioned are highlighted

This makes it easy to see where characters speak together (e.g. where does Kunti throw away Karna? Where does she meet him again? Did Draupadi really love Karna before her wedding? Was Arjuna really her favourite? Whom does Krishna favour? etc.) By clicking on the section, you can read the full text of that section.

The second question is, which characters are most closely related? Measuring closeness of characters is a difficult thing to do, even for humans. Fortunately, with text, we can rely on a proxy: how often are two characters found within a few words of each other.

If we take Draupadi as a benchmark character and check how often various people are mentioned within a few words of her, here’s what the picture looks like:

mahabharatha-draupadi-closeness

Each row has the name of the character (along with aliases). The first column shows the number of times they’re mentioned within 50 words of her. The next shows how many times they’re mentioned within 100 words of her. And so on. (All within the same section.)

A visual inspection suggests that many characters start fading off at a distance of 200 words, so perhaps 200 might be a reasonable boundary to consider. (This is arbitrary. But based on our subsequent analysis, we find that this parameter does not impact the visual result too much.)

By plotting a network of their closeness, one can get some insights about the structure of the tale.

mahabharatha-network

Yudhishthira is clearly at the centre of the plot. Arjuna, surprisingly, isn’t. Apart from his close relationship with Krishna and Bhishma, his interaction with other characters is not as well spread out (despite his popularity in the epic.) Contrary to popular opinion, Bhima is mentioned quite often, and is fairly well-networked. Nakula and Sahadeva remain peripheral characters. Gandhari is nearly outside of the network, except for her connection with her husband Dhritarashtra, sister-in-law Kunti, and brother-in-law Vidura (with whom she seems to converse much more than with her husband.)

Another way of looking at this picture is through a correlation matrix.

mahabharatha-matrix

This shows each pair of characters and the number of times they occur within 200 words of each other. The closeness between Nakula and Sahadeva is very obvious; so are Drona & Kripa; Dhritharastra & Vidura; Arjuna & Krishna. Draupadi is mentioned with Dhrishtadhyumna more than anyone else.

You can also see the blocks breaking up into two clusters of sorts – on the bottom right are the primary characters. They interact a lot with each other. In the middle are secondary characters, who again interact amongst themselves; and then there are the narrators on the top left. This is in line with the Mahabharatha discussing several side-plots with secondary characters in parallel with the main plot. The story of Dhrishtadhyumna, of Satyaki, Nakula and Sahadeva’s conversations, etc are examples of these. In fact, in a larger scatterplot, you can see many more tales emerge, such as Nala & Damayanti; Nahusha & Yayati; Uma & Daksha; Vasishta & Vishwamitra; Chitrasena & Vikarna; Virata & Uttara; Dhrishtadhyumna & Shikhandin; Parva & Sambhava; even Ravana & Vali.

If you are interested in seeing the full correlation matrix with all major and minor characters, please reach us at contact@gramener.com.