How did the Internet predict #StateElections: Deciphering the future of exit polls in India

The good ole’ polls didn’t seem to have got it quite right during the recent state elections in India. It’s surprising how most polls were greatly off the mark when it came to predicting the outcomes! And with this latest round of elections, the pollsters in the country have taken a serious hit. Considering the not so effective performance of the pollsters, quite a few ‘hangdog prognosticators’ are now actively looking for alternate approaches to predicting the election data and re-think their sources of information. But, one doesn’t really need to look so far to find these alternate information sources as they have been right in front of everyone’s eyes all along: the hyper-active interactions and enthusiasm on the Web!

So while the data scientists and political economists are busy scrambling what really went wrong with the predictions of the exit polls, we took up our extensive research digging deep into the digital space to find out what people really posted, shared, and liked across different platforms. We looked at the major websites on the Internet to measure their users’ expressions, namely: Google, Facebook, Twitter and YouTube. Although these are not the only places where Indians are the most active in sharing their views and opinions, they are most certainly the highest involved out here. Using the original API of all the websites, we collected a pool of searches, mentions, hashtags and any other information available. And then using our in-house state-of-the-art analytics tools, we carried out a detailed sentiment analysis around the major political parties and their leaders to identify their respective standing in the digital sphere.

Here’s how we conducted our research

Through the official API as well as our advanced programming interface application, we compiled a huge sample of data, which explicitly included mentions of one or more of the main candidates or parties. We then segregated the information state wise. For this, we use geotags, also popularly known as geolocation metatags.

And once we had our data pool ready, we carried out a detailed sentiment analysis in terms of either positive, negative or neutral. This was a supervised sentiment analysis, meaning that we weren’t just relying upon the machine AI. We combined human-efforts with computer-enabled applications to avoid any possible noise and also to comprehensively grasp nuances and indirect references. We used a team of tech as well as politically-savvy experts for coming up with an opinion-mining algorithm, which could replicate manual data tagging.

Looking at how the polls didn’t go right in the US presidential elections in 2016, experts were sensing similar loopholes for our state elections too. It’s purely because, what these traditional exit polls are lacking is an accurate reflection of the animated mobilization of the people’s feelings and emotions on the Web. It’s no secret that Indians are less shy on the Internet, especially on social media, than they are when they are approached by the pollsters. And to come up with such a research, it was important to create an effective interplay between human intelligence and machine learning. We used human brain for what it’s good at and AI for what machines excel at. And the result, as you can see in our analysis in the infographic, is a better representation of people’s sentiments.

So, is social media a replacement for the traditional exit polls?

We looked where the pollsters haven’t been looking so far. Getting closer to the young and the active web users, who are not necessarily just the cosmopolitans, which are the most convenient samples for the pollsters! So, yes in future, the exit poll experts will need to increasingly dive into hyper chaotic online chatter streams to come up with more reliable predictions. This is because, the Web is nothing but a simple and straight forward representation of what people are saying about the political parties and their leaders. Every post or search represents an emotion, which is more likely to be converted later on into a vote.

But, the Internet alone is not enough to predict the election results. It can’t see it all. And that is why, we still need the exit polls, but more accurate ones preferably from here on.

As we near the next round of state elections, the pollsters must understand that people’s opinion has taken a drastic shift because of the Internet. And they need to use these extremely vital signals of public sentiments to study and predict politics in India.