Our technology helped to visualise hundreds of thousands of tweets and the topics they concerned, picking up unknown trending topics and providing new insights into trending topics of conversation.
Customer

A polling company.


Problem

When polling companies need to find out what people are talking about, they traditionally polling method involves send out a survey and ask people to respond. But this is like putting the cart before the horse, because you have to know what the important issues are in advance in order to ask questions about them. To solve this, the polling company asked Faculty to find out what certain key ‘influencers’ on Twitter were talking about by grouping tweets into topics.


Solution

After scraping Twitter to obtain tweets, we had to separate hundreds of thousands of them into topics. We first converted the tweets into vectors using GloVe vectorisation, and then used UMAP (a dimensionality reduction algorithm) to reduce these vectors to two dimensions so that we could visualise the tweets. Finally, we used HDBScan (a clustering algorithm) to group tweets into topics. An example of this is shown below:

This created well-separated clusters of distinct topics. For example, the green cluster in the top right corner of the picture, above, concerns tweets about US politics. The picture is from an app we made for the client that allowed it to explore the tweets and their content; hovering the mouse above a point allowed the user to see the content of the tweet. This would not have been possible without AI, since the clusterings we produced typically had hundreds of thousands of tweets in them. This visualisation provided a systematic way for someone to look through the tweets.


Impact

The topic analysis generated by AI allowed the client to visualise hundreds of thousands of tweets and the topics they concerned. This allowed it to pick up previously unknown trending topics that would not have been found through normal polling techniques, and provided new insights into trending topics of conversation.