Finally we get social – How we created our first Twitter dashboard

Creating a Twitter dashboard is not an easy task. With Twitter analytics being the cynosure of every web marketer, creating a dashboard that fulfills all their needs is a deeply involving task. A media outlet, generally being a one-way broadcast, would really like to understand whether it is having a meaningful conversation on Twitter. A literary reviewer would like to know how many times his beautiful quote has been favorited or retweeted. With all of these in mind, we decided to take the plunge ourselves and create a Twitter dashboard of our own – FusionTweet! We’ve managed to keep it intuitive and informative, and are very happy about the delicate balance we’ve achieved. Let us walk you through the process of how we created FusionTweet.

Metrics that make sense

To start with, we needed to narrow down the metrics to the ones that really make sense and are useful in making Twitter-related decisions. We figured that having the tweet count of a particular Twitter user, say @barackobama for a particular period of time is very useful. It gives a clear idea of how engaging he has been with his followers during that time period. And so is the number of times that other users mention him on Twitter, which gives an idea of how much his followers engage with him. We also wanted users to compare their Twitter activity with friends and competitors. So we decided to bring in a competitive analysis of the tweets, replies and followers for different Twitter handles.

Twitter API wasn’t enough

Up next, we were faced with the task of getting the data we wanted without having to authenticate the user. We faced a few roadblocks while gathering the data – a big one being unable to get historical data using Twitter’s API. So we turned to other API services like Topsy and Twittercounter. Out of these, Topsy has the most exhaustive data, and is rightfully called as Twitter search without an expiration date. However, even in a marketplace buzzing with Twitter APIs, we could not get our hands on the historical follower count of a Twitter user for more than the last 2 weeks (if anyone could help us out here, we’d be grateful). Topsy has been cataloging all kinds of Twitter data on its servers, and they are reported to have over a petabyte of Twitter data on their servers. Topsy provided us with most of the data we wanted. Their otter API has many useful queries, and constructing each query was a breeze. Their [cciel lang=”javascript”]/searchhistogram[/cciel] API call was especially useful, and it allowed us to easily slice up the returned data in intervals. We used it to get data on tweets [cciel lang=”html”]from:barackobama[/cciel], and tweets [cciel lang=”html”]@barackobama[/cciel]. It can be used to do a lot more, just browse through their docs to learn more. We began poking around the Twitter API with the help of Apigee.com. It has a wonderful console where you can run queries and experiment with the Twitter API. And we also loved the part where we could switch the API response from XML to JSON, so understanding the API’s response gets much easier by just reading it.

Apigee was very useful for managing Twitter’s rate limiting. Using Apigee, you can create a proxy to query the regular Twitter API. We created a simple query using their services, and all it took was a minute. We used this method to query the basic profile summary for FusionTweet – screen name, profile image URL, followers, following and listed counts.

So there we are! After a couple of days of coding, we finally got the dashboard up and running. If you go through the source, you’ll find that most of the code here is to “get” the data, than to create the charts themselves.

Design and development thoughts

Comparing your Twitter handle with friends and competitors visually was the main priority while designing FusionTweet. We decided to have a default of 3 handles to compare, and a maximum of 6.

We faced an interesting visualization problem when Twitter handles of completely different scales were compared. For example, if you compare @techcrunch to someone that has really low Twitter usage, then the lines in the chart would just be stuck to the ceiling and the floor. For situations exactly like these, we have the Logarithmic Line Chart. So we put in the log line chart to plot the tweets, replies and follower count. While the log scale represents the wide variation in data very easily, not everyone will have the exactitude to read through the scale and will end up misinterpreting the data. Since we want everyone to use the dashboard, not just the scientifically-inclined, we decided to stick to the simple line chart itself. And finally, we had to find a way to show that the drop-down menu with time periods is applicable to only the first 2 charts, not the third one. We tried a variety of solutions, ranging from a drop-down for each of the first 2 charts to having tabs. But none of them were tasteful enough, and we finally added two horizontal lines enclosing the drop-down and the first 2 charts.

A few best practices that we used on the FusionCharts chart object is to escape the data URL passed in the [cciel lang=”javascript”]setXMLUrl[/cciel] function. If we don’t, the FusionCharts Visual Debugger would throw a warning. And on the backend, in our [cciel lang=”xml”]<chart>[/cciel] element, we would like to highlight a really useful attribute.

[cciel lang=”xml”]connectnulldata=’1′[/cciel] – If a user has not tweeted at all on some day, then the line chart would be totally disconnected for that day. Setting this attribute to 1 ensures that the line dips down to the X-axis, and doesn’t show a disconnect between the data.

Scaling FusionTweet

We spoke to the wonderful guys at Topsy and TwitterCounter, and asked them for higher grade access to their APIs. The current free limit is 10,000 calls in 1 day for Topsy and 100 calls per hour for TwitterCounter, we were sure to cross them once the dashboard goes live. TwitterCounter has increased our limit to 500 counts per hour with some kind words (we’ll bug you soon for more), and Topsy would require us to buy higher grade access, which would give us 1 million calls in a day.

Fun!

This was our first hand at the omnipresent Twitter. It was a lot of fun developing it, and comparing our follower counts with the rest of the team. Do check out FusionTweet and let us know how you like it.