The web view of the data has been upgraded to provide more information. The Des Moines Twitter Trends website now provides a clearer view of the data collected. It provides current trends for the past few hours, the past day, and the past week. All of the terms are now links to Twitter Searches.

I added some new information, such as recent links, and current view of the latest Twitter user’s avatars in a grid.

I also created my own theme, and display instead of using a third party Wordpress Theme.

P.S.
Here was the write up for the first version: http://www.dotcodedump.com/2009/07/finding-local-twitter-trends.html

Bookmark It

Local or Targeted trends on Twitter is not something that Twitter provides. So I decided to create my own solution. For this project I used Perl, Twitter API via Net::Twitter, Yahoo Term Extraction Service, and SQLite3 to store data. This technique is a quick and dirty way to perform a simple data analysis on Twitter posts, and extract current conversation trends.

To view the most recent output from the process please go to my site: Des Moines Twitter Trends. (Updated for V2)

Process Overview:

1. Get New Local Twitter Posts
2. Clean and Filter Data
3. Extract Significant Terms
4. Rank Term Occurrence
5. Record Term and Tweet Data
6. Consume Historical Data

Process Details:

Get New Local Twitter Posts

I used the Perl Net::Twitter API interface to download new Twitter posts. This was freely available via CPAN. This provided an easy way to interface with the Twitter API. I created a new account, and registered my application with Twitter. I used this account information, and Net::Twitter to download Twitter Posts via the Twitter API.

Code for Defining Net::Twitter, and downloading tweets:

#$user: Twitter Account Username
#$password: Twitter Account Password
#$consumer_key: Twitter Application Consumer Key
#$consumer_key: Twitter Application Consumer Secret

my $nt = Net::Twitter->new(
traits => [qw/API::REST API::Search OAuth/],
username => $user,
password => $password,
consumer_key => $consumer_key,
consumer_secret => $consumer_secret
);

#q: Search Query. I am using none
#geocode: geographical coordinates for search
#rpp: results per page

my $statuses = $nt->search({ q=> $search, geocode=> $geocode, rpp=>$rpp});
$results = $statuses->{results};

The Net::Twitter object constructor requires the account, and application information. Then a search request is made. The response will be similar to the JSON object that is returned via the search api. To access the posts, we have to call “result” property which contains all the Twitter posts.

Clean and Filter Twitter Posts

This step is pretty straightforward. We take the results from our previous step, and we loop through each post, and run a set of search and replace regular expressions against the post data. We are specifically looking to remove things such as URLs, HTML characters, and other data specific to Twitter such as the abbreviation RT.

Extract Significant Terms

For this step we will need to compile the cleaned data into a large string. Once the data is compiled we can post it to the Yahoo Term Extraction Service (YTES). Again there is a free CPAN module for the YTES. Before using the YTES I had to sign up for a Yahoo Developer API Key. The provided module is very programmer friendly, just plug and play.

Accessing the Yahoo Term Extraction Service:

#appid: Yahoo Application/API key
#context: content to extract from
my $yte = WebService::Yahoo::TermExtractor->new( appid => $appid, context => $text);

my $terms = $yte->get_terms;

Define a yahoo term extractor object with You API key and a context. Then call the get_terms function, and it returns an array extracted terms.

Rank Term Occurrence

Next we rank the output from the last step. For each term we loop through every tweet, and check to see if it occurs. If it does we add a point to the term.

Record Term and Tweet Data

Next we save the list of terms and their counts to a historical data store. We also save the Twitter posts. These are saved so we can filter out already sampled posts.

Consume Historical Data

Finally we pull a list of the top trends, and output it as a Twitter Post. We use a simple SQL query to pull a list of trends by their occurrence in descending order. Then using Net::Twitter to post it to an account.

Net::Twitter Update Code

my $result = $nt->update(“$text”);

Sample Output:

Using the data store from June 25th, it produced this data from Des Moines, IA the scripts determined the following terms were ranked highly:

michael jackson, farrah fawcett, iphone, risingstars, king of pop

This was pretty easy to validate, because news of Micheal Jackson and Farrah Fawcett dominated news coverage. Also there was the a Republican event called Rising Stars in Iowa that week.

This is not “true” data mining it is a bastardized process. It does have some bugs, but it tends to be very accurate when definite trends appear.

Des Moines Twitter Trends.

Bookmark It

The Google App Engine does lock you down to using Python. I took this as a challenge to delve into Python. The App Engine does not use a traditional relational database, instead it uses BigTable built on Google File System. The App Engine is also scalable, and completely hosted by Google.

This is checklist to get an application running on Google App Engine:

• Sign up for an Application (Requires a Google Account)
• Download the SDK (Requires Python 2.5)
• Create an application using the template provided in SDK
• Upload your application to the AppEngine

Some notes on the App Engine programming, you must use python. You are limited to standard Python 2.5. They provide some extensions, and a limited version of Django. You can upload other “pure Python” extensions.

View my simple Python CGI application for displaying the Twitter trends API call.

My view of the App Engine is it is a service that has potential. The service requires you to download the SDK and work for the command line, so if you are not comfortable with this then it probably is not the service for you. I will probably continue to play with the App Engine, becuase it has some real potential, and I want to learn more about working BigTable.

Once the service is more refined it could be a really smooth way to develop highly scalable web service. Look out for this one as a major player in the cloud market if they decide to take it mainstream with other languages beyond Python.

Here is the source code, also this was my first real dive into Python, beyond just toying with the interpreter (I love it!!!1!)

#!/usr/bin/env python

import wsgiref.handlers

from google.appengine.ext import webapp#url fetchingfrom google.appengine.api import urlfetchfrom datetime import datetime#use the simplejson library, lucky it was in djangofrom django.utils import simplejsonimport time

#Output the html ick, but it's just a exampleoutput = '<html><head></head><body>'output += '<h1>Twitter Trends</h1>'output += '<ul>'

# Make the API call to twitterurl = "http://search.twitter.com/trends.json"result = urlfetch.fetch(url)

#Status code ok?if result.status_code == 200:json = simplejson.loads(result.content)#Loop through the trends listfor i in json['trends']:output += '<li>'output += '<a href="' + i['url'] + '">'output += i['name']output += '</a></li>'

#Finish out the requestoutput += '</ul>'output += '</body></html>'

#The primary handler, just outputs outputclass MainHandler(webapp.RequestHandler):def get(self):self.response.out.write(output)

#Defines the calls through the applicationdef main():application = webapp.WSGIApplication([('/', MainHandler)],                                  debug=True)wsgiref.handlers.CGIHandler().run(application)

if __name__ == '__main__':main()

Bookmark It

A quick how to:

To start out you must import the api using your Google API key.

<script src="http://www.google.com/jsapi?key=[Your Google API Key]"></script>

This is script was taking from the Google AJAX API Playground, I just added in the call for Twitter, the full script is below.

First the Google feed object is instantiated with the call to the Twitter API via RSS.

var feed = new google.feeds.Feed("http://twitter.com/statuses/user_timeline/ianlintner.rss");

The Google feed API is great way to access and work with remote feeds, and APIs using just JavaScript. This also an asynchronous call, so it should play well with other AJAX solutions.

The full listing, with my changes.

<!--Copyright (c) 2008 Google Inc.

You are free to copy and use this sample.License can be found here: http://code.google.com/apis/ajaxsearch/faq/#license-->

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="content-type" content="text/html; charset=utf-8"/><title>Google AJAX Search API Sample</title><script src="http://www.google.com/jsapi?key=[Google Key]"></script><script type="text/javascript">/**  How to load a feed via the Feeds API.*/

google.load("feeds", "1");

// Our callback function, for when a feed is loaded.function feedLoaded(result) {  if (!result.error) {    // Grab the container we will put the results into    var container = document.getElementById("content");    container.innerHTML = '';

    // Loop through the feeds, putting the titles onto the page.    // Check out the result object for a list of properties returned in each entry.    // http://code.google.com/apis/ajaxfeeds/documentation/reference.html#JSON    for (var i = 0; i < result.feed.entries.length; i++) {      var entry = result.feed.entries[i];      var div = document.createElement("div");      div.appendChild(document.createTextNode(entry.title));      container.appendChild(div);    }  }}

function OnLoad() {  // Create a feed instance that will grab twitter  var feed = new google.feeds.Feed("http://twitter.com/statuses/user_timeline/ianlintner.rss");

  // Calling load sends the request off.  It requires a callback function.  feed.load(feedLoaded);}

google.setOnLoadCallback(OnLoad);</script></head><body style="font-family: Arial;border: 0 none;"><div id="content">Loading...</div></body></html>

Bookmark It