As programmers we forget the fact that many of the problems we encounter every day have already been solved. We don’t need to write a sort algorithm, create a new type of data structure, or write an encryption library from scratch.  We try not to repeat ourselves, but we have no problem repeating every one else.

I know that programmers love acronyms, play on words, and guidelines. Therefore, I am proposing the DRAW principal. Don’t Repeat Anyone’s Work.

This is defined as: Not solving generic and common problems that have already been solved. This may take many forms, it could be as simple as using a common library, or it may be more in depth such as reading the code of an open source application, or even looking up a general form of an algorithm.

This is a problem that many programmers have faced for a long time, and we have complained about it for a long time here, here, and here. Sometimes we think we can solve a problem very quickly, sometimes we don’t know some one has already solved it, we may even think we can do a better job, or just maybe we are doing it for hack value. Whatever the reason we have all, took the time to solve a problem that has been solved before, that could have been handled with a quick Google search.

Many times Google will return an answer you are looking for. Last week I had to parse a comma delimited list, passed as a string from from a web service I was integrating into a .NET application. Below is the Google search I used, which return a number of results. Parsing CSV data is something that has been solved thousands of times before. It was that straightforward.

Part of being a good software developer is knowing when to be lazy, and find an already existing solution and when you have to hand code something. The basics of this principal is being knowledgeable about the language you are using, and being able to research problems, and leverage the resources of the developer community at large.

Footnote this is play on the DRY Principle

Bookmark It

Programming in assembly requires you to think in bits and bytes. You must allocate your own memory. You have to move data into registers and directly perform arithmetic and logical operations on bits and bytes of data, rather than just defining a for each loop and iterating through objects. Using assembly is in effect directly telling the processor which instructions to execute on which data. You learn how to do the heavy lifting rather than having a high level language do it for you.

Assembly programming can help shore up your knowledge of existing languages. One area where assembly is very effective is understanding memory management. Even though memory management is now handled by modern language with garbage collectors. In assembly we have to explicitly define how much memory we need. We may define 64 bytes for an array, rather than just defining a variable for a list and initializing the object without ever knowing how big that list is. Understanding how memory is allocated can help us minimize the overall foot print of all of our applications and be more efficient.

Beyond this experience with assembly allows you to experience how processing happens. How expensive it is to move things to and from memory and processor. How expensive and slow accessing non memory data such as disk or network resources. It gives you a better feel for bottlenecks in applications. This knowledge becomes extremely valuable when optimizing applications, and understanding how and when to make time and space trade offs.

Overall assembly programming showed me the whole picture of programming from typing in code, compiling the code, and finally running an executable.  This solidified many of the concepts I had learned, how algorithms work and why some are orders of magnitude more efficient, and how data structures are allocated and processed.  These higher level concepts directly improved my knowledge of programming in high level languages, and allowed me to make wiser technical decisions because of it.

Bookmark It

The programming languages you dine on is based in the same principals as your culinary choices. These decisions are based in your culture. Many factors which contribute to your technical culture: where you learn, where you work, and where you play. Reading these cultural factors can help us understand why a person or an organization picks the languages they do.

Some of us choose us prefer a buffet of languages, and like to sample a little bit of every language. Some choose to stick with a couple core choices and savor every little piece of a language. On the other hand you have your diet specific programmers, such as you free software-gens, and the script-arians.

Sometimes our jobs require us to use a specific language, or the technology requires it. Utility is largest factor in choosing languages. Secondly what you learned and perfected you programming is a close second to choosing which language to use. Tertiary factors are general community factors, such as what are the current trends, what would you like to learn. All of these factors relate to how we choose food, what’s available to us to buy, what our families have eaten, and what we are recommended.

On a side note. The programming languages you choose to dine are signals to other programmers. Just as you culinary choices are signals to other people. These signs, and signals many times are more rooted in stereo type. So many programmers may learn a specific language to affect their appearance to others, just as some people make dining choices to impress other people.

Finally as I discussed earlier the biggest factor in choosing a programming language is the utility, and what we cut our teeth on. Just as many times primary food choices are based on utility, and then what we learned from our families. Understanding these factors can help us grow as programmer, and understand what programming languages we consume.

Bookmark It

The web view of the data has been upgraded to provide more information. The Des Moines Twitter Trends website now provides a clearer view of the data collected. It provides current trends for the past few hours, the past day, and the past week. All of the terms are now links to Twitter Searches.

I added some new information, such as recent links, and current view of the latest Twitter user’s avatars in a grid.

I also created my own theme, and display instead of using a third party Wordpress Theme.

P.S.
Here was the write up for the first version: http://www.dotcodedump.com/2009/07/finding-local-twitter-trends.html

Bookmark It

the degree to which a set of inherent characteristics fulfills requirements — ISO 9000 : 2000

The definition is interesting, because it makes no statements about composition or any other property. The only statement it makes is about the fulfillment of requirements.

Software exists because it fulfills a need. These needs are translated into requirements. Secondly software is expected to have certain qualities by it’s users. These are requirements which are many times unwritten. If software does not fulfill most or all the requirements it will be interpreted as poor quality.

Here are some examples of unwritten requirements:

Bugs should be kept to a minimum.
Software should perform well.
Software should be very usable.

Unwritten requirements are not easy to fulfill, and they are ambiguous and subjective. Clearly defining these requirements and expectations. This process is integral to developing software that fulfills users needs and expectations.

Misunderstanding quality is dangerous, and leads to false beliefs. These beliefs cause developers to ignore parts of the software development picture. The only way to consistently build quality software is to define not only the functional requirements, but also the unwritten requirements and expectations.

Bookmark It

Local or Targeted trends on Twitter is not something that Twitter provides. So I decided to create my own solution. For this project I used Perl, Twitter API via Net::Twitter, Yahoo Term Extraction Service, and SQLite3 to store data. This technique is a quick and dirty way to perform a simple data analysis on Twitter posts, and extract current conversation trends.

To view the most recent output from the process please go to my site: Des Moines Twitter Trends. (Updated for V2)

Process Overview:

1. Get New Local Twitter Posts
2. Clean and Filter Data
3. Extract Significant Terms
4. Rank Term Occurrence
5. Record Term and Tweet Data
6. Consume Historical Data

Process Details:

Get New Local Twitter Posts

I used the Perl Net::Twitter API interface to download new Twitter posts. This was freely available via CPAN. This provided an easy way to interface with the Twitter API. I created a new account, and registered my application with Twitter. I used this account information, and Net::Twitter to download Twitter Posts via the Twitter API.

Code for Defining Net::Twitter, and downloading tweets:

#$user: Twitter Account Username
#$password: Twitter Account Password
#$consumer_key: Twitter Application Consumer Key
#$consumer_key: Twitter Application Consumer Secret

my $nt = Net::Twitter->new(
traits => [qw/API::REST API::Search OAuth/],
username => $user,
password => $password,
consumer_key => $consumer_key,
consumer_secret => $consumer_secret
);

#q: Search Query. I am using none
#geocode: geographical coordinates for search
#rpp: results per page

my $statuses = $nt->search({ q=> $search, geocode=> $geocode, rpp=>$rpp});
$results = $statuses->{results};

The Net::Twitter object constructor requires the account, and application information. Then a search request is made. The response will be similar to the JSON object that is returned via the search api. To access the posts, we have to call “result” property which contains all the Twitter posts.

Clean and Filter Twitter Posts

This step is pretty straightforward. We take the results from our previous step, and we loop through each post, and run a set of search and replace regular expressions against the post data. We are specifically looking to remove things such as URLs, HTML characters, and other data specific to Twitter such as the abbreviation RT.

Extract Significant Terms

For this step we will need to compile the cleaned data into a large string. Once the data is compiled we can post it to the Yahoo Term Extraction Service (YTES). Again there is a free CPAN module for the YTES. Before using the YTES I had to sign up for a Yahoo Developer API Key. The provided module is very programmer friendly, just plug and play.

Accessing the Yahoo Term Extraction Service:

#appid: Yahoo Application/API key
#context: content to extract from
my $yte = WebService::Yahoo::TermExtractor->new( appid => $appid, context => $text);

my $terms = $yte->get_terms;

Define a yahoo term extractor object with You API key and a context. Then call the get_terms function, and it returns an array extracted terms.

Rank Term Occurrence

Next we rank the output from the last step. For each term we loop through every tweet, and check to see if it occurs. If it does we add a point to the term.

Record Term and Tweet Data

Next we save the list of terms and their counts to a historical data store. We also save the Twitter posts. These are saved so we can filter out already sampled posts.

Consume Historical Data

Finally we pull a list of the top trends, and output it as a Twitter Post. We use a simple SQL query to pull a list of trends by their occurrence in descending order. Then using Net::Twitter to post it to an account.

Net::Twitter Update Code

my $result = $nt->update(“$text”);

Sample Output:

Using the data store from June 25th, it produced this data from Des Moines, IA the scripts determined the following terms were ranked highly:

michael jackson, farrah fawcett, iphone, risingstars, king of pop

This was pretty easy to validate, because news of Micheal Jackson and Farrah Fawcett dominated news coverage. Also there was the a Republican event called Rising Stars in Iowa that week.

This is not “true” data mining it is a bastardized process. It does have some bugs, but it tends to be very accurate when definite trends appear.

Des Moines Twitter Trends.

Bookmark It

There are thousands of blogs on software development. Many of these blogs demonstrate, promote, or even proselytize tools or methodologies. The large amount of information provides developers with a large choice of tools and ideas. Searching for proper ideas and tools is very time consuming.

Software developers put knowledge in high regard. Learning is even more valuable. A good software developer should find a balance, between learning and doing. Programming is not philosophy. Making and creating is far more valuable to good programmers. Good is what works, and works well, not what is academically perfect.

Finding tools and even using tools can become a distraction. Learning when your tools are getting in the way is skill, which is underdeveloped, in many programmers. A perfect example is how programmers can get into holy flame wars over a TEXT EDITOR! This is just one example of where the meta-product, becomes more important than the product, but there are many others.

Ironically I am advocating learning how to ignore programming noise, in a blog.

The Lintner Code Complexity Analysis Test: Next time you start incorporating, tools, methodologies, and other ideas into your code base. Ask yourself could an average developer jump in and get the basic flow, and meaning of the code? If no, you may want to rethink your approach.

Personally I advocate simplicity, and minimalism when it makes sense, and some level of complexity is necessary. Tools create complexity. Tools require knowledge to operate, and they will change how your process works. When choosing tools, you must be aware of how they will effect your software development ecosystem over time.

Technologies, tools, ideas, or just plain noise can get in the way of creating, and making. Many times a small project or tasks can be finished the without tools, in the time it would take to set up, and use complex tool. There are many examples of high quality software that was created without code analysis tools, unit tests, and even a structured development methodologies. Next time you start working on a new project, you may want too evaluate your tools too see if they are truly valuable or just wasteful.

Bookmark It

When making changes to a section of code, causes other bugs, sometimes recurring, to pop up. I.E. You fix one bug, and three other bugs pop up after you release.

Have you ever worked on code where troubleshooting bugs was like playing a game whack a mole. I have worked on a few applications that seem to have taken this idea to heart.

If you encounter an application that not very stable, many time your options for dealing with it are limited. You just have to work within the constraints of the system. Previously I have mentioned, defensive development as a technique to cope with difficult system. This is an example of controlling what you can, the new code, and fixes you introduce into the system.

When dealing with difficult systems your best weapon is going to be communication. Make everything as clear as possible. Make your new code readable, understandable, and most importantly comment what you changed, and why you needed to change it. Keep documentation on the system. Document critical, and time consuming tasks. If a process took you a week to update, document it, and it will take the next guy half as long.

The most important technique for dealing with difficult systems is asking questions, and making sure you understand what people are asking. Many times “whack a mole” conditions are caused by mis-communication, and lack of knowledge.

Maintaining difficult systems is a reality for many developers. This is as important, or more important of a development skill as creating new applications. Many time maintenance programming is far more difficult than new development, because you have to work outside of your code “comfort zone”.

Bookmark It

Recursion

If you are using recursion there may be a better way, or I like to say just because it fits into recursive paradigm, it doesn’t mean it HAS to. There are many times I have found myself ending up doing something recursively, and regretting. My word of warning: “It starts with a couple of recursive functions, and then you wake up 3 days later in the middle of New Mexico in playing Russian roulette with Bill Richardson. Recursion has it’s uses, but with it you probably are going to incur a performance cost that may not be worth it.

Nesting Logic

Any time you start nesting logic beyond 1 or 2 levels this should be a warning sign that something could be a miss. This is where debugging is generally going to start getting difficult. If you are having difficulty coding a section, and it involves nesting of logic this a point where you need to step back and get a second opinion on how to do it. I am by no means saying don’t do it, but watch

Really Long Functions

Writing really long functions is generally a sign of two things, either there is a lot of logic to parse through, or you may be doing something poorly. This may also be an indicator that something else outside of your current scope is amiss. This is generally a point to ask questions, and thoughts on how to program a specific section.

Maze Like Program Flow

If you are constantly searching for functions, and logic, and your program looks like a maze this is probably a point where you should take a step back. This means that either there is some really sloppy code, or there is very high level of systematic complexity. If the complexity does not match the task you are trying to achieve this may a point to step back and take a look at the big picture, and possibly get a different perspective on the situation.

There are many other sets of guidelines, many other people have created. This a small list of warning or “yield” signs that I use regularly. This reflects my general bias for trying to make things as simple as possible, and reducing complexity where ever possible.

Bookmark It

A quick how to:

To start out you must import the api using your Google API key.

<script src="http://www.google.com/jsapi?key=[Your Google API Key]"></script>

This is script was taking from the Google AJAX API Playground, I just added in the call for Twitter, the full script is below.

First the Google feed object is instantiated with the call to the Twitter API via RSS.

var feed = new google.feeds.Feed("http://twitter.com/statuses/user_timeline/ianlintner.rss");

The Google feed API is great way to access and work with remote feeds, and APIs using just JavaScript. This also an asynchronous call, so it should play well with other AJAX solutions.

The full listing, with my changes.

<!--Copyright (c) 2008 Google Inc.

You are free to copy and use this sample.License can be found here: http://code.google.com/apis/ajaxsearch/faq/#license-->

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="content-type" content="text/html; charset=utf-8"/><title>Google AJAX Search API Sample</title><script src="http://www.google.com/jsapi?key=[Google Key]"></script><script type="text/javascript">/**  How to load a feed via the Feeds API.*/

google.load("feeds", "1");

// Our callback function, for when a feed is loaded.function feedLoaded(result) {  if (!result.error) {    // Grab the container we will put the results into    var container = document.getElementById("content");    container.innerHTML = '';

    // Loop through the feeds, putting the titles onto the page.    // Check out the result object for a list of properties returned in each entry.    // http://code.google.com/apis/ajaxfeeds/documentation/reference.html#JSON    for (var i = 0; i < result.feed.entries.length; i++) {      var entry = result.feed.entries[i];      var div = document.createElement("div");      div.appendChild(document.createTextNode(entry.title));      container.appendChild(div);    }  }}

function OnLoad() {  // Create a feed instance that will grab twitter  var feed = new google.feeds.Feed("http://twitter.com/statuses/user_timeline/ianlintner.rss");

  // Calling load sends the request off.  It requires a callback function.  feed.load(feedLoaded);}

google.setOnLoadCallback(OnLoad);</script></head><body style="font-family: Arial;border: 0 none;"><div id="content">Loading...</div></body></html>

Bookmark It