Before I go on with this post, I need to lay my preferences down on the proverbial table. I don't loathe Java, but I definitely lean towards more dynamically typed languages for my own projects. I see Java as a corporate language that really has very little application in open source (which is where I spend most of my time). I'm not going to say anything like "Java sucks" but I will say "I don't much care for Java."

I've been doing a lot of JSP/Servlet work recently. It's more for research than for production use. As someone who's done GOBS of web development, I didn't think it'd be too difficult. Writing the code, getting the syntax under my belt, all these things were rather simple. I've just grown use to the idioms used in web development as a whole, and so that wasn't too difficult (Eclipse made it easier as well, once I found a good JSP plugin).

I think the biggest problem to learning a language is running the code itself. My 12-year-old brother was up and running with python pretty quickly, having Ubuntu's Synaptic installer take care of installation, and then using the python interpreter. C, C++ and Java are a little more complicated. They require knowledge of rather complicated tools, provided you're doing anything more than "Hello World!" There are compilers, linkers, etc. Shoot, in all fairness, even writing your own python modules requires knowledge of how import works.

So I'm writing my Java web app, and I'm getting Tomcat all set up to work. Then I realize that there's more to deploying this app then I originally thought. Now I've got to write an ant build file to automate builds (because compiling all these classes and moving them all to the appropriate directories just isn't practical). After that, I've got to deploy it to the Tomcat server. The barrier to entry is just too high.

When I first started playing with Django, the one thing that really grabbed me was the development server that ships with Django. No mod_python/mod_wsgi/mod_fcgi/whatever to mess with just to see if I wanted to invest more time into the tool. When I started with jQuery and ExtJS, I had the javascript terminal in Firebug to play around with the API and find its strengths and weaknesses.

I'm not saying that a language should necessarily be "so easy a caveman could do it," but I do think that there needs to be a low barrier to entry. If it gets complicated later on, thats fine. Shoot, make some really simple, easy to follow tutorials on how to use the tools. Make explanations on the really simple things, like how library paths are structured, etc. Create tools that don't require 200 different command line flags to compile anything more complicated then your standard "Hello World!"

I'm a dynamically typed language fan boy. I've dabbled in all sorts of languages, but got my first development job working with PHP and Javascript, graduated to Python, and then did the C/C++ and Java stints as well. One thing I'm quite used to doing is writing print statements for debugging. In large C/C++ or Java environments, I found this quite impractical, and eventually became quite intimate with debugging tools in those languages. Until recently, however, I continued using print statements for working with scripted languages like Python. Now I use pdb for almost every problem I encounter in python.

A while ago, I battled a nasty Entertainer bug. It was repeatable, but not consistently. I would start the backend, and the it would start indexing videos from the configured video folders. Then, for no reason, the backend would die. I was able to trace the problem back to the VideoThumbnailer, which would be my baby. Crap. So I started adding print statements, so that I could see if the thumbnailer was consistently dying on the same video (thinking it might have been a codec problem with GStreamer). What I found was that the backend rarely died in the same spot, and since the thumbnailer didn't create thumbnails for videos that already had thumbnails, once it got past a video it had failed on before, it never struggled on it again. However, with the multi-threaded environment of entertainer, it was quite difficult to debug.

Eventually, I followed the yellow brick road to see the Google. I typed in 'python debugger' and found the first page of results full of pdb articles. "Oh no," I thought, "anything but the confusing and difficult pdb!" Alas, once I forced myself to work with it, I found that it was very simple to use (and that the pydev eclipse plugin makes it so much more complicated than it needs to be). I'm still only using a small subset of the things that pdb can do, but frankly, I can't see a need for anything more than what I used it for (setting a breakpoint, reading local variables, seeing a backtrace, and stepping through code).

First, I decided where I wanted to have pdb stop and give me a debugging console (setting the breakpoint). When that was decided, I added the following code:

import pdbpdb.set_trace()

That's it! Now run the code, and you'll find that instead of complete execution, you'll be at the (Pdb) prompt. Above your prompt, you should see the pdb.set_trace() call that created the console. I'm not one of those people who fears a command prompt, so if you are, I apologize. However, this isn't nearly as complicated as a shell prompt. You've got only a few keys to worry about:

• n (next) - Hitting 'n' at this prompt will execute the current line of code and go to the next line.
• c (continue) - If you've gotten to the point where you know your bug is fixed, and you want to just close the debugger, hit 'c' to just continue executing the program. This will complete execution, provided that you have no more calls to pdb.set_trace()
• s (step into) - If you're at a function or method call on the prompt, and you'd like to see what's going on inside that function, stepping into it will allow you to follow execution into the function. If you don't want to see what's going on in the function, 'n' is your character.
• l (list) - Wait. Where was I in the code again? Yea, I guarantee that'll happen to you. Hitting 'l' will give you a chunk of code in the area you are, with a nice little -> where you are currently.
• bt (backtrace) - Okay, so now I know where I am, but how did I get here? 'bt' gives you a backtrace, showing you the start of execution, and all the function and method calls that got you where you are today.
• p (print) - Okay, so what's the value of variable foo? 'p foo' will print it's value, allowing you see how the variable is changing as you walk through the code.
• q (quit) - Ah! Found the bug. Let's quit this session and fix it!
• enter (repeat last command) - Tired of hitting 'n' over and over? How 'bout just using 'n' once, and then hitting 'Enter' over and over. It's a bigger key, and less likely to get fat fingered. I don't use it, but it's there if you want it.

Needless to say, debuggers are the tools that would make your life easier if you just learned how to use them. Since my experience with the debugger, I've chatted with a few pythonistas and developers from other languages, and to my surprise have received varying responses. I would say that if you've spent more than 5 minutes writing print statements and running your code, do yourself a favor and fire up the debugger. There might be a learning curve now, but you'll be a better developer for it.

I've been trying to get lirc support into Entertainer the last few days, and while SOME of pylirc is still able to work, it's incredibly old. I contacted to current maintainer about this, and haven't heard back from yet. Since I need to move quickly to get this module updated, I have mirrored the Sourceforge CVS repository and converted it into a Mercurial repository. It can be found at:

http://hg.ironlionsoftware.com/pylirc2

Please note that I don't intend to fork the project or anything like that. I just plan to continue it's maintenance, specifically for Entertainer, but for the use of anyone who would like to have python bindings to lirc.

I've been posting a lot about test driven development recently, and I got an email from a reader who asked me to go into detail on how I've been going about discovering and working with tests. I think this type of post would have helped me develop the "test first" mindset so much earlier in my career, so I'm sitting down to actually detail exactly how I'm learning about this world that I've always heard about, but only now have visited. I guess an alternate title for this post could be "Python Unit Testing 101"

Because I try to spend at least 10 hours a week on Entertainer, and it's already got a pretty good test base that I have lots of experience in (because I wrote a large chunk of it), I think it would in everyone's best interest if I used a new Entertainer test as an example. If you would like, you can check out the code and see exactly what I've been doing by looking in trunk/src/tests. To run the tests, just navigate to the test directory, and type ./run_tests.py You'll see the tests grind away, telling you which ones passed and which ones didn't (hopefully, all of them are passing when you check out the code...)

Abstraction - The Seed of an Idea

I've been maintaining (or trying to maintain) the thumbnailers, ImageThumbnailer and VideoThumbnailer. They both inherit from an empty abstract class Thumbnailer. Recently, when trying to hunt down a bug, I noticed similar code in both VideoThumbnailer and ImageThumbnailer that could easily be rafactored to an already existing Thumbnailer class. For reference, I actually found this while hunting down a bug in ImageThumbnailer, and writing a test to make sure the bug stayed squished. As I planned out exactly what the Thumbnailer abstract class needed to look like, I started thinking about the test I needed to write first.

The abstract Thumbnailer class looked like this: thumbnailer.py

Planning - Writing the Test

The most important thing to remember about TDD is that your tests describe the way your code should behave, not the way it does behave. For instance, if you find a place where the code should be refactored to behave differently, write the test that way, make sure it fails in the way you expect, and then you can refactor the code to pass the test. That's what I did with the ThumbnailerTest. Since there wasn't any code/logic in the Thumbnailer class, all the tests technically should fail. But I had a few things in mind in refactoring:

1. The Thumbnailer should have logic to tell what type of thumbnail it will create, so it can set it's destination path properly. The types are determined by what folders exist in the thumbnails config directory (currently image, video, and recording). It should throw an exception if it's an invalid type.
2. The Thumbnailer should only try to thumbnail an image or video that exists. It should throw an exception otherwise.
3. It should make sure that the classes that should be overridden by the child classes exist. I'll use a fun little technique for this that I learned only recently.

Obviously, I now have my tests. Let's look at item one. I like to break that test up into two tests, one that tests that each type can be properly parsed and recognized as valid, and one test that throws junk data as the type, and ensures that it throws an exception. I created my test by inheriting from unittest.TestCase, creating a setUp() method, containing only a debug flag (which you can use to make sure your test is working...), and then I start creating my tests.

The result of these tests is the following: Thumbnailer_test.py

On A Tangent - The Test Runner

Before we run the new test in the test suite, take the time to look at the run_tests.py script. run_tests.py

You'll notice that the script searches through the current directory for python files (except those in the blacklist), and using the unittest.TestLoader and unittest.TestSuite classes, loads the those files as modules, and then runs the TestCase child objects. The output of this script is the results from the tests. It's nothing terribly complicated, but then again, if your tests or test harness are complicated, you probably ought to change the implementation.

An alternative to the script used in Entertainer is python-nose. By installing it, you can run each test file yourself by doing nosetests <filename>. It will automatically find the unittest.TestCase instances in your file, and run them, printing the results to the terminal.

The Test Is Failure

Back to the tests at hand, you'll notice they all fail. This is good! If they passed, something is definitely wrong with your tests. Look at the errors for the failed tests and make sure they are what you expect. For instance, if you're trying to call a method that doesn't exist, make sure you've got an AttributeError exception, etc. If they are failing the way you expect, then you can consider your tests to be working properly. Now your job is to hack on the class you're testing until it passes the tests.

While you're writing this class, hopefully you're starting to see the value of the test. It was when I was working on the VideoThumbnailer that I started realizing how great TDD is. Instead of firing up the UI every time I made a small change to the VideoThumbnailer, I just ran the test to see if it was generating thumbnails. So instead of making a change, firing up the ui, navigating to the videos, and checking for thumbnails (and at the time, the UI was in a great state of flux at the time, and was only working for me every once in a while. Having the tests meant making a change, running the test runner to see if the test passed. No dependence on the UI working or anything of the sort.

Your Code Works Because the Test Says It Does

After a quick iteration of work (because you've got the test to run instead of the use case), you've now got working code. You know it's working because your test is passing. You know your test is working because you wrote it, and watched it fail exactly the way it should.

The Thumbnailer class now looks like this: thumbnailer.py

Now you can feel safe in committing your refactored code. Because I already had tests for ImageThumbnailer and VideoThumbnailer written well, the tests required no changes in order to test the code I removed to the abstract class. The general rule is that your class' interfaces should be simple, and shouldn't have to know about the "guts" of another class. Because it was constructed well, the tests worked without a problem.

As you write tests, remember that you're writing code still. I've seen lots of people write the tests and see them pass, and then work on the code, and the tests fall out of maintainability. When you make drastic changes to your code, make the tests reflect that, or rather, make the tests reflect the changes first. When you have tests, your codebase is 2-4 times the size of the code that actually gets deployed, so you've got more code to maintain. However, hopefully the maintainence of the code gets easier as the testing gets better (and if it doesn't, something is wrong with either you, or your codebase).

On Friday, I decided to plan a sprint for the weekend. It required a little more coordination than I originally expected. With the developers strewn from Finland and Poland, to the UK, to Maryland, and finally, myself, in good ol' -7 GMT, there's a 9 hour difference. This required me to get up quite early on Saturday, and spend a whole weekend hacking on Entertainer.

After a whole weekend working with two other developers, and very good up-and-coming developer/tester, and the occasional testers dropping into the IRC channel to say hi, file a bug, and then leave, we got a lot accomplished. I was finally able to complete a working VideoThumbnailer class (which meant reading every gstreamer doc, and tons of blogs, and still learning from trial and error). Joshua Scotton was able to complete the news reader portions, with support for Liferea feeds, etc. Lauri has successfully written cd playback, and is hoping to get DVD playback in soon. We also now have 10x the tests than we had on Friday evening (from 4 to 41), with all but two passing, and four of which were tests that revealed bugs (and the two that aren't passing are among them). A total of 1,728 new lines of code were introduced into the codebase, over 25 commits. Documentation was updated and confirmed to be detailed. 10 new bugs were filed, and 7 reported bugs were closed. A large amount of code was refactored using our newfangled test framework, and we were able to find a few issues with architecture and coding practices, optimizing as we went.

Overall, the sprint was an absolute success. We were able to talk to eachother, and determine what was required for a 0.1 "preview" release, and what we could push off for another version. I made some great progress on converting all of our current ASCII string data to UTF-8, although there is still a bit of room for improvement (and I'll be working on that off-and-on for the next while).

I'm hoping that we can sprint once a month on Entertainer, and really get a neat, polished app soon.

I've slowly been reading the Python IAQ, a compilation of "infrequently asked questions" for Python. This fascinated me when I first stumbled over it because I had just learned that prefixing a member attribute of an object with '__' made the attribute pseudo-private. I'd never used this idiom before, so I didn't know, and it only surfaced when I was trying to write unit tests for code that I hadn't actually written. My curiousity at what else I had been missing out on got the better of me, and I started reading the article a bit at a time compiling another project I'm working on.

Some of it I grew tired of, because it was trying to make Python into C/C++/Java/pick-your-favorite. One my favorite things about Python is its idioms and differences with other languages, so I wasn't thrilled about making Python look like Java. But after reading through those first few questions, I found quite a few little tricks, like implementing an abstract class in python, which I've always thought about. It uses a few neat little tricks in python to do things that otherwise wouldn't be simple.

It's a good read. Go read it. Do it. Do it now.

I've been working on a new project, called Entertainer, which is aiming to be similar to MythTV, but with a featureset devoted more to end users than MythTV is. It's got a pretty slick interface (although some of the interface code is based on an alpha library right now), and has been quite educational for me, especially since I've been working specifically with the gstreamer python bindings.

Recently, I've been working on a VideoThumbnailer for Entertainer using gstreamer. While I worked on it, another developer broke the actual GUI code (or rather, a widget from a library Entertainer's GUI depends on broke), which completely halted me from testing it (Entertainer is pre-alpha). This was okay, because I had only been reading documentation on the Gstreamer, and hadn't quite gotten to code yet. However, before I got to coding, I decided that I needed a way to the thumbnailer, and with the frontend broken, I couldn't very well do that. So I decided to write a unit test.

After stubbing out the unit test, I got to work. Originally, I was using nose and then eventually just creating my own test runner, to eliminate dependencies (we already have a ton of them...). I put a lot of work into the test framework, making sure output was clean and everything.

It was only after all of this, and I was working on the thumbnailer and running the unittest to see if it worked that I realized I was doing test-driven development. It was something I had always read about, but had never done. I will swear by it now. It's made my life so simple. Even now that the GUI in Entertainer is working again, I don't much care to use it to test. I've got a unittest that tells me all I need. If I need more, I'll modify the unittest.

What's even better is that the rest of the Entertainer developers are also getting on the bandwagon. We even have a developer who's never written python before that will be writing the tests for a lot of the code we have. How cool?

I stopped reading Slashdot two years ago. Compared to news sites that are more "web 2.0" compliant, and driven more by the community, Slashdot consists of the same rotten community comments of Web 2.0, without having anything that can merit "news." However, while exploring more Google Reader features, I stumbled upon these prepackaged feed lists that Google compiles. I added a few, and lo and behold, Slashdot's feed was in there.

Then I read this article about Python 3000's lack of compatibility. Did I miss something? How is this news? Guido van Rossum announced this quite some time ago. I severely doubt Google, with so many of python's developers as employees, is being taken by surprise at this news, or any other respectable Python shop.

Needless to say, Slashdot had a good run of two days in my reader, but I removed it, so that I don't have quite so much "noise" in my reader.

I've been working in a Java environment recently. I've found that no matter how much Java I think I knew in school, it must not have stuck with me. I'm having to relearn data structures in Java, and now that I know a thing or two, I've come to realize that Java's data structures are hugely verbose and rather difficult to work with sometimes. Of course, this is because I've been spoiled recently working in more "fun" languages, like my python...

Consider the following situation. I had two maps that needed to be compared. If you're not a Java guy, think key->value structure. Anyway, I solved the problem by creating a Set (think array) or all the keys that were different between the two, and then showing the results. Here's the java result:

    Set<String> changes = null;
        Iterator it = map1.keySet().iterator();
        while(it.hasNext()) {
                String key = (String)it.next();
                if(map2.containsKey(key)) {
                        if(!map1.get(key).equals(map2.get(key))) {
                                changes.add(key);
                        }
                } else {
                        changes.add(key);
                }
        }                

In python, I just do this:

changes = []
for key in dict1.keys():
    if key in dict2:
        if dict1[key] != dict2[key]:
            changes.append(key)
    else:
        changes.append(key)

Now, I may be nit-picking, but you tell me which one is more clear and concise. Sometimes, being verbose isn't such a good thing after all... (I'm thinking of a specific person who talks way too much)

After looking through my Google Analytics results, I noticed that a few hits to my site were people looking for a howto on integrating django-tagging, which I have been wanting to write for quite some time. While I'm quite experienced working with SQL schemas and the like, there were some ORM concepts that I had never been exposed to until I started using Django (it was my first framework featuring an ORM). Things like Generics confused me, although I knew how to do something similar in plain ol' SQL, it took me a bit to grasp the concepts. So I figure a good way to demonstrate their uses, along with demonstrating integrating django-tagging into an app would be a fine use of my precious blog space...

Please keep in mind that I am not a Django expert, nor am I a django-tagging expert. My experience with django-tagging has been adding it to my blog posts. While I've considered making an entire category hierarchy-type site with tags, I haven't actually implemented it. All I can demonstrate on the basics. You'll have to learn the rest on your own. Also, it's important to point out that I have been using the svn version of django-tagging, although recently, a 0.2 version was released. I just set up my project's svn to use externals and keep me up to date on django-tagging.

First thing's first, you need a model. I'm going to use a dumbed down version of the model I use for blog posts. Lay out your model like this:

from django.db import models

class BlogPost(models.Model):

    title = models.CharField(max_length=30)
    body = models.TextField()
    date_posted = models.DateField(auto_now_add=True)

Now you have a pretty simple blog class. After a syncdb, you can fire up the admin app and start blogging! Er, kinda. You won't have any way for a user to see your posts, but you'll have posts in the database. However, you've got no tags! How would you implement tags into your new fangled blog? Easy. Download the tagging module and install it (I usually just copy the appropriate files to my $PYTHONPATH). Then it's actually quite simple, and you'll kick yourself for not figuring this out. Modify your blog model, and add this:

from django.db import models
from tagging.fields import TagField

class BlogPost(models.Model):

    title = models.CharField(max_length=30)
    body = models.TextField()
    date_posted = models.DateField(auto_now_add=True)
    tags = TagField()

However, you're going to want a method in your model that will allow you to iterate through the tags, called get_tags. I also have a set_tags method, and I know there was a reason I added that, I just can't remember what it was... (a good case for why you should always comment your code). So modify your model so that it now looks like this:

from django.db import models
from tagging.fields import TagField
from tagging.models import Tag

class BlogPost(models.Model):

    title = models.CharField(max_length=30)
    body = models.TextField()
    date_posted = models.DateField(auto_now_add=True)
    tags = TagField()
    
    def set_tags(self, tags):
        Tag.objects.update_tags(self, tags)

    def get_tags(self, tags):
        return Tag.objects.get_for_object(self)      

Now you've got a full blown model with tagging built in. Make sure you've got tagging installed in your INSTALLED_APPS, blow out your database, and syncdb again. Hooray! You'll notice on my blog that I have the tags shown at the top of each post, which is accomplished by with the following template code:

{% for tag in blogpost.get_tags %}
  <a href="/blog/tag/{{tag}}" alt="{{tag}}" title="{{tag}}">{{tag}}</a>
{%endfor%}

I know there are template tags for many of the standard operations for django-tagging, but I haven't used any of them. I noticed that most of them are for tag clouds by model or object, and I would like a tag cloud period, with links to pages on the site with various items that are tagged with the given tag. I have just been far too lazy to actually implement it yet, but there is a weekend coming up...

Edit: I've found some better ways to implement django-tagging, so I've made some changes to this tutorial

A very good colleague and friend of mine were discussing development today. Many of my geek friends are sysadmins, and, while that entails some programming, the conversations are usually heavily weighted in other matters, so this discussion was very refreshing for me.

My friend has most of his experience in Java, and much of my Java experience has been from school, so I asked him a question that no one has been able to answer for me: "Why is it better to have something implement an interface in java rather than just inherit from a base class?" Much of his response is stuff that I had heard already, e.g. multiple interfaces can be inherited (which I don't care for, because I think there's better solutions than multiple inheritance), allowing inheritance from a abstract class and then implementing an interface. All of that was nice, but then he said something that I knew, but hadn't really let sink in. We write code for other developers, not for the machine. The compiler makes sure that your code is readable by machine. If that's what your problem is, fire your compiler, not the developer.

My friend then went on to talk about the concept of Design by Contract, which he originally introduced to me months ago (and I have since elaborated on the concept in my personal time). It was then that he used a word that I hadn't considered using before: culture. He said developers need to get into the habit of developing a culture of programming.

Having lived outside of my native country, and having experienced many different cultures, I can definitely appreciate different cultures and ideas. A perfect example: On Sunday, I like to watch the Denver Broncos play American football. A friend of mine who lives in London prefers the Manchester United. When I lived in the Carribean, everyone watching cricket, and I mean everyone. I love soccer and cricket more than I like American football, but it's a different culture. We Americans are very business oriented, quick and to-the-point, while many other cultures are far more "people oriented," and enjoy knowing about a person long before they sit down to work with them.

What does culture have to do with programming? Simple. My culture is no better or worse than any other culture, but it's something I'm used to. Is it perfect? No. Is it universal? Nope. Does it work in this situation? Yup. It's the same with programming. Having worked as an independent contractor, I've had to learn to integrate into many different development teams. This means that I learn the culture of the team. One PHP team can be much different than another, with different practices, different methodologies, and different implementation ideas. Sometimes, there are better ideas than others, but when you start a project in one culture, it's sometimes easiest for developers to stay with that culture, even if another would have better results in X area or Y implementation. It makes the code easier to read, because that "culture" has been developed.

What are some of the ideas you have used in past development cultures that you find helpful? Is it a mishmash of various Agile practices? Was it a consensus of the team to develop a "best practices" guide, and published in a document or on a wiki? What works for you, and what doesn't?

If you've ever used IPython, you'll know that it's the cat's meow. It's interactive shell adds some nice features that aren't available in the default python shell, and it's helped me in debugging on countless times. The IPython crew is preparing their IPython 0.8.2 Release. Help out the OSS world by testing the new IPython and reporting bugs into their Trac System. You're not doing anything better this weekend anyway.

Happy Thanksgiving! While my wife was cooking Thanksgiving dinner (more like lunch for us...), I finally got around to messing around with a Logitech Dual Action gamepad I've had laying around. It's basically a PlayStation 2 controller with a USB connection on the end. It's got two joysticks, a D-pad, four right thumb buttons, 4 top controller buttons and two buttons in the middle of the controller.

After doing a modprobe joydev, I used a utility called jstest to test the joystick device. Then, it came time to start messing with the device. So out came python, and I found that pygame already had support for reading from a joystick. I had toyed with pygame before, but had never done anything specific. I wanted to see exactly how to read and react to the joystick events. Since I'd never really worked with joysticks before, I wrote a python script that only reads the joystick events and print the events (with a little bit of manipulation to make it human readable).

#!/usr/bin/env python

import pygame
#from pygame import joystick, event

pygame.init()
#joystick.init()
j = pygame.joystick.Joystick(0)
j.init()
print 'Initialized Joystick : %s' % j.get_name()
try:
    while True:
        pygame.event.pump()
        for i in range(0, j.get_numaxes()):
            if j.get_axis(i) != 0.00:
                print 'Axis %i reads %.2f' % (i, j.get_axis(i))
        for i in range(0, j.get_numbuttons()):
            if j.get_button(i) != 0:
                print 'Button %i reads %i' % (i, j.get_button(i))
except KeyboardInterrupt:
    j.quit()

A few caveats I would like to mention that I ran into as I was writing this script. First, I tried to import pygame.joystick only. This allowed me to initialize the joystick, but it wasn't reading events. After reading some documentation, I found that pygame's joystick interface requires pygame.event.pump() to read events. However, only importing pygame.event and pygame.joystick still doesn't work, because event requires the pygame.video interface to be intialized. It lead me to see that pygame really must be imported altogether, and making one single call to pygame.init() allows interfacing with everything.

The entire reason I started this was to interface with an Arduino and control a robotic arm. This will involve pyserial, so I'll be posting more detailed code snippets later that will bring both python libraries together (and hopefully implement a good example of good coding patterns... :)

Bill de Hora has written a great post entitled Python v Ruby Celebrity Deathmatch. Please go read that article, and then come back here. There will be a test...

Many of my posts are about Python. I love python. End of story. I've dabbled in Ruby, and what I've deduced from conversations with many people smarter than I, I've deduced that Ruby has never really made me jump up and sign because it's a pretty good equivalent to Python. As far as features go, they are pretty equivalent. I'm adding my approval to Bill's post.

That is all.

Everyone else has blogged about this, but there is now an official Python LinkedIn Group. Go, join it. While you're at it, you can probably also take a look at my LinkedIn profile.

As a sidenote, I'm starting to prototype some code for a small startup I'd like to, um, start up. --:o) Because of this, I've been reading up on startups, businesses and management (which I know very little about). As a result, I stumbled over Cool Software, a Digg clone that Intel set up to showcase new startups. Not sure what Intel's business strategy is, and Cool Software is pretty young, but it shows promise. Go check it out.

I've finally gotten around to finishing up the third version of The Iron Lion. It's been quite a pleasure to work with Django, and I've seen the light of a good python framework. I feel pretty confident in the work I've done for this site, and as such, I've decided to publish the source for it. There are some out of date tests and docs in there (which is only because it's my own project, so I've been lax there.

The next step for me is to integrate that nice django tagging module, as well as using epydoc to document my code and the doctest and unittest modules from the standard library to get proper code coverage.

The source can be found here

I have recently been fascinated with the sorting. Specifically, I've been working on processing log files into nice HTML pages with graphs and the like. My first conquest in sorting was working with my Squid proxy server access logs. As I've worked through it, I've found that python has incredible resources for sorting. As I worked through the text processing, sorting through and finding the most popular sites, busiest clients, etc. I found a python module that has intrigued me called bisect

With bisect, I can create a list, sort it, and then add new elements to the sorted list in its given location. In C, this is a pretty cumbersome job, and even though it may be faster, the resulting code is much more complicated. As a matter of fact, these log files are quite large (sometimes > 1GB), and the server is only a 1GHz server, and it's still fast enough.

import bisect, random

random.seed(random.randint(1, 10))

l = []
for i in range(1, 50):
  new = random.randint(1, 200)
  bisect.insort(l, new)
  print '%d %d' % (new, l)

The above code results in a randomly generated list of numbers. However, as the numbers are inserted into the list, they are inserted at their proper index with the sort, so as not to corrupt the already sorted list. Try it out in a python shell and you'll see that the list never actually gets corrupted.

Edit: In regards to my comment about C vs. Python, there is apparently an equivalent C implementation of bisect available. If that C implementation is found, it overrides the Python module. I guess it really can be as fast as C, huh?

I've been intrigued with the concept of testing software for quite some time. Unfortunately, the team I work with at my employer is so small that we hardly have time to write tests for everything we've needed. This means that my experience with testing software has been solely on personal, non-critical projects. I've read books and blog posts and entire mailing list threads on the topic of application testing with fascination. The one thing I've seen lacking is information on testing web applications. It seems the mentality of web applications is that since web applications are all written in Java, we can just use any of the Java testing suites. I hope you see the problem....

Yesterday, I stumbled upon Grig Gheorghiu's blog post about his new O'Reilly e-book An Introduction to Testing Web Applications with twill and Selenium which I went out and purchased, mostly because the price was right. I figured if the book sucked, it was a $10 loss. I'd never purchased an e-book before, but it was a subject that appeared to be something I'd definitely be interested in.

I've played with Selenium before, but I didn't find it to be incredibly useful, and I could never get the Selenium Remote Control to work right. I'd heard about twill, but I had never actually experienced it. After having downloaded and written tests in twill, I can see it's exactly what I've been spending a lot of time writing : a DSL to wrap Python's mechanize module for web testing. The limitation I had with twill was with pages that were AJAX driven, because it couldn't interpret Javascript. That's why the combination of twill and Selenium is so great. Because Selenium runs in your browser (it's got a Firefox extension...), it allows the browser to execute the Javascript, thus giving Selenium access to dynamically created DOM elements.

While I haven't quite finished the whole "book" yet, I'd highly recommend it to anyone who's tired of writing new features for their web applications, and fearing that they may be breaking other functionality at the same time. After subscribing to the testing-in-python mailing list, I cannot stress enough the importance of testing, testing, and more testing.

sed and awk are powerful command line tools. There have been many times when they've done a great job of providing me the information I needed when I needed it.. The problem is that I can never remember the syntax or usage for exactly what I want to do, which is always pretty basic.

Today, I stumbled across PyLine which allows me to use python syntax for grabbing the information I need from any file I have. Since this morning when I put it in my $HOME/bin folder, I've used it twelve times, 13 if you count my history and using pyline with grep and wc -l to filter out my count. Take PyLine for a spin, and leave that other stuff for the heavy lifting.

I followed Jeff Croft's Howto to install Django with Dreamhost, my hosting provider. This works well, and, while I didn't follow his instructions to the letter, I followed it close enough to know how the underpinnings worked. I've gone through now three revisions of my software, each time, being careful not to step on my Django installation (which is trivial). As you can see from my site, it works well.

Today I ran into a snafu. Dreamhost uses Analog 6.0 to provide simple web statistics. I've been having a hard time implementing my Google Analytics middleware with Django, so I decided I'd use the Analog stuff for now. However, when I navigate to the statistics url page, the FCGI dispatcher I use with Django grabs the request, and handles it as a 404 error. After reading an entirely too much about Apache redirects, I finally figured it out. If you follow Jeff Croft's .htaccess instructions, your file should look like this:

RewriteEngine On
RewriteBase /
RewriteRule ^(media/.*)$ - [L]
RewriteRule ^(admin_media/.*)$ - [L]
RewriteRule ^(dispatch\.fcgi/.*)$ - [L]
RewriteRule ^(.*)$ dispatch.fcgi/$1 [L]

The trouble occurs because you must specify what url strings aren't handled by the dispatch.fcgi file, and everything else will be handled by it. So I added one line to the .htaccess file that would fix the Dreamhost stats for me, so it now looks like this:

RewriteEngine On
RewriteBase /
RewriteRule ^(media/.*)$ - [L]
RewriteRule ^(admin_media/.*)$ - [L]
RewriteRule ^(dispatch\.fcgi/.*)$ - [L]
RewriteCond %{REQUEST_URI} ^/stats/(.*)$ [OR]
RewriteRule ^(.*)$ dispatch.fcgi/$1 [L]

Thank goodness for stats! The result is the knowledge that The Iron Lion is growing very rapidly! Soon there will be more features here, and I'll also be releasing another tarball of the entire site.