When I first approached Lauri about working on Entertainer, he had released Entertainer on Launchpad. Not long after I started working on the project, Lauri had problems getting bazaar to work with Launchpad, and so we move to Google Code. We've been there since then.

However, recently, while working on the new backend-refactoring stuff, I found that svn was just too lacking in allowing me to merge in continuing development on trunk into my refactoring branch. It then got to the point where I dreaded the actual merging back into trunk. I think much of this was because I had been using distributed version control systems consistently in the work I was doing for my clients, and really enjoyed it.

After chatting with the rest of the developers, we decided we should move to Launchpad. With the option of having vcs-imports, it was a good option, for many reasons. After all, the idea of Launchpad is to have all open source projects in one place anyway.

So, the basis of this blog post is this: Entertainer can now be found at http://www.launchpad.net/entertainer. Mentoring is available to anyone who would like to contribute...

A very dear friend of mine and I had a conversation yesterday. Somewhere in the conversation, he said (and I'm paraphrasing) "I'm going to make my own encryption algorithm, and it's going to rock." This struck me as odd so I responded "Why would you try to do that when so many really good encryption algorithms already exist?" By asking this, I didn't doubt his ability to write an encryption algorithm, per se, but more that the AES algorithm was the life's work of Joan Daemon and Vincent Rijmen, two doctors of cryptography. His response was that he just wanted to tinker, which is a valid response, and one that I'm sure every developer has had at some point in their career (and hopefully still have some variant of it).

So I thought long and hard about that. It's good to tinker. I remember the first time I wrote a linked list in C. Advanced concepts like pointers suddenly just made sense. I understand how you could make an array that changed its size. It was a to me. I optimized it as much as I could, and I still have that very code on my system today. Today, I have no need for it, and haven't looked at it since.

Why? Because I know how it works now. I can use someone else's linked list code, one that I don't have to maintain, and one that has had much more work than I'm willing to put into it to make it fast and efficient. I've moved on from that. I could spend my time focusing on how to make my own data structures and make them efficient, or I could make real software using someone else's library, and devote my ingenuity to that.

While still talking to my friend, I used the analogy of building a birdhouse (and did my best not to let my high school shop class experiences taint the analogy). If you write up plans for a birdhouse, and then write up plans for a hammer, and then write up plans for a nail, and then write up plans for a saw, then you'll have quite a mound of work to do. Chances are, you'll probably end your birdhouse project with a half-made hammer, and lots of grand plans. Everyone does this at least once or twice in their career, and if you're a developer, and you claim you don't have a single half-made project somewhere, I will say that you are a liar to supplement your developer title. --:-)

I no longer recompile my linux kernels from vanilla, or even the distro's patched version. I don't need to. The speed difference isn't all that dramatic, and frankly, there's really no purpose. But I know how to do it, should I need it, and I understand how the kernel works. It's not magic to me anymore.

Matt Layman has a blog now! Welcome to the blogosphere Matt.

Before I go on with this post, I need to lay my preferences down on the proverbial table. I don't loathe Java, but I definitely lean towards more dynamically typed languages for my own projects. I see Java as a corporate language that really has very little application in open source (which is where I spend most of my time). I'm not going to say anything like "Java sucks" but I will say "I don't much care for Java."

I've been doing a lot of JSP/Servlet work recently. It's more for research than for production use. As someone who's done GOBS of web development, I didn't think it'd be too difficult. Writing the code, getting the syntax under my belt, all these things were rather simple. I've just grown use to the idioms used in web development as a whole, and so that wasn't too difficult (Eclipse made it easier as well, once I found a good JSP plugin).

I think the biggest problem to learning a language is running the code itself. My 12-year-old brother was up and running with python pretty quickly, having Ubuntu's Synaptic installer take care of installation, and then using the python interpreter. C, C++ and Java are a little more complicated. They require knowledge of rather complicated tools, provided you're doing anything more than "Hello World!" There are compilers, linkers, etc. Shoot, in all fairness, even writing your own python modules requires knowledge of how import works.

So I'm writing my Java web app, and I'm getting Tomcat all set up to work. Then I realize that there's more to deploying this app then I originally thought. Now I've got to write an ant build file to automate builds (because compiling all these classes and moving them all to the appropriate directories just isn't practical). After that, I've got to deploy it to the Tomcat server. The barrier to entry is just too high.

When I first started playing with Django, the one thing that really grabbed me was the development server that ships with Django. No mod_python/mod_wsgi/mod_fcgi/whatever to mess with just to see if I wanted to invest more time into the tool. When I started with jQuery and ExtJS, I had the javascript terminal in Firebug to play around with the API and find its strengths and weaknesses.

I'm not saying that a language should necessarily be "so easy a caveman could do it," but I do think that there needs to be a low barrier to entry. If it gets complicated later on, thats fine. Shoot, make some really simple, easy to follow tutorials on how to use the tools. Make explanations on the really simple things, like how library paths are structured, etc. Create tools that don't require 200 different command line flags to compile anything more complicated then your standard "Hello World!"

I'm a dynamically typed language fan boy. I've dabbled in all sorts of languages, but got my first development job working with PHP and Javascript, graduated to Python, and then did the C/C++ and Java stints as well. One thing I'm quite used to doing is writing print statements for debugging. In large C/C++ or Java environments, I found this quite impractical, and eventually became quite intimate with debugging tools in those languages. Until recently, however, I continued using print statements for working with scripted languages like Python. Now I use pdb for almost every problem I encounter in python.

A while ago, I battled a nasty Entertainer bug. It was repeatable, but not consistently. I would start the backend, and the it would start indexing videos from the configured video folders. Then, for no reason, the backend would die. I was able to trace the problem back to the VideoThumbnailer, which would be my baby. Crap. So I started adding print statements, so that I could see if the thumbnailer was consistently dying on the same video (thinking it might have been a codec problem with GStreamer). What I found was that the backend rarely died in the same spot, and since the thumbnailer didn't create thumbnails for videos that already had thumbnails, once it got past a video it had failed on before, it never struggled on it again. However, with the multi-threaded environment of entertainer, it was quite difficult to debug.

Eventually, I followed the yellow brick road to see the Google. I typed in 'python debugger' and found the first page of results full of pdb articles. "Oh no," I thought, "anything but the confusing and difficult pdb!" Alas, once I forced myself to work with it, I found that it was very simple to use (and that the pydev eclipse plugin makes it so much more complicated than it needs to be). I'm still only using a small subset of the things that pdb can do, but frankly, I can't see a need for anything more than what I used it for (setting a breakpoint, reading local variables, seeing a backtrace, and stepping through code).

First, I decided where I wanted to have pdb stop and give me a debugging console (setting the breakpoint). When that was decided, I added the following code:

import pdbpdb.set_trace()

That's it! Now run the code, and you'll find that instead of complete execution, you'll be at the (Pdb) prompt. Above your prompt, you should see the pdb.set_trace() call that created the console. I'm not one of those people who fears a command prompt, so if you are, I apologize. However, this isn't nearly as complicated as a shell prompt. You've got only a few keys to worry about:

• n (next) - Hitting 'n' at this prompt will execute the current line of code and go to the next line.
• c (continue) - If you've gotten to the point where you know your bug is fixed, and you want to just close the debugger, hit 'c' to just continue executing the program. This will complete execution, provided that you have no more calls to pdb.set_trace()
• s (step into) - If you're at a function or method call on the prompt, and you'd like to see what's going on inside that function, stepping into it will allow you to follow execution into the function. If you don't want to see what's going on in the function, 'n' is your character.
• l (list) - Wait. Where was I in the code again? Yea, I guarantee that'll happen to you. Hitting 'l' will give you a chunk of code in the area you are, with a nice little -> where you are currently.
• bt (backtrace) - Okay, so now I know where I am, but how did I get here? 'bt' gives you a backtrace, showing you the start of execution, and all the function and method calls that got you where you are today.
• p (print) - Okay, so what's the value of variable foo? 'p foo' will print it's value, allowing you see how the variable is changing as you walk through the code.
• q (quit) - Ah! Found the bug. Let's quit this session and fix it!
• enter (repeat last command) - Tired of hitting 'n' over and over? How 'bout just using 'n' once, and then hitting 'Enter' over and over. It's a bigger key, and less likely to get fat fingered. I don't use it, but it's there if you want it.

Needless to say, debuggers are the tools that would make your life easier if you just learned how to use them. Since my experience with the debugger, I've chatted with a few pythonistas and developers from other languages, and to my surprise have received varying responses. I would say that if you've spent more than 5 minutes writing print statements and running your code, do yourself a favor and fire up the debugger. There might be a learning curve now, but you'll be a better developer for it.

This weekend, all the Entertainer developers got together and sprinted again. This month's sprint had TONS more work done, as we pushed through to get closer to a 0.1 release. I wouldn't say we'll see that 0.1 release in the next month, but we are much closer now.

Joshua worked on getting a working weather module into Entertainer. The 'Weather' option in the main screen has always led to a blank screen. Currently, the 'Weather' option is gone from the main screen, but only because the UI is being worked on, and the flow that will be followed to set up weather is being worked out. However, the backend code has been completed, and it's ready to be used once those issues are worked out.

Lauri refactored TONS of the UI code, to make it more readable (I will agree it is more readable now), and the UI now features navigation tabs to make things a little clearer to the user. The UI has been a source of much deliberation between developers since one of the main goals of Entertainer is to make it "so easy a caveman can do it." Lauri has done a wonderful job at paying attention to the small details that push Entertainer that extra mile.

Matt has also been working tirelessly to get the clutter API figured out, and has been fixing bugs and writing more tests. It's getting to the point that he owns more of the test framework than anyone else, and he probably knows more of the codebase than anyone now, since he's been writing most of the unit tests to bring our coverage up.

I've been working tirelessly on a side branch, out of everyone's way, so my work wasn't seen much. The current backend for Entertainer has needed some love for a quite a while, so I've been making a LARGE rafactoring effort there. The new backend will feature an ORM, so that it's easier to write tests against our database, as well as easier to work communicate with the database. It'll also feature a refactored and redesigned schema that will allow for quicker access to the database.

As for development stats, we saw 38 commits overall (revs 276-314), comprising of 1,837 new lines of code, and 1,723 changed lines of code. Most of that code was refactoring, so we only saw one new test all weekend (although I believe the release of a certain video game may have hampered development on that front.... :)

I like to dabble in the "user experience" a lot, but when I say "dabble," I mean "No one in the world would EVER let me influence user experience so I pretty to do it on my own projects in order to feel like I'm contributing something. I have Smashing Magazine in my feed reader, among others. I just don't have the eye for the "design" part of the whole equation. I'm not a command line zealot either, but I don't know how to recreate the "feel" of an app that I really like. So I continue to dabble.

This morning, I was reading through the feed reader, and just went on a link stroll through the internet after clicking through links to other articles. I stumbled on a blog that I wish were still around, called Creating Passionate Users. I found it because Jeff of Coding Horror was blogging about why Passionate Users is no longer around. While I didn't much care for the Jeff's recent post (not that it isn't important, just that it's not something that is directly on my mind), it led me to a few links that I found were directly inspiring to not only user experience, but to general life: Jeff's Who Needs Talent When You Have Intensity and Kathy's How To Be An Expert

I've posted what I think it takes to be a good developer before. I've worked with some good developers, and I've worked with some bad developers, and I've worked with developers in between. The colleagues that I really have to dig to remember are mostly the ones that write code as a 9-5 job. This isn't necessarily a bad thing, but I can tell you that you don't do a job that you LOVE the same way you do a job that you start at 9am and end a 5pm. I've worked with people who claim coding is their passion, but don't seem to take pride in their work (and in all but one case, I have to clean up that work, the one case being I was laid off in time to miss that maintenance...) But after all is said and done, just like Tenacious D is musically horrible yet entertaining, the good developers are those with passion.

How do I have passion? On my bedside table, there currently sits a copy of "Buffy the Vampire Slayer Omnibus Vol 2" along with "JSP, JSF, and Tomcat", "Pro Javascript Techniques", and "Java Generics." Buffy usually gets read once a week or so. I write code at work at all, and then I come home and write more code. I usually like to take on a new language or a new framework every three months or so. I find niches that need to be filled. And what makes an expert, as Kathy points out in her post, is someone who is always looking for ways to improve. I recently endeavored on the quest to re-engineer the backend of Entertainer, because it had some valid issues, and because I knew I could make it better.

The moral of the story is that I need to stop "dabbling" in understand user experiences, and make it my passion, so that it will start showing through in the products I create or help to create. You should do the same for the things you want to do.

I've been trying to get lirc support into Entertainer the last few days, and while SOME of pylirc is still able to work, it's incredibly old. I contacted to current maintainer about this, and haven't heard back from yet. Since I need to move quickly to get this module updated, I have mirrored the Sourceforge CVS repository and converted it into a Mercurial repository. It can be found at:

http://hg.ironlionsoftware.com/pylirc2

Please note that I don't intend to fork the project or anything like that. I just plan to continue it's maintenance, specifically for Entertainer, but for the use of anyone who would like to have python bindings to lirc.

My grandfather and I share the same name. This wouldn't be a problem if he hadn't been the Assistant Dean of Agriculture at Oklahoma State University, written books, and had lots of stuff written about him. I've known quite a few people now that have asked me about my time at OSU, even though I never attended there (and they obviously didn't read why Paul Hummer was at OSU in the first place...)

Because of this, I decided to create my own landing page for Google to find, index, and finally show teh internets exactly who I am. You find information about me here: Paul Hummer

I've been posting a lot about test driven development recently, and I got an email from a reader who asked me to go into detail on how I've been going about discovering and working with tests. I think this type of post would have helped me develop the "test first" mindset so much earlier in my career, so I'm sitting down to actually detail exactly how I'm learning about this world that I've always heard about, but only now have visited. I guess an alternate title for this post could be "Python Unit Testing 101"

Because I try to spend at least 10 hours a week on Entertainer, and it's already got a pretty good test base that I have lots of experience in (because I wrote a large chunk of it), I think it would in everyone's best interest if I used a new Entertainer test as an example. If you would like, you can check out the code and see exactly what I've been doing by looking in trunk/src/tests. To run the tests, just navigate to the test directory, and type ./run_tests.py You'll see the tests grind away, telling you which ones passed and which ones didn't (hopefully, all of them are passing when you check out the code...)

Abstraction - The Seed of an Idea

I've been maintaining (or trying to maintain) the thumbnailers, ImageThumbnailer and VideoThumbnailer. They both inherit from an empty abstract class Thumbnailer. Recently, when trying to hunt down a bug, I noticed similar code in both VideoThumbnailer and ImageThumbnailer that could easily be rafactored to an already existing Thumbnailer class. For reference, I actually found this while hunting down a bug in ImageThumbnailer, and writing a test to make sure the bug stayed squished. As I planned out exactly what the Thumbnailer abstract class needed to look like, I started thinking about the test I needed to write first.

The abstract Thumbnailer class looked like this: thumbnailer.py

Planning - Writing the Test

The most important thing to remember about TDD is that your tests describe the way your code should behave, not the way it does behave. For instance, if you find a place where the code should be refactored to behave differently, write the test that way, make sure it fails in the way you expect, and then you can refactor the code to pass the test. That's what I did with the ThumbnailerTest. Since there wasn't any code/logic in the Thumbnailer class, all the tests technically should fail. But I had a few things in mind in refactoring:

1. The Thumbnailer should have logic to tell what type of thumbnail it will create, so it can set it's destination path properly. The types are determined by what folders exist in the thumbnails config directory (currently image, video, and recording). It should throw an exception if it's an invalid type.
2. The Thumbnailer should only try to thumbnail an image or video that exists. It should throw an exception otherwise.
3. It should make sure that the classes that should be overridden by the child classes exist. I'll use a fun little technique for this that I learned only recently.

Obviously, I now have my tests. Let's look at item one. I like to break that test up into two tests, one that tests that each type can be properly parsed and recognized as valid, and one test that throws junk data as the type, and ensures that it throws an exception. I created my test by inheriting from unittest.TestCase, creating a setUp() method, containing only a debug flag (which you can use to make sure your test is working...), and then I start creating my tests.

The result of these tests is the following: Thumbnailer_test.py

On A Tangent - The Test Runner

Before we run the new test in the test suite, take the time to look at the run_tests.py script. run_tests.py

You'll notice that the script searches through the current directory for python files (except those in the blacklist), and using the unittest.TestLoader and unittest.TestSuite classes, loads the those files as modules, and then runs the TestCase child objects. The output of this script is the results from the tests. It's nothing terribly complicated, but then again, if your tests or test harness are complicated, you probably ought to change the implementation.

An alternative to the script used in Entertainer is python-nose. By installing it, you can run each test file yourself by doing nosetests <filename>. It will automatically find the unittest.TestCase instances in your file, and run them, printing the results to the terminal.

The Test Is Failure

Back to the tests at hand, you'll notice they all fail. This is good! If they passed, something is definitely wrong with your tests. Look at the errors for the failed tests and make sure they are what you expect. For instance, if you're trying to call a method that doesn't exist, make sure you've got an AttributeError exception, etc. If they are failing the way you expect, then you can consider your tests to be working properly. Now your job is to hack on the class you're testing until it passes the tests.

While you're writing this class, hopefully you're starting to see the value of the test. It was when I was working on the VideoThumbnailer that I started realizing how great TDD is. Instead of firing up the UI every time I made a small change to the VideoThumbnailer, I just ran the test to see if it was generating thumbnails. So instead of making a change, firing up the ui, navigating to the videos, and checking for thumbnails (and at the time, the UI was in a great state of flux at the time, and was only working for me every once in a while. Having the tests meant making a change, running the test runner to see if the test passed. No dependence on the UI working or anything of the sort.

Your Code Works Because the Test Says It Does

After a quick iteration of work (because you've got the test to run instead of the use case), you've now got working code. You know it's working because your test is passing. You know your test is working because you wrote it, and watched it fail exactly the way it should.

The Thumbnailer class now looks like this: thumbnailer.py

Now you can feel safe in committing your refactored code. Because I already had tests for ImageThumbnailer and VideoThumbnailer written well, the tests required no changes in order to test the code I removed to the abstract class. The general rule is that your class' interfaces should be simple, and shouldn't have to know about the "guts" of another class. Because it was constructed well, the tests worked without a problem.

As you write tests, remember that you're writing code still. I've seen lots of people write the tests and see them pass, and then work on the code, and the tests fall out of maintainability. When you make drastic changes to your code, make the tests reflect that, or rather, make the tests reflect the changes first. When you have tests, your codebase is 2-4 times the size of the code that actually gets deployed, so you've got more code to maintain. However, hopefully the maintainence of the code gets easier as the testing gets better (and if it doesn't, something is wrong with either you, or your codebase).