A Plug for Moleskine

First off, I need to say something about my method for taking notes here. Prior to leaving for my trip, I picked up two Moleskine notebooks that have unlined pages in them. I'd seen them used by many of my colleagues and friends, but never really saw why they had such hype behind them. After looking through my notes, there was something about the Moleskine that absolutely changed the way I write notes. The diagrams, blurbs, and quotes I wrote down last week were in a completely different and amazing format that allowed me to actually capture a lot of the thoughts I usually just try to reference and hope I remember.

On Collaboration

Collaboration has been on my mind for a while, and there are a lot of thoughts I have that are arranging themselves into blog post as we speak. I'm amazed at how much Canonical and its employees are thinking about collaboration. There was an overwhelming amount of questions like "How can we make X better?" and "What if we did Y like this? Would it be easier?"

Launchpad and Bazaar are two humongous examples of us trying to make collaboration easier. Launchpad has created this workflow for handling your code, doing merges of other people's code, getting bugs fixed, getting your code translated, releasing tarballs, and allowing people to easily install software into Ubuntu based machines. Launchpad has a rich code review system, which I am convinced is the best way to make open source software better.

The Ayatana Project excites me beyond my ability to express. Free software is generally written, tested, and (unfortunately) designed by geeks. This usually means that to find a tool with great design and usability is a gem. To have a team that is completely and totally devoted to making free software easy to use for my mother is amazing. Take your app (you know, the one you think is just amazing), get someone's attention in #ayatana on Freenode, and ask them (ever so nicely) what you can do to make it stellar. Do it. Do it now.

UbuntuOne is going to end up being a great service, and I would venture to say it will be a reference guide to the web services of the future. No data lock-in, open protocols, standards-based implementations, and the sky is the limit. UbuntuOne isn't about file sharing (although that is a cute first feature). It's about sharing data in any form, and as it fills out, it really will make collaboration even more social.

Finally, I wrote an open ended question to myself to think about, and thought that maybe I'd throw it out here as well. For context, Jono Bacon was talking about community (y'know, plugging his book and all that :), and I wrote "Open source thrives on a meritocracy - how can we prevent feelings of entitlement?" I see this a lot in open source communities: people earn their "commit rights" and then start behaving like everyone owes them something. Collaboration is about peers, not about hierarchies.

Getting Organized

A highlight of the talks I attended was a talk by Jono Bacon on recognizing and avoiding burnout. It was on eye opener for me. I spend a lot of time writing code, and I have now recognized that at some level I'm getting burned out by it. I'm doing a lot of things at the same time, and doing them absolutely terribly.

The biggest thing that hit me was that my methods for staying organized are terrible, and they aren't working. When this happens, one may be working hard, but not accomplishing much of anything. I've since restructured what my daily planning is like, and will report when I have data on its success or failure.

The Culture of Canonical

This was my greatest experience at AllHands. The nature of Canonical is that most of us work from home. This means that we can easily miss out on the social aspects of co-located work, like going out for a beer (Coke for me thanks) after work, shooting Nerf guns at each other, etc. However, because we work with these people on a daily basis, we form these incredibly strong bonds, so much that when we actually meet on this physical plane, we're old friends that would bend over backwards for one another.

Daniel Holbach epitomizes the Canonical culture at the very core with his hugs. I think of my direct manager more as my friend than as my manager. I enjoy seeing the rest of my team in person, because they are all really great people. There is absolutely no difference in social status between Mark Shuttleworth himself and the newest hire. We all work together. There isn't this idea of "What I do is more important than what you do" but more that "What you do is so helpful that I want to make sure what I do is helpful in return" Canonical is really about Ubuntu, the product, and the meaning of brotherhood and working together.

Shared repositories

If you work on a project on a regular basis, you really need to create a shared repository for your project. The basic idea of a shared repository is that you keep all your revisions in a pool together. For instance, you have a local mirror of trunk, and since you share those revisions among your branches, when you create a branch from that local mirror, there's very little overhead to creating the branch, since the revisions don't need to be copied.

I keep my various shared repositories in a folder called ~/Projects/repos with each project having its own folders there. When I want to hack on a new project, like, say, Loggerhead, I have the following process:

$ mkdir ~/Projects/repos/loggerhead
$ cd ~/Projects/repos/loggerhead
$ bzr init-repo --no-trees .
$ bzr branch lp:loggerhead

Now, every time I branch from ~/Projects/repos/loggerhead/loggerhead into ~/Projects/repos/loggerhead/, I don't actually get any new revisions, just a new branch using the old revisions. I don't think this really saves on space too much, but it makes the overhead of branching much cheaper.

But wait! Why --no-trees? How the heck do you work if you don't have a working tree? Ah, so now we move on to my next point.

Lightweight checkouts

So I have two branches in ~/Projects/repos/loggerhead, the mirror of trunk, and a feature branch we just created. Neither that feature branch nor the local trunk have anything in them except the .bzr folder. No working tree. So let's back up a dir to ~/Projects where I keep my working trees. So now I'm going to create a folder to house my Loggerhead working trees.

$ mkdir ~/Projects/loggerhead
$ cd ~/Projects/loggerhead
$ bzr checkout --lightweight ~/Projects/repos/loggerhead/loggerhead
$ bzr checkout --lightweight ~/Projects/repos/loggerhead/feature-branch

Now I have working trees to work with. When I commit, those committed revisions go back to my branch, and in effect, to my shared repository. Now my working tree is completely separate from the branch itself. I can easily delete the working tree when I'm done with it without losing the branch, and when I decide to delete the branch, I still have those revisions hanging around (in most cases, they've also been merged into trunk).

I can also treat that lightweight checkout of the trunk mirror to branch from, merge, commit, etc. as if it was the branch itself, but now there is a clear separation between where I work and where I store my revision data.

Using cbranch

So we've got two bzr commands to type, the branch command and the lightweight checkout command. If you use bzrtools you can do this in one easy step by setting a few bzr settings in your ~/.bazaar/locations.conf This is what I have set for my directory structure/layout:

[/home/rockstar/Projects]
cbranch_target = /home/rockstar/Projects/repos
cbranch_target:policy = appendpath

I also want to set an alias in ~/.bazaar/bazaar.conf to give me lightweight checkouts.

[ALIASES]
cbranch = cbranch --lightweight

Now, when I'm ~/Projects/loggerhead I can just bzr cbranch loggerhead new-feature-branch and bzr will automatically create a branch in ~/Projects/repos/loggerhead and then create the lightweight checkout in ~/Projects/loggerhead bound to that branch.

The importance of bzr whoami

One of the really important things that lots of people forget is to start off with is setting your name is Bazaar. This can be done really easily by doing bzr whoami "Your Name <email@example.com>" It's important that you set this properly, or your revisions aren't going to get your name on them. The email address you set here also needs to be an email that Launchpad knows about (a confirmed address). This way, you get the karma for the revisions, and revision listings in a branch page will link back to your profile.

Set a Development Focus

Because we're talking about a distributed version control system, there are going to be many branches in your project. It's important to point people glancing at your project to the "trunk" of your project, where all the new fun development it. Once you've pushed up your first branch, it's good to set it as the development focus. Now, new contributors will know where to branch from when they want to, well, contribute.

Setting a development focus also tells Launchpad itself what branch is your main branch. For instance, because I have the development focus of Entertainer to lp:~entertainer-releases/entertainer/trunk, every time I do a bzr branch lp:entertainer Launchpad knows to give me the correct branch. It's an alias that can save you a lot of time. I often find myself doing bzr branch lp:randomproject just to get the most up to date source.

A quick note about series branches. When you decide to release your software (because you're releasing software often, right?), you'll want to keep a branch of that version. That's what's called a series branch. It'll have point release milestones attached, and when you release those point releases, they'll come out of this branch. The development focus is a glorified series branch. So, in effect, if you create a series branch called myproject-1.1 in the myproject project, you can use bzr branch lp:myproject/myproject-1.1 to get your myproject-1.1 series branch.

Since branches are owned by people, you might want to consider creating a team that owns your development focus, and adding people to that team. This keeps your project moving when you go on vacation for two months. It's just a good idea.

Workflow decisions

Now, you need to decide what your workflow is like. Some projects only have one person working on them. If that's your project, then all you need is one branch, commit, push to Launchpad. That's relatively simple. I do this with a few personal projects.

If your project has more than one person, then you've got to make other arrangements. What I suggest is the concept of feature branches. Create a branch specifically for a feature or a bugfix, hack it all together, and merge it into your trunk. This is the optimal solution, and, as far as I'm concerned, the only solution.

It's a good idea to see my former post on good Launchpad/Bazaar settings This will take the hard work out of pushing, especially with feature branches. What I usually do is keep a mirror of the development focus branch on my system (with a cron job to update it at regular intervals). When I start a new branch, I branch from that mirror, and start hacking. I try to push at normal intervals, but sometimes I just space that.

Merge Proposals and Code Reviews

So now you're creating feature branches, because, well, that's the workflow you've chosen. You've written your tests (because you tested first, right?), your feature is ready to land (or so you think). At this point, you should propose it for merging into your development focus branch. No matter how your project handles reviews, it's good that someone at least blesses the revisions that are to proposed to land in trunk.

So your branch goes through the review process, comments are made, and eventually, all required reviewers approve of your changes. Now it's time to actually get your branch landed. Someone in the team that owns your development focus will need to merge your branch into their local copy of the "trunk" and then push it up. Once that's happened, Launchpad will mark the merge proposal and your feature branch is also marked as merge. Go hack on another branch!

For Reviewers

So there are a few different pages that help to know about if you're a reviewer for a project. The first is the +activereviews page. If I navigate over to the Entertainer active reviews page you'll see that I have some landing work to do. This also applies to people. If I navigate to my own active reviews page you can see all the branches that I have submitted for review.

What if, however, I want to see all the reviews I've done for someone else's branch, and what the status of those branches is. Maybe I set my review vote to "Needs Fixing" and now I need to see if that fixing has occurred. That's when you'll want to check out your +requestedreviews page (as of this writing, currently only available in beta, due out in a week). If we go take a look at mine, you'll see that I have some work to do.

Bazaar branch formats

As Bazaar features get implemented, new formats become a result (so as not to risk breaking old, tried and true formats). As of Bazaar 1.6, a feature called "stacking" was introduced, that allowed bzr branches to only have the history specific to that branch, and just keep a reference to the trunk branch. This is great if you have LOTS of history for your project, and pushing seems to take forever. If you upgrade your branch to format 1.6 by doing bzr upgrade --format=1.6 you'll now be able to push stacked branches.

If you're using the Bazaar PPA (and you really should be using it), you'll have a 1.9 format available. It's faster, supports stacking, and is awesome all around. It's not the default format, so you'll have to either create your branch with bzr init --format=1.9 or bzr upgrade --format=1.9.

Import branches

Distributed version control drastically reduces the barrier to entry for hacking on a project. With a VCS like CVS or Subversion, the project usually works as a meritocracy, and so you have to "prove yourself" before you get access. What if a project you really want to hack on won't give you commit access to their SVN/CVS repo? Create a project in lp, and import the the SVN or CVS branch into a bazaar branch. Then all you need do is branch from that branch, and hack on. When you're done, you diff between the current import and your branch, and send that in as a patch to the project itself. The difference is that you have a versioned copy of the making of that patch, instead of having a single checkout.

Comments and suggestions are welcome. I'll do my best to keep this post up to date with new things, and once it's baked, I'll publish to a wiki somewhere.

Getting a Branch object

In order to operate on your branch, you'll need to create a Branch object. This will be a programmatic handle to your bzr branch itself. So let's start by getting the branch itself.

from bzrlib.branch import Branch

my_branch = Branch.open('/path/to/branch')

Now we can operate on the branch with python. Let's say we want to see what config variables are set on this branch. We'll want to get the configuration of the branch, and then examine its contents. In this example, we'll check the username you'll be committing to bzr with (because you hopefully ran bzr whoami.

my_branch_config = my_branch.get_config()
print my_branch_config.username()

Simple local branching

Let's say you want to programatically create a local copy of a remote branch.

from bzrlib.branch import Branch

remote_branch_url = 'bzr+ssh://bzr.eventuallyanyway.com/code/foo-bar-baz'

remote_branch = Branch.open(remote_branch_url)
local_branch = remote_branch.bzrdir.sprout(
    '/code/foo-bar-baz').open_branch()

Getting a diff

Programmatically, I think the best use for understanding the Bazaar API is to diff the changes in your current working tree. You'll do this by creating a WorkingTree object and operating on it.

from bzrlib.workingtree import WorkingTree

tree = WorkingTree.open('/code/foo-bar-baz')

Now you've got a working tree. Let's get a diff between it's current state and its last committed state. If it's changed, commit the changes with a commit message of "Committed changes"

delta = tree.changes_from(tree.basis_tree())
if delta.has_changes:
    tree.commit('Committed changes')

All in all, I think the greatest strength of Bazaar is its easy extensibility, and easy entry into working with its API is proof.

Creating a merge proposal with email

Looking back on the general Launchpad branch configs we used, we need to add one more line so that Bazaar knows the email address that Bazaar's "bundles" should go to. So, let's edit our ~/.bazaar/locations.conf to show the following:

[/home/rockstar/Projects/repos]
push_location = lp:~rockstar/
push_location:policy = appendpath
public_branch = lp:~rockstar/
public_branch:policy = appendpath
submit_to = merge@code.launchpad.net

The line added is the submit_to line. This means that Bazaar is going to send its bundles to merge@code.launchpad.net. This is the email address that parses the bundle, and derives the the source branch (the branch to be merged) and the target branch (the branch to merge into).

Now we need to specify the submit branch. This requires some specific configs for each project, so we'll need to add a new section. For this example, we're going to use Entertainer since Bazaar's development doesn't use Launchpad for it's review process. I'm going to add another section specific to Entertainer.

# Entertainer
[/home/rockstar/Projects/repos/entertainer]
submit_branch = lp:entertainer

This sets all branches in ~/Projects/entertainer to submit to the branch lp:entertainer which happens to be the current development focus of Entertainer. Note: The target branch MUST be a Launchpad branch. The source branch doesn't really need to be, and Launchpad is smart enought to create a Remote branch (a branch that Launchpad only keeps a reference of).

Finally, we need to tell Bazaar what your preferred email client is. So crack open ~/.bazaar/bazaar.conf and let's add one more line to the [DEFAULT] section.

mail_client = evolution

Wait. You don't want to use evolution to send your email? Well, you can choose between thunderbird, evolution, kmail, thunderbird, mutt, emacsclient (although I don't know how that emacsclient works), or editor. I use editor, but that requires some more config set up (and it's beyond the scope of this post). Also, if you want to use Gmail, see BzrSendWithGmail or use James Henstridge's bzr-imapclient plugin to use GMail to send the merge-directive (see https://lists.ubuntu.com/archives/bazaar/2009q1/051602.html. As of Bazaar 1.11, bzr also now supports claws as a mail_client as well.

Now, hacka hack on your branch, and when it's all ready, make sure it's pushed to Launchpad, and then do bzr send --no-bundle. This will create a bundle and open up your mail client for you to detail what your branch does. Fill that out, and send the email. It'll be processed, and the merge proposal will be created, using the Default Reviewer for the target branch as the reviewer.

Reviewing a branch through email

This is where things get more interesting. Those subscribed to the branch (as branch code reviewers should be) will get an email with the creation of the merge proposal, and an email with the patch (these two emails will be consolidated shortly).

So a reviewer cracks open the email and looks through the patch. They merge the branch into their local mirror of trunk and run the tests (because you have tests for code, right?). The reviewer makes their inline comments to the patch in the email reply. The To: in their reply should be something like mp+1234@code.launchpad.net.

If you (as reviewer) were to send this email now, it'll be parsed as a comment to the merge proposal. What if you want to give an official review vote? What if you want to set the merge proposal status to Approved? This is the clever part. For example, I review a branch and I think it needs to be set to Needs Fixing. Within the email, I would type:

vote needs_fixing

Note the leading space there. It's important. Now, GPG sign your email, and send it off. You've just given your vote. After someone decides to fix the issues that arose from your review, you can reply to the response email with:

vote approve
status approved

Send this with a GPG signed email, and you've updated the merge proposal to Approved as well as "voted" approve. Now, if you can merge the branch, merge it, otherwise, nudge someone who can.

Hooray! You've successfully navigated the hairy process of code review without having to hassle with the web UI at all. Now THAT'S a service worth working with. There are similar commands to work with bugs as well, and by learning to use these commands, you can create a workflow with Launchpad that works best for you.

Update: Thanks to some great comments, I've noted some of extras.