Google AppEngine was launched to the public today and it looks very cool. I was fortunate enough to get an invitation almost immediately after I signed up on the waiting list. The API has a strong Django smell, this will most likely be very interesting.

Meh. For the first time ever this blog received a fair amount of visits. And what happens?

Stupid me decides to upgrade and forgets to reset an environment variable which makes Django believe it's running in a debug environment; tables, files, everything was in the "wrong" location.

The next time I'll be more careful...

Earlier this week I described the way I organize my web site projects in Bazaar in multiple branches and how each branch is organized in directories. I promised a follow-up the next day, but since I'm a lazy, lazy man I totally forgot it.

This post will describe two scripts that glue those four directories I showed you the last time together and form my development environment. In this post we will play around in the RSSEpisodes.com source code and create a new branch for a new feature that I have been planning.

I start out by creating a new branch for this feature:

$> bzr branch rssepisodes-trunk rssepisodes-gossip
$> ls
rssepisodes-calendar/   rssepisodes-gossip/   rssepisodes-links/   rssepisodes-regroup/
rssepisodes-trunk/

Bazaar creates a branch of the main developement tree and I can immediately cd into this new branch and start hacking:

$> cd rssepisodes-gossip 
$> source environment.sh 
$> manage.py runserver
Validating models...
0 errors found

Django version 0.97-pre-SVN-7209, using settings 'rssepisodes.settings_dev'
Development server is running at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Now that's what I call agile! The only important part here is that I read the script environment.sh. This script must be executed from the root of my new branch since it uses the $PWD environment variable when setting its variables:

$> cat environment.sh
#!/bin/bash 
export PYTHONPATH="$PYTHONPATH:$PWD/python"
export PATH="$PATH:$PWD/scripts"
export DJANGO_TEMPLATE_PATH="$PWD/templates"
export DJANGO_STATIC_ROOT="$PWD/media"
export DJANGO_SETTINGS_MODULE="rssepisodes.settings_dev"

Pretty straightforward. In fact this setup is very straightforward. There's only a small bit of semi-magic going on in manage.py, which resides in scripts/ and that directory is now in my $PATH. Here's that small amount of magic:

$> cat scripts/manage.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-

from os import getenv
from django.core.management import execute_manager
if __name__ == "__main__":
    settings_module = getenv('DJANGO_SETTINGS_MODULE')
    settings = __import__(settings_module, fromlist=[settings_module.split('.')[-1]])
    execute_manager(settings)

This modified version of the default manage.py imports the correct settings module by using the environment variable that we set above. To accomplish this we need to invoke the __import__ function instead of the standard Python statement.

Now, you may have noticed the last time that I have two distinct settings modules; settings_prod.py and settings_dev.py. The former is a standard settings module, only renamed to emphasize that it is used in production. The latter actually imports everything from the production settings and only reassigns a few values. The interesting part is what I do with the template paths:

$> cat python/rssepisodes/settings_dev.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-

import os
from settings_prod import *

DEBUG = True
TEMPLATE_DEBUG = DEBUG

# overridden DATABASE settings go here

STATIC_ROOT = os.getenv('DJANGO_STATIC_ROOT')
TEMPLATE_DIRS = (
    os.getenv('DJANGO_TEMPLATE_DIRS'),
)

I use the environment variable DJANGO_TEMPLATE_DIRS set above as the template directory, and since this project also uses the static file server for development, I set the STATIC_ROOT variable which is used in urls.py when DEBUG is True.

And that is all. Now I really need to get some hacking done, I want this feature to go live tonight.

Djangofriendly by Ryan Berg is the latest and greatest site by and for Djangonauts. Great looks and a great idea! I will definitely check it out before I launch my next project.

I have previously written about how I overcome A Small Orange's poor implementation of FastCGI and Django for this site. I'm still running this blog with static pages generated by that script.

Aside from that quirk, my experience with A Small Orange has been great; they respond to support requests within few hours and they are always willing to help. They could have provided us with more storage, which I hear is cheap these days, but I guess that is my fault for being cheap as well.

My night-time project RSSEpisodes.com is hosted by VPSLand and it has been reliable, fast and VPSLand has been a fantastic hosting provider. RSSEpisodes has colophon describing its current setup.

Most of us use some kind of SCM tool for our websites, Django powered site or not. A few months ago I switched from the old and trusted Subversion to Bazaar. The transition was easy and gradually I have moved all of my websites to Bazaar.

The primary reason for changing to Bazaar was its extensive support for branch and merge tracking. I now develop each new feature for my websites in a new and separate branch and thanks to Bazaar's merge history I never have to keep track of what has been merged where.

It has made my life so much easier!

I had to make a few changes to the way I organize my projects and today I can easily develop multiple branches at the same time without changing any code or settings. The last point was extremely important to me, I want to move between branches as painlessly as possible, without changing any code at all in the new branch.

I'll try to give a rough overview of my current setup using the code tree for RSSEpisodes.com and in the next few days I will show you the scripts that glue it together.

All my branches live in a top level repository called rssepisodes-repo. The version currently in production always lives in trunk and at the time of writing I have three branches of trunk:

$> ls
rssepisodes-calendar/   rssepisodes-links/   rssepisodes-regroup/
rssepisodes-trunk/

The three branches each focus on a particular feature and that feature is developed and tested in its own branch before it is merged into trunk.

The regroup branch focuses on making my templates simpler by implementing a few custom template tags. If we take a closer look at it we see how I structure my websites:

$> cd rssepisodes-regroup
$> ls
media/  python/  scripts/  templates/  environment.sh  readme.txt

Remember, the regroup was branched of trunk, so all my branches are organized in the same manner. As we see, the top level directory of a branch consists of four directories and a shell script.

The four directories cover everything I need:

• media -- This directory contains the media files for my application. Images, stylesheets, Javascripts and so on. On my production machine the web server has an alias pointing to this directory, locally I use static_root in urls.py.

• python -- This directory contains the Python code that operates on my site. Applications that are specific to this site has a package in this directory.

  This directory is added to the Python load path via the PYTHONPATH environent variable by environment.sh.

• scripts -- In here I place various scripts that I write to manage my web site and most importantly a customized manage.py.

  This directory is added to my PATH by environment.sh.

• templates -- This is my templates directory. It does not belong in my Python path in my opionion (django-admin startproject places it in the root of your project, which is in your Python path. Blech.)

The python/ directory is where my Python code lives. In here I stuff Python modules (.py files) and the Python packages for my Django applications.

As mentioned, this directory is added to the Python load path. In addition to this directory I usually have ~/python/ where I keep Django itself and applications such as django-registration and django-tagging, and other shared Python libraries.

Now, let's take a look at that python/ directory:

$> ls python
rssepisodes/     shows/    tags/    tidy.py

It contains three Python packages and a module. The interesting package is rssepisodes -- this is the Django application that runs my website:

$> ls python/rssepisodes
__init__.py      settings_dev.py     settings_prod.py     urls.py

As we see, this package only contains urls.py plus development and production settings. The remaining packages, shows and tags, are developed as stand-alone Django applications, even though they are highly specialized and not very reusable.

Tomorrow I will write about the scripts that makes this set up extremely easy to live with, namely environment.sh and manage.py, and hopefully I will write about branch-specific databases later this week.

Update: Read part 2 of this post.

I recently had more than one hour of downtime following a crash on RSSEpisodes.com, my Django powered spare-time developed website. I am still investigating the exact cause for the crash, but what I learned was that the FastCGI process was killed and it was never restarted.

No FastCGI process running means no website.

To ensure that this does not happen again, I have set up monitoring of the FastCGI process, using djb's Daemontools. Daemontools comes with a tool called supervise which monitors a daemon and restarts it if it dies. Exactly what I want and super-easy. Here's how I set it up.

I'm using Gentoo on this particular server and here are the three steps I needed to perform to install Daemontools:

# emerge daemontools
# rc-update add svscan default
# /etc/init.d/svscan start

Daemontools creates a directory /service -- in here you create one directory for each service you want monitored. In this case it is only the Django web application.

However, once svscan (the daemontools daemon) sees a directory below /service it tries to supervise it, so we will create the web application service directory somewhere else and then create a symbolic link to the appropriate location. In the examples below this link is named webapp.

The svscan daemon tries to execute a file named run in each subdirectory of /service, so we will create a simple shell-script that invokes our FastCGI process

# mkdir ~/service
<create the file ~/service/run>

The only two file that you need to create is given below. The script, run, sets the PYTHONPATH environment variable and the executes manage.py runfcgi. Notice that it calls exec so that it executes in the current process instead of forking a new one. It also calls setuidgid as a security measure so that the webapp process executes as a non-privileged user.

This is ~/service/run:

#!/bin/sh
export PYTHONPATH=$HOME/extra/python/path
export WEBAPP=$HOME/path/to/webapp
exec setuidgid username $HOME/$WEBAPP/manage.py runfcgi daemonize=false host=localhost port=4040

The important part of this script is daemonize=false. If you don't specify this the Python process will return immediatly. Daemontools will recognize that the process it started has died and of course restart it.

Now we create a symbolic link in /service that points to ~/service, I named it webapp. As soon as Daemontools sees this link, it should happen immediately, it runs the run script. We can use the svstat utility to monitor the process' uptime:

# ln -s ~/service /service/webapp
# svstat /service/*
/service/webapp: up (pid 12078) 5 seconds

If manage.py dies it will restart automatically within a second!

Whenever we want to restart the FastCGI process we simply kill it:

# kill 12078
<process is restarted by svnscan>

You do this every time you update your code.

We give manage.py a host name and port number. These values should match whatever your webserver expects. Since I'm using lighttpd to power RSSEpisodes I have something like this in my lighttpd.conf:

fastcgi.server = (
    "/django.fcgi" => 
    ((
    "host" => "127.0.0.1",
    "port" => 3333,
    "check-local" => "disable",
    ))
)

alias.url += (
    "/media/" =>  "/path/to/django/contrib/admin/media/",
)

url.rewrite-once = (
    "^(/media.*)$" => "$1",
    "^(/.*)$" => "/django.fcgi$1",
)

That was all. Now that my Django process is securly monitored I can investigate what caused the downtime in the first place.

I was browsing through documentation for various systems that fall under my administration at work today. I found the following quote in the preface of the documentation for a obscure and complex system:

This document is dedicated to Satan, no doubt the creator of both screen scraping as well as the [...] system.

It made me laugh :)

Today I stumbled upon a great article about the Python debugger, pdb. As the author points out, finding easy-to-read introductions to pdb can be quite hard

Unfortunately, most discussions of pdb are not very useful to a Python newbie -- most are very terse and simply rehash the description of pdb in the Python library reference manual.

I'm a big fan of debuggers. I've previously worked with C# and the Visual Studio.NET debugger repeatedly impressed me. Jumping around code while changing it, deleting it and even moving backwards is quite impressive and it was a valuable tool for me at the time.

However, reading about pdb today made me realize that I've never actually needed that tool for Python.

I usually write Python with a bottom-up design as described by Paul Graham. Bottom-up programming combined with the interactive shell works pretty well for me.

The results are well-defined, small functions and such functions are easy to test. In fact I often start out by writing the doctests for my functions. This is easy when programming bottom-up.

The result is that I never need a debugger. I never have the need to step into and inspect a variable, change it and retry.

Graham has an interesting note on that other kind of program design, emphasis mine:

If some component of a program grows beyond the stage where it's readily comprehensible, it becomes a mass of complexity which conceals errors as easily as a big city conceals fugitives. Such software will be hard to read, hard to test, and hard to debug.

This made me think; could it be that the complexity in the .NET debugger is necessary because of the way we write .NET programs? I.e that the programs written in .NET are hard to debug and thus a powerful debugger is required?

I think I know the answer.

Earlier today James Bennett shared his opinions on parts of the problems that users run into when trying to deploy modern web applications on traditional shared hosting providers such as Dreamhost and A Small Orange. The long-lived processes needed for modern web frameworks like Ruby on Rails and Django, are often killed after a short timeout.

I have experienced this problem myself, on this blog. This site is hosted by A Small Orange, a hosting provider that is actually listed as a Django friendly web host. The Django FastCGI process receives a termination signal after a very short time, which means that almost every request to this site needs to restart the process.

The result? Well, in practice I'm running Django as a good old CGI application. And what does that mean? Almost every visitor to this site has to wait a good 5 seconds before the first byte is returned.

Earlier this week I stumbled upon StaticGenerator, a simple tool that generates static pages from your Django models and URLs. So tonight I thought I'd give it a spin and see if it had any effect on my problem. And it sure did!

StaticGenerator generates index.html files in a directory tree corresponding to the URLs for every model object you throw at it. By adding a single line to my .htaccess and following the simple steps provided by StaticGenerator, this site is now blazingly fast!

Since my URLs are already nice and clean I added a simple rule: Only reroute to the Django FastCGI process if index.html at the given path does not exist. My .htaccess before:

RewriteRule ^/(media.*)$ /$1 [QSA,L,PT]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ django.fcgi/$1 [QSA,L]

And after:

RewriteRule ^/(media.*)$ /$1 [QSA,L,PT]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME}/index.html !-f
RewriteRule ^(.*)$ django.fcgi/$1 [QSA,L]

Easy as pie!

Looking back on my first Django powered website and seeing how its code has evolved, I can almost pinpoint to the exact day that I really understood what Django is all about.

For me Django is about the loose coupling between a stand-alone application and a Django project. As the project evolved, the code was refactored to make use of some absolutely fantastic applications, and as my own code went to /dev/null, my website gained feature after feature. In a few hours I had tossed out my buggy code and replaced it with django-registration, django-contact-form and django.contrib.comments, and I was amazed.

Not only was the resulting code much, much cleaner, but I now had features that I previously did not think I wanted.

For me the key to really understanding Django was about making use of available code and experiencing how Django encourages the use of loosely coupled designs and code re-use.

Using code coverage tools can be a fun way to help you keep the spirit while writing (sometimes) boring tests but in some situations 100% coverage can seem like a gigantic waste of time. However that doesn't mean that measuring code coverage is a waste. Take for example this simple coverage.py script combined with this script which outputs colored HTML of your code. When used correctly and applied carefully, the two can be a powerful tool that helps you ensure that at least the important parts of your code is tested thoroughly!

Earlier this year Siddharta Govindaraj wrote two great posts that showed how to integrate the above scripts with your Django code. Unfortunately they won't work if you use the latest and greatest Django development version, so I have hacked together a new and improved version of his simple test runner.

Here is how you use it:

1. Get the Python modules mentioned above and put them somewhere in your Python load path. I like to keep things simple so I keep them in my Django project folder alongside my settings.py etc. You need coverage.py, coverage_color.py and coverage_tests.py.

2. Edit your settings.py and add the following line:

   TEST_RUNNER = 'coverage_tests.run_test'

3. Add each module that you want to test to the COVERAGE_MODULES list in settings.py. For example:

   COVERAGE_MODULES = ['myproject.blog.views', 'myproject.blog.models']

And that's it. The next time you run manage.py test you will see a simple summary like this:

Name                        Stmts   Exec  Cover
-----------------------------------------------
rssepisodes.main.forms          9      9   100%
rssepisodes.main.managers      69     64    92%
rssepisodes.main.models       144    106    73%
rssepisodes.main.views         75     60    80%
-----------------------------------------------
TOTAL                         297    239    80%

And even more importantly: take a look at the files generated in build/coverage/; your code, in nicely formatted and colored HTML. Yeah!

I will try my very best to keep that script linked to above in sync with the Django development version. Contact me if you're having any trouble.