Fabric and Celery can be friends!

Recently, I've been working on a project that requires a lot of remote host management. It's Django based, and uses Celery and Fabric for task and SSH connection management, respectively. There are, however, a few gottchas that are applicable both in my situation and really in any other situation where you want to use Fabric as an API.

Dynamic Hosts

The first big step was being able to support dynamic hosts. The solution for this is simple. Inside any method in which we are grabbing host information and want to connect to that host via SSH through Fabric, we define an internal method with the @hosts decorator from the fabric API.

from fabric.api import hosts
from celery import task

@task()
def my_celery_task():
    username, host = grab_my_host_info()
    host_string = "%s@%s" % (username, host)

    @hosts(host_string)
    def my_fab_task():
        run("ls")

    execute(my_fab_task)

Exception Handling

This took a bit more research, but ended up not being as bad as I thought. Fabric has a fail-fast execution policy. Thus, if absolutely anything happens, it aborts, via abort().

This is a huge problem with celery, as what this ends up doing is killing your worker. The solution for this is two fold, and both are settings we enable on Fabric's global fabric.api.env variable.

These settings are skip_bad_hosts and warn_only. Both of these allow for custom error handling by the programmer. Note that in both cases you will need to watch the return value of any call for the error. While most fabric calls like run(), cd() and get() will return with a *.failed attribute, execute will not.

Execute will return a dictionary whose keys are the different host strings the task was supposed to run on, along with an exception as the value if one occurred. Error handling could look something like this, modifying to our previous example.

from fabric.api import hosts, env
from celery import task

env.skip_bad_hosts = True
env.warn_only = True

@task()
def my_celery_task():
    username, host = grab_my_host_info()
    host_string = "%s@%s" % (username, host)

    @hosts(host_string)
    def my_fab_task():
        run("ls")

    try:
        result = execute(my_fab_task)
        if isinstance(result.get(host_string, None), BaseException):
            raise result.get(host_string)
    except Exception as e:
        print "my_celery_task -- %s" % e.message

Replacing the print statement with your favourite logger, of course.

Env Context Management

Now, it might not be the best thing in the world to go and specify hard env variable values in the script like that. You can also use the settings context manager, either decorating the fabric task, or by using a with statement.

Cool links

Here are some links that helped me figure all this stuff out.

http://awaseroot.wordpress.com/2012/06/05/fabric-wrappers-and-more-error-handling/

http://docs.fabfile.org/en/1.4.0/usage/env.html

http://docs.fabfile.org/en/1.4.0/usage/execution.html