Python: Creating Your Project Structure

I love using Python for creating command-line applications which require just a bit more logic than writing a bash-script.

Python has excellent libraries for parsing command-line arguments and running other shell commands, while at the same time you can take advantage of the powerful object-oriented language. Additionally you can verify and document your application with the Python's unit testing framework.

The Example Application for this text can be found in github.com/mrako/python-example-project.

You can also read the article in mrako.com

Structuring your Application

In my experience the best folder structure for a Python project is to put your executables in a bin folder and your project into a your project name folder (I will use name project in this text). This way you can keep your core functionalities separated and reusable. It is also a standard for other applications.

In project folder you should have main.py as the main access point to your application. Your functionalities should then go to lib and your unitests to tests. This way the skeleton of your project should look like:

+ bin
  - project
+ project
  - main.py
  + lib
  + tests

Now your application can be executed by running:

$ bin/project <parameters>

Separating Parameters, Shell Commands and Functionalities

As in all object-oriented programming you should aim to separate your concerns. It's often forgotten in Python applications because reading the command-line parameters, handling options and running other shell commands is very easy.

Parsing the Command-Line Options

Create a class which defines and collects the command-line parameters. Python provides optparse, for which you can very easily define the options and behavior:

usage = 'bin/project'
parser = OptionParser(usage=usage)
parser.add_option('-x', '--example',
                  default='example-value',
                  dest='example',
                  help='An example option')

Now you have created a parser which reads a target value to the example variable by running bin/project -x <target value> or bin/project --example <target value>

Running other Shell Commands

If you want to create an application which depends on other shell commands you should separate the shell execution in its own class. This way your core functionalities can be easily re-used in other environments or applications, and you can more easily collect logs, errors and exceptions that come from external sources. I recommend you to use the Python subprocess for your external shell commands.

Create a classes for the process execution and the process exceptions (see process-class in the example project).

The Core Functionalities

In your project/lib/project.py you can now implement the core functionalities. Since this is an example I've only included receiving options(which have been separated from collecting the command-line arguments) and running the date command through your Process-class. See project.py in the example project.

Running your app

Call your project's main.py from the bin/project executable:

#!/bin/bash

BINPATH=`dirname $0`
python "$BINPATH/../project/main.py" $@

Don't forget to change the access rights for execution:

$ chmod 755 bin/project

In main.py you then collect the command-line options and run your application:

import sys

from lib import Project
from lib import Options

if __name__ == '__main__':
    options = Options()
    opts, args = options.parse(sys.argv[1:])
    v = Project(opts)

    print v.date()

Finally you are ready to run the application:

$ bin/project <arguments>

Testing (and Documenting)

As already written in Unit Testing in Python and Short Introduction to Unit Testing, I cannot emphasize the importance of testing enough. With unit tests you can guide the development, verify your functionalities and document the behavior of your application.

Add your tests in the project/tests folder (see tests-folder in the example project). I recommend you to use nose for running your tests.

Testing Command-Line Arguments

import unittest

from lib import Options

class TestCommandLineArguments(unittest.TestCase):
    def setUp(self):
        self.options = Options()

    def test_defaults_options_are_set(self):
        opts, args = self.options.parse()
        self.assertEquals(opts.example, 'example-value')

Running the Tests

$ nosetests
.....
Ran 5 tests in 0.054s

OK

Related protips:

Flatten a list of lists in one line in Python