Pyramid based project management with zc.buildout

What is the zc.buildout? According to the documentation, it is:

A system for managing development buildouts. Buildout is a project designed to solve two problems:

  • Application-centric assembly and deployment.

  • Assembly runs the gamut from stitching together libraries to create a running program, to production deployment configuration of applications, and associated systems and tools (e.g. run-control scripts, cron jobs, logs, service registration, etc.)....

  • Repeatable assembly of programs from Python software distributions.

  • Buildout puts great effort toward making program assembly a highly repeatable process, whether in a very open-ended development mode, where dependency versions aren’t locked down, or in a deployment environment where dependency versions are fully specified….

And it sounds just great, I definitely love zc.buildout because of this, but let’s see how it works with an example. Let’s say you want to build a simple website based on pyramid web framework. Pyramid has a nice tutorial on how to set up environment, install pyramid, and create pyramid package template, I  would like to do the same thing, but via zc.buildout.

Simplest use case 

First thing you need is to create a buildout configuration file that is usually called buildout.cfg, and it requires to have main section [buildout] to be defined, everything else is optional (though it would not make much sense having a configuration with only one section, because it would be useless actually; usually buildouts have more than 10 sections). The [buildout] section is actually the glue that brings everything  together, and it has a lot of options, but for now we are interested only in ‘parts’ option. It tells which other sections should be included into buildout and all listed sections must have “recipe” option to be defined.  So the most primitive buildout configuration would look like this:

[buildout]
parts =
   pyramid

[pyramid]
recipe = zc.recipe.egg:script
eggs =
   pyramid
dependent-scripts = true
interpreter = python

 As you can see this buildout configuration has only one part called [pyramid], let’s take a closer look at it. First of all, it uses zc.recipe.egg:script recipe, which is actually a name of this part installer. This recipe has pretty big set of options, but in this case only three of them concern me:

  • eggs - a list of packages I would like to have installed in part scope

  • dependent-scripts - tells if there is need to place packages console scripts into bin folder or not

  • interpreter - python interpreter which includes part scope eggs

But to build this configuration you will first need to bootstrap your buildout and you can do it with the following command:

$ wget https://bootstrap.pypa.io/bootstrap-buildout.py
$ python bootstrap-buildout.py

It will create bin and eggs directories in your project folder, and generate the buildout script. So  now you can build you buildout:

$ bin/buildout

That is is all, now you can create pyramid project as described in tutorial:

$ bin/pcreate -s alchemy my_project

Developing with zc.buildout 

zc.buildout makes development process much easier, and it’s awesome! 

Getting project source code

Let’s say you put the package you just created on github and now want to start developing it. Well, mr.developer extension will help you with that:

[buildout]
parts =
   pyramid
extensions = mr.developer
auto-checkout =
   my_project

[pyramid]
recipe = zc.recipe.egg:script
eggs =
   pyramid
dependent-scripts = true
interpreter = python

[sources]
my_project = git git@github.com:my/my_project.git 

After rebuilding buildout the src/my_project directory will appear with your project source code. BTW, you can specify as many packages as you want.

Packages source code inspecting

Sometimes there is a need to take a look at some package source code, andomelette recipe will make it easier. Add it to your buildout config and navigate through part/omelette files as if it was python module namespace: 

parts =
  ...
  omelette

[omelette]
recipe = collective.recipe.omelette
eggs = ${pyramid:eggs}

BTW, the option ‘eggs = ${pyramid:eggs}’ means that to part/omelette only packages from ‘pyramid’ section scope will be linked, no packages from other sections will appear there.

Code linting, testing and documentation building

The next thing I will do in my attempt to show more useful buildout tricks is not actually the best approach for development, but for demonstration purposes it’s fine.

Everyone knows that it's nice to have in mind a couple of things during developing process:

  • Ugly non pythonic code does not worthy

  • Not tested not working

  • No documentation no use

Well, I would like make buildout to care about these things instead of me. Let’s say that package my_package becomes pretty big with a lot of different tests, including:

  • doc-string tests based on sphinx (the fastest)

  • package unit tests and integration tests based on nose

  • acceptance or use-case tests based on robotframework (the slowest)

If you are using git as a CVS then using git-hooks for running tests is absolutely great solution to make sure that you don’t wouldn’t commit or push code that will break everything. So let’s make buildout to set up environment for testing and git hooks.

First of all, buildout should install all dependencies for code testing and linting: 

parts =
       ....
       tests_requirments

[tests_requirments]
recipe = zc.recipe.egg:script
dependent-scripts = true
eggs =
     pylint
     pep8
     nosetests
     sphinx
     coverage
     robotframework-selenium2library
     ${pyramid:eggs}

Then let’s create pre-commit hook that will lint code and run doc tests:

parts =
       ....
       pre-commit

[pre-commit]
recipe = collective.recipe.template
input = inline:
  #!/bin/sh
  cd ${buildout:directory}
  bin/pep8 hangout_api &&
  bin/pylint hangout_api --ignore=tests &&
  bin/sphinx-build -b doctest src/my_package/docs/ build
output = ${buildout:directory}/src/my_package/.git/hooks/pre-commit
mode = 755

 As you can see this time collective.recipe.template is used, and there is a lot of different recipes around for different purposes! Just check Buildout Recipes list, or search for it on pypi or even write your own recipe if you need.

And finally before making a push let’s test everything:

parts =
       ....
       pre-push

[pre-push]
recipe = collective.recipe.template
input = inline:
  #!/bin/sh
  set -e
  cd ${buildout:directory}
  bin/pep8 my_package
  bin/pylint my_package --ignore=tests
  bin/sphinx-build -b doctest src/my_package/docs/ build
  bin/nosetests my_package --no-skip --with-xcoverage --cover-package=my_package --cover-tests
  bin/pybot src/my_package/tests/robotframework
output = ${buildout:directory}/src/my_package/.git/hooks/pre-pushmode = 755

Going to production with zc.buildout

Creating separated configurations

The great thing about buildout is that you can have a lot of different configurations for different purposes. So far we have created a simple buildout configuration for development purposes, but now it's time to add a production configuration as well.

Let’s change buildout structure for a start, at the beginning it looked like this:

buildout_folder:
- bootstrap-buildout.py
- buildout.cfg

and let’s make it to look like this:

buildout_folder/
- bootstrap-buildout.py
- buildout.cfg
- profiles/
- - base.cfg
- - production.cfg
- - development.cfg

Change buildout.cfg to look like this:

[buildout]
extends =
     profiles/production.cfg
#      profiles/devel.cfg

BTW, the other great thing about buildout is that it supports configurations inheritance,  just list the a configuration you want to inherit in “extends” list and you got it!

In production configuration we do not need to have all that stuff  for testing and source code checking, so let’s move it to “profiles/devel.cfg:

[buildout]
extends = profiles/base.cfg
extensions += mr.developer
show-picked-versions = true
auto-checkout = my_project
parts +=
  omelette
  tests_requirments
  pre-commit
  pre-push[sources]
my_project = git git@github.com:my/my_project.git

[omelette]
recipe = collective.recipe.omelette
eggs = ${pyramid:eggs}

[tests_requirments]
recipe = zc.recipe.egg:script
dependent-scripts = true
eggs =
     pylint
     pep8
     nosetests
     sphinx
     coverage
     robotframework-selenium2library
     ${pyramid:eggs}

[pre-commit]
recipe = collective.recipe.template
input = inline:
 #!/bin/sh
 cd ${buildout:directory}
 bin/pep8 hangout_api &&
 bin/pylint hangout_api --ignore=tests &&
 bin/sphinx-build -b doctest src/my_package/docs/ build
 output = ${buildout:directory}/src/my_package/.git/hooks/pre-commit
 mode = 755

[pre-push]
recipe = collective.recipe.template
input = inline:
   #!/bin/sh
   cd ${buildout:directory}
   bin/pep8 my_package &&
   bin/pylint my_package --ignore=tests &&
   bin/sphinx-build -b doctest src/my_package/docs/ build &&
   bin/nosetests my_package --no-skip --with-xcoverage --cover-package=my_package --cover-tests &&
   bin/pybot src/my_package/tests/robotframework
output = ${buildout:directory}/src/my_package/.git/hooks/pre-push
mode = 755

As you can see it uses profiles/base.cfg as a parent configuration, and you may notice that “extensions” and “parts” options of buildout have notation with “+=”, which means that parent configuration options would be extended with new values, but not be overwritten.

A bit of security and packages sources

The base.cfg is a main configuration file, so it should be as simple as possible. Actually, it would look like the example I started with, but with a few differences:

[buildout]
parts =
  pyramid

find-links =
  http://mydist.com

allow-hosts =
  *.python.org
  mydist.com

[pyramid]
recipe = zc.recipe.egg:script
eggs =
  pyramid
  my_package
dependent-scripts = true
interpreter = python

The ‘find-links’ option extends the locations list of distributions search with ‘mydist.com’, and ‘allow-hosts’ option sets the hosts list where zc.buildout will look for distributions.

Fixing packages versions and adding crontab config

What am I expecting from production configuration? I would like to have at least two things:

  • a full control over used packages versions;

  • site automatic start after system reboot

So for achieving these goals production.cfg would look like:

[buildout]
extends = base.cfg
versions = versions
allow-picked-versions = false
newest = false
parts +=
   crontab

[crontab]
recipe = z3c.recipe.usercrontab
times = @reboot
command = ${buildout:directory}/bin/pserve production_config.ini --deamon

[versions]
my_package = 0.1
pyramid = 1.5.2
zc.recipe.egg = 2.0.1
…

There is a ‘crontab’ part which uses z3c.recipe.usercrontab recipe and will run pserve command with ‘--deamon’ option on system start. Also there is a “versions” option which tells zc.buildout to take packages versions list from [versions] section. And the most important options here is “newest = false” which means that buildout will never take the a newest version of package, event if there is one; and the “allow-picked-versions = false” which means that you won't be able to build a buildout before you list all the packages versions which are used in your project in [versions] section.

That’s all. I hope this post was useful and helped you understand the power of zc.buildout.

Prev Post Next Post