Automated deployment using Fabric

Pootle can be deployed using Fabric automation scripts. There are other methods to deploy Pootle, but using Fabric with these scripts allows automated deployments and simplifies maintenance tasks and the upgrade to newer versions.

The sample scripts bundled with Pootle allow you to deploy a Pootle server using a Python virtualenv, running on a Apache server with mod_wsgi using MySQL as database server on Debian-based systems. These sample scripts can be modified to perform different deployments.

To see a comprehensive list of all Fabric commands available to deploy Pootle check the Fabric commands reference.

Preparing the remote server

Before performing an automated deployment using Fabric, make sure the server where Pootle is going to be deployed has the required software.

Installing required software on the remote server

Before proceeding, install the following software (if absent) on the remote server:

  • Python 2.5, 2.6, or 2.7
  • python-pip
  • Git distributed version control system
  • Apache web server
  • MySQL database server
  • OpenSSH server
  • C compiler (to install Pootle’s Python dependencies – can be removed later)

Note

Currently only Debian-based (e.g. Ubuntu) servers are supported.

Note

If you have problems installing the dependencies during the bootstrap you are probably missing other packages needed to build those third-party Python modules. For example, lxml needs development files for libxml2 and libxslt1 (as well as the C compiler mentioned above).

Note

Also consider installing optional packages for optimal performance.

Hardware requirements

Check the Hardware requirements on Installation docs.

Preparing Fabric deployment

Before performing a deployment you will need to install some software on the local computer and configure the necessary settings to connect to the remote server.

Installing required software on the local computer

The first step is to install the necessary software on the local computer.

Note

We strongly recommend using a virtual environment (virtualenv). Check the Setting up the Environment docs for information about virtualenvs.

$ pip install Fabric

Getting Pootle Fabric files

Pootle is bundled with sample scripts for deploying using Fabric. The relevant files are:

  • fabfile.py
  • Files inside the deploy/ directory

You can get those files from the Pootle GitHub repository. The rest of the Pootle files are not necessary for this kind of deployment.

Setting up Fabric

The deploy/ directory contains sample files that can be used in combination with the fabfile.py file for deploying Pootle servers.

There are two different deployment environments. Each one has a directory inside deploy/:

  • Staging environment: deploy/staging/ directory
  • Production environment: deploy/production/ directory

This way server administrators can separate their testing and real-world Pootle servers.

For deploying a Pootle server using one of the environments it is necessary to put some configuration files in place:

  • deploy/pootle.wsgi WSGI script that will be used to run Pootle.
  • deploy/ENVIRONMENT/fabric.py Module with settings that will be used in Fabric.
  • deploy/ENVIRONMENT/settings.conf Pootle-specific settings for the server (it will override the defaults). For example, the settings for connecting to the database will go here.
  • deploy/ENVIRONMENT/virtualhost.conf Apache VirtualHost configuration file.

In the previous paths ENVIRONMENT is the directory name for the chosen environment (production or staging).

All the settings defined in the deploy/ENVIRONMENT/fabric.py module will populate the Fabric env dictionary, making the configuration keys available in the deploy/ENVIRONMENT/settings.conf and deploy/ENVIRONMENT/virtualhost.conf files. You can use basic Python string formatting to access the configuration values.

Note

Sample configuration files are provided for reference under the deploy/ directory. Put them in the desired environment directory, and adapt them to your needs before running any Fabric commands.

Check pootle/settings/90-local.conf.sample to see settings that you might want to use in deploy/ENVIRONMENT/settings.conf.

Note

If it is necessary you can adapt the deploy/pootle.wsgi file to meet your needs.

Once you make your changes to the settings you are ready to run the Fabric commands.

Note

For security, please make sure you change the db_password setting – using the example one could make your server vulnerable to exploits. The db_password setting is used both to properly configure Pootle, as well as to set up the database user access for the deployment.

The db_root_password setting, on the other hand, is only used to configure the MySQL options file, if you choose to do this, and is only needed when creating the database (if the normal user does not have the necessary permissions). Leaving this with default setting will have no security impact.

How to run Fabric commands

In order to run Fabric commands for Pootle it is necessary that the directory containing the fabfile.py file and the deploy subdirectory be included in the PYTHONPATH. If it is not, then add it using:

$ export PYTHONPATH=`pwd`:$PYTHONPATH

The fabric commands need to know the type of environment in which they are going to work, e.g. if the deployment will be for the production environment. The Fabric commands for Pootle support two environments: production and staging. To select the environment for running a command just add it before the command like this:

$ fab production bootstrap  # Use the 'production' environment
$ fab staging bootstrap     # Or use the 'staging' environment

Note

It is necessary to install Fabric in order to be able to run the fab command.

Provide arguments

Some commands do accept arguments – the argument name is followed by a colon (:) and the value for the argument (with no spaces):

$ fab production load_db:dumpfile=backup_mysql.sql  # Call load_db providing a database dump to load

The previous call runs the load_db command providing the value backup_mysql.sql for its dumpfile argument.

Tweak the environment

One possible use for arguments is to tweak the environment when setting it, before calling the commands:

$ fab production:branch=stable/2.5.0 bootstrap  # Run bootstrap for a branch

In the previous example bootstrap is run after setting the environment using production but changing the branch to work on, to be the value stable/2.5.0 passed to the branch argument.

Run several commands in a row

It is possible to run several commands in a row with a single call:

$ # Run several commands in a row using a single call to fab
$ fab production:branch=stable/2.5.0 bootstrap create_db load_db:dumpfile=backup_mysql.sql

The previous call will run production followed by bootstrap, create_db and load_db, in that exact order.

Note

If you want to know more about Fabric, please read its documentation.

See the Fabric commands reference for a comprehensive list of all Fabric commands available for deploying Pootle, with detailed descriptions of each command.

Configuring passwordless access

While it is not required, it is much easier to perform deployment operations without interactive prompts for login, sudo, or MySQL database passwords:

  • You can eliminate the need for an SSH login password by adding your public SSH key(s) to the ~/.ssh/authorized_hosts file of the user on the remote server.

  • You can eliminate the need for sudo passwords by adding in the /etc/sudoers.d/ directory on the remote server, a file (with permissions mode 440) containing the line:

    username ALL = (ALL) NOPASSWD: ALL

    where username must be replaced with the user configured in the fabric.py settings file.

  • You can eliminate the need for MySQL passwords by configuring the database password(s) in the fabric.py settings file, running the mysql_conf fabric command to create a MySQL options file for the remote user:

    $ fab production mysql_conf  # Set up MySQL options file

    and then modifying the fabric.py settings file to un-comment the alternate value for db_password_opt (and optionally db_root_password_opt, if db_root_password is configured).

Typical Usage Example

A typical usage example is included here in order to provide a more easy to understand example on how to use this deployment method and the available commands.

Bootstrapping the environment

You can install the Pootle software, configuration files, and directory tree(s) with the bootstrap command.

$ export PYTHONPATH=`pwd`:$PYTHONPATH
$ fab production:branch=stable/2.5.0 bootstrap  # Install Pootle 2.5

Setting Up the Database

When setting up the database there are several possible scenarios:

  • If creating a new database from scratch:

    $ fab production create_db  # Creates Pootle DB on MySQL
$ fab production update_config  # Uploads the settings
$ fab production setup  # Creates the DB tables and populates the DB

  • If creating a blank database and populating with a (local) database backup:

    $ fab production create_db  # Creates Pootle DB on MySQL
$ fab production load_db:dumpfile=backup_mysql.sql # Populate DB from local dump

    Note

    The dumpfile (for load_db and dump_db) is local to the system where Fabric runs, and is automatically copied to/from the remote server.

  • If updating a previous version database (possibly just loaded with load_db) to the latest version of the schema:

    $ fab production update_config  # Uploads the settings
$ fab production setup  # Updates the DB to the latest version

Enabling the web server

$ fab production:branch=stable/2.5.0 deploy