Devops scripting in Python

posted: November 10, 2018

tl;dr: Tips on how to use Python to write better devops scripts...

I’ve used bash for decades, and normally favor using ubiquitous tools, especially for devops tasks. Almost always a devops script is developed on one machine, perhaps a developer’s laptop, and run on potentially numerous other machines. Those other machines may have different OS flavors and versions, and different software packages installed, so choosing a ubiquitous, lowest common denominator script interpreter such as bash can avoid headaches when the script is run elsewhere.

A short, simple bash script that runs a few commands can be easy to write, but bash starts to suffer as script size and complexity grows. Bash syntax shows its age in comparison to other more modern languages, and is less readable. Fewer new programmers enter the workplace schooled in bash. The maintenance of non-trivial bash scripts can be a challenge.

Python is a great modern alternative to bash for devops scripts of moderate to high complexity, for these reasons:

Python is a more powerful language than bash with fewer quirks. This can be very helpful when doing more sophisticated script tasks, such as transforming data.
Python comes “batteries included” with devops-oriented modules in the standard library, reducing the need to install other dependencies.
It’s easier to handle error conditions and exceptions in Python.
Python makes it easier to write scripts that have command line options with help, and robust validation of input and dependencies.
Younger programmers often are schooled in Python already, or can pretty easily pick it up due to its quick learning curve.

Python is becoming more ubiquitous, although Apple needs to start pre-installing Python3 in MacOS. Aaron Christianson’s article Replacing Bash Scripting with Python goes into more technical depth about the shortcomings of bash. I won’t bash bash any further, but will instead offer my own advice on writing Python devops scripts.

I usually import these standard Python libraries into most of my devops scripts:

argparse: built-in command line argument parser, with typing, default values, and an easy-to-build help system
os: provides information about the operating system environment, including environment variables, and executes file system commands
subprocess: used to run other commands and installed programs
sys: provides access to the command line arguments and other Python interpreter variables

My best practices for writing Python devops scripts:

You’ll avoid a few headaches and quirks by using Python3. You’ll also avoid the Python2 end-of-life in 2020. If a target machine doesn’t have Python3, install it.
Minimize dependencies. The breadth of the Python standard library helps here, and I’d recommend sticking to the standard library wherever possible.
Use command line arguments and environment variables to pass needed information to the script, rather than prompting the user. This helps with automation and running the script remotely. Command line help is easy to add with the argparse library; it is literally just a keyword argument in the add_argument method.
Check for the presence of other needed system dependencies first, before entering the main body of work for the script. If a dependency is not present on the machine, it can be installed automatically. But if it is a something that the user may want to think twice about before installing on the machine, the script can fail with a detailed message to the user about what needs to be installed.
Try to anticipate (and test, during script development) everything that can possibly go wrong at each step in the script. What if a sub process being invoked fails? What if there is bad or missing data? A large percentage of the lines of the script should be checking for errors and catching exceptions. Provide the detailed error messages that you always wished you had when running some else’s program.
Print step-by-step messages as the script executes, to provide visual evidence to the user that the script is running. Do this especially for steps that may take a while to complete.
Clean up after yourself by restoring the machine to its original state, except for any installed packages that should persist long term. The tempfile Python standard library can be used to automatically clean up temporary files and/or directories after exiting a context manager.