I do some consultation work with a Université de Montréal lab writing open-source qMRI software. The software is yet to be released, but we're currently writing up some example Bash scripts so users have an idea of how the software can be used. Bash is extremely powerful as a scripting language, but because of its unfettered filesystem access, it can lead us into some tricky situations. Of course, it's always better to avoid pits than to climb out of them after having fallen in, so I set out to make a short slide deck. The topic? Best practices that can be followed in Bash to avoid those pitfalls. I recorded my talk and it was uploaded to the lab's YouTube channel.
In case you don't have 14 minutes and 33 seconds to spare, I'll summarize some points from my talk here.
Use ShellCheck. It checks syntax and semantics. Any issue it detects in the source code raises a warning. In the tool's output in the terminal or in its webapp, it points exactly where the problem is, how to fix it, and links to a wiki page explaining further about that exact pitfall. You can easily integrate this automated checking into PR checks on GitHub through either GH Actions or include it in a Travis build, which I've done for shimming-toolbox.
It's always nice to have a guidebook to reference when it comes to stylistic best practices. Google has done that for most of the common production languages including Bash. The Google Shell Style Guide solidifies some of the ambiguities with Bash to improve the maintainability of Bash scripts.
I can't understate the importance of testing. Unfortunately, there isn't any easy way to automatically set up tests for Bash scripts (if you know of one, my email is at the bottom of the page!), so we have to do things manually. Create a toy example script with some dummy files that you're not afraid of overwriting, and ensure that the script behaves as expected. When deciding on test cases, you should think about the following questions:
These are not comprehensive, but I've picked out a few that I think are really important to keep in your toolbelt.
1#!/bin/bash
On macOS, this points to the builtin Bash (aka an old version), which may not have the features that you're expecting.
1#!/usr/bin/env bash
On all POSIX systems, this points to whichever Bash is first in the PATH (including Homebrew or MacPorts Bash) or aliased.
Note: the example and the consequences of this in the video is inaccurate: it works correctly with both single and double brackets. Special thanks to Dr. Joseph D'Silva for teaching me this example in COMP 206 at McGill University.
1#!/usr/bin/env bash23if [ -r $1 ]4then5 ls -l $16else7 echo "Error $1 does not exist, refusing to execute ls"8fi
The intent of this script is to ls -l
the file provided to the script as an argument. If you neglect to supply the script with an argument when you run it,
it still tries to execute the ls -l
clause of the structure.
1#!/usr/bin/env bash23if [ -r $1 ]4then5 ls -l $16else7 echo "Error $1 does not exist, refusing to execute ls"8fi
The else
clause is executed in this case as intended when no argument is supplied to the script. This is because double-brackets do some basic variable checking first before proceeding. See here for further details. The GNU Bash Reference manual also goes into exhaustive detail on this (scroll down to [[…]]
).
1foo=`ls`
Backticks are bad - nesting them requires backslash escaping:
1\`
1foo="$(ls)"
The $(command)
format doesn't change when nested, and is therefore much easier to read if you have to go into 2 or 3 levels of nesting.
1st_shim -fmap fieldmap.nii [-coil-profile $SHIM_DIR/coils/siemens_terra.nii] -mask mask.nii -physio XX -method {volumewise, slicewise}
1st_shim -fmap "fieldmap.nii" [-coil-profile "${SHIM_DIR}/coils/siemens_terra.nii"] -mask "mask.nii" -physio XX -method {volumewise, slicewise}
Note the use of curly braces ({}
) for quoting a variable within a string.
When you write a Bash script (or any source code file, really), write a nice header comment about:
This is related to writing good documentation.
Speaking of return codes, you can use them in Bash, just like with C/C++. 0 indicates success, nonzero indicates failure. If you anticipate multiple kinds of failures, use different codes for them! For example, you might use 1 for a missing file, and 3 for a module runtime crash. Note that the return code won't be printed, but is visible when the user runs echo $?
in the terminal. $?
is a special variable for the status of the last-executed executable.
Hint: you can use this to catch a command failure inside your script too...
Suppose you want to write a script that takes a single file as an argument, copies it into a temporary directory, appends _copied
to the original's filename, the removes the temporary directory. You know there's a possibility the cp
command could fail if the copied file is excessively large and you're nearing your disk quota, but you want the cleanup function to be executed regardless of a successful copy. Here's how you would do so:
1#!/usr/bin/env bash23FILE=$145cleanup(){6 rm -rf "temp"7}89mkdir "temp"1011cp "$FILE" "temp/"1213CP_CODE=$?1415if [[ "CP_CODE" -ne 0 ]]16then17 "Copy failed with $CP_CODE"18 cleanup19 exit "$CP_CODE"20fi2122mv "$FILE" "${FILE}_copied"23cleanup24exit 0
There are some really convenient file tests built into Bash for different states:
$FILENAME
exists: [[ -e $FILENAME ]]
$FILENAME
is readable: [[ -r $FILENAME ]]
$FILENAME
is executable: [[ -x $FILENAME ]]
You can read more examples at https://devhints.io/bash, which also has a wealth of other Bash recipes.
So that's the gist of my talk. Here are all the links I've taken this material from.I encourage you to spend an hour and read all of them - bookmark them and reopen them whenever you have to write more Bash scripts.
Above all, practice with some toy examples in a dummy directory with files you're not afraid of overwriting.
Best of luck with your Bash scripting!
Questions? Comments? Write to me at [email protected].