Bash programming tips and terminology

The code is the word! Examples of "good" bash scripting:

• syno
• kops-gce

Use Shellcheck

Shellcheck is a small utility that checks the style and posix compliance of your bash scrips. Use it as follows:

brew install shellcheck
shellcheck my_new_script

Shellcheck GitHub page, try it out!

Protecting a script from being invoked from somewhere else

Inside a bash script, pwd will echo the invoking shell working directory. Imagine this situation:

Imagine you have this script:

#!/usr/bin/env bash

# Get own directory
DIRECTORY=$(cd `dirname $0` && pwd)
echo $DIRECTORY
echo $(pwd)

Imagine you place the script in a subdirectory, like this:

~:              Your home dir
~/install:      The dir where you're script will be located

Now, let's see what happens when you invoke the script from two different places:

~ $ cd install
~/install $ ./script.sh
/Users/javiercm/install
/Users/javiercm/install

All is good, as expected. But what if you invoke the script from your home directory?

~ $ cd
~ $ ./install/script.sh
/Users/javiercm/install
/Users/javiercm

What is happening is that the pwd command inside the script is printing the working directory of the script's parent shell, and this is wrong if you're referencing paths inside the script relative to the script location. So be careful, and use the ${DIRECTORY} variable instead.

Using POSIX mode

Use set -o posix in your scripts. Implications of POSIX mode are well documented here.

Safer bash scripts

See this post: Safer bash scripts with set -euxo pipefail good practices in general. What I do now is I start all my scripts with this header:

#!/usr/bin/env bash

set -CeE
set -o pipefail
if [[ "${BASH_VERSINFO:-0}" -lt 4 ]]; then
  cat <<EOF >&2
WARNING: bash ${BASH_VERSION} does not support several modern safety features.
This script was written with the latest POSIX standard in mind, and was only
tested with modern shell standards. This script may not perform correctly in
this environment.
EOF
  sleep 1
else
  set -u
fi

Different options are:

• set -C: Prevent output redirection using >, >&, and <> from overwriting existing files.
• set -e: Exit immediately if a pipeline (see Pipelines), which may consist of a single simple command (see Simple Commands), a list (see Lists), or a compound command (see Compound Commands) returns a non-zero status.
• set -E: If set, any trap on ERR is inherited by shell functions, command substitutions, and commands executed in a subshell environment. The ERR trap is normally not inherited in such cases.
• set -o pipefail: If set, the return value of a pipeline is the value of the last (rightmost) command to exit with a non-zero status, or zero if all commands in the pipeline exit successfully.
• set -u: This option causes the bash shell to treat unset variables as an error and exit immediately. Unset variables are a common cause of bugs in shell scripts, so having unset variables cause an immediate exit is often highly desirable behavior. Sometimes you’ll want to use a ${a:-b} variable assignment to ensure a variable is assigned a default value of b when a is either empty or undefined. The -u option is smart enough to not cause an immediate exit in such a scenario.

Using main()

Always use main "${@}" and a main() {} function in your bash scripts to embed any kind of logic. main() at the same time should invoke other functions encapsulating specific logic. Keep global vars if you want outside main:

#!/usr/bin/env bash

GLOBAL_VAR="${GLOBAL_VAR:-"value"}"

step1() {
  true
}

step2() {
  true
}

main() {
  step 1
  step 2
}

main "${@}"

Use bash in path for shebang

Whenever possible, start with this #!/usr/bin/env bash shebang instead of #!/bin/bash. Yes, there is no POSIX defined location for env, but in the other hand, it is quite common in Mac OS X systems to have upgraded bash via brew and have this newer version enabled and easily accessible through env.

Stopping a script for a watch command and resuming when a string is found

This is a typical use case when doing software installs. A bash script deploys a yaml file on a Kubernetes cluster, and you issue a watch command waiting for some pods to be ready.

The more general and powerful expect tool can be used for this. Imagine you want a script to pause until a specific file name named 'When I say stop, continue' appears in a directory. Expect will allow this to happen:

#!/usr/bin/env bash
expect <(cat <<'EOD'
set timeout 120
spawn watch "ls -l"
expect "When I say stop, continue"
EOD
)
echo "Success"

If the files does not exist, the script will pause 2 minutes waiting for the file to appear. If the file appears (touch "When I say stop, continue"), the script will continue and print success.

Source files relative to the script dir

Make sure your paths stay full by knowing where are you running your script for. This is the snippet that works best:

readonly SCRIPT_NAME="${0##*/}"
SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd)

Parameters and variables

A parameter is an entity that stores values. It can be a name, or a special character. A variable is a parameter noted by a name.

More at GNU Bash documentation

Checking proper bash version

This checks if Bash version is greater than 4. If not, and we have set -eenabled, the script will exit, copying the HereDoc into the standard error. If yes, it will do set -u, which treats unset variables and parameters other than the special parameters @or * as an error when performing parameter expansion. An error will be written to the standard error and a non-interactive shell will exit.

set -e
if [[ "${BASH_VERSINFO:-0}" -lt 4 ]]; then
  cat << EOF >&2
WARNING: bash ${BASH_VERSION} does not support several modern safety features.
This script was written with the latest POSIX standard in mind, and was only
tested with modern shell standards. This script may not perform correctly in
this environment.
EOF
  sleep 1
else
  set -u
fi

Source-only scripts

For the bash scripts that should only be sourced, it is advisable to use as shebang:

#!/bin/false
# shellcheck shell=bash

Last comment will enable shellchecking this script the same.

Setting default values for variables

This assigns to VARSAMPLE the value default-value if DEFAULTVAR was previously unset or null. It does not touch the DEFAULTVAR value.

VARSAMPLE="${DEFAULTVAR:-"default-value"}"

An example of this is setting a default value for a variable if the variable has not been assigned a value before:

VARSAMPLE="${VARSAMPLE:-"default-value"}"

The following code assigns to DEFAULTVAR the value default-value if DEFAULTVAR was previously unset or null, and then the value of DEFAULTVAR is assigned to VARSAMPLE:

VARSAMPLE="${DEFAULTVAR:="default-value"}"

This example sets the value of BASEURL if previously unset or null and also assigns a full path tot the FULLURL variable:

FULLURL="${BASEURL:="default-value"}/filename"

Making a variable both local and readonly

Let's make VARSAMPLE both local and read-only:

sample_func() {
  VARSAMPLE="sample value"
  local -r VARSAMPLE
}

Using associative arrays (dictionaries)

Let's declare and populate an associative array:

declare -A userdata
userdata[name]="Javier"
userdata[city]="Madrid"
#Also, declare all at once
declare -A userdata=( [name]="Javier" [city]="Madrid" )

Iterate over the array:

for i in "${!userdata[@]}"
do
  echo "key  : $i"
  echo "value: ${array[$i]}"
done

Unsetting the entire array:

unset  "userdata[*]"

Use a info function to display messages

Include this in your scripts; the var SCRIPT_NAME should be global in the script just in case we want to reuse it somewhere:

SCRIPT_NAME="${0##*/}"

info "This is an info message"

info() {
  echo "${SCRIPT_NAME}: ${1}"
}

"${@}" or "${*}"

"${@}" is the same as "${*}", but for "${@}" each parameter is a quoted string, that is, the parameters are passed on intact, without interpretation or expansion. When "${*}" is used then the command-line arguments are joined into one string using the first character of IFS as the "glue" between each argument (by default, IFS is a white space but can be altered).

Assuming $IFS is c:

Syntax	Effective Result
"${@}"	"\(1" "\)"2 "$3"
"${*}"	"\(1c\)2c$3"

&& and ||

Consider the effect of the && and || operators in the following function:

run() {
  { "${@}" && return 0; } || true
  echo "Failed executing ${*}"
  return 1
}

Inside the function, let's focus on the first line. The curly braces are a way to just group (like parenthesis in other programming languages) commands. The right side of && will only be evaluated if the exit status of the left side is zero (i.e. true). In this case "${@}" is a full bash liner passed for execution (example cd newdir). If executions succeeds, the exit status will be 0 (true), so then the function will return 0.

|| is the opposite of &&: it will evaluate the right side only if the left side exit status is non-zero (i.e. false). In this case, this is avoiding non-trapped execution failures and letting the function control what;s going to happen in the case of non successful execution.

Changing shell options inside the scope of a function

Imagine you want to change how the shell behaves through shopts, but only in the scope of a function. One example could be getting the shell to ignore caps for a case statement inside a function that's processing cli options.

For this, you can use TRAP. TRAP is used to trap signals and other events, it follows the format TRAP [argument] [SIGNAL]. In the previous example, let's do something like this:

parse_args() {
  trap "$(shopt -p nocasematch)" RETURN
  shopt -s nocasematch
}

This reads as follows: first, $(shopt -p nocasematch) is executed. shopt -p prints the current value of the nocasematch setting in an executable for. For example, it will return something like shopt -u nocasematch (which means that nocasematch is unset). This is, in a way, storing the current nocasematch value before proceeding to alter it.

Then, we have the trap statement. This sets up a handler for when the signal RETURN is executed. RETURN is executed when the function itsel finishes executing. When that happens, the handler shopt -u nocasematch will execute, effectively restoring the value of nocasematch.

The key thing here is that the "$()" part is processed when the line is read, and that execution in turn returns a command that is at the same time used as a handler for when the RETURN signal is trapped.

Process Substitution

Todo

Still to be properly documented and reviewd, here as a reminder.

Using process substitution here (see http://www.gnu.org/software/bash/manual/html_node/Process-Substitution.html#Process-Substitution)

Shifting array elements

Sometimes, you want to shift through array elements the same you shift throught positional arguments with shift. Consider this script, that shifts one element the array to the rights:

#!/bin/bash

foo=('foo bar' 'foo baz' 'bar baz')
echo ${foo[@]}
foo=("${foo[@]:1}")
echo ${foo[@]}

The script prints:

foo bar foo baz bar baz
foo baz bar baz

Handling positional parameters

Parameter(s)	Description
$0	the first positional parameter, equivalent to argv[0] in C, see the first argument
$FUNCNAME	the function name (attention: inside a function, $0 is still the $0 of the shell, not the function name)
$1 … $9	the argument list elements from 1 to 9
${10} … ${N}	the argument list elements beyond 9 (note the parameter expansion syntax!)
$*	all positional parameters except $0, see mass usage
$@	all positional parameters except $0, see mass usage
$#	the number of arguments, not counting $0

Understanding bash redirection

See these three valuable URLs:

• GNU Bash documentation on redirection
• Bash One-Liners Explained, Part III: All about redirections
• Differences between >&2 and &>2