This post appeared originally in our sysadvent series and has been moved here following the discontinuation of the sysadvent microsite
In bash, we often use redirects (that is <
and >
) to get output from
a command to a file, or input from a file to a command. But sometimes,
commands takes two or more files as input. Then our ordinary scheme
does not work anymore.
Classical problem: Diff output from two commands
Let’s say you want to diff(1)
the output of two commands. For example,
compare the contents of two directories. You may run the two commands,
and redirect the output to files, then diff the files, and finally
remove the files. Awkward.
ls dir1 | sort > file1
ls dir2 | sort > file2
diff -u file1 file2
rm file1 file2
Since diff can take stdin as one input via the special filename ‘-‘, we might cut down to one file, but this is still awkward.
ls dir1 | sort > file1
ls dir2 | sort | diff -u file1 -
rm file1
Bash has (of course) a better solution: Process substitution, that is, treat the output (or input) of commands as files. Enter the process substitution operators:
>(command list) # Input
<(command list) # Output
Now, let us solve our diff challenge with a simple one-liner:
diff -u <( ls dir1 | sort) <( ls dir2 | sort )
Neat, isn’t it? I use this all the time!
Bonus: Avoid sub-shell scripting
The following bash shell loop is a pitfall often missed, leading to subtle bugs that are hard to spot. Pipe to a while loop runs in a sub-shell, so global variables goes out of scope when they are changed inside the loop.
#!/bin/bash
global=0
echo "Outside loop, global=$global"
for n in 1 2 3; do echo $n; done | \
while read i; do
global=$i
echo "Inside loop: global=$global"
done
echo "Outside loop, global=$global again :-("
Using command substitution, we avoid this elegantly:
#!/bin/bash
global=0
echo "Outside loop, global=$global"
while read i; do
global=$i
echo "Inside loop: global=$global"
done < <( for n in 1 2 3; do echo $n; done )
echo "Outside loop, global=$global still :-)"
Just-Make-toolbox
make
is a utility for automating builds. You specify the source and the build file and make
will determine which file(s) have to be re-built. Using this functionality in make as an all-round tool for command running as well, is considered common practice. Yes, you could write Shell scripts for this instead and they would be probably equally good. But using make
has its own charm (and gets you karma points).
Even this ... [continue reading]