Dynamically Building Commands Using xargs, bash, and foreach

One of my favorite features of the find command is the -exec option. This option lets you execute a command using the path of each file found by the find command. For example:

$ find /home/user1 -type f -exec md5sum '{}' \;

6383e3def0fae45426d48186b0568b2b /home/user1/file1.txt
bacbbc96f0c76277cb51df3737460042 /home/user1/file2.txt
.
.
.

In this example find / -type f will recursively search for files in the system starting in the /home/user1 directory. Find will execute the md5sum command against each file that is found and the result is sent to standard out.

Unfortunately most commands do not have an option like -exec, but you can approximate this behavior using some redirection and command line magic.

The xargs Command

Suppose you had the following text file with a list of file paths:

/home/user1/file1.txt
/home/user1/file2.txt

file_list.txt

Now imagine that you wanted to create an MD5 hash of each of the files listed in file_list.txt. The md5sum command does not natively support an option to do this, but you can use the xargs command to get it done:

$ cat file_list.txt | xargs md5sum

6383e3def0fae45426d48186b0568b2b /home/user1/file1.txt
bacbbc96f0c76277cb51df3737460042 /home/user1/file2.txt

Here we are piping the output of the cat command (contents of file_list.txt) into the xargs command. The xargs command then executes the md5sum command providing each line of input as an argument. This effectively creates an MD5 hash for each file listed in file_list.txt.

You can also eliminate the use of the cat command and pipe by simply redirecting file_list.txt into standard in of xargs:

$ xargs md5sum < file_list.txt

6383e3def0fae45426d48186b0568b2b /home/user1/file1.txt
bacbbc96f0c76277cb51df3737460042 /home/user1/file2.txt

Now imagine you had the file below, passwords.txt, and you wanted to create an MD5 hash of each of the strings contained in the file.

password
1234
password1

passwords.txt

Again, this is not something that md5sum natively supports, and this time the same same xargs technique also fails:

$ xargs md5sum < passwords.txt

md5sum: password: No such file or directory
md5sum: 1234: No such file or directory
md5sum: password1: No such file or directory

Note that md5sum throws an error because it is expecting a file path as an argument, not a string. The md5sum command can create a hash of a string, but it needs to come into the command via standard in, not as an argument:

$ echo 'hash this' | md5sum

d2e196667eeb24381125d3d4230d8bfb -

You can construct a dynamic command pipeline to address this by exercising additional options available in xargs:

$ xargs -I '{}' bash -c "echo {} | md5sum" < passwords.txt

286755fad04869ca523320acce0dc6a4 -
e7df7cd2ca07f4f1ab415d457a6e1c13 -
10b222970537b97919db36ec757370d2 -

Here we first use the xargs -I option to specify a replace string. This string will be replaced by the xargs command with each line of input from the passwords.txt file.

Next xargs will execute bash -c, which will spawn a new shell and execute whatever commands are given to it. In this case "echo {} | md5sum" will be executed in the new shell, but not before the replace string ( {} ) is replaced by xargs with the input line from the passwords.txt file.

The result is the MD5 hash of each of the words in the passwords.txt file.

One-liner Bash Loop

Another option for accomplishing this same task in an bash loop. First, here is what it would look like as a standard bash script:

#!/bin/bash

while IFS="" read LINE
do
        echo $LINE | md5sum
done

md5.sh

The md5.sh script uses a while loop to reach standard in one line at a time. Each time the loop is executed it stores the current line in the variable LINE. It then uses the echo command to send the line to the standard input of md5sum.

With some carefully placed semicolons you can do this exact same logic as a one-liner rather than from a script file:

$ while IFS="" read LINE; do echo $LINE | md5sum; done < passwords.txt

286755fad04869ca523320acce0dc6a4 -
e7df7cd2ca07f4f1ab415d457a6e1c13 -
10b222970537b97919db36ec757370d2 -

Foreach Script

To simplify the process of building commands from input streams I have created a bash script, fe, that does the work for you. This is similar to foreach constructs seen in many modern programming and scripting languages. Here is the same example as before using the fe script:

$ ./fe 'echo {} | md5sum' < passwords.txt

286755fad04869ca523320acce0dc6a4 -
e7df7cd2ca07f4f1ab415d457a6e1c13 -
10b222970537b97919db36ec757370d2 -

To use the fe script just provide the command pipeline you would like to execute inside of single-quotes. The script will automatically replace any instance of {} with the line read from standard in. Your command pipeline can be as complex as you wish, and you can also use the replace string multiple times. Here is an example that includes multiple pipes, redirection, and multiple replacements:

$ ./fe 'echo {} | md5sum | cut -d" " -f1 > {}.txt' < passwords.txt

You can also use the -n option with fe to display the constructed commands to the screen rather than executing them. This is very useful when constructing commands to ensure you are building them as expected.

$ ./fe -n 'echo {} | md5sum | cut -d" " -f1 > {}.txt' < passwords.txt

echo password | md5sum | cut -d" " -f1 > password.txt
echo 1234 | md5sum | cut -d" " -f1 > 1234.txt
echo password1 | md5sum | cut -d" " -f1 > password1.txt

You can download the fe script from our GitHub repository. To make it even more useful be sure to add fe to your path and make it executable so the script can be accessed from anywhere on your command line.

Conclusion

The ability to construct dynamic command pipelines is an extremely important CLI skill. Mastering it will allow you to create powerful one-liners where mere mortals would need an entire script.