This is the third article in the “CLI tricks” series. This time we are going to cover UNIX-specific pipes, streams and redirections, explaining what they are for and how they can help us in every day console tasks.

1. The Pipes

Pipe is one of the powerful techniques commonly used in Unix systems and inextricably related to console environment. It consists mainly in chaining small, concise commands into bigger, more complex expressions. It boils down to data streaming in which data output from one program are redirected to the input of another program. That is why textual data streams in Unix systems are in common use. And just these ideas were behind Unix’s processing paradigm.

Commands are joined by “|” operator. A typical pattern looks like this:

command1 | command2 | command3

To show the pattern in action I have invented an example in which the output from ls command will be send to the input of wc -l program. -l parameter makes the “wc” program to print on the screen a total number of files and directories found in current directory.

adam@laptop:~/Documents/polishlinux.org/examples$ ls
new.txt  example.txt  all_about_console.txt
adam@laptop:~/Documents/polishlinux.org/examples$ ls | wc -l
3

Pipes are extensively used in search operations applied to system logs. Of course, they make use of regular expressions as well. Pipes are also handy in looking for files (more on this in further parts of the guide).

2. The Streams and Redirections

As I said at the beginnings of this guide, Unix shells make extensive use of data streams: input streams, output streams, and error streams. They can be redirected from a device which supports them to other device or to a file, e.g. instead of error streams being sent to a screen they can be sent to a file. All the streams have attached an unique descriptor or an unique number:

• stdout - standard output (descriptor 1) - mainly monitor’s screen,
• stderr - standard error (descriptor 2) - mainly monitor’s screen,
• stdin - standard input (descriptor 0) - mainly computer’s keyboard.

There exists another operator which is used for redirection. It is “greater than” character associated by descriptor number - descriptor >. If the file chosen as a target does not exist it will be created. Of course, if we’ve been granted permission to write in the given directory. If the file exists it will be overwritten (when having the rights to do so). To avert the “disaster” we should use another descriptor “>>” which appends the redirected data to the end of the file.

Consecutive examples will show in practice what was said above - error stream will be redirected to a file, standard output will be redirected to a file, and standard input will be redirected from keyboard to “from a file” (descriptor <).

adam@laptop:~/Documents/polishlinux.org/examples$ cat \\
non-existent_file.txt 2> error.txt

adam@laptop:~/Documents/polishlinux.org/examples$ cat error.txt
cat: non-existent_file.txt: No such file or directory
adam@laptop:~/Documents/polishlinux.org/examples$ cat > command.txt
ls -l
(Ctrl+D)

adam@laptop:~/Documents/polishlinux.org/examples$ bash < command.txt
total 8
-rw-r--r-- 1 adam adam 55 2007-06-26 14:04 error.txt
-rw-r--r-- 1 adam adam  0 2007-06-10 12:00 new.txt
-rw-r--r-- 1 adam adam  5 2007-06-26 14:07 command.txt
-rw-r--r-- 1 adam adam  0 2007-06-04 00:00 example.txt
-rw-r--r-- 1 adam adam  0 2007-06-05 12:00 all_about_console.txt
adam@laptop:~/Documents/polishlinux.org/examples$

Redirections are frequently used in tasks run in the background, when there is no need to read messages currently. It is worth showing how to get rid of the messages.

adam@laptop:~/Documents/polishlinux.org/examples$ cat \\
non-existent_file.txt 2> /dev/null

Because all Unix devices are represented as files there is a possibility to redirect messages to system’s “trash can”, namely, /dev/null device.

3. xargs

Another command which is inseparable from the ideas of streams and pipes is xargs. It receives text stream and parse it according to set up criteria (null or end of line character), and then sends the separated parts successively as parameters to the next command. xargs is mostly used with following commands: find, locate, and grep.

adam@laptop:~$ ls | grep report | xargs -i cp {} Documents/

A new grep command was applied here but we will describe it later. Thanks to the -i option, all files whose names contain the “report” string will be copied to Documents directory. Brace brackets will be replaced by the stream parts, here by file names, during processing of the expression.

xargs also make frequent use of -0 parameter (or null equivalent). It denotes another parse delimiter. It is useful in case of file names containing space characters.

Attention! The -0 (null) option is available in GNU version of xargs command.

This article is part of the Command line tricks series.
Go back to the previous article: File management in Linux »
Go further to: Regular expressions and search patterns »

Translated by P2O2, Proof-read by trashcat