Input/Output Streams and Redirection

Up until now, all of the commands that we’ve used have accepted input directly from you via command-line arguments. While this is sufficient for many tasks, it does greatly limit the amount of larger-scale data processing that you can do. If we want to perform some more sophisticated data processing, and really unlock the power of the command line interface, we’ll need to begin to work with more sophisticated forms of input and output.

The sort command

As an example of a command that is most useful when using files for input and output, we will consider sort, which we’ve already seen before, briefly, in the first chapter of this tutorial.

The simplest use of the sort command is without any arguments. At first glance, this will not appear to do anything.

% sort

In reality, sort is currently waiting for some data to process. When running sort in this way, the input is expected to come from the keyboard. So you’ll need to manually type some text into the terminal. Like most filters, sort is line oriented, so enter a few lines of text. Then, signal that you are done with ^Z. sort will then sort your input and report the results back to you.

% sort
hello
goodbye
test
and more
^Z

and more
goodbye
hello
test

Of course, this particular example of sort is not very useful. For one thing, it requires you to manually type the data that you want sorted. For another, it reports the result of sorting directly on the command line. It would be better if you could read the data automatically from a file, and then output the results directly to a file. That’s what we’ll do next.

Standard Input and Standard Output

Every process that is launched on a computer is launched with three files automatically open: standard input (stdin), standard output (stdout), and standard error (stderr). By default, standard input is connected to the input buffer of your terminal, and standard output and standard error are connected to the output buffer. That is why, in the sort example above, it accepts input from whatever you type, and reports the results of the sorting directly on the terminal.

The shell will allow you to change this default mapping using input and output redirection. If you’d like for a command’s standard input to come from a file, rather than the terminal, you want to use the input redirection operator: <. To make the output go to a file instead of the terminal, you can use the output redirection operator: >.

Input Redirection

For example, let’s create a file using notepad and then use sort to sort it. First, create a file called tosort.txt in Notepad with the following contents,

b
c
d
g
k
l
q
u
a
l
A
1
2
ad
L

To demonstrate input redirection first, run sort like so,

% sort < tosort.txt
1
2
a
A
ad
b
c
d
g
k
l
l
L
q
u

You should see the sorted contents of the file appear on the terminal after your call to sort.

Notice that the command no longer paused for input. This is because the redirection of standard input caused it to read from the file, instead of the terminal input buffer. Every file contains an implicit EOF (^Z) at the end of it, so the command read each line from the file, encountered the EOF, and ran the sort–all without any explicit user-interaction required. You can imagine how useful this is for process very large files.

% sort tosort.txt

Output Redirection

The above command displayed the results of sorting directly on the terminal. But if our file is sufficiently large, it might be more useful to create a new file containing the sorted results. We could copy and paste from the terminal into a Notepad window, but that is overly manual (not to mention, some very large files may completely fill the terminal’s output buffer and be cut off).

We can tell the shell to redirect the command’s output to a file like so,

% sort < tosort.txt > sorted.txt

Note that we have specified both an input and an output redirect in the same command. This is perfectly valid.

The command will no longer produce output on the terminal, as we’ve redirected its output from the output buffer to a file. However, we can easily see the results of our sort by using type,

% type sorted.txt
1
2
a
A
ad
b
c
d
g
k
l
l
L
q
u

Error Redirection

The third standard file that is opened is called standard error. Despite the name, it is used for more than error messages. A lot of information messages are written to standard error, as are prompts written by commands (as opposed to the shell). Standard error is used so that messages from commands that are using output redirection remain visible on the terminal, or to allow you to write errors to a different location than output. For example, should sort report an error for some reason, you don’t necessarily want that error message being written to the same file as the sorted data.

As an example of a carguments, will write the name of each file to stderr.ommand that writes data to standard error, the type command, when provided multiple files as arguments, will write the name of each file to stderr. For example,

% type t1.txt t2.txt t3.txt

t1.txt


this is file 1
with two lines

t2.txt


and this is file2

t3.txt


Any, finally, file
3 is here.

We can verify that these headers are in fact going to stderr by redirecting stdout to a file.

% type t1.txt t2.txt t3.txt > merged.txt

t1.txt



t2.txt



t3.txt

This allows us to see file headers when using type interactively, while also allowing us to use type to merge files together, without introducing any extra lines into the merged file.

If we wanted to redirect the error messages, leaving the file contents going to the terminal, we can do that with the standard error redirection operator, 2>, like so,

% type t1.txt t2.txt t3.txt 2> headers.txt
this is file 1
with two lines
and this is file2
Any, finally, file
3 is here.

% type headers.txt

t1.txt



t2.txt



t3.txt

We can also send both stdout and stderr to the same file if you want. To do this, first redirect stdout, and then follow up with 2>&1, which will tell the shell to send stderr to the same spot as stdout,

% type t1.txt t2.txt t3.txt > headers.txt 2>&1

Note that you must put the output redirect first, and then the error redirect, or it will not work properly.

Conclusion

In this chapter, we discussed using input and output redirection to make a command read input from a file, or write output to a file (or both). In the next chapter, we’ll follow up on this by discussing more commands which work well with input and output redirection, as well as a related concept called a pipeline, which allows us to link several such commands together.