Input/Output Streams and Redirection
Up until now, all of the commands that we’ve used have accepted input directly from you via command-line arguments. While this is sufficient for many tasks, it does greatly limit the amount of larger-scale data processing that you can do. If we want to perform some more sophisticated data processing, and really unlock the power of the command line interface, we’ll need to begin to work with more sophisticated forms of input and output.
The sort command
As an example of a command that is most useful when using files for input and
output, we will consider
sort, which we’ve already seen before, briefly, in
the first chapter of this tutorial.
The simplest use of the
sort command is without any arguments. At first
glance, this will not appear to do anything.
sort is currently waiting for some data to process. When running
sort in this way, the input is expected to come from the keyboard. So you’ll
need to manually type some text into the terminal. Like most filters,
line oriented, so enter a few lines of text. Then, signal that you are done
sort will then sort your input and report the results back to you.
% sort hello goodbye test and more ^Z
and more goodbye hello test
Of course, this particular example of
sort is not very useful. For one thing,
it requires you to manually type the data that you want sorted. For another, it
reports the result of sorting directly on the command line. It would be better
if you could read the data automatically from a file, and then output the
results directly to a file. That’s what we’ll do next.
Standard Input and Standard Output
Every process that is launched on a computer is launched with three files
automatically open: standard input (stdin), standard output (stdout), and
standard error (stderr). By default, standard input is connected to the input
buffer of your terminal, and standard output and standard error are connected
to the output buffer. That is why, in the
sort example above, it accepts
input from whatever you type, and reports the results of the sorting directly
on the terminal.
The shell will allow you to change this default mapping using input and output
redirection. If you’d like for a command’s standard input to come from a file,
rather than the terminal, you want to use the input redirection operator:
To make the output go to a file instead of the terminal, you can use the output
For example, let’s create a file using notepad and then use
sort it. First, create a file called
tosort.txt in Notepad with the
b c d g k l q u a l A 1 2 ad L
To demonstrate input redirection first, run
sort like so,
% sort < tosort.txt 1 2 a A ad b c d g k l l L q u
You should see the sorted contents of the file appear on the terminal after your call to sort.
Notice that the command no longer paused for input. This is because the redirection of standard input caused it to read from the file, instead of the terminal input buffer. Every file contains an implicit EOF (^Z) at the end of it, so the command read each line from the file, encountered the EOF, and ran the sort–all without any explicit user-interaction required. You can imagine how useful this is for process very large files.
Many commands which process files do not actually require input redirection of this form, and will actually allow you to specify a filename directly as an argument instead. For example, you could’ve executed the above command with an argument, rather than a redirection, like so,
% sort tosort.txt
and gotten the same result. Input redirection is probably the least commonly used form of redirection for this reason. But it’s still worth being aware of it.
The above command displayed the results of sorting directly on the terminal. But if our file is sufficiently large, it might be more useful to create a new file containing the sorted results. We could copy and paste from the terminal into a Notepad window, but that is overly manual (not to mention, some very large files may completely fill the terminal’s output buffer and be cut off).
We can tell the shell to redirect the command’s output to a file like so,
% sort < tosort.txt > sorted.txt
Note that we have specified both an input and an output redirect in the same command. This is perfectly valid.
The command will no longer produce output on the terminal, as we’ve redirected
its output from the output buffer to a file. However, we can easily see the
results of our sort by using
% type sorted.txt 1 2 a A ad b c d g k l l L q u
The third standard file that is opened is called standard error. Despite the name, it is used for more than error messages. A lot of information messages are written to standard error, as are prompts written by commands (as opposed to the shell). Standard error is used so that messages from commands that are using output redirection remain visible on the terminal, or to allow you to write errors to a different location than output. For example, should sort report an error for some reason, you don’t necessarily want that error message being written to the same file as the sorted data.
As an example of a carguments, will write the name of each file to stderr.ommand that writes data to standard error, the
type command, when provided multiple files as arguments, will write the name of each file to stderr. For
% type t1.txt t2.txt t3.txt t1.txt this is file 1 with two lines t2.txt and this is file2 t3.txt Any, finally, file 3 is here.
We can verify that these headers are in fact going to stderr by redirecting stdout to a file.
% type t1.txt t2.txt t3.txt > merged.txt t1.txt t2.txt t3.txt
This allows us to see file headers when using
type interactively, while also
allowing us to use
type to merge files together, without introducing any
extra lines into the merged file.
If we wanted to redirect the error messages, leaving the file contents going
to the terminal, we can do that with the standard error redirection operator,
2>, like so,
% type t1.txt t2.txt t3.txt 2> headers.txt this is file 1 with two lines and this is file2 Any, finally, file 3 is here. % type headers.txt t1.txt t2.txt t3.txt
We can also send both stdout and stderr to the same file if you want. To do
this, first redirect stdout, and then follow up with
2>&1, which will tell
the shell to send stderr to the same spot as stdout,
% type t1.txt t2.txt t3.txt > headers.txt 2>&1
Note that you must put the output redirect first, and then the error redirect, or it will not work properly.
In this chapter, we discussed using input and output redirection to make a command read input from a file, or write output to a file (or both). In the next chapter, we’ll follow up on this by discussing more commands which work well with input and output redirection, as well as a related concept called a pipeline, which allows us to link several such commands together.