F4.2 · LESSON 2 OF 6

Streams and pipes

Navigate a project and describe how its interpreter, dependencies and configuration are selected.

THE IDEA

Three streams, not one

Every command line program has three channels. It reads from standard input (stdin, stream 0), writes its results to standard output (stdout, stream 1) and writes errors and warnings to standard error (stderr, stream 2). On screen, stdout and stderr look the same, but they are separate streams you can steer independently.

stdout (1) goes to stderr (2) goes to

$ cat notes.txt missing.txt

On your screen

hello from notes
cat: missing.txt: No such file or directory

Try sending stdout to a pipe while leaving stderr on the screen.

One command, two streams. Choose where each goes and see exactly what ends up where.

The operators

cmd > file       stdout to a file (replaces it)
cmd >> file      stdout appended to a file
cmd 2> file      stderr to a file
cmd 2>&1         stderr to wherever stdout is going
cmd1 | cmd2      stdout of cmd1 becomes stdin of cmd2

Pipelines: small tools in a row

A pipe feeds one command's stdout straight into the next command's stdin, so you can build a result from small steps. Here is a log with 12 lines. Each stage below shows what it passes on, so you can see the data shrink from 12 lines to 5 and then to a single number.

cat app.log

12 lines out

2024-05-01 09:00:01 INFO server started
2024-05-01 09:00:05 INFO user login ok
2024-05-01 09:01:12 ERROR disk full
2024-05-01 09:02:30 INFO request /home
2024-05-01 09:03:44 ERROR disk full
2024-05-01 09:04:02 WARN slow query
… 6 more

grep ERRORreceives the previous stage's stdout

5 lines out

2024-05-01 09:01:12 ERROR disk full
2024-05-01 09:03:44 ERROR disk full
2024-05-01 09:06:09 ERROR auth failed
2024-05-01 09:08:41 ERROR disk full
2024-05-01 09:09:00 ERROR timeout

wc -lreceives the previous stage's stdout
1 line out
```
5
```

cat app.log | grep ERROR | wc -l, one stage at a time.

You are not limited to counting. Add steps to answer a different question, such as which error is most common:

grep ERROR app.log

5 lines out

2024-05-01 09:01:12 ERROR disk full
2024-05-01 09:03:44 ERROR disk full
2024-05-01 09:06:09 ERROR auth failed
2024-05-01 09:08:41 ERROR disk full
… 1 more

cut -d' ' -f3-receives the previous stage's stdout

5 lines out

ERROR disk full
ERROR disk full
ERROR auth failed
ERROR disk full
… 1 more

sortreceives the previous stage's stdout

5 lines out

ERROR auth failed
ERROR disk full
ERROR disk full
ERROR disk full
… 1 more

uniq -creceives the previous stage's stdout

3 lines out

      1 ERROR auth failed
      3 ERROR disk full
      1 ERROR timeout

sort -rnreceives the previous stage's stdout

3 lines out

      3 ERROR disk full
      1 ERROR timeout
      1 ERROR auth failed

Cut off the timestamp, group identical lines, then sort by count.

Check yourself first

Predict first. cat missing.log | wc -l: the file does not exist. What happens?

Chapter context

A terminal runs commands in a current working directory. An environment makes the program's interpreter, dependencies and configuration explicit so a successful run can be repeated elsewhere.

TRY IT YOURSELF

You are in ~/logs. The panel on the right previews every stage of whatever pipeline you are typing, before you press Enter.

Count the ERROR lines using a pipe (one command, answer 5)
Save just the ERROR lines into errors.txt with redirection
Add a final line --- end --- to errors.txt without erasing what is there
Rank the distinct error messages by how often they occur, most frequent first
Send an error message to a file instead of the screen
Prove a pipe carries only stdout: pipe a failing command into wc -l and notice what still reaches your screen

learner@laptop

Type a command and press Enter. Tab completes names, ↑ recalls history.

learner@laptop:~/logs$

Live pipeline preview

Type a pipeline like cat app.log | grep ERROR | wc -l to see each stage.

Stuck? Try one of these

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does a normal pipe automatically include stderr?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

A reproducible run can be described as (source revision, interpreter, dependency resolution, input data, configuration). Equal source code alone is insufficient. For pipelines, compare the records emitted at each stage rather than assuming every command preserves row counts.

Worked example

A project processes 12 lines, filters to 5 matching lines, then counts them. The final count is 5, not 12. In a path puzzle, joining /work/project with data and train.csv identifies the input without changing the real filesystem.

Reference code

The following example uses the Python standard library and does not call a remote service.

from pathlib import PurePosixPath
path = PurePosixPath('/work/project') / 'data' / 'train.csv'
assert str(path) == '/work/project/data/train.csv'
lines = ['keep' if i < 5 else 'drop' for i in range(12)]
assert sum(line == 'keep' for line in lines) == 5

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

A virtual environment is not a security sandbox. A lockfile does not preserve external input datasets. Ignoring a secret after committing it does not undo exposure.