Pipes, Search & Text

The Unix philosophy — small tools, composed

T1 introduced the pipe character | as a way to chain commands. T2 is where pipes go from a trick you've heard of to a tool you reach for automatically.

Unix was designed around one idea: each program does exactly one thing, does it well, reads from standard input, and writes to standard output. No program knows or cares what comes before or after it in a pipeline. That narrow contract is what makes composition possible. A tool written in 1975 pipes into a tool written last year because they both speak the same language: text.

When you write cmd1 | cmd2 | cmd3, the shell wires the output of cmd1 directly into the input of cmd2, and cmd2's output into cmd3. No temp files. The commands run concurrently — as cmd1 produces output, cmd2 is already consuming it. This is the engine that makes one-liners like "top 10 most frequent words in a file" a 30-second job.

Before the tools: the three standard streams every process has from birth.

• stdin (0) — where a process reads input from. By default, your keyboard.
• stdout (1) — where a process writes normal output. By default, your terminal.
• stderr (2) — where a process writes errors and diagnostics. Also your terminal by default, but a separate stream from stdout — so you can redirect them independently.

You'll meet those stream numbers again when you redirect stderr (2>) later in this module.

grep — search inside files

grep searches for a pattern in files — or in piped input — and prints every line that matches. It's one of the most-reached-for tools in the Unix toolkit.

$ grep "error" app.log              # lines containing "error"
$ grep -i "error" app.log           # -i: case-insensitive (Error, ERROR, error)
$ grep -r "TODO" src/               # -r: recursive — search a whole directory tree
$ grep -n "error" app.log           # -n: show line numbers with each match
$ grep -v "DEBUG" app.log           # -v: invert — lines that do NOT match
$ grep -c "error" app.log           # -c: count matching lines (just the number)
$ grep -E "error|warn" app.log      # -E: extended regex — match "error" OR "warn"
$ grep -F "1.2.3" version.txt       # -F: fixed string — no regex, literal match

Flags combine freely: grep -rn "TODO" src/ searches recursively and shows line numbers. grep -iv "debug" app.log case-insensitively excludes debug lines.

Regex, just enough

Regular expressions are a mini-language for describing patterns. You don't need to be an expert — you need enough to read and write basic patterns without being surprised.

.       # any single character (except newline)
*       # zero or more of the preceding (e.g. "ab*" matches "a", "ab", "abb", ...)
^       # anchor to start of line (e.g. "^Error" only matches at line start)
$       # anchor to end of line (e.g. "\.log$" — line ends with ".log")
[abc]   # character class — matches one of a, b, or c
[0-9]   # range — matches any single digit
\.      # escaped dot — literal period (unescaped, . means "any character")

In practice for T2, you mainly need -E for "either/or" patterns (grep -E "error|warn") and -F when you want literal strings. Deeper regex is covered in context when you need it — don't memorise; recognise and look up.

find — locate files

find walks a directory tree and prints files that match criteria you specify. It's how you answer "where is the config file I changed last week?" or "how many log files are in this tree?"

$ find . -name "*.txt"              # files named *.txt from current dir
$ find ~/projects -name "*.py"      # Python files under ~/projects
$ find . -type f                    # files only (not directories)
$ find . -type d                    # directories only
$ find . -size +1M                  # files larger than 1 megabyte
$ find . -mtime -1                  # modified in the last 24 hours
$ find . -name "*.log" -exec rm {} \;   # delete every .log file found

The -exec flag runs a command on each matched file: {} is a placeholder for the matched path, and \; terminates the expression. Before running -exec rm, always dry-run with -print first to see exactly what you'd be deleting: find . -name "*.tmp" -print.

wc — counting

wc (word count) counts lines, words, and bytes in its input. You'll reach for it constantly to sanity-check what's in a file or pipeline.

$ wc -l app.log                     # count lines
$ wc -w essay.txt                   # count words
$ wc -c data.bin                    # count bytes
$ cat *.log | wc -l                 # total lines across all log files
$ grep "error" app.log | wc -l      # count lines that matched

Note the distinction between grep -c "error" app.log and grep "error" app.log | wc -l — they give the same number for a single file, but -c works per file only while the pipe form works on anything, including multi-step pipelines.

sort and uniq — order and deduplicate

sort sorts lines of text. By default it sorts alphabetically; add flags for other orderings:

$ sort names.txt                    # alphabetical ascending
$ sort -r names.txt                 # reverse alphabetical
$ sort -n sizes.txt                 # numeric sort ("10" after "9", not before)
$ sort -u words.txt                 # sort and remove duplicates in one step

uniq removes adjacent duplicate lines. Paired with sort, it handles deduplication across an entire file:

$ sort words.txt | uniq             # deduplicate (sort first, collapse adjacent dupes)
$ sort words.txt | uniq -c          # prefix each unique line with its count
$ sort words.txt | uniq -d          # show only lines that appear more than once

The canonical idiom — count how often each value appears, most frequent first:

$ sort access.log | uniq -c | sort -rn

Read it left to right: sort the log (identical lines become adjacent) → count adjacent duplicates with uniq -c (prepends each unique line with its count) → sort numerically in reverse (largest count first). Result: a frequency table of every unique line, ranked by occurrence. You'll use this exact pipeline constantly.

cut — slice columns

cut extracts specific fields from each line. Essential for structured text like CSVs and tab-separated files:

$ cut -d',' -f1 data.csv            # column 1 from comma-separated file
$ cut -d',' -f1,3 data.csv          # columns 1 and 3
$ cut -f2 report.tsv                # column 2 from tab-delimited file (tab is the default)
$ cut -d':' -f1 /etc/passwd         # usernames (colon-delimited)

-d sets the delimiter character; -f picks the field number(s). For anything more complex — quoted CSV fields, computed columns, restructured output — reach for awk, which you'll meet in T4.

Streams and redirection, deeper

T1 covered > (write to file) and >> (append). Now the rest — specifically stderr, which trips everyone up eventually.

$ cmd > output.txt              # redirect stdout to a file
$ cmd 2> errors.txt             # redirect stderr to a file (stdout still to terminal)
$ cmd > output.txt 2>&1        # stdout to file, then merge stderr into stdout
$ cmd > output.txt 2>/dev/null  # stdout to file, discard stderr
$ cmd 2>/dev/null               # silence all errors, stdout to terminal
$ cmd >/dev/null 2>&1          # discard everything (fully silent)

The numbers are file descriptors: 1 = stdout, 2 = stderr. 2>&1 means "redirect fd 2 to wherever fd 1 currently points." Order matters: write the stdout redirect first, then 2>&1 — reversing them doesn't work the way you'd expect.

/dev/null is the kernel's black hole — anything written there is discarded immediately, no disk I/O. Reading from it gives you an empty stream. You'll most often use it to silence noisy commands you're running for their side effects.

Putting it together — real pipelines

This is where the tools pay off. Three worked examples, each built left-to-right so you can see how a pipeline grows.

Example 1 — top 10 most frequent words in a file

$ cat essay.txt | tr -s ' ' '\n' | sort | uniq -c | sort -rn | head -10

Left to right: print the file → translate spaces to newlines (one word per line, tr -s squeezes multiple spaces) → sort alphabetically (identical words become adjacent) → count adjacent duplicates → sort by count largest-first → keep only the top 10. Six tools; each does one thing.

Example 2 — count ERROR lines in a log

$ grep -i "error" app.log | wc -l

Simple two-step. To see the error lines and the count, use tee to split the stream: grep -i "error" app.log | tee /dev/stderr | wc -l — lines print to the terminal via stderr, count appears at the end via stdout.

Example 3 — 5 largest files in a directory tree

$ find . -type f -exec du -sk {} \; 2>/dev/null | sort -rn | head -5

Find all files, run du -sk on each (prints size in kilobytes + path), silence permission errors with 2>/dev/null, sort numerically largest-first, show the top 5. The -k flag on du gives kilobytes — portable across macOS and Linux.

# 1
find . -name *.txt

# 2
uniq access.log

# 3
grep "3.14" pi.txt

# 4
for f in $(ls *.txt); do
  wc -l "$f"
done

# 5
grep -P "\d+" file.txt

# 6
find . -name "*.tmp" -exec rm {} \;