Unix philosophy
Doug McIlroy's core belief: Write programs that do one thing and do it well. Write programs to work together. The power is in composition, not in features.
"This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface."
Small tools, loosely joined. The opposite of monolithic design.
McIlroy articulated the Unix philosophy in the Bell System Technical Journal (1978):
Each program should do one thing. Not two. Not "one thing plus some related things." One thing.
Not this:
# A tool that does everything
supertool --compress --encrypt --upload --notify --cleanup file.txt
This:
# Compose single-purpose tools
gzip file.txt | gpg -e | aws s3 cp - s3://bucket/file.txt.gz.gpg && notify "done"
Programs should be designed to connect. Output of one becomes input of another.
Requirements for composition:
"Write programs to handle text streams, because that is a universal interface."
Text is:
Not this:
# Binary format requiring special tools
myprogram --output-binary data.bin
special-reader data.bin
This:
# Text that any tool can process
myprogram | grep "ERROR" | wc -l
McIlroy's famous response to Knuth's "literate programming" solution shows the philosophy in action.
The problem: Read a file, find the N most common words, print them in decreasing frequency.
Knuth's solution: 10+ pages of literate Pascal, a beautiful essay with a custom data structure.
McIlroy's solution:
tr -cs A-Za-z '\n' |
tr A-Z a-z |
sort |
uniq -c |
sort -rn |
sed ${1}q
Six lines. Uses existing tools. Immediately understandable to any Unix user.
"Knuth has shown us here how to program intelligently, but I claim that his exercise is not software engineering."
Engineering is not just writing correct code. It's solving problems with appropriate tools. Sometimes that means writing nothing new at all.
"Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new features."
Not this:
class DataProcessor:
def load(self): ...
def validate(self): ...
def transform(self): ...
def filter(self): ...
def aggregate(self): ...
def format(self): ...
def output(self): ...
# 2000 lines, handles everything
This:
# Separate programs, composed
load data.csv | validate | transform | filter | aggregate | format > output.json
"Rule of Silence: When a program has nothing surprising to say, it should say nothing."
Not this:
$ cp file.txt backup/
Copying file.txt to backup/...
Checking permissions...
Creating backup directory if needed...
Copy complete!
Success!
This:
$ cp file.txt backup/
$
No news is good news. Only speak on error or when asked (-v).
Exit immediately on error. Don't try to continue. Return non-zero exit codes.
#!/bin/bash
set -e # Exit on error
set -u # Error on undefined variables
set -o pipefail # Fail if any pipeline command fails
# Now failures are obvious, not hidden
"Rule of Optimization: Prototype before polishing. Get it working before you optimize it."
Build the pipeline first. Optimize only the slow parts.
The ideal Unix tool is a filter: reads stdin, writes stdout, configurable via arguments.
# Good: Works as filter
cat data.txt | mytool --option | head
# Good: Also works with file arguments
mytool --option data.txt | head
# Bad: Requires interactive prompts
mytool
> Enter filename:
> Enter option:
Follow existing conventions:
-h / --help for help-v / --verbose for verbosity-q / --quiet for silence-o / --output for output file- for stdin/stdout-- to end option parsingWork without configuration. Sane defaults for everything.
# Should just work
mytool file.txt
# Options are optional
mytool --format=json --limit=10 file.txt
Before committing a tool or interface, ask:
Apply these checks:
cat data | step1 | step2 | step3 > result
cat data | tee >(step1 > result1) >(step2 > result2) > /dev/null
step1 data && step2 || handle_error
cat files.txt | xargs -P4 -I{} process {}
Use a different skill when:
distributed (statelessness, idempotency, failure handling)optimization (profiling, cache behavior, data-oriented)data-first (kernel style, data structures)simplicity (Go Proverbs, interfaces)clarity (readability, naming)McIlroy is the CLI/Unix tool skill—use it when building filters, utilities, and pipeable commands.
"The notion of 'intricate and beautiful complexities' is almost an oxymoron. Unix programmers vie with each other for 'simple and beautiful.'" — Doug McIlroy