Day 3 — Iterators & Tests

Day 2 left logwise parsing honestly and filtering correctly, but it still couldn’t summarize — count per level, total the matches, and shrug off the occasional bad line in a good file. That’s an aggregation problem, and aggregation in Rust means iterators: lazy, composable, zero-cost pipelines built from filter, map, and fold. Today we write the analyze step with them, then do the thing that turns a script into software — prove it works with unit and integration tests — and finally ship a real installed binary.

Iterators: a pipeline, not a loop

An iterator is any type that can produce a next() value until it’s empty. The standard library hangs dozens of adapters off that one idea, and they read top-to-bottom as a data pipeline:

   lines ─▶ filter(non-blank) ─▶ map(parse) ─▶ keep Ok ─▶ fold(into counts)
            └── keep some ──┘    └ transform ┘  └ select ┘  └── reduce to one value ┘

Each adapter takes a closure — an inline function that can capture variables from its surroundings. |line| line.level >= Level::Warn is a closure; so is the accumulator we’ll fold with. The crucial property: adapters are lazy. filter and map do nothing until a consumer (fold, collect, count, a for loop) pulls values through. That laziness is what lets a long chain run in a single pass with no intermediate collections.

The analyze step, for real

Here is the heart of src/analyze.rs. It takes any iterator of &str lines, parses them, and returns a Report (a Summary plus the matching lines):

   pub fn analyze<'a>(
       lines: impl Iterator<Item = &'a str>,
       filter: &Filter,
       strict: bool,
   ) -> Result<Report, LogError> {
       let mut parsed: Vec<LogLine> = Vec::new();
       let mut total_lines = 0usize;
       let mut malformed = 0usize;

       for (idx, raw) in lines.enumerate() {
           if raw.trim().is_empty() {
               continue;                       // blank lines count toward nothing
           }
           total_lines += 1;

           match parse_line(raw) {
               Ok(line) => parsed.push(line),
               Err(source) => {
                   if strict {
                       return Err(LogError::Parse { line_no: idx + 1, source });
                   }
                   malformed += 1;             // lenient mode: tally and move on
               }
           }
       }

       // Per-level histogram, built by FOLDING an accumulator array over the lines.
       let counts = parsed
           .iter()
           .fold([0usize; Level::COUNT], |mut acc, line| {
               acc[line.level.index()] += 1;
               acc
           });

       // The lines the user asked for: FILTER, then clone the keepers into a Vec.
       let matches: Vec<LogLine> = parsed
           .iter()
           .filter(|line| filter.matches(line))
           .cloned()
           .collect();

       let summary = Summary {
           total_lines,
           parsed: parsed.len(),
           malformed,
           matched: matches.len(),
           counts,
       };
       Ok(Report { summary, matches })
   }

Three iterator ideas earn their keep here:

• fold reduces a whole sequence to one value by threading an accumulator through it. We seed it with a fresh [0; 5] array and, for each line, bump the slot for its level — line.level.index() is the Trace→0 … Error→4 mapping, a small helper on Day 1’s Level enum in parser.rs. One pass, one array, the histogram falls out.
• filter keeps the lines where the closure returns true. The closure |line| filter.matches(line) captures the outer filter by reference — that’s a closure doing what a plain function can’t.
• cloned().collect() turns the borrowed &LogLines the filter yielded into owned LogLines in a fresh Vec. collect is the consumer that finally drives the lazy chain.

Why the for loop for parsing but iterators for counting? Because the parse step has an early return (--strict bails on the first bad line), and that’s clearer as a loop than as an iterator gymnastics. Use the tool that reads best; iterators are an option, not an obligation.

The Summary, and printing it

Summary is a plain data struct, and a Display impl renders the table — including the aligned histogram you saw in the overview, courtesy of f.pad:

   impl fmt::Display for Summary {
       fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
           writeln!(f, "{:<11} : {}", "total lines", self.total_lines)?;
           writeln!(f, "{:<11} : {}", "parsed", self.parsed)?;
           if self.malformed > 0 {
               writeln!(f, "{:<11} : {}", "malformed", self.malformed)?;
           }
           writeln!(f, "{:<11} : {}", "matched", self.matched)?;
           writeln!(f, "by level:")?;
           for level in Level::ALL {
               writeln!(f, "  {:<5} : {}", level, self.counts[level.index()])?;
           }
           Ok(())
       }
   }

Level::ALL is the five-element array defined on Level in parser.rs; iterating it keeps the output in severity order. Run the finished tool:

   $ cargo run -- samples/app.log
   total lines : 16
   parsed      : 15
   malformed   : 1
   matched     : 15
   by level:
     TRACE : 1
     DEBUG : 2
     INFO  : 6
     WARN  : 3
     ERROR : 3

By the numbers — iterators that allocate nothing

The histogram fold writes into one fixed [usize; 5] array — 5 × 8 = 40 bytes on the stack, zero heap allocations, for a file of any size. Contrast the “obvious” eager style, where each stage materializes a Vec: for a 1,000,000-line file, a parse→filter→map pipeline built with intermediate collections would allocate roughly three temporary Vecs of ~1,000,000 elements — tens of megabytes copied and freed for nothing. The lazy chain fuses into a single pass: Σ allocations = 0 for the fold, and exactly one for the collect that keeps the matches. Same readability as the eager version, a fraction of the memory traffic.

Under the hood — why iterators are “zero-cost”

map(...).filter(...).fold(...) looks like it builds three layers of objects. It doesn’t. Each adapter is a tiny struct that lazily wraps the previous one and implements Iterator::next by calling inward. Because logwise is generic over impl Iterator, the compiler monomorphizes the chain — it generates one concrete specialization with every next call inlined — and the optimizer flattens the whole tower into the same machine code you’d write by hand as a for loop. That’s the meaning of zero-cost abstraction: the abstraction is free at runtime; you pay only at compile time. It’s also why a --release build matters — the optimizer is what collapses the layers:

   cargo build --release          # optimized; collapses the iterator chain, omits debug overflow checks
   ./target/release/logwise samples/app.log

Prove it works: tests

A CLI you can’t test is a CLI you’re afraid to change. Rust builds testing into the toolchain — no framework to install. A #[cfg(test)] module compiles only under cargo test, so tests ship in the same file as the code they cover, with access to its private items. From src/analyze.rs:

   #[cfg(test)]
   mod tests {
       use super::*;

       const SAMPLE: &str = "\
   2026-06-26T09:00:01Z INFO server starting up
   2026-06-26T09:00:02Z DEBUG loaded config
   2026-06-26T09:00:05Z WARN cache miss rate high
   2026-06-26T09:00:12Z ERROR failed to open blob

   this line is not valid
   2026-06-26T09:00:25Z INFO request GET /health 200";

       #[test]
       fn counts_levels_and_skips_blanks_and_malformed() {
           let report = analyze(SAMPLE.lines(), &Filter::default(), false).unwrap();
           let s = &report.summary;
           assert_eq!(s.total_lines, 6);      // 7 lines minus 1 blank
           assert_eq!(s.parsed, 5);
           assert_eq!(s.malformed, 1);
           assert_eq!(s.counts[Level::Error.index()], 1);
       }

       #[test]
       fn level_filter_keeps_that_level_and_above() {
           let filter = Filter { min_level: Some(Level::Warn), ..Default::default() };
           let report = analyze(SAMPLE.lines(), &filter, false).unwrap();
           assert_eq!(report.summary.matched, 2);   // WARN + ERROR
           assert!(report.matches.iter().all(|l| l.level >= Level::Warn));
       }

       #[test]
       fn strict_mode_fails_on_the_first_bad_line() {
           let err = analyze(SAMPLE.lines(), &Filter::default(), true).unwrap_err();
           assert!(matches!(err, LogError::Parse { .. }));
       }
   }

assert_eq! checks values; assert! checks a bool; matches! checks that a value fits a pattern without binding it. Each #[test] fn runs in isolation, and a panic (a failed assert) marks it failed. The parser, cli, and error modules carry their own #[cfg(test)] blocks the same way — unit tests live next to the unit.

Integration tests: run the real binary

Unit tests check functions; an integration test checks the program a user actually runs. Files in tests/ are compiled as separate crates that link your binary. Cargo even hands you the binary’s path in the CARGO_BIN_EXE_logwise env var, so tests/cli.rs launches it as a subprocess:

   use std::process::Command;

   fn run(args: &[&str]) -> (bool, String, String) {
       let out = Command::new(env!("CARGO_BIN_EXE_logwise"))
           .args(args)
           .output()
           .expect("failed to launch logwise");
       (out.status.success(),
        String::from_utf8_lossy(&out.stdout).into_owned(),
        String::from_utf8_lossy(&out.stderr).into_owned())
   }

   #[test]
   fn summarizes_the_sample_log() {
       let log = concat!(env!("CARGO_MANIFEST_DIR"), "/samples/app.log");
       let (ok, stdout, _) = run(&[log]);
       assert!(ok);
       assert!(stdout.contains("parsed      : 15"));
       assert!(stdout.contains("ERROR : 3"));
   }

   #[test]
   fn missing_file_reports_an_error() {
       let (ok, _, stderr) = run(&["does-not-exist.log"]);
       assert!(!ok);                                    // non-zero exit
       assert!(stderr.contains("could not read log file"));
   }

This exercises the whole stack — clap, file I/O, the error chain, the exit code — exactly as the shell sees it. Now run everything:

   $ cargo test
   running 21 tests
   test result: ok. 21 passed; 0 failed; 0 ignored; ...
   running 6 tests
   test result: ok. 6 passed; 0 failed; 0 ignored; ...

Twenty-one unit tests across the modules, six integration tests driving the binary — green.

It actually happened — reused code that nobody re-tested

On 4 June 1996, the maiden flight of the Ariane 5 rocket veered off course and self-destructed about 40 seconds after launch, destroying a payload worth on the order of hundreds of millions of dollars. The trigger was an inertial-guidance routine reused unchanged from Ariane 4: it converted a 64-bit floating-point velocity into a 16-bit integer, and Ariane 5’s faster flight produced a value that overflowed the 16-bit slot. The deeper failure was process: the reused module was never re-tested against Ariane 5’s flight profile. Rust can’t write your test matrix, but it leans against both halves of this story — in debug builds an integer overflow panics instead of silently wrapping, narrowing as conversions are explicit and visible, and cargo test makes re-testing reused code the cheap default. The lesson logwise practices: when behavior matters, encode it in a test that runs every build.

Ship it

A release build plus one command installs logwise onto your PATH:

   cargo install --path .       # builds --release and copies the binary to ~/.cargo/bin
   logwise samples/app.log --level error --verbose

From here it’s a tool, not a tutorial. Natural next features — a --json output mode, real regex patterns (add the regex crate), CSV input (add csv) — are all small additions to the same shape: parse into types, aggregate with iterators, prove it with tests.

What you built, and what the compiler protected you from

The thread one last time: what does building this force you to understand — and what is Rust protecting you from? Aggregating a file forced you to reach for iterators and closures — and the payoff was a pipeline that’s both readable and allocation-free, because the compiler fuses lazy adapters into one optimized loop. Testing forced you to encode behavior as checks that run every build, the exact discipline whose absence sank Ariane 5. Across three days the same answer kept returning: the compiler enforces one-owner and shared-XOR-mutable so you can’t leak, race, or silently ignore a failure — and building logwise made each of those abstract guarantees something you felt.

You’ve shipped a real Rust binary from a blank cargo new. The next project keeps the iterators and adds the thing those ownership rules were really built for — fearless concurrency: Project 2 · kvlite, a key-value store →.

Check your understanding

1. Iterator adapters like filter and map are described as “lazy”. What does that mean, and which kinds of call actually drive the work?
2. In the histogram fold, what is the accumulator, what does the closure do on each line, and how many heap allocations does building the whole [usize; 5] take?
3. logwise parses lines with a for loop but counts and filters with iterators. Give the concrete reason the parse step resists a pure-iterator rewrite.
4. What’s the difference between a #[cfg(test)] unit-test module and a file in tests/? What does each one get to see and exercise?
5. “Zero-cost abstraction” — what does the compiler do to a filter(...).fold(...) chain so it costs no more at runtime than a hand-written loop, and why does --release matter for that?

Show answers

1. Lazy means filter/map build a small wrapper and do no work until something pulls values through. The work is driven by a consumer: fold, collect, count, sum, or a for loop. No consumer, no computation (and no intermediate collections).
2. The accumulator is a [0usize; 5] array seeded into fold; for each line the closure does acc[line.level.index()] += 1 and returns acc. Building the entire histogram takes zero heap allocations — the array lives on the stack regardless of file size.
3. The parse step has an early return: in --strict mode the first malformed line must abort the whole run with LogError::Parse. That control flow is clear as a for loop with a return, but awkward to express through lazy adapters — so a loop is the right tool there.
4. A #[cfg(test)] module compiles only under cargo test, lives inside the module it tests, and can touch its private items (unit testing). A file in tests/ is a separate crate that links the public binary/library and exercises it from outside — for logwise, it launches the actual binary and checks stdout, stderr, and exit code (integration testing).
5. The chain is generic, so the compiler monomorphizes it (one concrete specialization with every next inlined) and the optimizer fuses the adapters into a single loop — identical machine code to a hand-written for. --release turns the optimizer on; a debug build leaves the layers (and extra checks) in place, so the “zero cost” only fully materializes when optimized.