Day 6 · Persistence & Errors

After Day 5 the store is fast and safe to share — and still entirely in RAM, which means a crash or a restart wipes it. A “database” that forgets everything on exit isn’t one. Today we make writes durable: every mutation is appended to a log on disk before it’s applied, and on startup we replay that log to rebuild the map exactly as it was. Replaying a file of records is a job iterators were born for, and the moment real I/O enters the picture we finally need a real error type — so we build one with thiserror.

Two ways to persist, and why we pick the log

There are two classic strategies, and a real system often uses both:

Strategy	What’s on disk	Restart cost	Write cost	Data loss on crash
Snapshot	the whole map, dumped periodically	fast (load one file)	spiky (dump everything)	everything since the last snapshot
Append-only log (WAL)	every mutation, in order	slower (replay all)	cheap (append one line)	only the un-flushed tail

Redis ships exactly these two: RDB snapshots and the AOF (append-only file). kvlite uses the log, because it’s the one that teaches the most and loses the least: each write is a cheap append, and durability is decided per-write. Its weakness — the file grows forever and replay gets slower — is real, and we’ll name the fix (compaction) at the end.

The append-only log

The rule that makes it an append-only log: we never edit the file in place. Every SET and DEL adds one line to the end; an overwrite is just a newer line that wins during replay. The format is the same line grammar the Day 7 server will speak:

   SET name ada          # key, then the rest of the line is the value (spaces OK)
   SET name ada lovelace # an overwrite: just a later line
   DEL temp

Writing a record is “encode, append, flush”:

   pub fn append(&mut self, cmd: &Command) -> Result<()> {
       writeln!(self.writer, "{}", cmd.encode())?;   // BufWriter over an append-mode File
       self.writer.flush()?;                          // push our buffer to the OS
       Ok(())
   }

And the discipline that makes it safe is write-ahead: in Db::set, we append the SET record to the log first, and only then mutate the in-memory map. If the process dies between the two, the write is already on disk and replay re-applies it. The log is the source of truth; RAM is a fast cache of it.

   client SET k v
        │
        ▼
   ① append "SET k v" to log  ──fsync/flush──▶  disk     (durability point)
        │
        ▼
   ② store.set(k, v)                                       (now visible to reads)
        │
        ▼
   OK

Replay is an iterator pipeline

On startup we rebuild the map by reading the log top to bottom and applying each record. This is a textbook iterator job — BufReader::lines() yields one record at a time, lazily, so we never load the whole file into memory:

   pub fn replay(path: impl AsRef<Path>) -> Result<(MemStore<String, String>, usize)> {
       let mut store = MemStore::new();
       let file = match File::open(&path) {
           Ok(f) => f,
           // First boot: no log yet. An empty store is the right answer, not an error.
           Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok((store, 0)),
           Err(e) => return Err(KvError::Io(e)),
       };
       let reader = BufReader::new(file);
       let mut applied = 0;
       for line in reader.lines() {          // the iterator
           let line = line?;                 // each item is a Result — ? propagates a read error
           if line.trim().is_empty() { continue; }
           Command::parse(&line)?.apply(&mut store)?;   // parse, then apply to the map
           applied += 1;
       }
       Ok((store, applied))
   }

Because records are applied in order, replay is automatically last-write-wins: SET a 1 then SET a 2 leaves a = 2, and a later DEL a removes it. The map after replay is byte-for-byte the state the writes described — that’s the whole contract. (apply is just a small match that calls set or delete on the store, so the same Command type both writes the log and rebuilds from it.)

Under the hood — why append-only is the fast choice

Appending isn’t just simple, it’s the I/O pattern disks like best. An append is a sequential write: the next bytes go right after the last, so even spinning disks avoid seeks and SSDs hit their happy path. Editing records in place would mean random writes scattered across the file — far slower, and not crash-safe (a half-written record corrupts existing data). This is the core insight behind write-ahead logging, formalized in the ARIES recovery algorithm (Mohan et al., 1992) and used by essentially every serious database: write the change to a sequential log before touching the real data structures, so you can always recover by replaying the log. kvlite is a tiny instance of a very deep idea.

The durability spectrum: flush is not fsync

Here is the honest, load-bearing subtlety. self.writer.flush() pushes bytes out of our program’s buffer into the operating system’s cache. That survives a process crash (a panic, a kill) — the OS still has the data and will write it out. It does not necessarily survive a power loss, because the data may still be sitting in the OS page cache, not yet on the physical disk. Forcing it all the way down requires file.sync_all() (an fsync), which is much slower. So there’s a spectrum, and you choose where to sit:

Policy	Call per write	Survives process crash	Survives power loss	Speed
buffer only	nothing	no	no	fastest
flush each write	flush()	yes	not guaranteed	fast
fsync each write	sync_all()	yes	yes	slow

kvlite flushes per write (the middle row), so a clean process exit or a panic never loses an acknowledged write — and the chapter is honest that a power cut might lose the very last writes. Redis exposes this same choice as appendfsync always | everysec | no, with everysec (fsync about once a second) as the default sweet spot.

By the numbers — the “everysec” data-loss window

Pick the popular middle setting: fsync the log roughly once per second instead of on every write. The most you can lose to a power cut is one second of acknowledged writes:

   loss window  = 1 second
   write rate   = 50,000 writes/sec      (a busy small store)
   worst case   = 1 s × 50,000 /s ≈ 50,000 writes potentially lost

That’s the price of speed: per-write fsync would lose ~0 but cap you near disk-sync latency (often a millisecond-plus each). “Lose at most a second” buys you orders of magnitude more throughput. There is no free durability — only a dial, and you set it knowing the numbers.

It actually happened — “fsyncgate” (2018)

In 2018 PostgreSQL developers discovered that fsync durability was subtler than everyone assumed: on Linux, if a writeback failed, the kernel could drop the dirty page and clear the error flag, so a later, retried fsync() would return success even though the data had been lost. Databases that assumed “retry fsync on error” were silently exposed to data loss. The episode — now known as “fsyncgate” — forced coordinated changes across PostgreSQL (which switched to PANIC on fsync failure), other databases, and the Linux kernel. The lesson for kvlite’s chapter: durability is not “I called flush”; it’s a contract with the OS and the hardware that you must understand to trust. Naming flush vs fsync plainly is the first step.

A real error type with thiserror

Real I/O fails: the disk is full, the file is unreadable, a log line is corrupt. Day 4’s store could mostly use Option; now functions genuinely need to say what went wrong. A library defines its own error enum so callers match on its meaning — “the key was invalid”, “the log line was unparseable” — not on whatever low-level type happened to bubble up. thiserror makes that nearly free:

   #[derive(Debug, thiserror::Error)]
   pub enum KvError {
       #[error("invalid key {key:?}: keys must be non-empty and contain no whitespace")]
       InvalidKey { key: String },

       #[error("protocol error: {0}")]
       Protocol(String),

       #[error("io error: {0}")]
       Io(#[from] std::io::Error),   // #[from] generates From<io::Error>

       #[error("lock poisoned: a thread panicked while holding the store lock")]
       Poisoned,
   }

Two derives do the work the brief asked for. #[error("...")] writes the Display text for each variant (so eprintln!("{e}") reads well). #[from] on the Io variant generates From<io::Error> for KvError — which is what lets a bare ? on any I/O call turn an io::Error into the right KvError with no mapping code:

   let file = File::open(path)?;   // io::Error -> KvError::Io, automatically, via ?

That ? is the same operator from Project 1, now load-bearing across a whole library: every fallible function returns Result<T, KvError>, errors convert and propagate on their own, and the binary prints one clean line at the top. Notice too that InvalidKey is validated before anything is logged — a malformed key never reaches the disk, so a corrupt key can’t poison the log we’ll replay tomorrow.

Build it: survive a restart

The Day 6 deliverable is a store that remembers. The crate’s test (db.rs) is the proof: write in one session, drop the Db, reopen the same log, and read it back.

   {  // session 1
       let (db, _) = Db::open(&path)?;
       db.set("name", "ada")?;
       db.set("lang", "rust is great")?;   // value with spaces survives
       db.delete("name")?;
   }  // Db dropped — simulated shutdown
   {  // session 2: same log
       let (db, replayed) = Db::open(&path)?;
       assert_eq!(replayed, 3);                          // SET, SET, DEL
       assert_eq!(db.get("lang")?, Some("rust is great".into()));
       assert_eq!(db.get("name")?, None);                // the DEL replayed too
   }

You can see it by hand with the REPL: cargo run -- repl, SET name ada, QUIT; then run it again and GET name — VALUE ada, restored from disk. Peek at kvlite.wal and you’ll see your commands in plain text.

The honest loose end: the log grows forever — every overwrite and every deleted key stays on disk, and replay gets slower over time. The standard fix is compaction (what Redis calls AOF rewrite): periodically write a fresh snapshot of the current state and start a new, shorter log — combining the two strategies from the top of this page. We won’t build it, but you now know exactly what it would do and why.

That’s the recurring thread again: what does building this force you to understand, and what is the compiler protecting you from? Persistence forces you to understand that “saved” is a spectrum, not a boolean — and Rust’s Result/? machinery makes sure every disk failure along that spectrum is a value you must handle, never an exception you forgot. Next, we put the store on the network.

→ Next: Day 7 · TCP Server · Prev: Day 5 · Concurrency · Back to the Project 2 overview

Check your understanding

1. Contrast a snapshot with an append-only log on two axes: write cost and worst-case data loss on a crash. Why does kvlite choose the log?
2. What does “write-ahead” mean in Db::set, and what bad outcome does doing the log append before the in-memory update prevent?
3. Replay reads the log with BufReader::lines() and applies records in order. Why does that ordering give you last-write-wins for free, including deletes?
4. Explain the difference between flush() and sync_all() (fsync). Which crash does each protect against, and which does kvlite’s flush-per-write not fully protect against?
5. What two things do thiserror’s #[error("...")] and #[from] attributes generate, and how does #[from] make a bare ? on an io::Error work?

Show answers

1. A snapshot has spiky write cost (dump the whole map) and loses everything since the last snapshot on a crash. An append-only log has cheap per-write cost (append one line) and loses only the un-flushed tail. kvlite picks the log because each write is a cheap sequential append and durability is decided per write, minimizing loss — at the cost of slower replay and unbounded growth.
2. “Write-ahead” means the mutation is appended to the on-disk log before the in-memory map is changed. If the process dies between the two steps, the write is already durable and replay re-applies it — so an acknowledged write is never lost to a mid-operation crash.
3. Records are applied in the order they were written, so a later SET overwrites an earlier value and a later DEL removes it — the final state equals the last operation on each key. No extra bookkeeping is needed; sequential replay is last-write-wins.
4. flush() pushes bytes from the program’s buffer into the OS cache — surviving a process crash/panic but not necessarily a power loss. sync_all() (fsync) forces the data all the way to the physical disk — surviving power loss too, but much slower. kvlite flushes per write, so it does not fully protect against a power cut losing the most recent writes still in the OS cache.
5. #[error("...")] generates the Display implementation (the human-readable message) for each variant; #[from] generates a From<io::Error> for KvError conversion. Because ? calls From::from on the error it propagates, that generated conversion turns an io::Error into KvError::Io automatically — so a bare ? on any I/O call just works.