Day 6 · Persistence & Errors

After Day 5 the store is fast and safe to share — and still entirely in RAM, which means a crash or a restart wipes it. A “database” that forgets everything on exit isn’t one. Today we make writes durable: every mutation is appended to a log on disk before it’s applied, and on startup we replay that log to rebuild the map exactly as it was. Replaying a file of records is a job iterators were born for, and the moment real I/O enters the picture we finally need a real error type — so we build one with thiserror.

Today touches four files: we expand src/error.rs (the Day 4 stub grows to carry real I/O and protocol errors), create src/log.rs (the write-ahead log) and src/db.rs (the durable store), and wire both new modules into lib.rs. Build order follows the dependencies: error → log → db.

Two ways to persist, and why we pick the log

There are two classic strategies, and a real system often uses both:

Strategy	What’s on disk	Restart cost	Write cost	Data loss on crash
Snapshot	the whole map, dumped periodically	fast (load one file)	spiky (dump everything)	everything since the last snapshot
Append-only log (WAL)	every mutation, in order	slower (replay all)	cheap (append one line)	only the un-flushed tail

Redis ships exactly these two: RDB snapshots and the AOF (append-only file). kvlite uses the log, because it’s the one that teaches the most and loses the least: each write is a cheap append, and durability is decided per-write. Its weakness — the file grows forever and replay gets slower — is real, and we’ll name the fix (compaction) at the end.

First, expand the error type

Real I/O fails: the disk is full, the file is unreadable, a log line is corrupt. Day 4’s store could get away with an almost-empty KvError; now functions genuinely need to say what went wrong. A library defines its own error enum so callers match on its meaning — “the key was invalid”, “the log line was unparseable” — not on whatever low-level type happened to bubble up. Replace the Day 4 stub src/error.rs with the full version:

   use std::io;

   /// Everything that can go wrong inside kvlite.
   #[derive(Debug, thiserror::Error)]
   pub enum KvError {
       /// A key was empty or contained whitespace. The append-only log is
       /// line-oriented and splits on spaces, so a key with a space in it could
       /// not be replayed unambiguously — we reject it at the door instead.
       #[error("invalid key {key:?}: keys must be non-empty and contain no whitespace")]
       InvalidKey { key: String },

       /// A client (or a corrupt log line) sent something we could not parse.
       #[error("protocol error: {0}")]
       Protocol(String),

       /// The underlying file/socket I/O failed. `#[from]` lets `?` convert an
       /// `io::Error` into this variant automatically.
       #[error("io error: {0}")]
       Io(#[from] io::Error),

       /// A lock was poisoned: another thread panicked while holding it. We surface
       /// it rather than re-panicking so a server can log and keep serving.
       #[error("lock poisoned: a thread panicked while holding the store lock")]
       Poisoned,
   }

   /// Crate-wide convenience alias, so signatures read `Result<T>` not
   /// `Result<T, KvError>`.
   pub type Result<T> = std::result::Result<T, KvError>;

Two derives do the work. #[error("...")] writes the Display text for each variant (so eprintln!("{e}") reads well). #[from] on the Io variant generates From<io::Error> for KvError — which is what lets a bare ? on any I/O call turn an io::Error into the right KvError with no mapping code:

   let file = File::open(path)?;   // io::Error -> KvError::Io, automatically, via ?

That ? is the same operator from Project 1, now load-bearing across a whole library: every fallible function returns Result<T> (i.e. Result<T, KvError>), errors convert and propagate on their own, and the binary prints one clean line at the top.

The append-only log

The rule that makes it an append-only log: we never edit the file in place. Every SET and DEL adds one line to the end; an overwrite is just a newer line that wins during replay. A mutation is a Command, and the same type both encodes to a log line and parses back from one. This is the top of the new src/log.rs:

   use std::fs::{File, OpenOptions};
   use std::io::{BufRead, BufReader, BufWriter, Write};
   use std::path::Path;

   use crate::error::{KvError, Result};
   use crate::store::{MemStore, Store};

   /// A single durable mutation. `GET` never appears here — reads change nothing,
   /// so logging them would be pure waste.
   #[derive(Debug, Clone, PartialEq, Eq)]
   pub enum Command {
       Set { key: String, value: String },
       Del { key: String },
   }

   impl Command {
       /// Render the command as one log line (no trailing newline; the writer adds
       /// it). The inverse of `parse`.
       pub fn encode(&self) -> String {
           match self {
               Command::Set { key, value } => format!("SET {key} {value}"),
               Command::Del { key } => format!("DEL {key}"),
           }
       }

       /// Parse one log line back into a `Command`. Also used by the network
       /// protocol, which is why a bad line is a `Protocol` error.
       pub fn parse(line: &str) -> Result<Command> {
           let line = line.trim_end_matches(['\r', '\n']);
           let mut parts = line.splitn(3, ' ');
           match parts.next() {
               Some("SET") => {
                   let key = parts
                       .next()
                       .ok_or_else(|| KvError::Protocol("SET needs a key".into()))?;
                   let value = parts
                       .next()
                       .ok_or_else(|| KvError::Protocol("SET needs a value".into()))?;
                   validate_key(key)?;
                   Ok(Command::Set {
                       key: key.to_string(),
                       value: value.to_string(),
                   })
               }
               Some("DEL") => {
                   let key = parts
                       .next()
                       .ok_or_else(|| KvError::Protocol("DEL needs a key".into()))?;
                   validate_key(key)?;
                   Ok(Command::Del {
                       key: key.to_string(),
                   })
               }
               Some(other) => Err(KvError::Protocol(format!("unknown command {other:?}"))),
               None => Err(KvError::Protocol("empty command".into())),
           }
       }

       /// Apply this command to an in-memory store. Replaying the whole log is just
       /// calling this for every record in order.
       pub fn apply(&self, store: &mut MemStore<String, String>) -> Result<()> {
           match self {
               Command::Set { key, value } => {
                   store.set(key.clone(), value.clone())?;
               }
               Command::Del { key } => {
                   store.delete(key)?;
               }
           }
           Ok(())
       }
   }

   /// Reject keys we could not faithfully store in the line-oriented log.
   pub fn validate_key(key: &str) -> Result<()> {
       if key.is_empty() || key.chars().any(char::is_whitespace) {
           return Err(KvError::InvalidKey {
               key: key.to_string(),
           });
       }
       Ok(())
   }

The format is the same line grammar the Day 7 server will speak — SET <key> <value> and DEL <key>, one record per line — and because keys may not contain whitespace, splitting on the first one or two spaces round-trips unambiguously. validate_key is the guard that keeps that promise: a malformed key is an InvalidKey error before it can reach the disk.

Writing a record is “encode, append, flush”. Append the Wal type to log.rs:

   /// The append-only writer. Wraps a buffered, append-mode file handle.
   #[derive(Debug)]
   pub struct Wal {
       writer: BufWriter<File>,
   }

   impl Wal {
       /// Open (creating if absent) the log for appending. Existing contents are
       /// kept — we never truncate; that is what "append-only" means.
       pub fn open(path: impl AsRef<Path>) -> Result<Wal> {
           let file = OpenOptions::new()
               .create(true)
               .append(true)
               .open(path)?;
           Ok(Wal {
               writer: BufWriter::new(file),
           })
       }

       /// Append one record and `flush` it.
       pub fn append(&mut self, cmd: &Command) -> Result<()> {
           writeln!(self.writer, "{}", cmd.encode())?;   // BufWriter over an append-mode File
           self.writer.flush()?;                          // push our buffer to the OS
           Ok(())
       }
   }

And the discipline that makes it safe is write-ahead: in Db::set (below), we append the SET record to the log first, and only then mutate the in-memory map. If the process dies between the two, the write is already on disk and replay re-applies it. The log is the source of truth; RAM is a fast cache of it.

   client SET k v
        │
        ▼
   ① append "SET k v" to log  ──fsync/flush──▶  disk     (durability point)
        │
        ▼
   ② store.set(k, v)                                       (now visible to reads)
        │
        ▼
   OK

Replay is an iterator pipeline

On startup we rebuild the map by reading the log top to bottom and applying each record. This is a textbook iterator job — BufReader::lines() yields one record at a time, lazily, so we never load the whole file into memory. Finish log.rs with replay:

   /// Rebuild a store by replaying a log file. Returns a fresh `MemStore` plus the
   /// number of records applied. A missing file is *not* an error — it just means
   /// "first run", so we hand back an empty store.
   pub fn replay(path: impl AsRef<Path>) -> Result<(MemStore<String, String>, usize)> {
       let mut store: MemStore<String, String> = MemStore::new();

       let file = match File::open(&path) {
           Ok(f) => f,
           // First boot: no log yet. An empty store is the correct answer.
           Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok((store, 0)),
           Err(e) => return Err(KvError::Io(e)),
       };

       let reader = BufReader::new(file);
       let mut applied = 0usize;
       for line in reader.lines() {          // the iterator
           let line = line?;                 // each item is a Result — ? propagates a read error
           if line.trim().is_empty() {
               continue;
           }
           Command::parse(&line)?.apply(&mut store)?;   // parse, then apply to the map
           applied += 1;
       }
       Ok((store, applied))
   }

Because records are applied in order, replay is automatically last-write-wins: SET a 1 then SET a 2 leaves a = 2, and a later DEL a removes it. The map after replay is byte-for-byte the state the writes described — that’s the whole contract. Notice apply reuses the same Command type that wrote the log, so one type both serializes a mutation and rebuilds from it.

Under the hood — why append-only is the fast choice

Appending isn’t just simple, it’s the I/O pattern disks like best. An append is a sequential write: the next bytes go right after the last, so even spinning disks avoid seeks and SSDs hit their happy path. Editing records in place would mean random writes scattered across the file — far slower, and not crash-safe (a half-written record corrupts existing data). This is the core insight behind write-ahead logging, formalized in the ARIES recovery algorithm (Mohan et al., 1992) and used by essentially every serious database: write the change to a sequential log before touching the real data structures, so you can always recover by replaying the log. kvlite is a tiny instance of a very deep idea.

The durability spectrum: flush is not fsync

Here is the honest, load-bearing subtlety. self.writer.flush() pushes bytes out of our program’s buffer into the operating system’s cache. That survives a process crash (a panic, a kill) — the OS still has the data and will write it out. It does not necessarily survive a power loss, because the data may still be sitting in the OS page cache, not yet on the physical disk. Forcing it all the way down requires file.sync_all() (an fsync), which is much slower. So there’s a spectrum, and you choose where to sit:

Policy	Call per write	Survives process crash	Survives power loss	Speed
buffer only	nothing	no	no	fastest
flush each write	flush()	yes	not guaranteed	fast
fsync each write	sync_all()	yes	yes	slow

kvlite flushes per write (the middle row), so a clean process exit or a panic never loses an acknowledged write — and the chapter is honest that a power cut might lose the very last writes. Redis exposes this same choice as appendfsync always | everysec | no, with everysec (fsync about once a second) as the default sweet spot.

By the numbers — the “everysec” data-loss window

Pick the popular middle setting: fsync the log roughly once per second instead of on every write. The most you can lose to a power cut is one second of acknowledged writes:

   loss window  = 1 second
   write rate   = 50,000 writes/sec      (a busy small store)
   worst case   = 1 s × 50,000 /s ≈ 50,000 writes potentially lost

That’s the price of speed: per-write fsync would lose ~0 but cap you near disk-sync latency (often a millisecond-plus each). “Lose at most a second” buys you orders of magnitude more throughput. There is no free durability — only a dial, and you set it knowing the numbers.

It actually happened — “fsyncgate” (2018)

In 2018 PostgreSQL developers discovered that fsync durability was subtler than everyone assumed: on Linux, if a writeback failed, the kernel could drop the dirty page and clear the error flag, so a later, retried fsync() would return success even though the data had been lost. Databases that assumed “retry fsync on error” were silently exposed to data loss. The episode — now known as “fsyncgate” — forced coordinated changes across PostgreSQL (which switched to PANIC on fsync failure), other databases, and the Linux kernel. The lesson for kvlite’s chapter: durability is not “I called flush”; it’s a contract with the OS and the hardware that you must understand to trust. Naming flush vs fsync plainly is the first step.

Db: thread-safe and durable

Now glue the two halves — the Day 5 SharedStore for concurrent RAM access, and the Wal for durability — into the store the server will actually use. This is the whole of src/db.rs:

   use std::path::Path;
   use std::sync::{Arc, Mutex};

   use crate::error::{KvError, Result};
   use crate::log::{self, Command, Wal};
   use crate::store::SharedStore;

   #[derive(Clone)]
   pub struct Db {
       store: SharedStore<String, String>,
       /// The log is a single OS file handle, so it cannot be shared by value — we
       /// serialize all appends through one `Mutex`. Reads never touch it.
       wal: Arc<Mutex<Wal>>,
   }

   impl Db {
       /// Open a database backed by the log at `path`. Replays the existing log to
       /// rebuild state, then opens the same file for appending.
       pub fn open(path: impl AsRef<Path>) -> Result<(Db, usize)> {
           let path = path.as_ref();
           let (mem, replayed) = log::replay(path)?;
           let wal = Wal::open(path)?;
           let db = Db {
               store: SharedStore::from_store(mem),
               wal: Arc::new(Mutex::new(wal)),
           };
           Ok((db, replayed))
       }

       /// Set a key. Validates, appends `SET k v` to the log, *then* updates RAM.
       pub fn set(&self, key: &str, value: &str) -> Result<()> {
           log::validate_key(key)?;
           let cmd = Command::Set {
               key: key.to_string(),
               value: value.to_string(),
           };
           // Durability first: get the write on disk before we acknowledge it.
           self.wal.lock().map_err(|_| KvError::Poisoned)?.append(&cmd)?;
           self.store.set(key.to_string(), value.to_string())?;
           Ok(())
       }

       /// Read a key. Pure in-memory; the log is not touched.
       pub fn get(&self, key: &str) -> Result<Option<String>> {
           self.store.get(&key.to_string())
       }

       /// Delete a key. Returns `true` if the key existed. Logs `DEL k` first.
       pub fn delete(&self, key: &str) -> Result<bool> {
           log::validate_key(key)?;
           let cmd = Command::Del {
               key: key.to_string(),
           };
           self.wal.lock().map_err(|_| KvError::Poisoned)?.append(&cmd)?;
           Ok(self.store.delete(&key.to_string())?.is_some())
       }

       pub fn len(&self) -> Result<usize> {
           self.store.len()
       }

       pub fn is_empty(&self) -> Result<bool> {
           Ok(self.len()? == 0)
       }
   }

Db is Clone: every clone shares the same Arc-wrapped store and the same Mutex-guarded log, so handing a clone to each connection thread on Day 7 is cheap and correct. Notice set and delete both do the write-ahead dance — log first, RAM second — and both validate the key before logging, so a corrupt key can never poison the log we’ll replay next boot.

Finally, wire the two new modules into lib.rs:

   pub mod db;      // add
   pub mod error;
   pub mod log;     // add
   pub mod store;

   pub use db::Db;                       // add
   pub use error::{KvError, Result};
   pub use log::Command;                 // add
   pub use store::{MemStore, SharedStore, Store};

Build it: survive a restart

The Day 6 deliverable is a store that remembers. db.rs carries the proof — write in one session, drop the Db, reopen the same log, and read it back:

   #[cfg(test)]
   mod tests {
       use super::*;
       use std::sync::atomic::{AtomicU64, Ordering};

       // A unique temp path per call, so parallel tests never collide. std only.
       fn temp_wal(tag: &str) -> std::path::PathBuf {
           static N: AtomicU64 = AtomicU64::new(0);
           let n = N.fetch_add(1, Ordering::Relaxed);
           std::env::temp_dir().join(format!("kvlite-{}-{}-{}.wal", tag, std::process::id(), n))
       }

       #[test]
       fn survives_a_restart() {
           let path = temp_wal("restart");
           let _ = std::fs::remove_file(&path);

           {   // session 1: write, then drop the Db (simulated shutdown)
               let (db, replayed) = Db::open(&path).unwrap();
               assert_eq!(replayed, 0);
               db.set("name", "ada").unwrap();
               db.set("lang", "rust is great").unwrap();   // value with spaces survives
               db.set("temp", "x").unwrap();
               db.delete("temp").unwrap();
           }

           {   // session 2: reopen the SAME log; replay must rebuild the exact state
               let (db, replayed) = Db::open(&path).unwrap();
               assert_eq!(replayed, 4);                         // SET, SET, SET, DEL
               assert_eq!(db.get("name").unwrap(), Some("ada".into()));
               assert_eq!(db.get("lang").unwrap(), Some("rust is great".into()));
               assert_eq!(db.get("temp").unwrap(), None);       // the DEL replayed too
               assert_eq!(db.len().unwrap(), 2);
           }

           let _ = std::fs::remove_file(&path);
       }

       #[test]
       fn rejects_invalid_keys_before_logging() {
           let path = temp_wal("badkey");
           let _ = std::fs::remove_file(&path);
           let (db, _) = Db::open(&path).unwrap();
           assert!(db.set("bad key", "v").is_err());
           // The bad write must not have reached the log: replay sees nothing.
           let (db2, replayed) = Db::open(&path).unwrap();
           assert_eq!(replayed, 0);
           assert!(db2.len().unwrap() == 0);
           let _ = std::fs::remove_file(&path);
       }
   }

Run cargo test. You can also see it by hand once the Day 7 binary exists: cargo run -- repl, SET name ada, QUIT; then run it again and GET name — VALUE ada, restored from disk. Peek at kvlite.wal and you’ll see your commands in plain text.

The honest loose end: the log grows forever — every overwrite and every deleted key stays on disk, and replay gets slower over time. The standard fix is compaction (what Redis calls AOF rewrite): periodically write a fresh snapshot of the current state and start a new, shorter log — combining the two strategies from the top of this page. We won’t build it, but you now know exactly what it would do and why.

That’s the recurring thread again: what does building this force you to understand, and what is the compiler protecting you from? Persistence forces you to understand that “saved” is a spectrum, not a boolean — and Rust’s Result/? machinery makes sure every disk failure along that spectrum is a value you must handle, never an exception you forgot. Next, we put the store on the network.

→ Next: Day 7 · TCP Server · Prev: Day 5 · Concurrency · Back to the Project 2 overview

Check your understanding

1. Contrast a snapshot with an append-only log on two axes: write cost and worst-case data loss on a crash. Why does kvlite choose the log?
2. What does “write-ahead” mean in Db::set, and what bad outcome does doing the log append before the in-memory update prevent?
3. Replay reads the log with BufReader::lines() and applies records in order. Why does that ordering give you last-write-wins for free, including deletes?
4. Explain the difference between flush() and sync_all() (fsync). Which crash does each protect against, and which does kvlite’s flush-per-write not fully protect against?
5. What two things do thiserror’s #[error("...")] and #[from] attributes generate, and how does #[from] make a bare ? on an io::Error work?

Show answers

1. A snapshot has spiky write cost (dump the whole map) and loses everything since the last snapshot on a crash. An append-only log has cheap per-write cost (append one line) and loses only the un-flushed tail. kvlite picks the log because each write is a cheap sequential append and durability is decided per write, minimizing loss — at the cost of slower replay and unbounded growth.
2. “Write-ahead” means the mutation is appended to the on-disk log before the in-memory map is changed. If the process dies between the two steps, the write is already durable and replay re-applies it — so an acknowledged write is never lost to a mid-operation crash.
3. Records are applied in the order they were written, so a later SET overwrites an earlier value and a later DEL removes it — the final state equals the last operation on each key. No extra bookkeeping is needed; sequential replay is last-write-wins.
4. flush() pushes bytes from the program’s buffer into the OS cache — surviving a process crash/panic but not necessarily a power loss. sync_all() (fsync) forces the data all the way to the physical disk — surviving power loss too, but much slower. kvlite flushes per write, so it does not fully protect against a power cut losing the most recent writes still in the OS cache.
5. #[error("...")] generates the Display implementation (the human-readable message) for each variant; #[from] generates a From<io::Error> for KvError conversion. Because ? calls From::from on the error it propagates, that generated conversion turns an io::Error into KvError::Io automatically — so a bare ? on any I/O call just works.