Day 5 · Concurrency

On Day 4 the store was single-threaded: one REPL, one &mut MemStore, no contention. A real KV store is hit by many clients at once. The instant two threads want to touch one HashMap — and at least one of them writes — you are staring down a data race, the single most-cited bug class in systems programming. Today we make the store safe to share, and the way Rust makes us do it is the headline feature of the whole language: fearless concurrency. The compiler will not let us write the race in the first place.

The bug we are preventing

A data race needs four things at once: two threads, shared data, at least one writer, and no synchronization. In C, all four are one careless line away, and the failure is the worst kind — intermittent, non-reproducible, corrupting memory silently. Here’s the shape:

   thread A           shared HashMap            thread B
   ────────           ──────────────            ────────
   read bucket  ◀───────  count: 5  ───────▶  read bucket
   write 6      ──────▶                ◀──────  write 6
                          count: 6                          one update lost,
                       (should be 7!)                       or worse: corrupt

In Rust you simply can’t hand a plain &mut MemStore to two threads — the borrow checker’s “shared XOR mutable” rule already forbids two mutable borrows. And you can’t dodge it with the single-threaded sharing tools either: Rc<RefCell<T>> (shared mutable state on one thread) is not Send, so the compiler refuses to move it into a thread at all. The error is blunt and early:

   error[E0277]: `Rc<RefCell<MemStore<..>>>` cannot be sent between threads safely

That refusal is the feature. To share across threads you must reach for tools that are safe to share — and Rust has exactly two families.

Tool 1: shared state behind a lock — Arc<RwLock<T>>

Two pieces, each doing one job:

• Arc<T> — atomically reference-counted shared ownership. Many threads can each hold an Arc to the same value; the value is dropped only when the last Arc does. Cloning an Arc just bumps an atomic counter — cheap, and thread-safe (that’s the “A”).
• RwLock<T> — a lock enforcing many readers XOR one writer at runtime. This is the exact same “shared XOR mutable” rule from The Rust Mindset, now applied to data shared between threads instead of within one.

Stack them and you get kvlite’s thread-safe handle:

   use std::sync::{Arc, RwLock};

   #[derive(Debug, Clone)]
   pub struct SharedStore<K, V> {
       inner: Arc<RwLock<MemStore<K, V>>>,
   }

   impl<K: Eq + Hash, V: Clone> SharedStore<K, V> {
       pub fn get(&self, key: &K) -> Result<Option<V>> {
           let guard = self.inner.read().map_err(|_| KvError::Poisoned)?;  // shared read lock
           Ok(guard.get(key))   // clone the value *inside* the lock...
       }                        // ...guard drops HERE, lock released

       pub fn set(&self, key: K, value: V) -> Result<Option<V>> {
           let mut guard = self.inner.write().map_err(|_| KvError::Poisoned)?; // exclusive write lock
           guard.set(key, value)
       }
   }

Three things to notice, because each is the compiler teaching you:

1. get takes &self, not &mut self — yet it can mutate the lock’s internals. That’s interior mutability: the RwLock moves the mutable-access check from compile time to runtime, which is the only way many threads can share one writable thing. The lock is the runtime stand-in for &mut.
2. The guard’s Drop releases the lock. You never call unlock(). When guard goes out of scope, its destructor frees the lock — ownership, the same rule from page one, now managing a lock instead of memory. Forgetting to unlock is structurally impossible.
3. We clone the value inside the lock and return the clone (this is why Day 4’s get returns owned V). The borrow into the map cannot escape the guard’s lifetime, so we copy out and let go. Holding a lock longer than necessary is how you build a slow server; the design hands the lock back immediately.

Mutex vs RwLock — pick by your read/write mix

Mutex<T> allows exactly one accessor at a time, reader or writer. RwLock<T> allows many readers or one writer. Use RwLock when reads dominate (a cache, a config, a KV store), Mutex when access is mixed or writes are frequent enough that the bookkeeping doesn’t pay off. One caution: std’s RwLock gives no fairness guarantee — its reader/writer priority is left to the OS, so a steady stream of readers can in principle starve a writer on some platforms. For a read-mostly store it’s the right call; just know it’s a real trade-off, not a free lunch.

By the numbers — why a read-mostly store wants RwLock

Take a burst of 16 concurrent GETs on a 16-core box:

   Mutex  : 1 accessor at a time     -> 16 reads run one-after-another  (serialized)
   RwLock : many readers concurrently -> up to 16 reads run at once      (≈ 16× parallel)

The win is an upper bound set by your core count and shrunk by any interleaved writers (a writer blocks all readers for its duration). But for the common cache pattern — reads vastly outnumbering writes — letting readers proceed together is the difference between a store that scales with cores and one that doesn’t.

Under the hood — Send, Sync, and how the compiler knows it’s safe

How does Rust decide Arc<RwLock<MemStore>> is shareable but Rc<RefCell<MemStore>> is not? Two marker traits the compiler auto-derives:

• Send — a type is safe to move to another thread.
• Sync — a type is safe to share by reference (&T) across threads. (T: Sync exactly when &T: Send.)

These propagate structurally: a struct is Send/Sync only if all its fields are. Rc is deliberately neither (its refcount is a plain, non-atomic integer — racing on it would corrupt the count), which is why moving one into a thread is a compile error. Arc is Send + Sync (atomic count), and RwLock<T> is Sync as long as T: Send + Sync. So Arc<RwLock<MemStore>> is Send + Sync — and thread::spawn requires its closure be Send, so the check happens automatically when you spawn. The data-race guarantee isn’t a lint or a runtime monitor; it’s the type system. Unsynchronized sharing is a type error.

Tool 2: don’t share at all — message passing with mpsc

The other way to be safe is to not share mutable state in the first place: give one thread sole ownership of the data and have everyone else send it messages. Rust ships std::sync::mpsc — a multi-producer, single-consumer channel — for exactly this. The store lives in one “owner” thread that needs no lock at all, because nothing else can reach it:

   use std::sync::mpsc;

   enum Cmd {
       Set(String, String),
       Get(String, mpsc::Sender<Option<String>>),  // carries a reply channel
   }

   let (tx, rx) = mpsc::channel::<Cmd>();

   // The owner thread: it alone holds the store.
   std::thread::spawn(move || {
       let mut store: MemStore<String, String> = MemStore::new();
       for cmd in rx {                       // iterate until every Sender is dropped
           match cmd {
               Cmd::Set(k, v) => { let _ = store.set(k, v); }
               Cmd::Get(k, reply) => { let _ = reply.send(store.get(&k)); }
           }
       }
   });

   // Any number of producers can clone `tx` and send commands.
   tx.send(Cmd::Set("name".into(), "ada".into())).unwrap();

This is the “do not communicate by sharing memory; share memory by communicating” philosophy. The channel itself moves ownership of each message from sender to receiver — so there’s never two owners of the same data, and never a lock to forget. kvlite uses the lock approach (it fits a read-mostly store and keeps the code small), but knowing both models is the point: shared-state-plus-lock and message-passing are the two pillars of std concurrency, and you’ll choose between them constantly.

It actually happened — Therac-25, a race condition that killed

Between 1985 and 1987 the Therac-25 radiation-therapy machine delivered massive overdoses to patients; several were killed or gravely injured. The root cause analyzed by Leveson and Turner (1993) was, in part, a race condition: a fast operator could edit treatment settings faster than a background task updated shared machine state, leaving the hardware in an inconsistent configuration the software believed was safe. It is the textbook case of concurrent access to shared mutable state with inadequate synchronization — the exact combination Rust’s Send/Sync rules make unrepresentable. A modern rewrite wouldn’t be magically correct, but that specific category of bug would have to get past a compiler that refuses to share mutable state without synchronization.

Honest trade-offs: locks don’t make you correct, only race-free

RwLock removes data races — torn memory, corrupt counts. It does not remove logic races. Watch:

   let n = store.get("hits")?.unwrap();   // lock taken and RELEASED
   store.set("hits", n + 1)?;             // lock taken again — but n is stale now!

Between the two calls, another thread can bump hits, and your set overwrites it — a classic lost update (a time-of-check-to-time-of-use race). The fix is to do the read-modify-write inside one lock (a single increment method that takes the write lock once) or use an atomic. The compiler can’t catch this for you — it’s about what you do under the lock, not whether you took it. Two other hazards round out the honesty:

• Deadlock — take two locks in different orders on two threads and both wait forever. Rust prevents data races, not deadlocks. Discipline (a fixed lock order, or just one lock) is on you.
• Poisoning — if a thread panics while holding a lock, the lock becomes poisoned; later .lock() calls return Err. kvlite maps that to KvError::Poisoned instead of re-panicking, so one bad request can’t take down the server.

Build it: concurrent access from threads

The Day 5 deliverable is hammering one SharedStore from many threads and proving nothing is lost. The crate’s integration test (tests/concurrency.rs) does exactly this:

   let store: SharedStore<String, u64> = SharedStore::new();
   let handles: Vec<_> = (0..8).map(|t| {
       let store = store.clone();              // Arc bump: same map, new handle
       std::thread::spawn(move || {
           for i in 0..1000 {
               store.set(format!("t{t}-k{i}"), 0).unwrap();
           }
       })
   }).collect();
   for h in handles { h.join().unwrap(); }
   assert_eq!(store.len().unwrap(), 8 * 1000);  // exact — no update lost

Eight threads, eight thousand distinct keys, exact final count. Run it with cargo test. Then change the threads to all write the same key while incrementing across two calls, and watch the lost-update race appear — the count comes out less than expected, with no panic and no compiler complaint. That contrast is the whole lesson: what does building this force you to understand, and what is the compiler protecting you from? It protects you absolutely from data races (they’re type errors) and not at all from logic races (they’re your design) — and knowing which is which is what makes you dangerous in the good way.

Next we make those writes survive a crash.

→ Next: Day 6 · Persistence & Errors · Prev: Day 4 · Traits & Generics · Back to the Project 2 overview

Check your understanding

1. Name the four ingredients of a data race. Which one does Arc<RwLock<T>> remove, and how?
2. Rc<RefCell<MemStore>> won’t compile when moved into a thread, but Arc<RwLock<MemStore>> will. In terms of Send/Sync, why is one rejected and the other accepted?
3. SharedStore::get takes &self yet mutates the lock’s internals and returns an owned clone of the value. Explain both: why &self is enough, and why it returns a clone rather than a &V.
4. Describe the lost-update race in the read-modify-write example. Why can’t RwLock prevent it, and what is the fix?
5. Contrast the two std concurrency models (shared-state-plus-lock vs mpsc message passing). Why does the message-passing owner thread need no lock?

Show answers

1. Two threads, shared data, at least one writer, and no synchronization. Arc<RwLock<T>> removes the “no synchronization” ingredient: Arc gives safe shared ownership, and RwLock enforces many readers XOR one writer at runtime, so reads and writes can never overlap unsafely.
2. Rc’s reference count is a non-atomic integer, so Rc is not Send — racing threads could corrupt the count — and thread::spawn requires a Send closure, so it’s rejected. Arc uses an atomic count (so it’s Send + Sync) and RwLock<T> is Sync when T: Send + Sync, making Arc<RwLock<MemStore>> safe to move/share; the compiler accepts it.
3. &self is enough because the RwLock provides interior mutability — it does the mutable-access check at runtime, so the method doesn’t need &mut self. It returns a clone because a &V would borrow data owned by the lock guard; the borrow can’t outlive the guard, so we copy the value out and release the lock immediately.
4. The code takes the lock for the get, releases it, then takes it again for the set — between the two, another thread can update the key, so the set writes a stale value and that other update is lost. RwLock only guarantees no data race per call; it can’t span two separate lock acquisitions. Fix: do the whole read-modify-write under one lock (one method holding the write lock) or use an atomic.
5. Shared-state: many threads share one value through Arc and coordinate with a lock. Message-passing: one thread owns the data and others send it commands over an mpsc channel. The owner needs no lock because nothing else can reach the data — there is no sharing to synchronize; the channel transfers ownership of each message instead.