Day 13 · Streaming and RAG

Day 12 got you a one-shot answer: send the whole request, wait, print the whole reply. That “wait” is the problem — a long answer can take many seconds, and a blank terminal for ten seconds feels broken even when it isn’t. Today you fix the feel with streaming, and then make askr smarter by teaching it to read your own notes first with a tiny RAG pipeline.

Part 1 — Streaming: the bytes arrive in pieces

Set "stream": true in the request body and the Messages API stops sending one JSON object. Instead it sends a sequence of Server-Sent Events (SSE) — a long-lived HTTP response whose body is a text stream of event:/data: lines, one chunk of the answer at a time:

   event: message_start
   data: {"type":"message_start","message":{...}}

   event: content_block_delta
   data: {"type":"content_block_delta","delta":{"type":"text_delta","text":"Moving "}}

   event: content_block_delta
   data: {"type":"content_block_delta","delta":{"type":"text_delta","text":"transfers "}}

   ...

   event: message_stop
   data: {"type":"message_stop"}

The text you want lives in the content_block_delta events, in delta.text. Everything else (message_start, pings, message_stop) you skip.

By the numbers — why streaming changes the feel, not the total time

Take a 600-token answer. At an illustrative output rate of ~50 tokens/second:

   total time     = 600 tokens / 50 tokens/sec = 12 seconds
   non-streaming  : 12 s of blank screen, then 600 tokens at once
   streaming      : first words in well under 1 s, then a steady flow for 12 s

The total generation time is the same. What changes is time-to-first-token — from ~12 s to under a second. Perceived latency is dominated by that first token, which is why every chat UI streams. (Token rates vary by model and load; the point is the shape, not the exact 50.)

Parsing a stream without corrupting it

A byte stream is not a String. Two real hazards the compiler makes you confront:

1. A chunk boundary can fall in the middle of a UTF-8 character. If you decode each raw network chunk with from_utf8 independently, a multi-byte character split across two chunks becomes garbage (or a replacement char).
2. An SSE line can be split across chunks, or two lines can arrive in one chunk.

The robust shape: accumulate bytes in a buffer, cut on the newline byte b'\n', and only decode a complete line — because a complete data: line is always valid UTF-8 on its own:

let mut stream = resp.bytes_stream();   // Stream<Item = Result<Bytes, _>>
let mut buf: Vec<u8> = Vec::new();
let mut answer = String::new();

while let Some(chunk) = stream.next().await {
    buf.extend_from_slice(chunk?.as_ref());

    while let Some(pos) = buf.iter().position(|&b| b == b'\n') {
        let line_bytes: Vec<u8> = buf.drain(..=pos).collect();
        let line = String::from_utf8_lossy(&line_bytes[..line_bytes.len() - 1]);
        if let Some(text) = parse_sse_line(line.trim_end()) {
            sink(&text);            // print it the instant it arrives
            answer.push_str(&text); // also accumulate the full answer
        }
    }
}

parse_sse_line is deliberately defensive — it returns Option<String>, treating anything it doesn’t recognize as “no text here” rather than panicking:

fn parse_sse_line(line: &str) -> Option<String> {
    let data = line.strip_prefix("data:")?.trim();
    if data.is_empty() || data == "[DONE]" { return None; }
    let value: serde_json::Value = serde_json::from_str(data).ok()?;
    if value.get("type")?.as_str()? == "content_block_delta" {
        let delta = value.get("delta")?;
        if delta.get("type")?.as_str()? == "text_delta" {
            return delta.get("text")?.as_str().map(str::to_string);
        }
    }
    None
}

This is the recurring thread, made concrete: what does building this force you to understand? That a network stream is untrusted, ragged bytes — and Rust won’t let you pretend otherwise. The ?-on-Option chain means a malformed line can never index past the end of a missing field; it just yields None and the loop moves on.

The sink: streaming as a callback

Notice complete_streaming takes a sink: &mut TokenSink<'_> — a &mut dyn FnMut(&str) + Send. Each delta is handed to the sink the moment it arrives. The CLI passes a sink that prints and flushes:

let mut sink = |text: &str| {
    print!("{text}");
    let _ = std::io::stdout().flush();
};
client.complete_streaming(&request, &mut sink).await?;

A test passes a sink that pushes into a String. The mock client emits its canned reply through the same sink, word by word — so the streaming path is exercised offline, with no SSE at all.

It actually happened — streaming became the default UX overnight

When ChatGPT launched on 30 November 2022, its token-by-token streaming set a new bar for how an AI product should feel. The model wasn’t faster than a batch response — the total time was the same — but showing words as they were generated made multi-second waits feel responsive instead of frozen. Within months, streaming went from a nice-to-have to the assumed default for any LLM interface. Your CLI gets the same win for the price of parsing an SSE stream correctly.

Part 2 — RAG: let the model answer from your text

A model only knows its training data. Ask it about your design doc, your meeting notes, last week’s decision — it can’t help. Retrieval-Augmented Generation (RAG) fixes that without any retraining: before you ask, you retrieve the most relevant passages from your own files and paste them into the prompt as context.

   your files ──► chunk ──► embed ──► [ vector store ]
                                            ▲
   question ───► embed ───────────────────►│  top-k by similarity
                                            ▼
                          retrieved passages + question ──► LLM ──► grounded answer

Four steps, and askr does each one:

1. Chunk — split each .txt/.md file into bite-sized passages (we prefer paragraph breaks, and pack overly long paragraphs word-by-word so nothing is cut mid-word).
2. Embed — turn each passage into a vector of f32s (more on the embedder tomorrow).
3. Retrieve — embed the question too, and find the top-k passages whose vectors are most similar.
4. Stuff — build a prompt that puts those passages above the question.

pub async fn build_store(embedder: &dyn Embedder, dir: &Path, max_chars: usize)
    -> Result<VectorStore>
{
    // read every .txt/.md file, chunk it, embed all chunks in one batch,
    // and load (chunk, vector) pairs into the store.
}

pub async fn retrieve<'a>(store: &'a VectorStore, embedder: &dyn Embedder, query: &str, k: usize)
    -> Result<Vec<Scored<'a>>>
{
    let q = embedder.embed_one(query).await?;
    Ok(store.top_k(&q, k))   // cosine similarity — Day 14
}

The final prompt is just a string with the retrieved context first and an instruction to stay grounded:

pub fn build_rag_prompt(query: &str, retrieved: &[Scored<'_>]) -> String {
    // "Answer using only the context below. If it's not there, say you don't know."
    // === CONTEXT ===  (each retrieved chunk, with its source + score)
    // === QUESTION === (the user's question)
}

That “if it’s not in the context, say you don’t know” line matters: RAG’s whole value is grounding the answer in retrieved text, so you instruct the model to lean on it.

Where RAG came from

The term and the technique come from Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” published at NeurIPS 2020. The core idea — pair a retriever with a generator so the model conditions on fetched documents instead of only its weights — is exactly what you’re building, just at a smaller, hackable scale.

Embedder is, like LlmClient, a trait — so the RAG pipeline is testable with a deterministic MockEmbedder and runs offline with a real-but-local HashingEmbedder. That’s tomorrow’s chapter: the vector store, the cosine math, and why Rust is a particularly good home for that hot loop.

Build it

1. Add stream: true, switch ClaudeClient to consume resp.bytes_stream(), and parse SSE lines into text deltas through the sink.
2. Print each delta immediately (flush stdout) so answers stream.
3. Add the Embedder trait, a chunk_text function, build_store, retrieve, and build_rag_prompt.
4. Wire a --rag <dir> flag: build the store, retrieve top-k, build the augmented prompt, then ask.
5. cargo run -- --rag sample-docs "why is Rust good for vector search?" — watch it retrieve, then stream.

Check your understanding

1. With "stream": true, what does the API send instead of one JSON object, and which event type carries the actual answer text?
2. Why does askr buffer raw bytes and only decode a line once it sees b'\n', instead of decoding each network chunk as it arrives?
3. Streaming doesn’t make the total response faster. What does it actually improve, and why does that dominate how the program feels?
4. List the four steps of the RAG pipeline in order, and say what each one produces.
5. Why does build_rag_prompt instruct the model to answer “using only the context below”?

Show answers

1. A sequence of Server-Sent Events — event:/data: lines streamed over a long-lived response. The answer text arrives in content_block_delta events, specifically in delta.text; other events (message_start, message_stop, pings) carry no answer text.
2. Because a network chunk boundary can fall in the middle of a multi-byte UTF-8 character (and SSE lines can split across chunks or arrive several-per-chunk). Decoding each raw chunk independently can corrupt a character; decoding only a complete line is safe because a full data: line is always valid UTF-8.
3. It improves time-to-first-token — the first words appear in under a second instead of after the whole answer is generated. Perceived latency is dominated by that first token, so a streamed answer feels responsive even though total generation time is unchanged.
4. Chunk (split files into passages) → Embed (turn each passage into an f32 vector) → Retrieve (embed the question, find the top-k most similar passages) → Stuff (build a prompt with those passages above the question). Output: passages, vectors, the top-k passages, and the final grounded prompt.
5. So the answer is grounded in the retrieved passages rather than the model’s training memory — the whole point of RAG. It also lets the model say “I don’t know” honestly when the retrieved context doesn’t contain the answer, instead of confabulating.

---

Prev: Day 12 · Calling the Claude API · Next: Day 14 · Vector Search →