Streaming

Streaming lets you show a response as it arrives instead of waiting for the whole thing. Call chat.stream(&mut messages) and iterate the result with futures::StreamExt. Each item is a StreamEvent, so you can react to text, reasoning, and tool activity as they happen.

Streaming needs the stream cargo feature, and the provider has to implement StreamProvider. stream is bound on that trait, so it appears on the default chat only for providers that can stream; a completion-only provider has no stream method, and calling it is a compile error rather than a runtime failure.

The simple form

When you use the default chat and the provider supports streaming, stream is available without any extra opt-in. Here is the shape from the Ollama example.

use chat_rs::{
    ChatBuilder, StreamEvent,
    ollama::OllamaBuilder,
    parts,
    types::messages::{self, content},
};
use futures::StreamExt;

let client = OllamaBuilder::new().with_model("llama3.2").pull().await?.build();
let mut chat = ChatBuilder::new().with_model(client).build();

let mut messages = messages::Messages::default();
messages.push(content::from_user(parts!["Tell me a short story."]));

let mut stream = chat.stream(&mut messages).await.map_err(|err| err.err)?;

while let Some(chunk_res) = stream.next().await {
    match chunk_res {
        Ok(StreamEvent::TextChunk(text)) => print!("{text}"),
        Ok(StreamEvent::ReasoningChunk(thought)) => print!("{thought}"),
        Ok(StreamEvent::Done(res)) => println!("\n[usage] {:?}", res.metadata),
        Ok(_) => {}
        Err(failure) => {
            eprintln!("\n[stream error]: {failure:?}");
            break;
        }
    }
}

There is no separate builder call to turn output streaming on. As long as the provider implements StreamProvider and the stream feature is enabled, a normal chat exposes stream right next to complete.

Sending input while it streams

To push more input while the model is still responding, switch the builder into the input-stream state with .with_input_stream(). The returned stream then carries a producer handle you can send to, for full-duplex exchanges. See Responses: input streaming for the full pattern.

StreamEvent variants

Every item the stream yields is a StreamEvent:

pub enum StreamEvent {
    TextChunk(String),
    ReasoningChunk(String),
    ToolCall(FunctionCall),
    ToolResult(FunctionResponse),
    Structured(serde_json::Value),
    Paused(PauseReason),
    Done(ChatResponse),
}

• TextChunk and ReasoningChunk are partial fragments. Concatenate them as they arrive.
• ToolCall fires when the model decides to call a tool, and ToolResult fires once that tool runs.
• Structured carries a whole serde_json::Value you can deserialize directly.
• Paused means a tool needs your action (approval, scheduling). The stream ends; resolve the tools on Messages and call stream again to continue.
• Done carries the same ChatResponse a non-streaming complete() call would have returned, so the final state stays equivalent either way.

When you also wire up tools, the loop surfaces tool lifecycle as it goes. This is handy for showing progress in a UI.

match event {
    StreamEvent::ToolCall(call) => println!("[calling {}]", call.name),
    StreamEvent::ToolResult(res) => println!("[{} finished]", res.name),
    _ => {}
}