Streaming, progress and cancellation

Streaming answers

askStream() / ask_stream() yield the answer’s text as the LLM writes it. They take the same options as ask().

TypeScript

for await (const delta of doc.askStream("Explain the introduction")) {
  process.stdout.write(delta);
}

Python

for delta in doc.ask_stream("Explain the introduction"):
    print(delta, end="", flush=True)
print()

A stream yields text only. To get citations with a streamed answer, collect the text and call extractCitations(text, hits) (TypeScript) or extract_citations(text, hits) (Python) with the passages from search(), or use ask(). Over HTTP, send "stream": true to /ask; see REST API.

Build progress

Chunks are embedded in batches of 64. The progress callback runs after each batch with the document’s source, the chunks embedded so far, and the total.

TypeScript

await collection.build({
  onProgress: ({ source, embedded, total }) => console.log(`${source}: ${embedded}/${total}`),
});

Python

collection.build(
    on_progress=lambda p: print(f"{p['source']}: {p['embedded']}/{p['total']}"),
)

A document whose saved index is reused reports no progress: it needs no embedding.

Cancelling a build

A cancelled build stops between embedding batches and keeps the previous index: the old index is replaced only once all chunks are embedded.

TypeScript

Pass an AbortSignal. The build throws an AbortError.

const controller = new AbortController();
setTimeout(() => controller.abort(), 30_000);

try {
  await collection.build({ signal: controller.signal });
} catch (err) {
  if ((err as Error).name !== "AbortError") throw err;
}

Python

Pass a threading.Event, and set it from another thread. The build raises BuildCancelledError.

import threading
from raglite import BuildCancelledError

cancel = threading.Event()
threading.Timer(30, cancel.set).start()

try:
    collection.build(cancel=cancel)
except BuildCancelledError:
    pass

Cancelling a request

TypeScript

ask() and askStream() take the same signal, which cancels the LLM request:

await doc.ask("Summarize the document", { signal: AbortSignal.timeout(20_000) });

Python

Stop reading from ask_stream() to stop a streamed answer. ask() has no cancel option.