Knowledge Base / RAG

What it is

The Knowledge Base is TurfAI's RAG layer: enable indexing on a document and it becomes searchable by meaning, not just keywords. Natural-language queries return top-K passages with similarity scores and an AI-synthesized answer carrying source citations (document title, snippet, score, and a signed URL to the source). For the concept, see RAG; this page covers the module.

When to use it

Use it whenever AI needs to answer from your documents — policy Q&A, support knowledge bases, contract lookups — or when an agent needs a search_documents tool. It also powers the Chatbot.

How it works

Process flow — index then query:

Call flow — a RAG query:

Sub-features

• Document indexing — async with status (queued → processing → completed, or failed), force_reprocess, chunk count (rag_chunk_count), embedding-model metadata.
• Semantic search — top-K with similarity scoring, optional reranking, configurable similarity threshold.
• Conversation sessions — persistent multi-turn chat with session ids, history, deletion, titles.
• Collections — group documents into named collections and scope queries to them.
• Two backends — Google File Search (default) and pgvector (legacy), selected per deployment.

Configuration parameters

RAG config has two layers: deployment/indexing config (set by ops via env vars, fixed for all queries) and per-query config (passed on each /api/rag/query call).

Indexing & backend (deployment-level)

Per-query parameters (POST /api/rag/query)

Tips & best practices

• Use a collection when you have a distinct knowledge domain (one product, one customer, one policy set). Scope queries with filters.collection_ids so unrelated documents can't leak into answers.
• Document hygiene: prefer machine-readable PDFs, DOCX, TXT, or Markdown; keep one topic per document; re-run with force_reprocess=true after meaningful edits so the chunk index matches the live file.
• Choosing a backend: start with Google File Search (API key only, embeddings handled by Gemini, ~90% cheaper, auto-scaling). Move to pgvector only when you need on-prem/full control of the vector store.
• Tuning similarity_threshold: raise it (e.g. 0.5–0.6) when answers pull in loosely related passages; lower it (e.g. 0.2–0.3) when relevant documents are being missed — especially for multi-lingual content. Turn on use_reranking before raising the threshold; reranking usually fixes "right docs, wrong order" without dropping recall.
• Citations: every answer ships sources[] with document_title, chunk_text, similarity_score, page_number, and a 1-hour signed_url. Surface these in the UI so users can verify the answer; if sources is empty, treat the answer as ungrounded.

Concrete examples

1. Enable RAG, then poll status

# Enable indexing on document 42
POST /api/documents/42/enable-rag
{ "force_reprocess": false }
# → { "status": "queued", "job_id": "rag-embed-42-1718800000000", "document_id": "42" }

# Poll until completed
GET /api/documents/42/rag-status
# → { "processing_status": "queued",     "chunk_count": 0 }   # then...
# → { "processing_status": "processing", "chunk_count": 0 }   # then...
# → { "processing_status": "completed",  "chunk_count": 37 }
# status values: not_started | queued | processing | completed | failed

2. Run a RAG query

POST /api/rag/query
{
  "query": "What is the notice period for terminating the MSA?",
  "top_k": 5,
  "similarity_threshold": 0.3,
  "use_reranking": true,
  "filters": { "collection_ids": [7] }
}

Response (shape from QueryResponse / Source):

{
  "answer": "The MSA requires 60 days' written notice to terminate for convenience [1].",
  "sources": [
    {
      "document_id": 42,
      "document_title": "Master Services Agreement.pdf",
      "chunk_text": "...either party may terminate this Agreement for convenience upon sixty (60) days' prior written notice...",
      "chunk_index": 12,
      "similarity_score": 0.71,
      "page_number": 8,
      "file_url": "gs://turfai-docs/42/msa.pdf",
      "signed_url": "https://storage.googleapis.com/turfai-docs/42/msa.pdf?X-Goog-Expires=3600..."
    }
  ],
  "confidence": 0.82,
  "session_id": "sess_a1b2c3",
  "processing_time_ms": 940
}

3. Multi-turn session — reuse the returned session_id on the next call to keep context:

POST /api/rag/query
{ "query": "And what about termination for cause?", "session_id": "sess_a1b2c3" }

The service resolves "what about..." against the prior turn. Omit session_id and a new session is auto-created; list/inspect/delete them via /api/rag/sessions.

Limitations

• Standalone images (JPEG/PNG) may not be supported by Google File Search — convert to PDF first.
• Google File Search auto-OCR of scanned PDFs is not explicitly documented by Google. Gemini's native vision usually reads them, but test before relying on it; in-boundary OCR for scanned PDFs is coming soon (today's fallback is TurfAI OCR → pgvector).
• Per-document file limits (File Search): up to 100 MB / 1000 pages; pages scaled to 3072×3072.
• Data Shield does not yet tokenise the RAG chat path (v0.5) — do not send untokenised PII through RAG chat expecting masking.
• Persistent / cross-call session scope beyond the current store is coming soon.

Troubleshooting

How to test

End-to-end happy path:

1. Upload a machine-readable PDF (DOCX/TXT/MD also fine).
2. Enable RAG: POST /api/documents/:id/enable-rag; expect status: queued.
3. Wait for completed: poll GET /api/documents/:id/rag-status until processing_status is completed and chunk_count > 0. If it lands on failed, read rag_error.
4. Query: POST /api/rag/query with a question whose answer is in the document.
5. Confirm a cited answer: the response has a non-empty answer and sources[] with the right document_title, a page_number, a similarity_score above your threshold, and a working signed_url.
6. Multi-turn: send a follow-up question with the returned session_id and confirm the answer respects the earlier turn's context.
7. Collection scope (optional): add the document to a collection and confirm filters.collection_ids returns it while excluding others.

Cross-links & roadmap

• Chatbot — public/embedded chat surface built on this layer.
• RAG concept — chunking, embeddings, and retrieval explained.
• Data Shield — note: RAG chat is not tokenised by Data Shield in v0.5.
• Coming soon: in-boundary OCR for scanned PDFs; persistent (cross-call) session scope.

Dependencies

• RAG query service (Python) — embeddings, vector search, conversation memory, citations.
• RAG embeddings worker — chunks + embeds documents off a Redis queue.
• Vector store — Google File Search or PostgreSQL + pgvector.
• LLM / embedding service — Gemini via Vertex AI (File Search) or TurfAI LLM service (pgvector).
• Document management + GCS — source documents and 1-hour signed URLs.

APIs used

See the synced feature reference for the full RAG surface.