Prompt Lab

What it is

Prompt Lab is the interactive workspace for designing, testing, and saving AI prompts. You test an extraction prompt against a real document, optionally define a structured JSON output schema, switch between models (Vertex AI, OpenAI, Anthropic), let AI enhance the prompt into RTIO structure, and save the result to the prompt library for reuse by tasks, agents, and squads.

Two related surfaces work together:

• Prompt Lab — the synchronous test-and-tune surface. It runs a single prompt against one document and returns the structured result immediately so you can iterate.
• Prompt Library — the persistent store of reusable prompts (the prompt content type). An agent references a saved prompt by its title via prompt_title; an extraction task node references one the same way.

When to use it

Use Prompt Lab whenever you are authoring or tuning a prompt before wiring it into an extraction task or an agent. It is where you iterate on prompt quality against real inputs instead of guessing — refine the wording, tighten the output schema, and compare models side by side until the extraction is reliable.

RTIO — Role, Task, Input, Output

RTIO is TurfAI's structured prompt format. Rather than one free-form blob of instructions, a prompt is organised into clear sections so the model knows who it is, what to do, what it is looking at, and how to answer:

• Role — the persona/expertise the model adopts ("You are an expert document extraction assistant…"). Stored as the system entry in the prompt's roles.
• Task — what to extract, in plain language. Stored as the user entry in roles, plus the discrete steps in tasks.
• Input — the document content. At run time the platform injects it; enhanced prompts use a {{document_content}} placeholder in the user role.
• Output — the required answer shape: outputFormat (text / json / markdown / structured) plus the output schema and exampleOutput that pin the JSON structure.

Structuring a prompt this way reduces ambiguity, which is what makes extraction repeatable across documents.

AI-powered RTIO enhancement

Prompt Lab can turn a rough prompt into polished RTIO automatically. You provide a title, your draft prompt, and (optionally) an output schema; the platform calls the LLM with a prompt-engineer meta-prompt and returns a structured { roles, tasks, instructions } object you can review and edit before saving.

There are two distinct generation paths in the platform:

1. Enhance RTIO (POST /api/prompt-lab/enhance-rtio) — interactive polishing inside Prompt Lab. Takes your prompt, output_schema, title, description; returns roles (system + user with a {{document_content}} placeholder), tasks (3–5 steps), and instructions (3–5 formatting/accuracy rules).
2. RTIO generation from classification — an automated pipeline task that builds a full prompt (title, description, section, roles, tasks, instructions, outputFormat, exampleOutput) from a document plus its classification result. Generated prompts are stamped level: "user" and version: "1.0.0" with a default confidence of ~0.85.

RTIO also shows up elsewhere: the agent prompt_title field (see Agents) and extraction nodes in the Node catalog both resolve a saved RTIO prompt by title.

How it works

Process flow — iterate then save:

Call flow — a structured test run. With an output schema, the controller wraps your prompt in an RTIO template (Role / Task / Output Format / Rules) before sending it; without one it sends the prompt as free-form:

Sub-features

• Interactive testing against a real document, returning the result synchronously.
• Output schema — supply a JSON shape; the prompt is wrapped in a structured RTIO template.
• Free-form mode — omit the schema to send the prompt as-is for exploratory extraction.
• Multi-model support — Vertex AI (vertex, default), OpenAI, Anthropic.
• AI enhancement — auto-polish a draft into RTIO { roles, tasks, instructions }.
• Save to library — convert a tested prompt into a reusable library entry.
• Prompt library — system (shared) vs user (private) prompts, with title, description, section, tags, and versioning.

Configuration parameters

Prompt Lab test run (/api/prompt-lab/execute)

RTIO enhancement (/api/prompt-lab/enhance-rtio) takes prompt (required), title (required), description, and output_schema, and returns roles, tasks, and instructions.

Saved prompt (the prompt library entry)

Reconciled against the prompt content-type schema. These are the fields a saved library entry carries:

Tips & best practices

• Write the Task before the Role. Decide exactly which fields you need, then add a Role that primes the model for that domain. A vague Role ("you are helpful") adds nothing; a specific one ("expert AP invoice analyst") measurably improves field accuracy.
• Always define an output schema for extraction. Without it the run is free-form and the model may return prose. With a schema, the controller wraps your prompt with explicit rules: return only valid JSON, use null for missing fields, use [] for empty arrays.
• Match field names in the prompt to the schema. If the schema key is invoice_total, say "invoice total" in the Task — divergent wording is the most common cause of empty fields.
• System vs user roles. Put stable, reusable framing (persona, global rules) in the system role; put the document-specific extraction ask in the user role. Enhanced prompts place a {{document_content}} placeholder in the user role.
• Library visibility: level. Keep work-in-progress prompts at level: user (private to the owner). Promote to level: system only once a prompt is stable and meant to be shared — a system prompt is visible to everyone, so churn there is disruptive.
• Versioning. Bump version when you change extraction behaviour rather than overwriting silently, so agents/nodes referencing the prompt by title have a traceable history. Use forked_from when branching an existing prompt instead of editing a shared one.
• Mind the review gate. A draft prompt is blocked from agent/workflow execution; move it through in_review to published before depending on it in production.

Concrete examples

An output schema for invoice extraction — supply this as output_schema on the test run:

{
  "vendor_name": "string",
  "invoice_number": "string",
  "invoice_date": "string (YYYY-MM-DD)",
  "due_date": "string (YYYY-MM-DD)",
  "currency": "string",
  "line_items": [
    { "description": "string", "quantity": "number", "unit_price": "number", "amount": "number" }
  ],
  "subtotal": "number",
  "tax": "number",
  "total": "number"
}

A system + user prompt pair (RTIO roles) as stored on a saved prompt:

{
  "title": "Extract Invoice Details",
  "section": "Accounts Payable",
  "outputFormat": "json",
  "roles": [
    {
      "role": "system",
      "prompt": "You are an expert accounts-payable analyst. You extract structured data from invoices precisely and never invent values."
    },
    {
      "role": "user",
      "prompt": "Extract the vendor, invoice number, dates, line items, and totals from the invoice below.\n\n{{document_content}}"
    }
  ],
  "tasks": [
    "Identify the vendor and invoice number",
    "Extract invoice date and due date",
    "Extract each line item with quantity, unit price, and amount",
    "Extract subtotal, tax, and total"
  ],
  "instructions": [
    "Return only valid JSON matching the output schema",
    "Use null for any field you cannot find",
    "Return an empty array for line_items if none are present",
    "Match numbers exactly as printed; do not recalculate"
  ]
}

A Prompt Lab test request (structured run against document 42 on Anthropic):

POST /api/prompt-lab/execute
{
  "document_id": 42,
  "prompt": "Extract the vendor, invoice number, dates, line items and totals.",
  "output_schema": { "vendor_name": "string", "total": "number", "invoice_date": "string" },
  "model_type": "anthropic"
}

The response includes result (the structured content), tokens_used, latency_ms, model, and the document it ran against.

Dependencies

• DMS prompt-lab API — execute and enhance-rtio controllers; enforces document ownership/visibility before running.
• DMS prompt service — library CRUD, level-based visibility, and the review/lint governance fields.
• LLM service — runs the extraction at /api/v1/extract across the selected provider.
• Document management — supplies the test document (the controller passes the document's GCS URL straight to the model, which handles OCR for scanned files internally).

Limitations

• Quality is prompt-dependent; complex extraction schemas may need several iterations.
• Provider availability depends on which LLM credentials the deployment has configured.
• The extraction call has a long timeout (up to ~10 minutes) for large documents — a single test run is not instantaneous on big files.
• The internal RTIO enhancement meta-prompt and confidence-scoring logic are not yet documented; treat enhanced output as a reviewable draft.
• Batch testing (running a prompt across many documents at once) is coming soon — it is not part of the current synchronous single-document Prompt Lab.

Troubleshooting

How to test

1. Create a draft prompt. In Prompt Lab, give it a title (e.g. Extract Invoice Details) and write a plain-language Task describing the fields you need.
2. Add an output schema. Paste the invoice schema above so the run is structured rather than free-form.
3. Test against a sample document. Pick a sample invoice and run POST /prompt-lab/execute with your prompt, output_schema, and a model_type. Inspect the returned JSON.
4. Refine. Where fields come back empty or wrong, adjust wording and re-run; switch model_type to compare providers on the same document.
5. (Optional) Enhance. Call enhance-rtio to polish the draft into structured roles/tasks/instructions, then review the result.
6. Save to the library. Persist the tuned prompt; set level (user while iterating, system once shared) and bump version on later changes.
7. Reference it. Point an agent's prompt_title or an extraction task node at the saved title, and confirm the workflow produces the same structured output as your Prompt Lab run.

Related

• Agents — consume saved prompts via prompt_title.
• Node catalog — extraction nodes reference library prompts.
• Task Node Mechanics — how nodes resolve inputs and prompts.
• Solution Packs — bundle prompts (pack_slug / solution_pack).
• Agentic automation and Governance — where prompts fit in the platform and the review gate.

Roadmap

• Batch testing across multiple documents — coming soon.
• Documented RTIO enhancement algorithm and confidence scoring — not yet documented.

APIs used