Data Shield

Synced from the TurfAI source on 2026-06-21.

Feature Status: Limited GA (v0.5) Last Updated: 2026-06-08

Overview

Data Shield tokenises PII before a workflow, agent, or extraction call reaches the underlying LLM provider. The LLM sees opaque tokens (EMAIL_84d0a133, SSN_c95eb795); the caller sees the original values restored in the response. Tokenisation is deterministic within a request scope (the same email maps to the same token across messages in one call) but isolated across scopes (the same email in two different calls maps to two different tokens).

Every tokenisation event is recorded in an immutable audit log with entity counts per type — never raw values.

v0.5 scope: L1 detection (regex + format validation) across 8 entity types, per-request ephemeral scope, three LLM entry points covered. Customer-held KMS, persistent document scope, and regulator evidence-pack export are v1.0 deliverables.

What gets tokenised

Entity types covered (L1, always on when shield enabled):

Type	Detection
`EMAIL`	RFC-5322 subset
`PHONE`	E.164 + common US/UK/IN local formats
`SSN`	US SSN (rejects 000 / 666 / 9xx prefixes)
`CREDIT_CARD`	12–19 digits + Luhn validation
`IP`	IPv4
`AADHAAR`	Indian Aadhaar (12 digits + Verhoeff checksum)
`PAN`	Indian PAN (5 letters + 4 digits + letter)
`IBAN`	International bank account number

Entry points covered:

POST /api/v1/chat — direct LLM chat (used by llm_task, agent_task ReAct loop)
POST /api/v1/extract — structured extraction (used by extraction_task and agent file-mode)
Workflow agent_task (chat + extract call sites in the ReAct loop)

NOT covered in v0.5 (explicit gaps):

RAG chat path — questions and retrieved context are NOT tokenised before LLM (v1.0)
Chatbot public chat — RAG-only path; user message and retrieved context are NOT tokenised (v1.0)
File contents — file_urls are sent to the LLM as-is; in-boundary OCR + pre-LLM tokenisation of document text is a v1.0 deliverable (DATA_SHIELD_IMPLEMENTATION_PLAN.md §1.1)
Free-text PII — names, locations, organisations (PERSON / LOC / ORG) require L3 NER, which lands in v1.0. v0.5 only catches the 8 L1 types above
Persistent document scope — v0.5 scope is per-request only; cross-call consistency requires v1.0 KMS-encrypted scope storage

How to turn it on

Per-node (workflow author)

WorkflowBuilder → select an llm_task or agent_task node → scroll to the "Data Shield" card → flip the switch. Optionally restrict to a subset of entity types via the chip row.

Per-workflow (workflow author)

WorkflowBuilder → Resilience tab → "Data Shield policy" card → "Require Data Shield on every LLM node." When required, the executor fails any workflow whose LLM nodes haven't opted in (terminal error DATA_SHIELD_POLICY_VIOLATION).

Per-pack (pack author)

Declare data_shield_policy in the solution-pack manifest. Every workflow instantiated from a pack template inherits the pack policy onto its activity row; the workflow author can refine per-instance.

Audit log

Every shielded LLM call writes one row to the audit log (action: data_shield.tokenise). The row contains:

Counts of entities by type ({EMAIL: 1, SSN: 1}) — never raw values
Layers invoked (['L1'] in v0.5)
Stage latencies ({tokenise_ms: 1, roundtrip_ms: 1461})
Retention timestamp (default 30 days, tunable via DATA_SHIELD_AUDIT_RETENTION_DAYS)
Correlation ID linking back to the workflow run

End users can view their own activity at /account/activity. Operators can search across tenants via the admin audit dashboard. Rows are immutable after write (Strapi lifecycle hook + Phase E hash-chain columns pre-wired for v1.0 export).

Failure modes

Failure	Behaviour	Rationale
Detector error at ingress	Fail-closed — return 503, block call	False negative = PII breach; better to fail loudly
Audit emit failure	Fail-open — chat succeeds, audit row missing, warning logged	Audit is observability, not correctness
Workflow policy `required=true` + node didn't opt in	Workflow fails terminal with `DATA_SHIELD_POLICY_VIOLATION`	Author declared the contract; executor enforces
Unknown token in LLM response	Pass through untouched	Avoid false substitution from hallucinated tokens

Kill switches

Variable	Effect
`DATA_SHIELD_DISABLED=true` (llm-service)	Bypass gateway entirely — raw payload to LLM, no audit row. Incident-response use only.
`DATA_SHIELD_AUDIT_DISABLED=true` (llm-service)	Tokenise as normal, skip audit emit. Use if audit DB is the bottleneck.
`DATA_SHIELD_AUDIT_RETENTION_DAYS=N` (DMS)	Override default 30-day retention on new audit rows.

Roadmap

Version	Adds
v0.5 (current)	L1 + 3 entry points + per-request scope + audit log
v0.6	Doc-batch action paths, chatbot publicChat path, Prompt Lab path
v1.0	Customer KMS vault, persistent document scope, RAG path coverage, L2 gazetteer, L3 NER, evidence-pack export, in-boundary OCR
v1.5	L4 commercial detector, L5 local-LLM detector, sensitivity-based routing

See architecture/data-shield/spec.md for the full specification and architecture/data-shield/pack-policy.md (forthcoming) for pack-manifest syntax.

Pricing & packaging

To be determined. Track in compliance/data-shield-pricing.md when finalised.

On this page