Data Shield

Synced from the TurfAI source on 2026-09-14 · 7c101c0.

Feature Status: GA (v3.0) Last Updated: 2026-07-05

Overview

Data Shield replaces sensitive values with opaque tokens before a workflow, agent, or extraction call reaches the underlying LLM provider. The LLM sees tokens (EMAIL_84d0a133, EMPLOYEE_ID_af7bbe5a); the caller sees the original values restored in the response. Tokenisation is deterministic within a request scope (the same email maps to the same token across messages in one call) but isolated across scopes (the same email in two different jobs maps to two different tokens). When the scope is destroyed at request end, the reverse map is destroyed too — the tokens become irreversible.

Every tokenisation event is recorded in an immutable audit log with entity counts per type — never raw values.

v3.0 scope: eight L1 detectors (regex + validators), L2 gazetteer (Aho-Corasick word lists), L3 NER (Microsoft Presidio) with customer- authorable extensions, tenant-scoped detector storage, three admin surfaces (Simple mode wizard / Advanced mode tabs / marketplace-style pack catalogue), live tokenisation demo, gateway-owned audit forwarding.

For the customer-facing five-minute version, see data-shield-one-pager.md.

Architecture

Data Shield runs as a separate internal-only service (data-shield-gateway) next to llm-service. In development it binds 127.0.0.1:9008 — kernel- enforced, unreachable from the LAN. In production it's a peer service on the same cluster (no ingress). llm-service reaches it over HTTP via a thin data_shield_client.py facade.

Workflow ─▶ llm-service ─▶ /v1/tokenise ─▶ gateway ─▶ LLM provider
                              (127.0.0.1:9008)
Workflow ◀─ llm-service ◀─ /v1/detokenise ◀─ gateway ◀─ LLM response
                                              │
                                              ▼
                                       /v1/audit/emit ─▶ DMS audit_log

The gateway hosts:

• L1 built-in detectors (regex + Luhn / Verhoeff / SSA-rule validators)
• L2 gazetteer engine (pyahocorasick, word-boundary matching)
• L3 Presidio analyzer (spaCy small model by default; large model via DATA_SHIELD_SPACY_MODEL env var for higher NER precision at +570 MB)
• Per-tenant detector cache (60s TTL, fail-soft on DMS unreachable)
• Audit emit path (gateway signs DMS service JWT — llm-service does not hold audit credentials, task #189)

The DATA_SHIELD_MODE env var switches llm-service between http (route via gateway, target end-state) and in_process (legacy path using the old in-line detector module — kept as rollback for one more release).

What gets tokenised

Built-in detectors (always on when shield enabled)

L1 — regex + format validators (highest priority):

L3 — Presidio NER (lower priority; L1 wins overlap):

Presidio's built-in recognizers for entity types already handled by L1 (EMAIL_ADDRESS, PHONE_NUMBER, US_SSN, IP_ADDRESS, CREDIT_CARD, IBAN_CODE) are suppressed at the analyzer layer — L1 always wins those overlaps by design, so running the L3 pass would just waste CPU.

Customer-authorable extensions

All three layers are tenant-scoped and can be extended without a deploy:

• L1 custom patterns — a regex + entity type name. Priority configurable (default 100 — beats built-in L3). Ideal for tenant- authored identifiers (ticket IDs, policy numbers, internal SKUs).
• L2 gazetteers — paste a word list (competitor names, product codes, banned vocabulary). Case-insensitive by default; word-boundary matched (no partial hits). Priority 75.
• L3 custom Presidio recognizers — a regex plus optional context words that boost the score when they appear near the match. Priority 50 by default; per-recognizer score threshold configurable.

Detector packs

Three domain packs ship pre-authored and install in one click per tenant. Install materialises the pack's definitions into the tenant-scoped tables (custom patterns / gazetteers / Presidio recognizers) and records what was created so uninstall can reverse cleanly.

Packs are super-admin-published globally; every tenant admin sees them in the Packs tab and can install / uninstall in-place. Edits or additions made after install are preserved on uninstall.

Overlap resolution

When two detectors match overlapping text, the higher-priority one wins. Ties break on longer span, then higher confidence. So a Luhn-validated CREDIT_CARD (L1, priority 100) always beats a Presidio PERSON guess (L3, priority 50) on the same span; a customer TICKET_ID regex at priority 100 wins over the L3 ORGANIZATION false positive.

Entry points covered

Shielded execution history no longer persists raw prompts. When data_shield.enabled is true, the _prompt_diagnostics field on executions carries the tokenised prompt, not the raw one (turfai@a0ba2a5). Historical rows (pre-fix) are unchanged — a Node backfill script for prod cutover is in the release_v3 upgrade runbook.

Files and media (honest limits)

Shielded workflows that send file_urls to multimodal LLMs (Vertex Gemini, GPT-4o with attachments) do not have their file contents scanned. The file goes to the LLM directly — the shield never reads it.

In v3.0, POST /api/v1/extract returns HTTP 400 with a clear error message when data_shield.enabled is true and file_urls is non-empty. Callers have two supported paths:

1. Extract text from the file first (via extraction_task on the unshielded path, or client-side OCR), then pass the extracted text through content — the extracted text will be shielded.
2. Disable Data Shield for that workflow and accept that files are not scanned.

Full strategy (v3.5 OCR-inside-the-shield-boundary + per-workflow attestation opt-out for multimodal fidelity cases) is in RELEASE_V3_FILE_MEDIA_SHIELD_STRATEGY.md.

Admin surfaces

Every tenant admin has three UIs to manage their shield:

/settings/data-shield — Simple mode (default)

• Quick starts card — pack tiles with one-click install (and an "Install all" button when 2+ packs are uninstalled)
• Masked in this workspace card — unified list of every detector the tenant has, with plain-language labels (EMPLOYEE_ID shown as "Employee ID", SWIFT_CODE as "SWIFT / BIC"). Enable toggle and remove per row.
• Add something to mask — two-field wizard: name + 2-3 example values. Server-side inference picks regex (when values look like a code with common prefix) or gazetteer (when values look like a fixed list). Live "try it on some text" preview before save.
• Installed quick starts card — per-pack uninstall.

Priority, layer name, raw regex are all hidden by default. A tenant admin can flip to Advanced any time — the preference persists in localStorage['ds-settings-mode'].

/settings/data-shield — Advanced mode

• Explainer alert — three-item plain-language definition of what patterns, word lists, and smart recognizers do and when to pick which.
• Packs tab (default) — full pack detail (vertical chip, version, publisher, detector count, description). Same install/uninstall.
• Custom Patterns tab — the raw form (name, entity_type, regex, priority, description). Browser-side regex compile check.
• Word Lists tab — entries editor accepting newline / CSV / JSON array paste with dedup + preview; case-insensitive toggle.
• Smart Recognizers tab — pattern + context words + score.

/data-shield/demo

• Side-by-side raw input vs tokenised output. Tokens shown as colour-coded badges (entity type → deterministic hash colour).
• Cross-vertical sample text pre-loaded so any tenant with any of the 3 packs installed sees masking fire immediately.
• Live entity_counts chips + tokenise latency.
• Recent shield events panel (polls /api/data-shield/recent-events every 5s) showing real audit events across the tenant. Native SSE arrives when the events stream lands as a follow-up slice.

How to turn shielding on

Per-node (workflow author)

WorkflowBuilder → select an llm_task, extraction_task, classification_task, summarization_task, agent_task or rag_query_task node → scroll to the "Data Shield" card → flip the switch. Optionally restrict to a subset of entity types via the chip row.

Per-workflow (workflow author)

WorkflowBuilder → Resilience tab → "Data Shield policy" card → "Require Data Shield on every LLM node." When required, the executor fails any workflow whose LLM nodes haven't opted in (terminal error DATA_SHIELD_POLICY_VIOLATION).

Per-pack (pack author)

Declare data_shield_policy in the solution-pack manifest. Every workflow instantiated from a pack template inherits the pack policy onto its activity row; the workflow author can refine per-instance.

Audit log

Every shielded LLM call writes one row (action: data_shield.tokenise) to the DMS audit_log table. In v3.0, the gateway signs its own internal service JWT and forwards the event — llm-service does not hold DMS credentials for this path.

Row contents:

• Entity counts by type ({EMAIL: 1, POLICY_NUMBER: 1}) — never raw values
• Layers invoked (['L1'] in practice today; L2/L3 wiring in place for future events)
• Stage latencies ({tokenise_ms: 25, roundtrip_ms: 2665})
• Model + provider (gemini-2.0-flash / vertex)
• Retention timestamp — default 30 days, tunable via DATA_SHIELD_AUDIT_RETENTION_DAYS
• Correlation ID linking back to the workflow run
• prev_hash / event_hash columns pre-wired for the Phase E evidence-pack chain (v3.5 populates them)

End users see their own activity at /account/activity. Operators search across tenants via the admin audit dashboard at /audit. Rows are immutable after write (Strapi lifecycle hook + C.5 PII redaction runs on beforeCreate). The /data-shield/demo events panel polls the last N events for the current tenant.

Failure modes

Kill switches and configuration

Roadmap

Related documents

• data-shield-one-pager.md — customer- facing five-minute version
• Sandbox docs/RELEASE_V3_PLAN.md — full release_v3 plan and slice history
• Sandbox docs/RELEASE_V3_FILE_MEDIA_SHIELD_STRATEGY.md — file and media handling strategy (v3 fail-fast + v3.5 roadmap)
• architecture/data-shield/spec.md — full technical specification

Pricing & packaging

To be determined. Track in compliance/data-shield-pricing.md when finalised.