TurfAI v2.0 — Operator Runbook

Synced from the TurfAI source on 2026-06-21.

Audience: SREs, on-call engineers, deploy operators Last updated: 2026-06-09 Scope: What changed at the operations layer in v2.0 — env vars, cron jobs, kill switches, incident playbooks. This is a delta on top of operations/env-variables.md and operations/deployment-checklist.md.

For customer-facing release notes, see releases/v2.0/release-notes.md.

1. What you must do before upgrade

These steps fail the deploy if missed.

1.1 Set required env vars (both repos)

The env-validator runs on Strapi boot and exits the process with a clear error if any of these are missing in NODE_ENV=production:

1.2 Set recommended-strong-default env vars

These have defaults but should be set explicitly in production for predictability:

1.3 SMTP configuration (release_2.0 C.4a)

Set these for branded transactional email. The plugin accepts both new and legacy names; new names take precedence:

Common gotcha: Most SMTP relays (Zoho, SendGrid, AWS SES) require From: to match the authenticated user. A mismatch produces 553 Sender is not allowed to relay emails. Verify with:

node dms/scripts/smoke-email-templates.js you@example.com

The script sends all 9 branded templates to the target address using raw nodemailer (does not require Strapi running).

2. New cron jobs

All run in DMS via config/cron-tasks.js. Check pm2 logs dms | grep cron if something looks off.

Manual triggers (under dms/scripts/):

• node scripts/run-token-usage-rollup-now.js [YYYY-MM-DD] — one-shot rollup with optional backfill date
• node scripts/run-quota-warner-now.js — one-shot warner pass
• node scripts/smoke-email-templates.js EMAIL — send all 9 branded templates to an address (smoke + cross-client check)

3. Kill switches

Set the env var, restart the affected service. Use only for incident response.

Verifying kill switches engage: llm-service/scripts/ds_kill_switch_verify.py — 4 scenarios with interactive prompts between env restarts.

4. New audit-log surfaces

All audit rows live in the single audit_log table (Wave 1.5). Immutable post-write via the M5.5 lifecycle hook; PII redacted via C.5 beforeCreate hook.

Querying: GET /api/admin/audit-log (Super-Admin only) — supports filters on action, actor_user, tenant, correlation_id, time range. End-user view: GET /api/audit-log/me — only the requester's own rows, action filter not exposed.

5. Common incident playbooks

5.1 Data Shield gateway returning 503s

Symptom: Workflows with Data Shield enabled start failing with 503 across the fleet.

Likely cause: A detector regex panicked (regex bomb, unicode edge case) and the gateway's fail-closed behaviour engaged.

Mitigation order:

1. Check pm2 logs llm-service | grep "data_shield ingress error" — should show the exception
2. Immediate stop-bleeding: DATA_SHIELD_DISABLED=true + restart llm-service. Workflows resume; raw payload reaches LLM; no audit rows during the outage.
3. Diagnose: isolate the offending request via correlation_id from the error logs; reproduce against staging
4. Restore: ship the detector fix, re-enable shield, run ds_kill_switch_verify.py baseline to confirm
5. Audit gap: the audit log will have a gap for the outage window. Document in the incident report.

5.2 INTERNAL_SERVICE_JWT_SECRET rotation

Why: Periodic rotation (recommended every 90 days) or post-incident.

Constraint: The secret is shared across DMS, processors, and llm-service. If they drift, internal calls fail.

Procedure:

1. Generate new secret: openssl rand -hex 32
2. Set new value in DMS env first (so it accepts both old + new JWTs during transition)
   • Requires code support for dual-secret rotation — NOT in v2.0; tracked in v2.1 backlog
   • v2.0 workaround: brief 2-minute window where minted tokens fail validation. Schedule for low-traffic window.
3. Set new value in processors + llm-service env
4. Restart all three services within 60 seconds of each other
5. Verify with ds_kill_switch_verify.py baseline — audit emit must succeed (proves JWT round-trip works end-to-end)

5.3 Audit emit failing (rows missing for shielded chat calls)

Symptom: Workflows with Data Shield enabled succeed, but no rows appear in audit_log for action=data_shield.tokenise.

Likely cause: JWT secret drift between llm-service and DMS, OR DATA_SHIELD_AUDIT_DISABLED=true left on from a prior incident.

Diagnose: pm2 logs llm-service | grep "audit:" — look for warn lines like audit emit failed: 401.

Fix:

• If 401: re-align INTERNAL_SERVICE_JWT_SECRET across services + restart
• If DATA_SHIELD_AUDIT_DISABLED is set: unset + restart llm-service

Important: Chat calls succeeded throughout the outage — Data Shield's correctness invariant is the tokenisation itself, not the audit row. The audit gap is documented in the incident report; replay is NOT possible (the request scope dies with the request frame).

5.4 Quota-warner not sending emails

Symptom: Customers crossing 80% don't get the soft-warn email.

Diagnose:

pm2 logs dms | grep "quota-warner"

Look for:

• [quota-warner] no caps configured — normal if no tenants/workflows have quotas set
• [quota-warner] email send failed — SMTP issue, see §5.5
• No log line at all → cron not running. Check strapi.cron registry.

Fix: node dms/scripts/run-quota-warner-now.js triggers a one-off pass. If that works manually, the cron is the issue — check Strapi logs at boot for cron registration errors.

5.5 SMTP delivery failing post-deploy

Symptom: pm2 logs dms | grep "Message failed" shows 553 Sender is not allowed to relay.

Likely cause: SMTP_FROM mismatch with SMTP_USER (the authenticated mailbox).

Fix: Set SMTP_FROM to exactly match SMTP_USER. Most relays require this for relay-policy compliance. Verify env reload by deleting + restarting the PM2 process (NOT pm2 restart — that preserves the cached process env):

cd dms && pm2 delete dms && pm2 start npm --name dms -- run develop

5.6 Workflow execution deadline killing legitimate long-running jobs

Symptom: Customer reports workflow failing with WORKFLOW_DEADLINE_EXCEEDED for runs that previously succeeded.

Diagnose: Check the workflow's execution_deadline_seconds. If unset (NULL), no deadline applies. If set, see if it's reasonable for the workflow's typical duration.

Fix: Either:

• Raise the per-workflow deadline (Resilience tab on the workflow builder)
• Set tenant-default to a higher value (Settings → Resilience → Default execution deadline)
• For one-off emergency: WORKFLOW_DEADLINE_DISABLED=true on processors + restart (kill switch)

5.7 GDPR export stuck in processing

Symptom: data_export_request row stays processing for more than 10 minutes.

Likely cause: Bundler worker crashed mid-build.

Self-healing: The _reclaimOrphans cron path flips orphans (>10min in processing) back to pending. Wait one tick.

Manual: If multiple cycles of reclaim fail, the bundle is hitting a real error. Check pm2 logs dms | grep "data-export-worker" for the underlying exception. Common cause: a corrupted entity's serialiser throwing. Fix the entity, retry.

5.8 Account-delete worker blocked by LEGAL_HOLD_USER_IDS

Symptom: User reports their delete request is executing for days without resolving.

Diagnose: Check the account-delete-request row. If error_message mentions LEGAL_HOLD_USER_IDS, the worker won't proceed.

Fix (legal sign-off required): Remove the user's ID from LEGAL_HOLD_USER_IDS env, restart Strapi, the worker picks the row up on next tick. Document the legal hold release in the incident log.

6. New observability hooks

6.1 Metrics endpoints

6.2 Grafana panels recommended

If you have a Grafana instance, the recommended panels:

• failure_events_total by classification (transient/terminal/user_error)
• failure_events_total by error_code (top-10) — flags spikes in specific failure modes
• data_shield_tokenise_ms p50/p95/p99 — should stay under 10ms p95
• data_shield_roundtrip_ms p95 — should be ≤ 5% above shield-off baseline
• quota_warnings_sent_total — should be small + non-zero
• audit_log_row_count — growth rate; spike = something looping

A wiring guide for the M2 + M3 stack is at dms/docs/SELF_SERVE_M2_INFRA_WIRING.md (sandbox repo).

6.3 Slack alert hooks

Set these to receive failures + service-status changes:

Failures fire one summary per (tenant, classification, 5-minute window) to avoid alert spam.

7. Backup + restore notes

No changes to backup/restore procedures in v2.0. The new tables added (audit_log, failure_events, token_usage_daily, workflow_drafts, service_health_probes, data_export_request, account_delete_request, chatbot_conversation, prompt_version) all backup with the standard Postgres dump.

GCS objects (data-export bundles) are NOT in the DB backup. They follow the GCS retention policy (7 days). If a customer needs to recover a deleted export beyond 7 days, regenerate by re-requesting the export.

8. Rollback procedure

v2.0 schema migrations are additive (no column drops, no row mutations on existing data). A rollback to v1.x is mechanically possible but operationally complex because of these one-way commitments:

Recommended rollback strategy: roll forward with a hotfix on release_2.0, not back to v1.x.

9. References

• Release notes (customer-facing)
• Env-variables inventory
• Deployment checklist
• Backup + restore runbook
• Data Shield feature · Data Shield spec
• DS load-test harness: llm-service/scripts/ds_load_test.py (backend repo)
• DS kill-switch verifier: llm-service/scripts/ds_kill_switch_verify.py (backend repo)
• Email-template smoke driver: dms/scripts/smoke-email-templates.js (backend repo)

Update this runbook with new incident playbooks as they're learned in production.