TurfAI Deployment Checklist

Synced from the TurfAI source on 2026-06-21.

Overview

This checklist ensures a complete and correct deployment of TurfAI to either local development environment or Google Cloud Run. Follow each step in order and verify completion before proceeding.

Last Updated: November 4, 2025 Target Environments: Local Development, Google Cloud Run Estimated Time: 45-60 minutes (first-time), 20-30 minutes (subsequent)

Pre-Deployment Phase
Infrastructure Setup
Service Deployment
Database Seeding
Verification
Post-Deployment
Rollback Procedures

Pre-Deployment Phase

1.1 Environment Preparation

Clone repository to local machine
Checkout correct branch/tag for deployment
Verify Git status is clean (no uncommitted changes)
Review recent changes in CHANGELOG or git log

1.2 Tools and Access

Required Tools:

gcloud CLI installed and authenticated
docker installed and running
psql client installed (for database verification)
redis-cli installed (for Redis verification)
node and npm installed (v18+ for DMS)
python installed (3.8+ for services)

Access Verification:

# Check gcloud authentication
gcloud auth list
gcloud config get-value project

# Check Docker
docker --version
docker ps

# Check database tools
psql --version
redis-cli --version

Verified gcloud authentication
Verified Docker is running
Verified database tools installed

1.3 Configuration Review

Review .env.example for new variables
Check deploy/.config/ for existing configuration
Review deployment scripts for recent changes
Verify documentation is up-to-date

Infrastructure Setup

2.1 Google Cloud Project Setup (Cloud Only)

Set Project:

export PROJECT_ID=your-project-id
gcloud config set project $PROJECT_ID

GCP project created and set
Billing enabled for project
Appropriate IAM permissions for deployment

2.2 Enable Required APIs

Run API enablement script:

bash deploy/infra/00-enable-apis.sh

Or manually enable:

gcloud services enable cloudsql.googleapis.com
gcloud services enable redis.googleapis.com
gcloud services enable storage-api.googleapis.com
gcloud services enable vpcaccess.googleapis.com
gcloud services enable run.googleapis.com
gcloud services enable containerregistry.googleapis.com
gcloud services enable aiplatform.googleapis.com
gcloud services enable vision.googleapis.com
gcloud services enable secretmanager.googleapis.com
gcloud services enable iam.googleapis.com
gcloud services enable logging.googleapis.com
gcloud services enable monitoring.googleapis.com

Checklist:

Verification:

gcloud services list --enabled --filter="name:cloudsql OR name:redis OR name:storage OR name:run OR name:vision OR name:aiplatform"

2.3 Create Cloud SQL Instance

Run script:

bash deploy/infra/01-create-cloud-sql.sh

Manual steps (if needed):

# Create instance
gcloud sql instances create turfai-db \
    --database-version=POSTGRES_15 \
    --tier=db-f1-micro \
    --region=us-central1

# Create database
gcloud sql databases create turfai_dms --instance=turfai-db

# Set password
gcloud sql users set-password postgres \
    --instance=turfai-db \
    --password=<generated-password>

# Store password in Secret Manager
echo -n "<password>" | gcloud secrets create turfai-db-password --data-file=-

Checklist:

Cloud SQL instance created (turfai-db)
Database created (turfai_dms)
PostgreSQL version 15
Password generated and stored in Secret Manager
Configuration saved to deploy/.config/cloudsql.env

Enable pgvector extension:

# Connect via Cloud SQL Proxy
cloud_sql_proxy -instances=PROJECT_ID:us-central1:turfai-db=tcp:5432 &

# Connect and enable extension
psql -h localhost -U postgres -d turfai_dms -c "CREATE EXTENSION IF NOT EXISTS vector;"

pgvector extension enabled
Verified: SELECT * FROM pg_extension WHERE extname='vector';

2.4 Create Redis Memorystore

Run script:

bash deploy/infra/02-create-redis.sh

Manual steps (if needed):

gcloud redis instances create turfai-redis \
    --size=1 \
    --region=us-central1 \
    --redis-version=redis_7_0

# Get internal IP
gcloud redis instances describe turfai-redis --region=us-central1 --format="value(host)"

Checklist:

2.5 Create Cloud Storage Bucket

Run script:

bash deploy/infra/03-create-storage.sh

Manual steps (if needed):

gsutil mb -l us-central1 gs://turfai-documents-$PROJECT_ID
gsutil uniformbucketlevelaccess set on gs://turfai-documents-$PROJECT_ID

Checklist:

GCS bucket created (turfai-documents-PROJECT_ID)
Region: us-central1
Uniform bucket-level access enabled
Configuration saved to deploy/.config/storage.env

2.6 Create Service Account

Run script:

bash deploy/infra/04-create-service-account.sh

Manual steps (if needed):

# Create service account
gcloud iam service-accounts create turfai-services \
    --display-name="TurfAI Services"

# Grant roles
gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:turfai-services@$PROJECT_ID.iam.gserviceaccount.com" \
    --role="roles/cloudsql.client"

gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:turfai-services@$PROJECT_ID.iam.gserviceaccount.com" \
    --role="roles/storage.objectAdmin"

gcloud projects add-iam-policy-binding $PROJECT_ID \
    --member="serviceAccount:turfai-services@$PROJECT_ID.iam.gserviceaccount.com" \
    --role="roles/aiplatform.user"

# Grant IAM signBlob permission for signed URLs
gcloud iam service-accounts add-iam-policy-binding \
    turfai-services@$PROJECT_ID.iam.gserviceaccount.com \
    --member="serviceAccount:turfai-services@$PROJECT_ID.iam.gserviceaccount.com" \
    --role="roles/iam.serviceAccountTokenCreator"

# Create key (for local development)
gcloud iam service-accounts keys create deploy/.config/service-account-key.json \
    --iam-account=turfai-services@$PROJECT_ID.iam.gserviceaccount.com

Checklist:

Service account created (turfai-services)
Role: cloudsql.client granted
Role: storage.objectAdmin granted
Role: aiplatform.user granted
Role: iam.serviceAccountTokenCreator granted
Service account key created (for local dev)
Configuration saved to deploy/.config/service-account.env

2.7 Create VPC Connector (Optional - for VPC access)

Run script:

bash deploy/infrastructure/02-create-vpc-connector.sh

VPC connector created (if needed for Redis access)

2.8 Generate Shared Secrets

Create shared secrets file:

cd deploy/.config

# Generate JWT secret (MUST be same across all services)
JWT_SECRET=$(openssl rand -base64 32)

# Generate Router API Key
ROUTER_API_KEY=$(uuidgen)

# Generate Strapi secrets
APP_KEY_1=$(openssl rand -base64 32)
APP_KEY_2=$(openssl rand -base64 32)
APP_KEY_3=$(openssl rand -base64 32)
APP_KEY_4=$(openssl rand -base64 32)
API_TOKEN_SALT=$(openssl rand -base64 32)
ADMIN_JWT_SECRET=$(openssl rand -base64 32)
TRANSFER_TOKEN_SALT=$(openssl rand -base64 32)

# Save to shared-secrets.env
cat > shared-secrets.env <<EOF
# Shared Secrets - Generated: $(date)
# CRITICAL: Keep secure, never commit to git

# JWT Authentication (MUST be identical across all services)
JWT_SECRET=$JWT_SECRET
STRAPI_JWT_SECRET=$JWT_SECRET

# Router API Authentication
ROUTER_API_KEY=$ROUTER_API_KEY

# Strapi Secrets
APP_KEYS="$APP_KEY_1,$APP_KEY_2,$APP_KEY_3,$APP_KEY_4"
API_TOKEN_SALT=$API_TOKEN_SALT
ADMIN_JWT_SECRET=$ADMIN_JWT_SECRET
TRANSFER_TOKEN_SALT=$TRANSFER_TOKEN_SALT
EOF

chmod 600 shared-secrets.env

Checklist:

Service Deployment

3.1 Deploy DMS (Document Management System)

Run deployment script:

bash deploy/dms/01-deploy-dms.sh

What it does:

Builds Docker image with Strapi
Pushes to Google Container Registry
Deploys to Cloud Run with all configuration
Sets up database connection
Configures storage and Redis
Exposes publicly accessible endpoint

Checklist:

DMS Docker image built successfully
Image pushed to GCR: gcr.io/PROJECT_ID/turfai-dms:latest
Cloud Run service deployed: turfai-dms
Health check passing: curl https://SERVICE_URL/_health
Service URL saved to deploy/.config/dms-deployment.env

Manual verification:

# Get service URL
gcloud run services describe turfai-dms --region=us-central1 --format="value(status.url)"

# Test health
curl https://SERVICE_URL/_health

# Check logs
gcloud run logs read turfai-dms --region=us-central1 --limit=20

Service URL obtained and accessible
Logs show no errors
Database connection successful

3.2 Seed DMS Database

IMPORTANT: Run this immediately after DMS deployment!

Run seeding script:

bash deploy/dms/02-seed-database.sh

What it creates:

Admin user (username: admin, password: TurfAIAdmin123!)
User roles and permissions
Sample prompts
Dashboard permissions
Initial configuration

Checklist:

Seeding script completed without errors
Admin user created
User roles created (Authenticated, Public, Admin)
Permissions configured for all roles
Dashboard permissions enabled
Sample prompts created (verify count)
Can login to admin UI: https://SERVICE_URL/admin

Manual verification:

# Connect to database
cloud_sql_proxy -instances=PROJECT_ID:us-central1:turfai-db=tcp:5432 &
psql -h localhost -U postgres -d turfai_dms

# Check admin user exists
SELECT id, username, email FROM users_permissions_user WHERE username='admin';

# Check roles
SELECT id, name FROM users_permissions_role;

# Check prompts
SELECT COUNT(*) FROM prompts;

# Exit
\q

Admin user exists in database
Roles exist (at least 3)
Prompts created (should be more than 3)

Test admin login:

# Try logging in via API
curl -X POST https://SERVICE_URL/api/auth/local \
  -H "Content-Type: application/json" \
  -d '{"identifier":"admin","password":"TurfAIAdmin123!"}'

3.3 Deploy LLM Service

Run deployment script:

bash deploy/llm/01-deploy-llm.sh

Checklist:

LLM Docker image built and pushed
Cloud Run service deployed: turfai-llm
Health check passing
Service URL saved to deploy/.config/llm-deployment.env
Vertex AI configuration working
Can query models: curl https://SERVICE_URL/health

3.4 Deploy RAG Query Service

Run deployment script:

bash deploy/rag/01-deploy-rag.sh

Checklist:

RAG Query Docker image built and pushed
Cloud Run service deployed: turfai-rag
Health check passing
Database connection successful
pgvector queries working
Service URL saved to deploy/.config/rag-deployment.env
Conversation tables created automatically

Manual verification:

# Check health
curl https://SERVICE_URL/health

# Should return: {"status":"healthy","database_connected":true,"vector_count":0}

Health check shows database connected
Tables created: conversation_sessions, conversation_messages

3.5 Deploy RAG Embeddings Worker

Run deployment script:

bash deploy/rag-embeddings/01-deploy-rag-embeddings.sh

Checklist:

RAG Embeddings Docker image built and pushed
Cloud Run service deployed: turfai-rag-embeddings
Health check passing
Redis connection successful
Database connection successful
OCR modules included (Google Vision, Tesseract)
Service URL saved to deploy/.config/rag-embeddings-deployment.env
document_embeddings table created automatically

Manual verification:

# Check logs for successful startup
gcloud run logs read turfai-rag-embeddings --region=us-central1 --limit=20

# Should see:
# - "PostgreSQL connection test successful (pgvector v0.8.0)"
# - "document_embeddings table created/verified"
# - "Database indexes created/verified"

Logs show successful schema creation
No errors in startup logs

3.6 Update DMS with RAG Query Service URL

IMPORTANT: DMS needs RAG Query Service URL after RAG deployment!

Option 1: Redeploy DMS (recommended if URL was placeholder):

bash deploy/dms/01-deploy-dms.sh
# Will auto-detect RAG URL from rag-deployment.env

Option 2: Update environment variables only:

bash deploy/dms/03-update-env-vars.sh

Checklist:

DMS updated with correct RAG_QUERY_SERVICE_URL
Verified: gcloud run services describe turfai-dms --format="value(spec.template.spec.containers[0].env)" | grep RAG_QUERY
URL points to RAG service: https://turfai-rag-*.run.app

3.7 Deploy Router Service

Run deployment script:

bash deploy/router/01-deploy-router.sh

Checklist:

Router Docker image built and pushed
Cloud Run service deployed: turfai-router
Health check passing
Redis connection successful
Service URL saved to deploy/.config/router-deployment.env
API key authentication working

3.8 Deploy Processor Service

Run deployment script:

bash deploy/processor/01-deploy-processor.sh

Checklist:

Processor Docker image built and pushed
Cloud Run service deployed: turfai-processor
Health check passing
Redis connection successful
Can process jobs from queue
Results publisher working (publishes to Redis)

Database Seeding

4.1 Verify Admin User

# Login to admin UI
open https://DMS_URL/admin

# Credentials:
# Username: admin
# Password: TurfAIAdmin123!

Can access admin UI
Can login with credentials
Dashboard loads correctly

4.2 Verify Permissions

Check in Strapi Admin UI:

Navigate to Settings → Users & Permissions Plugin → Roles
Check "Authenticated" role:
- Documents: find, findOne, create, update, delete
- Activities: find, findOne, create, update
- Workflow Executions: find, findOne, create
- Dashboard: all endpoints enabled
- Prompts: find, findOne
- RAG: query endpoint enabled
Check "Public" role:
- Health check endpoints enabled

4.3 Verify Sample Data

Check Prompts:

curl https://DMS_URL/api/prompts | jq '.data | length'

At least 10 prompts created (not just 3)
Prompts cover different categories
Prompts have proper formatting

If prompts are missing, re-run seeding:

bash deploy/dms/02-seed-database.sh

Verification

5.1 Service Health Checks

Run all health checks:

# DMS
curl https://turfai-dms-*.run.app/_health

# Router
curl https://turfai-router-*.run.app/health

# Processor
curl https://turfai-processor-*.run.app/health

# LLM Service
curl https://turfai-llm-*.run.app/health

# RAG Query Service
curl https://turfai-rag-*.run.app/health

# RAG Embeddings Worker
curl https://turfai-rag-embeddings-*.run.app/health

Checklist:

All health checks return 200 OK
DMS: {"status":"healthy"}
Router: Shows Redis connected
Processor: Shows Redis connected
LLM: Shows available providers
RAG Query: database_connected: true
RAG Embeddings: Shows PostgreSQL connected

5.2 Database Verification

# Connect to database
cloud_sql_proxy -instances=PROJECT_ID:us-central1:turfai-db=tcp:5432 &
psql -h localhost -U postgres -d turfai_dms

# Check tables exist
\dt

# Should see:
# - Strapi tables (users, documents, etc.)
# - document_embeddings
# - conversation_sessions
# - conversation_messages

# Check pgvector
SELECT * FROM pg_extension WHERE extname='vector';

# Exit
\q

Checklist:

All Strapi tables exist
document_embeddings table exists
conversation_sessions table exists
conversation_messages table exists
pgvector extension enabled

5.3 End-to-End RAG Test

Step 1: Upload Document

# Get JWT token
TOKEN=$(curl -s -X POST https://DMS_URL/api/auth/local \
  -H "Content-Type: application/json" \
  -d '{"identifier":"admin","password":"TurfAIAdmin123!"}' | jq -r '.jwt')

# Upload document
curl -X POST https://DMS_URL/api/documents \
  -H "Authorization: Bearer $TOKEN" \
  -F "files=@test-document.pdf" \
  -F "data={\"title\":\"Test Document\"}"

Document uploaded successfully
Document ID returned

Step 2: Enable RAG

# Enable RAG for document
curl -X POST https://DMS_URL/api/documents/DOCUMENT_ID/rag-enable \
  -H "Authorization: Bearer $TOKEN"

RAG enabled successfully
Status: queued or processing

Step 3: Wait for Processing

# Check status (may take 30-60 seconds)
watch -n 5 "curl -s https://DMS_URL/api/documents/DOCUMENT_ID/rag-status \
  -H 'Authorization: Bearer $TOKEN' | jq '.rag_processing_status'"

Status changes to processing
Status eventually becomes completed
rag_chunk_count > 0

Step 4: Query RAG

curl -X POST https://DMS_URL/api/rag/query \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "What is this document about?",
    "top_k": 5
  }' | jq '.'

Expected response:

{
  "answer": "...",
  "sources": [
    {
      "document_id": 1,
      "chunk_text": "...",
      "score": 0.85,
      "file_url": "gs://...",
      "signed_url": "https://storage.googleapis.com/..."
    }
  ],
  "query": "What is this document about?",
  "context_used": "..."
}

Checklist:

Query returns answer
Sources array present
Sources include signed_url (not null)
Signed URLs are accessible (HTTP 200)
Answer is relevant to document

5.4 Verify Signed URLs

# Extract signed URL from response
SIGNED_URL=$(curl -s -X POST https://DMS_URL/api/rag/query \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query":"test","top_k":1}' | jq -r '.sources[0].signed_url')

# Test signed URL
curl -I "$SIGNED_URL"

Checklist:

Signed URL is not null
Signed URL returns HTTP 200
Can download file via signed URL
URL expires after configured time (default 1 hour)

5.5 Check Logs for Errors

# Check all services for errors
for service in turfai-dms turfai-router turfai-processor turfai-llm turfai-rag turfai-rag-embeddings; do
  echo "=== $service ==="
  gcloud run logs read $service --region=us-central1 --limit=10 | grep -i error
done

No critical errors in logs
No authentication failures
No database connection errors
No Redis connection errors

Post-Deployment

6.1 Documentation

Update deployment status document
Record all service URLs
Document any manual configuration done
Update architecture diagrams if changed
Create deployment summary report

6.2 Security Hardening

Optional but Recommended:

# Disable admin access after initial setup
bash deploy/dms/03-update-env-vars.sh
# Set ENABLE_ADMIN_ACCESS=false when prompted

Admin access disabled (if in production)
Service account key secured
Secrets properly stored in Secret Manager
No secrets in code or logs

6.3 Monitoring Setup

Create Cloud Monitoring dashboard
Set up log-based metrics
Configure alerting for critical errors
Set up uptime checks for all services

6.4 Backup Configuration

Enable Cloud SQL automated backups
Configure backup retention (7-30 days)
Test backup restore procedure
Document backup/restore process

Rollback Procedures

7.1 Service Rollback

Roll back to previous revision:

# List revisions
gcloud run revisions list --service=SERVICE_NAME --region=us-central1

# Roll back to previous revision
gcloud run services update-traffic SERVICE_NAME \
  --to-revisions=PREVIOUS_REVISION=100 \
  --region=us-central1

Previous revision identified
Traffic routed to previous revision
Health check passing on previous revision

7.2 Database Rollback

Restore from backup:

# List backups
gcloud sql backups list --instance=turfai-db

# Restore from backup
gcloud sql backups restore BACKUP_ID \
  --backup-instance=turfai-db \
  --backup-instance=turfai-db

Recent backup identified
Database restored successfully
Services reconnected to database

7.3 Complete Rollback

If deployment fails completely:

Stop new services:

gcloud run services delete SERVICE_NAME --region=us-central1

Restore database:

gcloud sql backups restore BACKUP_ID --backup-instance=turfai-db

Restore previous deployment:
- Checkout previous git tag
- Run deployment scripts for previous version

Troubleshooting

Common Issues

Issue: Service won't start

Check logs: gcloud run logs read SERVICE_NAME --region=us-central1 --limit=50
Verify environment variables
Check database/Redis connectivity
Verify service account permissions

Issue: Database connection fails

Check Cloud SQL instance is running
Verify connection string format
Check database password in Secret Manager
Verify service account has cloudsql.client role

Issue: RAG not generating embeddings

Check Cloud Vision API is enabled
Verify RAG Embeddings worker logs
Check Redis queue has jobs
Verify LLM service is accessible

Issue: Signed URLs not working

Verify service account has iam.serviceAccountTokenCreator role
Check GCS bucket permissions
Review DMS logs for signed URL generation errors

Summary

Deployment Metrics

Total Services: 6 (DMS, Router, Processor, LLM, RAG Query, RAG Embeddings)
Infrastructure Components: 4 (Cloud SQL, Redis, GCS, Service Account)
Estimated Time: 45-60 minutes (first deployment)
Estimated Cost: $150-350/month

Key Success Criteria

✅ All 6 services deployed and healthy
✅ Admin user created and accessible
✅ Dashboard permissions enabled
✅ Sample prompts created
✅ Database schema complete
✅ End-to-end RAG pipeline working
✅ Signed URLs functioning
✅ No errors in service logs

Next Steps After Deployment

Change default admin password
Create additional users and test
Upload real documents and test workflows
Configure monitoring and alerting
Set up CI/CD pipeline
Plan for production hardening

Environment Variables: /docs/environment-variables-inventory.md
Deployment Lessons: /docs/deployment-lessons-learned-2025-11-04.md
Database Schema: /docs/database-schema-complete.md
Architecture: /CLAUDE.md
Deployment Status: /docs/deployment-status-2025-11-04.md

Document Version: 1.0 Last Updated: November 4, 2025 Maintainers: TurfAI Development Team

On this page