Skip to content

DEPLOYMENT_TESTING_PLAN.md

Latest commit

 

History

History
1069 lines (908 loc) · 28.4 KB

DEPLOYMENT_TESTING_PLAN.md

File metadata and controls

1069 lines (908 loc) · 28.4 KB
Raw

Complete Deployment & Testing Plan

Date: 2026-02-15 Branch: v1.2-auth-boundary Goal: Deploy monitoring stack and verify all 3 Grafana dashboards work with real data

Phase 1: Test Deployment (30 minutes)

Step 1: Start Docker Desktop (2 minutes)

# macOS: Open Docker Desktop application
open -a Docker

# Wait for Docker to start (watch for whale icon in menu bar)
# Verify Docker is running:
docker ps

# Expected: Should show running containers or empty list (not connection error)

Step 2: Start Monitoring Stack (5 minutes)

cd /Users/marshallnorriscott/Desktop/EFA-ARC/arc-core

# Start full stack with monitoring profile
docker compose --profile monitoring up -d

# This starts:
# - Redis
# - ARC API (with Data Lake writer)
# - Prometheus
# - Grafana
# - Alertmanager
# - Redis Exporter

# Wait for services to start (30-60 seconds)
sleep 60

# Verify all services are running
docker compose --profile monitoring ps

Expected Output:

NAME                STATUS        PORTS
arc-api             Up 1 minute   0.0.0.0:3000->3000/tcp
redis               Up 1 minute   0.0.0.0:6379->6379/tcp
prometheus          Up 1 minute   0.0.0.0:9090->9090/tcp
grafana             Up 1 minute   0.0.0.0:3001->3000/tcp
alertmanager        Up 1 minute   0.0.0.0:9093->9093/tcp
redis-exporter      Up 1 minute   0.0.0.0:9121->9121/tcp

Step 3: Verify API Health (2 minutes)

# Check API is responding
curl http://localhost:3000/health

# Expected: {"ok":true,"status":"healthy","timestamp":"..."}

# Check /metrics endpoint
curl http://localhost:3000/metrics | head -20

# Expected: Prometheus text format metrics
# TYPE arc_requests_total counter
# arc_requests_total{method="GET",path="/health",status="200"} 1
# ...

Step 4: Generate Test Data (10 minutes)

Since the Data Lake is currently empty, we need to generate sample events:

Option 1: Use Seed Script (Preferred)

# Run database seed script to populate test data
npm run db:seed

# This creates:
# - Test users (artists, collectors)
# - Master assets (artworks)
# - NFT minting requests
# - Transactions
# - Listings

Option 2: Manual API Calls

First, create an admin JWT token:

# Bootstrap admin user (first-time only)
curl -X POST http://localhost:3000/auth/password/bootstrap \
  -H "Content-Type: application/json" \
  -d '{
    "secret": "docker-bootstrap-secret",
    "person_id": "admin-bootstrap",
    "password": "Admin123!@#"
  }'

# Login to get JWT
ADMIN_JWT=$(curl -s -X POST http://localhost:3000/auth/token \
  -H "Content-Type: application/json" \
  -d '{
    "person_id": "admin-bootstrap",
    "password": "Admin123!@#"
  }' | jq -r '.access_token')

echo "Admin JWT: $ADMIN_JWT"

Then create test data:

# Create test artist
ARTIST_ID="artist-test-$(date +%s)"
curl -X POST http://localhost:3000/v1/ops/users \
  -H "Authorization: Bearer $ADMIN_JWT" \
  -H "Content-Type: application/json" \
  -d '{
    "person_id": "'$ARTIST_ID'",
    "identity_type": "artist",
    "full_name": "Test Artist",
    "email": "artist@test.com",
    "verified": true
  }'

# Create test collector
COLLECTOR_ID="collector-test-$(date +%s)"
curl -X POST http://localhost:3000/v1/ops/users \
  -H "Authorization: Bearer $ADMIN_JWT" \
  -H "Content-Type: application/json" \
  -d '{
    "person_id": "'$COLLECTOR_ID'",
    "identity_type": "collector",
    "full_name": "Test Collector",
    "email": "collector@test.com",
    "verified": true
  }'

# Trigger some transactions via webhook simulation
# (This will emit transaction.completed events)
for i in {1..5}; do
  curl -X POST http://localhost:3000/webhooks/stripe \
    -H "Content-Type: application/json" \
    -d '{
      "type": "payment_intent.succeeded",
      "data": {
        "object": {
          "id": "pi_test_'$i'",
          "amount": '$((100 * i))'00,
          "currency": "usd",
          "status": "succeeded",
          "metadata": {
            "transaction_id": "tx-test-'$i'",
            "from_person_id": "'$COLLECTOR_ID'",
            "to_person_id": "'$ARTIST_ID'"
          }
        }
      }
    }'
  echo "Created transaction $i"
  sleep 1
done

Option 3: Check for Existing Events

# If the database already has data, you can trigger event emission
# by performing actions via the API:

# List existing master assets
curl -s http://localhost:3000/v1/masterAssets \
  -H "Authorization: Bearer $ADMIN_JWT" | jq '.data | length'

# List existing transactions
curl -s http://localhost:3000/v1/ops/transactions \
  -H "Authorization: Bearer $ADMIN_JWT" | jq '.data | length'

# If you have existing data, events should start flowing to Data Lake

Step 5: Verify Data Lake Events (3 minutes)

# Check that events are being written to Data Lake
ls -lh data-lake/events/

# Expected directory structure:
# category=transaction/date=2026-02-15/page-000.jsonl
# category=artwork/date=2026-02-15/page-000.jsonl
# category=nft/date=2026-02-15/page-000.jsonl
# etc.

# Count total events
find data-lake/events -name "*.jsonl" -exec wc -l {} + | tail -1

# View sample transaction events
cat data-lake/events/category=transaction/date=$(date +%Y-%m-%d)/*.jsonl 2>/dev/null | jq | head -20

# Expected JSON structure:
# {
#   "event_id": "evt_...",
#   "event_type": "transaction.completed",
#   "event_category": "transaction",
#   "timestamp": "2026-02-15T...",
#   "actor": {...},
#   "payload": {...}
# }

Step 6: Verify Prometheus Scraping (3 minutes)

# Check Prometheus UI
open http://localhost:9090

# Navigate to: Status → Targets
# Or via API:
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | select(.labels.service=="arc-api")'

# Expected output:
# {
#   "discoveredLabels": {...},
#   "labels": {
#     "service": "arc-api",
#     "environment": "development"
#   },
#   "scrapeUrl": "http://api:3000/metrics",
#   "health": "up",
#   "lastScrape": "2026-02-15T...",
#   "lastError": ""
# }

# Query metrics via Prometheus
curl -s 'http://localhost:9090/api/v1/query?query=arc_data_lake_events_total' | jq '.data.result[] | {metric: .metric, value: .value[1]}'

# Expected: Array of metrics with event counts
# {
#   "metric": {
#     "event_type": "transaction.completed",
#     "event_category": "transaction"
#   },
#   "value": "5"
# }

Step 7: Import Grafana Dashboards (5 minutes)

# Open Grafana UI
open http://localhost:3001

# Login credentials:
# Username: admin
# Password: arc-dev

Manual Import:

  1. Navigate to Dashboards → Import
  2. Click Upload JSON file
  3. Select infra/grafana/dashboards/data-lake-revenue.json
  4. In datasource dropdown, select Prometheus
  5. Click Import
  6. Repeat for data-lake-sales.json and data-lake-nft.json

Verify Auto-Provisioning:

  1. Navigate to Dashboards → Browse
  2. Look for ARC folder
  3. Should contain 5 dashboards:
    • ARC API Overview
    • ARC API Errors
    • Data Lake - Revenue Analytics
    • Data Lake - Sales & Marketplace Analytics
    • Data Lake - NFT Performance

Step 8: Verify Dashboard Data (5 minutes)

Revenue Analytics Dashboard:

# Open dashboard
open http://localhost:3001/d/data-lake-revenue

# Check panels populate:
# 1. Daily Revenue Trend - Should show line chart (may be flat if all events are today)
# 2. Transaction Success Rate - Should show gauge (target >95%)
# 3. Average Transaction Value - Should show stat (e.g., $300)
# 4. 24h Transaction Volume - Should show stat (e.g., 5)
# 5. Transaction Failures - Should show line chart (hopefully 0)
# 6. Processing Time Percentiles - Should show graph

Sales & Marketplace Dashboard:

# Open dashboard
open http://localhost:3001/d/data-lake-sales

# Check panels:
# 1. Artworks Sold per Hour - Bar chart (may be empty if no artwork.purchased events)
# 2. Listing Activity - Line chart showing created vs cancelled
# 3. Artwork Lifecycle - Line chart showing created vs published
# 4. Certification Activity - Bar chart

NFT Performance Dashboard:

# Open dashboard
open http://localhost:3001/d/data-lake-nft

# Check panels:
# 1. Mint Success Rate - Gauge (should be >95% if mints are working)
# 2. 24h Mint Requests - Stat
# 3. 24h Mints Completed - Stat
# 4. 24h Mint Failures - Stat
# 5. Request → Complete Rate - Stat (conversion rate)
# 6. NFT Minting Funnel - Line chart (requested/completed/failed)
# 7. Failure Rate Trend - Line chart

Troubleshooting "No Data":

If panels show "No data":

  1. Check time range - Click time picker (top right), try "Last 7 days"
  2. Generate more events - Run the test data script again
  3. Check Prometheus has data - Run queries in Prometheus UI
  4. Verify metric names - Compare dashboard queries with /metrics output
  5. Check panel queries - Edit panel → Check PromQL query syntax

Phase 2: Configure Slack Alerts (30 minutes)

Step 1: Create Slack Webhook (10 minutes)

  1. Go to https://api.slack.com/apps
  2. Click Create New App → From scratch
  3. App Name: ARC Platform Alerts
  4. Workspace: Select your workspace
  5. Click Create App
  6. Navigate to Incoming Webhooks
  7. Toggle Activate Incoming Webhooks to ON
  8. Click Add New Webhook to Workspace
  9. Select channel: #arc-alerts-critical (create if doesn't exist)
  10. Click Allow
  11. Copy the webhook URL (starts with https://hooks.slack.com/services/...)

Step 2: Configure Alertmanager (5 minutes)

# Edit Alertmanager config
vim infra/alertmanager/alertmanager.yml

Update the Slack webhook URL:

global:
  slack_api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'

route:
  receiver: 'slack-critical'
  group_by: ['alertname', 'severity']
  group_wait: 10s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: critical
      receiver: 'slack-critical'
    - match:
        severity: warning
      receiver: 'slack-warnings'

receivers:
  - name: 'slack-critical'
    slack_configs:
      - channel: '#arc-alerts-critical'
        title: '🚨 {{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}*Alert:* {{ .Annotations.summary }}\n*Description:* {{ .Annotations.description }}\n{{ end }}'
        send_resolved: true

  - name: 'slack-warnings'
    slack_configs:
      - channel: '#arc-alerts'
        title: '⚠️ {{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}*Alert:* {{ .Annotations.summary }}\n{{ end }}'
        send_resolved: true

Step 3: Set Environment Variable (2 minutes)

# Create .env file if it doesn't exist
touch .env

# Add Slack webhook URL
echo "SLACK_WEBHOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK/URL" >> .env

Step 4: Restart Alertmanager (2 minutes)

# Restart to pick up new config
docker compose --profile monitoring restart alertmanager

# Check logs
docker compose --profile monitoring logs -f alertmanager

# Expected: "msg="Completed loading of configuration file""

Step 5: Test Alerts (10 minutes)

Method 1: Manual Alert (via Alertmanager API)

# Send test alert
curl -X POST http://localhost:9093/api/v1/alerts \
  -H "Content-Type: application/json" \
  -d '[
    {
      "labels": {
        "alertname": "TestAlert",
        "severity": "critical"
      },
      "annotations": {
        "summary": "Test alert from deployment verification",
        "description": "This is a test alert to verify Slack integration"
      }
    }
  ]'

# Check Slack channel #arc-alerts-critical for message

Method 2: Trigger Real Alert (High Failure Rate)

Create some failed transactions:

# Simulate failed transactions
for i in {1..3}; do
  curl -X POST http://localhost:3000/webhooks/stripe \
    -H "Content-Type: application/json" \
    -d '{
      "type": "payment_intent.payment_failed",
      "data": {
        "object": {
          "id": "pi_fail_'$i'",
          "amount": 10000,
          "currency": "usd",
          "status": "failed",
          "metadata": {
            "transaction_id": "tx-fail-'$i'"
          }
        }
      }
    }'
  sleep 1
done

# Wait 5-10 minutes for Prometheus to evaluate alert rules
# Check if HighTransactionFailureRate alert fires

# View active alerts in Prometheus
open http://localhost:9090/alerts

# View active alerts in Alertmanager
open http://localhost:9093/#/alerts

Method 3: Test via Alertmanager UI

  1. Open http://localhost:9093
  2. Click Silences → New Silence
  3. Create a test silence to verify config is loaded
  4. Check Slack for notification

Phase 3: Create Unified Executive Dashboard (45 minutes)

Step 1: Create Dashboard JSON (30 minutes)

# Create unified dashboard
cat > infra/grafana/dashboards/data-lake-executive.json << 'EOF'
{
  "annotations": {
    "list": []
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "id": null,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "description": "Total revenue in last 24 hours",
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "thresholds"
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {"color": "green", "value": null}
            ]
          },
          "unit": "currencyUSD"
        },
        "overrides": []
      },
      "gridPos": {
        "h": 4,
        "w": 6,
        "x": 0,
        "y": 0
      },
      "id": 1,
      "options": {
        "colorMode": "value",
        "graphMode": "area",
        "justifyMode": "auto",
        "orientation": "auto",
        "reduceOptions": {
          "values": false,
          "calcs": ["lastNotNull"],
          "fields": ""
        },
        "textMode": "auto"
      },
      "pluginVersion": "10.0.0",
      "targets": [
        {
          "datasource": {"type": "prometheus", "uid": "prometheus"},
          "editorMode": "code",
          "expr": "sum(increase(arc_data_lake_events_total{event_type=\"transaction.completed\"}[24h]))",
          "legendFormat": "Revenue",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "24h Revenue",
      "type": "stat"
    },
    {
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "description": "Transaction success rate",
      "fieldConfig": {
        "defaults": {
          "color": {"mode": "thresholds"},
          "mappings": [],
          "max": 100,
          "min": 0,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {"color": "red", "value": null},
              {"color": "yellow", "value": 85},
              {"color": "green", "value": 95}
            ]
          },
          "unit": "percent"
        },
        "overrides": []
      },
      "gridPos": {"h": 4, "w": 6, "x": 6, "y": 0},
      "id": 2,
      "options": {
        "orientation": "auto",
        "reduceOptions": {
          "values": false,
          "calcs": ["lastNotNull"],
          "fields": ""
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true
      },
      "pluginVersion": "10.0.0",
      "targets": [
        {
          "datasource": {"type": "prometheus", "uid": "prometheus"},
          "editorMode": "code",
          "expr": "100 * sum(rate(arc_data_lake_events_total{event_type=\"transaction.completed\"}[1h])) / (sum(rate(arc_data_lake_events_total{event_type=\"transaction.completed\"}[1h])) + sum(rate(arc_data_lake_events_total{event_type=\"transaction.failed\"}[1h])))",
          "legendFormat": "Success Rate",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Transaction Success Rate",
      "type": "gauge"
    },
    {
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "description": "NFT mint success rate",
      "fieldConfig": {
        "defaults": {
          "color": {"mode": "thresholds"},
          "mappings": [],
          "max": 100,
          "min": 0,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {"color": "red", "value": null},
              {"color": "yellow", "value": 85},
              {"color": "green", "value": 95}
            ]
          },
          "unit": "percent"
        },
        "overrides": []
      },
      "gridPos": {"h": 4, "w": 6, "x": 12, "y": 0},
      "id": 3,
      "options": {
        "orientation": "auto",
        "reduceOptions": {
          "values": false,
          "calcs": ["lastNotNull"],
          "fields": ""
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true
      },
      "pluginVersion": "10.0.0",
      "targets": [
        {
          "datasource": {"type": "prometheus", "uid": "prometheus"},
          "editorMode": "code",
          "expr": "100 * sum(rate(arc_data_lake_events_total{event_type=\"nft.mint_completed\"}[1h])) / (sum(rate(arc_data_lake_events_total{event_type=\"nft.mint_completed\"}[1h])) + sum(rate(arc_data_lake_events_total{event_type=\"nft.mint_failed\"}[1h])))",
          "legendFormat": "Mint Success",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "NFT Mint Success Rate",
      "type": "gauge"
    },
    {
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "description": "Artworks sold in last 24h",
      "fieldConfig": {
        "defaults": {
          "color": {"mode": "thresholds"},
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [{"color": "blue", "value": null}]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {"h": 4, "w": 6, "x": 18, "y": 0},
      "id": 4,
      "options": {
        "colorMode": "value",
        "graphMode": "area",
        "justifyMode": "auto",
        "orientation": "auto",
        "reduceOptions": {
          "values": false,
          "calcs": ["lastNotNull"],
          "fields": ""
        },
        "textMode": "auto"
      },
      "pluginVersion": "10.0.0",
      "targets": [
        {
          "datasource": {"type": "prometheus", "uid": "prometheus"},
          "editorMode": "code",
          "expr": "sum(increase(arc_data_lake_events_total{event_type=\"artwork.purchased\"}[24h]))",
          "legendFormat": "Artworks Sold",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "24h Artworks Sold",
      "type": "stat"
    },
    {
      "datasource": {"type": "prometheus", "uid": "prometheus"},
      "description": "Key platform metrics over time",
      "fieldConfig": {
        "defaults": {
          "color": {"mode": "palette-classic"},
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 10,
            "gradientMode": "none",
            "hideFrom": {"tooltip": false, "viz": false, "legend": false},
            "lineInterpolation": "smooth",
            "lineWidth": 2,
            "pointSize": 5,
            "scaleDistribution": {"type": "linear"},
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {"group": "A", "mode": "none"},
            "thresholdsStyle": {"mode": "off"}
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [{"color": "green", "value": null}]
          },
          "unit": "short"
        },
        "overrides": []
      },
      "gridPos": {"h": 8, "w": 24, "x": 0, "y": 4},
      "id": 5,
      "options": {
        "legend": {
          "calcs": ["mean", "max"],
          "displayMode": "table",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {"mode": "multi", "sort": "none"}
      },
      "targets": [
        {
          "datasource": {"type": "prometheus", "uid": "prometheus"},
          "editorMode": "code",
          "expr": "sum(rate(arc_data_lake_events_total{event_type=\"transaction.completed\"}[1h])) * 3600",
          "legendFormat": "Transactions/Hour",
          "range": true,
          "refId": "A"
        },
        {
          "datasource": {"type": "prometheus", "uid": "prometheus"},
          "editorMode": "code",
          "expr": "sum(rate(arc_data_lake_events_total{event_type=\"artwork.purchased\"}[1h])) * 3600",
          "hide": false,
          "legendFormat": "Artworks Sold/Hour",
          "range": true,
          "refId": "B"
        },
        {
          "datasource": {"type": "prometheus", "uid": "prometheus"},
          "editorMode": "code",
          "expr": "sum(rate(arc_data_lake_events_total{event_type=\"nft.mint_completed\"}[1h])) * 3600",
          "hide": false,
          "legendFormat": "NFT Mints/Hour",
          "range": true,
          "refId": "C"
        },
        {
          "datasource": {"type": "prometheus", "uid": "prometheus"},
          "editorMode": "code",
          "expr": "sum(rate(arc_data_lake_events_total{event_type=\"listing.created\"}[1h])) * 3600",
          "hide": false,
          "legendFormat": "Listings Created/Hour",
          "range": true,
          "refId": "D"
        }
      ],
      "title": "Platform Activity",
      "type": "timeseries"
    }
  ],
  "refresh": "30s",
  "schemaVersion": 38,
  "style": "dark",
  "tags": ["data-lake", "executive", "overview"],
  "templating": {"list": []},
  "time": {"from": "now-24h", "to": "now"},
  "timepicker": {},
  "timezone": "",
  "title": "Data Lake - Executive Overview",
  "uid": "data-lake-executive",
  "version": 1,
  "weekStart": ""
}
EOF

Step 2: Import Dashboard (5 minutes)

# Dashboard is already in provisioning directory, so it will auto-load
# Restart Grafana to pick it up:
docker compose --profile monitoring restart grafana

# Wait 30 seconds
sleep 30

# Or manually import via UI:
open http://localhost:3001/dashboard/import
# Upload: infra/grafana/dashboards/data-lake-executive.json

Step 3: Verify Dashboard (10 minutes)

# Open executive dashboard
open http://localhost:3001/d/data-lake-executive

# Verify panels:
# - 24h Revenue (stat, green)
# - Transaction Success Rate (gauge, >95%)
# - NFT Mint Success Rate (gauge, >95%)
# - 24h Artworks Sold (stat, blue)
# - Platform Activity (time series, 4 lines)

# All panels should show data (not "No data" or "N/A")

Phase 4: Add More Event Types (60 minutes)

Goal: Reach 56% coverage (18/32 events)

Currently at 44% (14/32 events). Need to add 4 more events.

Target Events (4 new):

  1. user.created - Track user registration
  2. certification.requested - Track certification pipeline start
  3. certification.rejected - Track certification failures
  4. admin.login - Track admin access patterns

Step 1: Implement user.created Event (15 minutes)

Currently no production user creation endpoint. Skip for now (users created via seed).

Step 2: Implement certification.requested Event (15 minutes)

# Edit master assets route
vim src/routes/masterAssets.ts

Find the certification request handler and add event emission:

// Around line X where certification is requested
import { emitCertificationRequested } from '../services/eventEmissionHelper.js';

// In certification request handler:
const certificationId = randomUUID();

// Emit certification.requested event
await emitCertificationRequested({
  certification_id: certificationId,
  master_asset_id: assetId,
  requested_by: actor.actor_person_id,
  request_timestamp: new Date().toISOString(),
});

// ... continue with certification logic

Step 3: Implement certification.rejected Event (15 minutes)

// In certification rejection handler:
import { emitCertificationRejected } from '../services/eventEmissionHelper.js';

// After rejection logic:
await emitCertificationRejected({
  certification_id: certificationId,
  master_asset_id: assetId,
  rejected_by: actor.actor_person_id,
  rejection_reason: reason,
  rejection_timestamp: new Date().toISOString(),
});

Step 4: Implement admin.login Event (15 minutes)

# Edit auth routes
vim src/routes/auth.ts

Add event emission to login handler:

// In POST /auth/token handler, after successful login:
import { emitAdminAction } from '../services/eventEmissionHelper.js';

// Check if user has admin capabilities
if (person.roles && person.roles.includes('admin')) {
  await emitAdminAction({
    action: 'admin.login',
    admin_person_id: person.person_id,
    ip_address: request.ip,
    user_agent: request.headers['user-agent'],
    timestamp: new Date().toISOString(),
  });
}

Step 5: Update Event Emission Helpers (If Needed)

# Check if helper functions exist
grep -n "emitCertificationRequested\|emitCertificationRejected" src/services/eventEmissionHelper.ts

# If not, add them:
vim src/services/eventEmissionHelper.ts

Add new helper functions:

export async function emitCertificationRequested(params: {
  certification_id: string;
  master_asset_id: string;
  requested_by: string;
  request_timestamp: string;
}): Promise<void> {
  const writer = getDataLakeWriter();
  if (!writer) return;

  const actor = createSystemActor();
  const event = createEvent('certification.requested', 'compliance', actor, {
    certification_id: params.certification_id,
    master_asset_id: params.master_asset_id,
    requested_by: params.requested_by,
    request_timestamp: params.request_timestamp,
  });

  await writer.writeEvent(event);
}

export async function emitCertificationRejected(params: {
  certification_id: string;
  master_asset_id: string;
  rejected_by: string;
  rejection_reason: string;
  rejection_timestamp: string;
}): Promise<void> {
  const writer = getDataLakeWriter();
  if (!writer) return;

  const actor = createSystemActor();
  const event = createEvent('certification.rejected', 'compliance', actor, {
    certification_id: params.certification_id,
    master_asset_id: params.master_asset_id,
    rejected_by: params.rejected_by,
    rejection_reason: params.rejection_reason,
    rejection_timestamp: params.rejection_timestamp,
  });

  await writer.writeEvent(event);
}

Step 6: Test New Events (10 minutes)

# Rebuild and restart API
npm run build:app
docker compose --profile monitoring restart arc-api

# Wait for API to restart
sleep 10

# Test certification.requested event
curl -X POST http://localhost:3000/v1/masterAssets/:id/certify \
  -H "Authorization: Bearer $ADMIN_JWT" \
  -H "Content-Type: application/json"

# Check Data Lake for new event
cat data-lake/events/category=compliance/date=$(date +%Y-%m-%d)/*.jsonl | grep certification.requested

# Test admin.login event
curl -X POST http://localhost:3000/auth/token \
  -H "Content-Type: application/json" \
  -d '{
    "person_id": "admin-bootstrap",
    "password": "Admin123!@#"
  }'

# Check for admin.login event
cat data-lake/events/category=admin/date=$(date +%Y-%m-%d)/*.jsonl | grep admin.login

Success Criteria

Phase 1: Deployment ✅

  • Docker compose starts all 6 services
  • API responds to /health
  • /metrics endpoint returns Prometheus format
  • Prometheus scrapes arc-api successfully (status: UP)
  • Grafana connects to Prometheus datasource
  • All 3 dashboards import successfully
  • Dashboards show data (not "No data")
  • At least 10 events in Data Lake

Phase 2: Slack Alerts ✅

  • Slack webhook configured
  • Alertmanager config updated
  • Test alert received in Slack
  • Alert format is readable
  • Resolved alerts sent to Slack

Phase 3: Executive Dashboard ✅

  • Dashboard JSON created
  • Dashboard imports successfully
  • All 5 panels populate with data
  • 30s auto-refresh works
  • Dashboard is visually appealing

Phase 4: Event Coverage ✅

  • 4 new events implemented
  • Event coverage reaches 56% (18/32)
  • New events appear in Data Lake
  • Prometheus metrics include new events
  • No regressions in existing events

Timeline

  • Phase 1: 30 minutes
  • Phase 2: 30 minutes
  • Phase 3: 45 minutes
  • Phase 4: 60 minutes
  • Total: 2 hours 45 minutes

Next Steps After Completion

  1. Performance Testing

    • Load test with 1000 events/sec
    • Verify Prometheus can keep up
    • Check Data Lake write performance

  2. Production Deployment

    • Deploy to staging first
    • Verify in production environment
    • Set up production Slack channels

  3. Continuous Improvement

    • Add more dashboards
    • Implement anomaly detection
    • Set up SLO monitoring

Ready to proceed! Start with Phase 1: Test Deployment.