# Disaster Recovery Procedures PAMP Group Holdings maintains a comprehensive disaster recovery (DR) strategy designed to meet strict recovery targets across the multi-cloud, multi-region infrastructure. The DR architecture leverages geographic redundancy across AWS, Azure, GCP, and on-premise Sangfor HCI clusters to ensure business continuity for all 7 Wave-1 countries. ## DR Targets ## Multi-Cloud DR Architecture AWS_PRIMARY CF --> AZURE_PRIMARY CF --> GCP_PRIMARY AWS_PRIMARY --> |Async Replication| AWS_DR_EU AWS_PRIMARY --> |Async Replication| AWS_DR_US AZURE_PRIMARY --> |Geo-Replication| AWS_DR_EU GCP_PRIMARY --> |Cross-Region Backup| AWS_DR_US SANGFOR --> |Backup to Cloud| AWS_PRIMARY CF --> |Failover Routing| AWS_DR_EU CF --> |Failover Routing| AWS_DR_US `} /> ## DR Tiers Different components have different DR strategies based on their criticality: ## Database Replication Strategy ## Failover Procedures ### Automated Failover (Tier 1 and Tier 2) Tier 1 and Tier 2 components use automated failover triggered by health check failures: DETECT{Primary Healthy?} DETECT --> |Yes| NORMAL[Continue Routing to Primary] DETECT --> |No — 3 consecutive failures| FAILOVER[Initiate Failover] FAILOVER --> DNS[Cloudflare DNS — Update weighted routing to DR region] FAILOVER --> DB_PROMOTE[Promote DR database replicas to primary] FAILOVER --> K8S_SCALE[Scale up DR Kubernetes deployments to full capacity] FAILOVER --> CACHE_WARM[Warm Redis cache from replicated data] DNS --> VERIFY[Verify DR region serving traffic] DB_PROMOTE --> VERIFY K8S_SCALE --> VERIFY CACHE_WARM --> VERIFY VERIFY --> |Healthy| NOTIFY[Notify SRE team — P1 incident channel] VERIFY --> |Unhealthy| MANUAL[Escalate to manual intervention] NOTIFY --> MONITOR[Monitor DR region — enhanced alerting] MONITOR --> PLAN_FAILBACK[Plan failback when primary recovers] `} /> ### Manual Failover (Tier 3) Tier 3 components require SRE-initiated failover: 1. **Detection** — Alertmanager triggers P1 alert for regional failure 2. **Decision** — On-call SRE confirms regional outage (not transient) 3. **Initiate** — SRE executes failover runbook via automation toolkit 4. **Database** — Promote warm standby replicas, verify data consistency 5. **Kubernetes** — Scale DR deployments from warm (25% capacity) to full 6. **DNS** — Update Cloudflare routing to direct traffic to DR region 7. **Verify** — Run smoke tests against DR endpoints 8. **Communicate** — Update status page, notify stakeholders ### Failback Procedure After the primary region recovers: 1. **Verify Primary** — Confirm primary infrastructure is fully operational 2. **Resync Data** — Replicate any data written to DR back to primary 3. **Validate Consistency** — Run data integrity checks across primary and DR 4. **Gradual Shift** — Use Cloudflare weighted routing to shift 10% → 25% → 50% → 100% traffic back to primary 5. **Monitor** — Enhanced monitoring for 24 hours post-failback 6. **Scale Down DR** — Return DR region to warm standby capacity 7. **Post-Mortem** — Document the incident and failover/failback performance ## DR Testing Protocol DR testing is conducted quarterly with increasing scope: ### DR Test Success Criteria Each DR test must validate: - **RTO Achieved** — Service restored within 20 minutes of failover initiation - **RPO Achieved** — Data loss is within 15 minutes of the failure point - **Data Integrity** — No data corruption or inconsistency after failover - **Functional Correctness** — All critical user flows (browse, checkout, payment) work correctly on DR - **Performance Parity** — DR region meets SLO targets (latency, throughput) - **Monitoring Coverage** — All dashboards and alerts function correctly in DR region ## Country-Specific DR Configuration ## Communication During DR Events ### Status Page A public status page (status.dukaexpress.com) is maintained via Atlassian Statuspage: - **Automated Updates** — Cloudflare health check failures automatically update component status - **Manual Updates** — On-call SRE posts updates every 15 minutes during active incidents - **Subscriber Notifications** — Email and SMS notifications to subscribed merchants and partners ### Internal Communication - **Slack #pamp-incidents** — Real-time incident coordination channel - **PagerDuty** — On-call escalation and acknowledgment tracking - **Bridge Call** — Zoom bridge auto-created for P1 incidents - **Executive Summary** — Sent to leadership within 1 hour of P1 incident declaration ## DR Automation Toolkit The DR process is supported by automation scripts and infrastructure-as-code: ```bash # DR failover toolkit commands # Initiate regional failover pamp-dr failover --region eu-west-1 --tier 2 --confirm # Check DR readiness pamp-dr status --all-regions # Promote database replica pamp-dr db-promote --service payment-svc --target eu-west-1 # Scale DR Kubernetes deployments pamp-dr k8s-scale --cluster pamp-dr-eu-west-1 --replicas full # Run DR smoke tests pamp-dr smoke-test --region eu-west-1 --suite critical-path # Initiate failback pamp-dr failback --region af-south-1 --gradual --steps 10,25,50,100 ```

DR Procedures