At 02:00 on November 22nd, we initiated the largest infrastructure migration in company history. 14.2 terabytes of user data, application state, and database snapshots needed to move from our Mumbai datacenter to distributed edge nodes across three continents.

The migration finished at 10:17. Services came back online at 10:23. By 11:00, we discovered the first major issue. By 13:00, we had identified six distinct problems. By 16:00, five were resolved. The sixth remains under investigation.

What Went Right

The bulk transfer completed 43 minutes ahead of schedule. Our pre-migration checksums matched post-migration verification across 99.97% of transferred files. Database replication synchronized within 2 minutes of target completion. Edge cache warming proceeded exactly as modeled.

Critical systems returned to operational status within the planned maintenance window. User-facing services experienced 8 hours 23 minutes of downtime against a projected 9 hours 15 minutes. Automated rollback triggers remained untouched. The happy path worked.

What Went Wrong

Issue 1: CDN Cache Poisoning
Our CDN provider cached a 503 error page during the migration window. This page then served to users for 47 minutes after services returned. We had configured TTL for static assets properly but failed to set cache control headers for error responses. Resolution: manual cache purge + updated error handling middleware.

Issue 2: Authentication Token Mismatch
Old server tokens remained valid for 15 minutes after migration due to JWT design. Users authenticated during migration got tokens that new servers rejected. This affected 217 active sessions. Resolution: implemented grace period in token validation logic.

Issue 3: Image Processing Queue Backlog
Background jobs for image optimization failed to resume properly. 4,000+ pending transformations sat in a dead queue. Our queue monitoring flagged this 2 hours post-migration. Resolution: restarted worker pools + manually processed backlog.

Issue 4: Webhook Delivery Failures
Outbound webhooks to third-party integrations failed for 90 minutes. Our webhook service had hardcoded references to old server IPs. Resolution: updated service discovery configuration + implemented retry logic for failed deliveries.

Issue 5: Search Index Lag
Search results showed stale data for 3 hours. Index rebuilding took longer than projected due to higher data volume than test environment. Resolution: parallelized indexing process + increased indexer instance count.

Issue 6: Email Delivery Delays (ONGOING)
Transactional emails queue at higher latency than baseline. Average delivery time sits at 4.2 minutes vs typical 1.1 minutes. Email service vendor confirms receipt but processing slower than expected. Currently investigating sender reputation scores and IP warming requirements for new server addresses.

Lessons Learned

1. Test error conditions, always. We tested happy path extensively. We failed to test error page caching behavior. This cost us 47 minutes of user-facing issues.

2. Grace periods matter. Distributed systems require time for consistency. Hard cutoffs create user-facing errors. Overlap windows absorb timing issues.

3. Queue observability saves time. Our custom queue monitoring caught the image processing issue quickly. Generic monitoring would have missed it for hours.

4. Hardcoded values will bite you. Every hardcoded IP, every environment-specific constant, every assumption about infrastructure—these become problems during migration. Service discovery exists for this reason.

5. Load testing at scale reveals different problems. Search indexing worked perfectly in staging. Production volume exposed performance bottlenecks that small datasets hide.