DNS Failover: Health Checks, Weighted Routing, Recovery Procedures

Tại Sao Điều Này Quan Trọng

DNS là điểm mấu chốt giữa clients và actual service location. Nếu primary region fails, DNS phải route clients to secondary region. Nhưng DNS failover không tức thời:

1. Health check phải detect failure (10–30 seconds)
2. DNS record phải update (varies by implementation)
3. Clients phải re-query DNS (respects TTL, 1–5 minutes)

Vì vậy, DNS failover RTO = health check detection time + DNS propagation time + client timeout.

---

Internal Model: How DNS Failover Works

Cloud Load Balancer (Global Load Balancing)

Architecture:

Anycast routing:
├─ Backend service in us-central1 (region A)
├─ Backend service in us-east1 (region B)
└─ Global load balancer distributes traffic

Health checks:
├─ Check A every 10 seconds (query service A)
├─ Check B every 10 seconds (query service B)
├─ If check fails 3 times → mark unhealthy

Failover logic:

Healthy endpoints: [A, B]
└─ Route traffic to both (load balance)

A fails health check:
T=0:00  Check fails (attempt 1, A doesn't respond)
T=0:10  Check fails (attempt 2, A still doesn't respond)
T=0:20  Check fails (attempt 3, A is marked unhealthy)

Healthy endpoints: [B]
└─ Route ALL traffic to B

When A recovers:
T=5:00  Check succeeds (A responds)
T=5:10  Check succeeds (A healthy again)

Healthy endpoints: [A, B]
└─ Route traffic to both again

Key point: Failover automatic and transparent. Clients don't need to re-query DNS.

Cloud DNS with Health Checks

Architecture (manual configuration):

cloud.example.com → weighted DNS routing

Routes:
├─ 100% to us-central1.example.com (if healthy)
├─ 100% to us-east1.example.com (if us-central1 unhealthy)

Health check:
└─ Monitors us-central1 endpoint every 30 seconds

Failover logic:

us-central1 healthy:
└─ DNS responds: 35.201.1.1 (us-central1 IP)

us-central1 health check fails (3 consecutive failures):
└─ DNS responds: 35.201.2.1 (us-east1 IP)

Clients using cached DNS:
├─ TTL = 300 seconds (5 minutes)
├─ Still use old IP for up to 5 minutes
└─ Might connect to failed region (timeout, retry)

Disadvantage: TTL delay. Clients may not failover until TTL expires.

---

Health Check Mechanisms

HTTP/HTTPS Health Checks

Check request:

GET /healthz HTTP/1.1
Host: service.example.com

Check response (healthy):

HTTP/1.1 200 OK
Content-Length: 4
OK

Configuration:

Health check interval: 10 seconds
Timeout: 5 seconds
Unhealthy threshold: 3 failures
Healthy threshold: 1 success (to recover)

Timeline:

T=0:00  Check 1: success ✓
T=0:10  Check 2: success ✓
T=0:20  Check 3: timeout ✗ (service slow)
T=0:30  Check 4: timeout ✗
T=0:40  Check 5: timeout ✗
        Marked unhealthy (3 failures)

Service recovers:
T=1:00  Check 6: success ✓
        Marked healthy (1 success to recover)

Total detection time: 40 seconds (failure) + 20 seconds (recovery) = 60 seconds

TCP/UDP Health Checks

Mechanism:

Try to establish TCP connection:
├─ If succeeds: healthy
├─ If timeout: unhealthy
└─ No HTTP request needed (faster)

Configuration: Same as HTTP (interval, timeout, thresholds).

Use case: Services without HTTP endpoint (databases, custom protocols).

GCP Health Check Best Practices

1. Choose right check interval:

Fast detection (high cost):
├─ Interval = 5 seconds
├─ Check every region every 5 sec = high API cost
└─ Good for: RTO < 1 minute

Normal detection (balanced):
├─ Interval = 10 seconds
├─ Default for most services
└─ Good for: RTO = 1–2 minutes

Slow detection (cost-effective):
├─ Interval = 30 seconds
├─ Fewer checks = lower cost
└─ Good for: RTO ≥ 5 minutes

2. Set appropriate timeout:

Timeout = 5 seconds
├─ If service responds in 5s, considered healthy
├─ Too low: false positives (slow service marked unhealthy)
├─ Too high: delayed failure detection

3. Unhealthy threshold:

Unhealthy threshold = 2–3
├─ Mark unhealthy after 2–3 consecutive failures
├─ Prevents flaky network from causing thrashing
├─ Lower threshold = faster detection

---

TTL (Time-To-Live) Impact

What is TTL?

TTL = how long client can cache DNS response.

DNS response from cloud.example.com:
35.201.1.1 (TTL = 300 seconds)

Client caches this:
T=0:00   Query DNS, get 35.201.1.1
T=0:05   Use cached response (no DNS query)
T=0:10   Use cached response (no DNS query)
...
T=4:59   Use cached response (last second of cache)
T=5:00   Cache expires, query DNS again

TTL Impact on Failover

Scenario: Primary fails at T=0:00

Clients with fresh DNS cache (just queried):
├─ Have IP of failed region (35.201.1.1)
├─ Try to connect → timeout/connection refused
├─ Retry with exponential backoff (1s, 2s, 4s, 8s, ...)
├─ Eventually retry DNS (after app timeout, ~30s)
├─ Query succeeds, get secondary IP (35.201.2.1)
└─ Total time: 30–90 seconds (depends on app retry logic)

Clients with old DNS cache (TTL about to expire):
├─ TTL expires, query DNS
├─ Get secondary IP immediately
├─ Connect to secondary (RTO = query latency + connection time)
└─ Total time: < 100ms (fast)

Average RTO = 10–60 seconds (across client base)

TTL Trade-Off

TTL = 60 seconds:
├─ Failover faster (clients re-query sooner)
├─ But: higher DNS query load (every 60 sec × clients)
└─ Cost: GCP DNS query volume scales with TTL

TTL = 3600 seconds (1 hour):
├─ Lower DNS query cost
├─ But: failover slower (clients use stale IP for up to 1 hour)
└─ Risk: during failover, some clients still try old region

Typical choice: TTL = 300 seconds (5 minutes)
├─ Balance: reasonable failover speed + acceptable DNS cost

Recommended TTL by RTO

RTO < 1 minute     → TTL = 60 seconds
RTO < 5 minutes    → TTL = 300 seconds
RTO < 30 minutes   → TTL = 600 seconds
RTO > 1 hour       → TTL = 3600 seconds

---

Global Load Balancer Failover

Active-Active Load Balancing

Architecture:

Global Load Balancer (anycast IP)
├─ Analyzes client request origin
├─ Routes to nearest healthy backend
└─ Transparent failover (no client-side awareness)

Backends:
├─ us-central1 instance group
├─ us-east1 instance group
├─ both same priority (equal load balancing)

Health checks:
├─ Monitor us-central1 group
├─ Monitor us-east1 group

Failover (automatic):

us-central1 healthy, us-east1 healthy:
└─ Route 50% clients to us-central1, 50% to us-east1

us-central1 fails:
T=0:00  Health check 1 fails
T=0:10  Health check 2 fails
T=0:20  Health check 3 fails, mark us-central1 unhealthy

New routing:
└─ Route 100% clients to us-east1

In-flight connections to us-central1:
├─ Already established, continue
├─ Eventually timeout (connection hangs)
├─ Client retry on timeout, route to us-east1
└─ Total: few hundred ms (connection timeout) + reconnect

Advantage: Automatic, no client awareness, transparent.

Disadvantage: In-flight connections might fail (client must retry).

Weighted Load Balancing

For gradual failover:

us-central1 unhealthy: weight = 0% (no traffic)
us-east1 healthy: weight = 100% (all traffic)

Or canary deployment:

us-central1 (new version): weight = 10% (test with 10% traffic)
us-east1 (stable version): weight = 90% (keep on stable)

---

Manual DNS Failover Procedure

For services not using Global Load Balancer:

Manual failover steps:

1. Detect primary failure (monitoring alert)
2. Confirm it's real (not false alarm)
3. Update DNS:
   - Change primary to secondary
   - Update TTL to low value (60 sec) for faster recovery
4. Monitor secondary health
5. Investigate and fix primary
6. DNS update to re-enable primary:
   - Change back to primary
   - Rebalance if needed

Example (Cloud DNS):

# Before failover (normal state)
gcloud dns record-sets list --zone=prod-dns-zone
# OUTPUT:
# NAME: cloud.example.com.
# TYPE: A
# TTL: 300
# RDATA: 35.201.1.1 (us-central1)

# Failover triggered:
gcloud dns record-sets update cloud.example.com. \
  --zone=prod-dns-zone \
  --rrdatas=35.201.2.1 \
  --ttl=60

# Update back to primary (after fix):
gcloud dns record-sets update cloud.example.com. \
  --zone=prod-dns-zone \
  --rrdatas=35.201.1.1 \
  --ttl=300

---

Health Check Patterns

Pattern 1: Simple HTTP Health Check

Service: api.example.com
├─ Backend: /api/health
├─ Healthy response: 200 OK, body "OK"
├─ Unhealthy response: 500+ status code

Configuration:
├─ Check interval: 10 seconds
├─ Timeout: 5 seconds
├─ Unhealthy threshold: 2 failures
└─ Healthy threshold: 1 success

Pattern 2: Dependency-Aware Health Check

Service: app.example.com
├─ Depends on: database, cache, message queue
├─ Health endpoint checks:
│  ├─ Database connectivity
│  ├─ Cache availability
│  ├─ Message queue connectivity
│  └─ Local service status
└─ Returns 200 only if ALL dependencies healthy

Benefit:
├─ If database down, health check fails
├─ Even though local service running
├─ Prevents routing traffic to broken instance

Pattern 3: Gradual Drain on Unhealthy

When service becomes unhealthy:
├─ New connections routed away (via load balancer failover)
├─ Existing connections: connection draining
├─ Close timeouts: 30 seconds to drain
├─ After 30 sec: forcefully close

Benefit:
├─ In-flight requests complete gracefully
├─ No abrupt connection termination
├─ Less client-side error handling needed

---

Validation: Testing Failover

Test 1: Health Check Responsiveness

Monthly:

# Check actual health endpoint
curl -v https://api.example.com/health

# Simulate failure (disable health endpoint)
# (manual or via feature flag)

# Verify health check detects within target time
# Expected: detected within 30 seconds
# Actual: 45 seconds (too slow, adjust threshold)

Test 2: DNS Failover Verification

Monthly:

# Query DNS before failover
nslookup cloud.example.com
# Result: 35.201.1.1 (primary)

# Trigger failover (simulate primary failure)
# Update DNS to secondary

# Query DNS after failover
nslookup cloud.example.com
# Result: 35.201.2.1 (secondary)

# Measure: time from failover trigger to DNS update
# Expected: < 1 minute for manual, < 30 seconds for automatic

Test 3: Full Failover Drill (Quarterly)

1. Disable primary region (or disable health checks)
2. Verify: load balancer routes to secondary
3. Verify: in-flight connections handled gracefully
4. Verify: new connections route to secondary
5. Measure: total failover time (detection + routing)
6. Re-enable primary, verify recovery

---

Anti-Patterns

Anti-pattern 1: "TTL Too High"

Symptom: TTL = 1 hour for critical service.

Problem:

• During failover, clients continue using stale IP for up to 1 hour
• RTO = 1 hour (not acceptable for critical service)

Right approach: TTL = 60–300 seconds for critical services.

Anti-pattern 2: "Health Check Too Sensitive"

Symptom: Unhealthy threshold = 1 (fails on first error).

Problem:

• Transient network glitch → failover triggered
• Cascading failures (unnecessary failover)
• Service flapping (constantly switching regions)

Right approach: Unhealthy threshold = 2–3 (require multiple failures).

Anti-pattern 3: "No Graceful Connection Draining"

Symptom: When health check fails, connections immediately closed.

Problem:

• In-flight requests fail (client sees 500 error)
• Client retry logic kicks in (thundering herd)
• Unnecessary errors during failover

Right approach: Enable connection draining (30–60 second timeout) so in-flight requests complete gracefully.

---

Summary

DNS failover combines:

1. Health checks: Detect failures (10–30 seconds)
2. DNS updates: Route to secondary region
3. TTL: Control client cache duration (affects failover speed)
4. Load balancer: Automatic routing (if using Global LB)

RTO depends on:

• Health check detection time
• DNS update time
• Client TTL cache expiration
• Client retry logic

Typical RTO: 10–60 seconds (with Global LB), 1–5 minutes (with manual DNS).

---

References

• Cloud Load Balancing Health Checks
• Cloud DNS Documentation
• Weighted Routing Policy
• Connection Draining