Zero-Downtime Deployment với Docker, Nginx và Spring Cloud

Copycopy code to clipboard
Client (HTTPS:443)
    │
    ▼
Nginx (Host OS) ──► proxy_pass 127.0.0.1:8888
    │
    ▼
Spring Cloud Gateway (:8888)
    │
    ▼
Eureka Discovery (:8761)
    │
    ├── SSO Service (:8000)
    ├── Category Service (:8001)
    ├── Payment Service (:8628)
    ├── Report Service (:8027)
    ├── Notification Service (:8023)
    └── ... (10+ services khác)

Copycopy code to clipboard
┌─────────────┐
        │    Nginx     │
        │  (upstream)  │
        └──────┬──────┘
               │
        ┌──────┴──────┐
        ▼             ▼
  ┌──────────┐  ┌──────────┐
  │   Blue   │  │  Green   │
  │  (v1.0)  │  │  (v1.1)  │
  │  :8888   │  │  :8889   │
  └──────────┘  └──────────┘
     active       standby

Copycopy code to clipboard
services:
  gateway-blue:
    build: ./gateway
    networks: [app-net]
    # Không cần ports mapping - Nginx truy cập qua Docker DNS
    environment:
      - SERVER_PORT=8888

  gateway-green:
    build: ./gateway
    networks: [app-net]
    environment:
      - SERVER_PORT=8888  # Cùng port, khác container IP

networks:
  app-net:
    driver: bridge

Copycopy code to clipboard
upstream gateway_backend {
    server gateway-blue:8888;
    server gateway-green:8888 backup;
}

Copycopy code to clipboard
services:
  gateway:          # Blue
    network_mode: host
    environment:
      - SERVER_PORT=8888

  gateway-green:    # Green
    network_mode: host
    environment:
      - SERVER_PORT=8889  # Port khác

Copycopy code to clipboard
upstream gateway_backend {
    server 127.0.0.1:8888;        # Blue (active)
    server 127.0.0.1:8889 backup; # Green (standby)
}

Copycopy code to clipboard
upstream gateway_backend {
    server 127.0.0.1:8888 backup; # Blue (giờ là standby)
    server 127.0.0.1:8889;        # Green (giờ là active)
}

Copycopy code to clipboard
Nginx ──► upstream {
            127.0.0.1:8888 (Blue)
            127.0.0.1:8889 (Green)  backup
          }
              │
              ▼
       Gateway (Blue hoặc Green)
              │
              ▼
         Eureka ──► 12 Backend Services
                    (giữ nguyên rolling update)

Copycopy code to clipboard
┌──────────────────────────────────────────────────────────┐
│                    ROLLING UPDATE FLOW                    │
├──────────────────────────────────────────────────────────┤
│                                                          │
│  1. POST /actuator/service-registry?status=OUT_OF_SERVICE│
│     → Eureka đánh dấu service là OUT_OF_SERVICE          │
│                                                          │
│  2. sleep 45s                                            │
│     → Chờ Gateway đồng bộ cache, ngừng route tới service │
│                                                          │
│  3. docker stop + docker rm (container cũ)               │
│     → Graceful shutdown, hoàn thành in-flight requests   │
│                                                          │
│  4. docker compose up -d (container mới)                 │
│     → Container mới khởi động                            │
│                                                          │
│  5. Health check polling (tối đa 6 phút)                 │
│     → 72 lần × 5s, chờ /actuator/health trả về UP       │
│                                                          │
│  6. Container mới tự đăng ký vào Eureka                  │
│     → Gateway bắt đầu route traffic đến service mới     │
│                                                          │
└──────────────────────────────────────────────────────────┘

Copycopy code to clipboard
#!/bin/bash
set -e

SERVICE_NAME=$1
PORT=${SERVICE_PORTS[$SERVICE_NAME]}

# Tạo file chặn cảnh báo healthcheck trong thời gian deploy
touch "/tmp/healthcheck_ignore_${SERVICE_NAME}"

# Bẫy lỗi: tự động thông báo nếu script crash
trap 'send_deploy_email "FAILED" "Script crash"; exit 1' ERR

# ── Bước 1: Git Pull + Build ──
cd $SERVICE_NAME && git fetch origin && git reset --hard origin/$BRANCH && cd ..
mvn -f common/pom.xml clean install -DskipTests -B
docker compose build $SERVICE_NAME

# ── Bước 2: Tìm container cũ đang chạy ──
OLD_CONTAINER=$(docker ps \
  --filter "label=com.docker.compose.service=$SERVICE_NAME" \
  --filter "status=running" --format "{{.Names}}")

# ── Bước 3: Deregister khỏi Eureka TRƯỚC khi stop ──
docker exec $OLD_CONTAINER curl -s -X POST \
  "http://localhost:$PORT/actuator/service-registry?status=OUT_OF_SERVICE"

# ── Bước 4: Chờ Gateway đồng bộ cache routing ──
sleep 45

# ── Bước 5: Dừng và thay thế container ──
docker stop $OLD_CONTAINER && docker rm $OLD_CONTAINER
docker compose up -d --scale ${SERVICE_NAME}=1 ${SERVICE_NAME}

# ── Bước 6: Chờ container mới healthy ──
for ((retry=1; retry<=72; retry++)); do
    HEALTH=$(docker exec $NEW_CONTAINER curl -s \
      "http://localhost:$PORT/actuator/health" || echo "failed")

    if echo "$HEALTH" | grep -q '"status":"UP"'; then
        echo "✅ Service $SERVICE_NAME đã UP!"
        break
    fi
    sleep 5
done

# Dọn dẹp
rm -f "/tmp/healthcheck_ignore_${SERVICE_NAME}"
send_deploy_email "SUCCESS" ""

Copycopy code to clipboard
name: Deploy Service
on:
  push:
    branches: [dev, production]

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: ${{ github.ref_name == 'production' && 'Production' || 'Dev' }}
    steps:
      # (Tùy chọn) Kết nối VPN nếu VPS nằm trong mạng nội bộ
      # - name: Connect to VPN
      #   uses: <vpn-provider>/github-action@v2
      #   with:
      #     credentials: ${{ secrets.VPN_CREDENTIALS }}

      - name: Deploy via SSH
        uses: appleboy/ssh-action@v1.0.3
        with:
          host: ${{ secrets.DEPLOY_HOST }}
          port: ${{ secrets.DEPLOY_PORT }}
          username: ${{ secrets.DEPLOY_USER }}
          key: ${{ secrets.DEPLOY_SSH_KEY }}
          timeout: 30m
          script: |
            cd /app && ./deploy-service.sh <service-name>

Copycopy code to clipboard
environment:
  # Cho phép Spring Boot hoàn thành các request đang xử lý trước khi tắt
  - SERVER_SHUTDOWN=graceful
  # Thời gian tối đa chờ các request hoàn thành (30 giây)
  - SPRING_LIFECYCLE_TIMEOUT_PER_SHUTDOWN_PHASE=30s

Copycopy code to clipboard
healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8888/actuator/health"]
  interval: 15s
  timeout: 5s
  retries: 3

Copycopy code to clipboard
server {
    listen 443 ssl;
    server_name gateway.example.com;

    ssl_certificate     /path/to/bundle.crt;
    ssl_certificate_key /path/to/private.key;
    ssl_protocols       TLSv1.2 TLSv1.3;

    # Security Headers
    add_header Strict-Transport-Security "max-age=31536000" always;
    add_header X-Frame-Options DENY always;
    add_header X-Content-Type-Options nosniff always;

    # Rate Limiting
    limit_req zone=api_limit_zone burst=100 nodelay;

    location / {
        proxy_pass http://127.0.0.1:8888;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Timeout cao cho các API xử lý lâu
        proxy_connect_timeout 300;
        proxy_send_timeout 300;
        proxy_read_timeout 300;
    }
}

Copycopy code to clipboard
Timeline deploy 1 service:

t=0s      POST OUT_OF_SERVICE → Eureka
t=0-45s   Gateway vẫn có thể route đến service cũ (cache chưa sync)
t=45s     docker stop (graceful shutdown 30s)
t=75s     Container cũ tắt hoàn toàn
t=75s     docker compose up (container mới bắt đầu khởi động)
t=75-135s Spring Boot startup (30-60s)
t=135s    Container mới UP, đăng ký Eureka
t=135-180s Gateway đồng bộ cache, bắt đầu route đến container mới

⚡ Window gap: ~75s đến ~135s (khoảng 1 phút)
   Trong khoảng này, không có instance nào phục vụ