Upgrade Scope

Upgrade Window

Plan the upgrade during a low-traffic window. If the environment has only one gateway instance, the upgrade usually requires a short service interruption. If the environment uses multiple gateway instances behind Nginx, Ingress, or a load balancer, upgrade nodes in batches and keep at least one healthy node serving traffic.

Before the upgrade, notify business owners of:

• Planned start time and end time.
• Expected impact on new requests and in-flight requests.
• Contact person during the upgrade.
• Rollback trigger conditions.

Important Limitations

Read this section before any operation.

• Database migrations are forward-only and append-only. Do not modify an already applied migration file, including comments.
• If a migration has changed table structure or data semantics, application-layer rollback may not be sufficient. Database restore may be required.
• Application-layer rollback applies to failures such as image startup failure, configuration file error, health check failure, and UI loading failure when no destructive schema or data change has been applied.
• Database restore applies to failures after schema or data migration when the old application version cannot read the new schema safely.
• When upgrading a production environment, prefer rolling back the entry layer first, then the application layer, and restore the database only as the last step.
• Do not reuse the production database for migration rehearsal. Use a copied database or a test database.
• During a grayscale upgrade, old and new versions may run at the same time only if the release notes explicitly state compatibility.

Applicable Boundaries

Check Current Version

Confirm the currently running package version, image tag, database migration version, and Console UI version. Record the current entry domain name, deployed nodes, and upstream model services.

Check Service Status

Confirm that gateway services, PostgreSQL, entry proxy, and upstream model services are healthy before upgrade. Do not start an upgrade while the existing environment is already abnormal.

Check Change Content

Read the release notes and identify whether the target version includes database migration, Route Policy changes, credential changes, Console UI changes, or LobsterPool-compatible entry changes.

Check Backup Conditions

Complete the following checks before the upgrade.

Back Up the Database

Back up the database before running any migration.

pg_dump -h {DB_HOST} -p {DB_PORT} -U {DB_USER} -Fc {DB_NAME} > zstack-api-router-$(date +%F-%H%M).backup

After backup, verify that the backup file is generated and readable.

pg_restore -l zstack-api-router-{TIME}.backup >/tmp/zstack-api-router-restore-list.txt

Back Up Deployment Directory

Back up at least the following files or directories:

• Application configuration file.
• Environment variable file.
• Docker Compose file, systemd unit, or Kubernetes manifest.
• TLS certificate and private key.
• Nginx, Ingress, or load balancer configuration.
• Script files used for offline deployment or local start.

Recommended backup directory:

sudo install -d -m 0750 /var/backups/zstack-api-router
sudo cp -a /etc/zstack-api-router /var/backups/zstack-api-router/etc-$(date +%F-%H%M)
sudo cp -a /opt/zstack-api-router /var/backups/zstack-api-router/app-$(date +%F-%H%M)

Back Up Entry Configuration

Back up Nginx, Ingress, load balancer, domain name, and TLS configuration. The entry configuration is usually the fastest rollback path during grayscale or production upgrade.

Verify Backup

Verify database backup metadata, file size, and read permission. Record the backup path in the upgrade record before any migration is executed.

Scenario

Use this procedure for a test environment, proof-of-concept environment, or single-node production environment that can accept a planned interruption.

Prerequisites

• Database backup, deployment directory backup, and entry configuration backup have completed.
• The target package or image has been obtained and verified.
• The upgrade window and rollback contact have been confirmed.

Procedure

1. Stop traffic or notify users.
2. Pull or import the new package.
3. Update configuration.
4. Run database migration.
5. Restart application components.
6. Perform upgrade verification.

Stop traffic or notify users:

If the gateway has no redundant instance, stop upstream traffic or notify users that model invocation may be interrupted during the upgrade.

Pull or import the new package:

For online environments, pull the new image:

docker pull {REGISTRY}/zstack-api-router:{TARGET_VERSION}

For offline environments, import the image:

docker load -i zstack-api-router-{TARGET_VERSION}.tar

Update configuration:

Compare the new example configuration with the current production configuration. Keep existing database, credential, domain name, TLS, and upstream model service settings, and add only required new configuration items.

Run database migration:

Run migration with the target version package. Do not manually edit historical migration files.

docker compose run --rm zr-migrate

If migration fails, stop immediately and keep the error log, migration version, and database backup path.

Restart services:

Restart the application components.

docker compose up -d zr-server zr-worker

Check service status:

docker compose ps
docker compose logs --tail=200 zr-server
docker compose logs --tail=200 zr-worker

Verification

After restart, verify health check, model invocation, Usage Log, Audit Log, Route Policy, quota, and metrics before restoring normal business traffic.

Scenario

This section applies to production environments with multiple gateway instances or an entry layer that can split traffic.

Prerequisites

• At least one old-version instance can continue serving traffic.
• The entry layer can remove or restore a gateway instance quickly.
• Database backup and configuration backup are complete.
• Release notes confirm whether old and new versions can coexist during grayscale.

Procedure

Production environments should use rolling upgrade when multiple gateway instances are available.

1. Remove one gateway instance from the entry layer.
2. Upgrade the application package or image on that instance.
3. Run the migration only once according to the release notes. If the release notes require migration before application startup, complete migration before replacing traffic.
4. Start the upgraded instance and confirm the health check is normal.
5. Send a small amount of traffic to the upgraded instance.
6. Verify model invocation, Route Policy, quota, Usage Log, Audit Log, and metrics.
7. Continue upgrading the remaining instances in batches.
8. Keep old-version images and configuration backups until the observation window ends.

During rolling upgrade, do not send production traffic to a new instance whose health check, model invocation, or audit writing has not been verified.

Verification

Verify the upgraded instance with grayscale traffic first. Check model invocation success rate, latency, route decision, Usage Log, Audit Log, quota, and upstream channel health.

Notes

Do not expand grayscale traffic if error rate, latency, audit writing, or usage statistics are abnormal.

Scenario

If the delivery package includes Console UI changes, perform the following checks after replacing static resources or restarting the UI service:

Procedure

1. Open the Console UI by using the external domain name.
2. Confirm that login and tenant switching are normal.
3. Open API Key, Channel, Model, Route Policy, Usage Log, Audit Log, Health Status, and Metrics pages.
4. Create or edit one non-production configuration item in a test tenant.
5. Refresh the browser and confirm the page still displays the latest data.

If the browser displays an old page, clear the static resource cache or confirm that Nginx, CDN, or Ingress cache has been refreshed.

Rollback

If only the Console UI is abnormal and backend APIs are healthy, restore the old static resources or old UI image first. Keep the backend version unchanged unless the release notes require backend rollback.

Scenario

When upgrading from an old database, complete the following checks before migration:

Procedure

• Confirm source database version and target version compatibility.
• Count API Keys, channels, models, Route Policies, Usage Log records, and Audit Log records.
• Check whether there is invalid JSON, duplicate name, or orphaned relationship data.
• Run migration rehearsal on a copied database.
• Record the migration duration and disk growth observed during rehearsal.

After migration, verify that key metadata can be queried from the Console UI and that new requests can be written to Usage Log and Audit Log.

Notes

Run migration rehearsal on a copied database before production migration. Do not use the production database for rehearsal.

Health Check

curl -f https://{GATEWAY_DOMAIN}/healthz

The health check should return a successful status. If the environment provides a readiness endpoint, confirm that database and upstream model service dependencies are ready.

Model Invocation Verification

Use a test API Key to call a test model.

curl https://{GATEWAY_DOMAIN}/v1/chat/completions \
  -H "Authorization: Bearer {API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5-7b-instruct",
    "messages": [
      {"role": "user", "content": "Return the word ok."}
    ]
  }'

Expected result:

• The request returns successfully.
• Usage Log records tenant, organization, model, token usage, latency, and result.
• Audit Log records credential, route decision, and operation result.

Route and Quota Verification

Verify that Route Policy, model permission, and quota still take effect:

• Use an allowed organization to call an allowed model. The request should succeed.
• Use an organization without permission to call the model. The request should be rejected.
• Use a test quota close to its limit. The gateway should reject requests after the quota is exhausted.

Monitoring Verification

Confirm that metric collection and alerting work normally:

• Request count, error count, latency, and upstream failure metrics are updated.
• Gateway health status is visible in the monitoring system.
• Logs are written to the expected directory and are included in log collection.

Rollback Principles

When an upgrade exception occurs, determine the rollback level based on the failure point.

Roll Back the Entry Layer

If only part of the traffic is abnormal, first remove the upgraded instance from Nginx, Ingress, or the load balancer, and route traffic back to the old healthy instance.

This method applies to:

• New version health check failure.
• New version has increased model invocation error rate.
• Console UI loading failure.
• Metrics or log collection failure on one instance.

Roll Back the Application Layer

If the new application fails to start or cannot serve traffic but the database migration has not changed schema or data semantics, restore the old image, application package, and configuration.

docker compose down
docker tag {REGISTRY}/zstack-api-router:{OLD_VERSION} zstack-api-router:current
docker compose up -d

After application rollback, re-run health check, model invocation, Usage Log, and Audit Log verification.

Restore the Database

If migration has changed schema or data semantics and the old application cannot work with the new schema, restore the database from the pre-upgrade backup during an approved maintenance window.

pg_restore -h {DB_HOST} -p {DB_PORT} -U {DB_USER} -d {DB_NAME} --clean --if-exists zstack-api-router-{TIME}.backup

Before database restore, stop all application instances that can write to the database to avoid mixed old and new data.

The New Version Cannot Start After Migration

Check database connection, configuration compatibility, missing environment variables, image tag, and application logs. If migration has completed, do not directly start the old version unless compatibility has been confirmed.

Migration Reports That a Historical Migration Has Been Modified

This usually indicates that an applied migration file has been changed. Do not edit applied migration files. Restore the correct migration file from the release package, confirm the checksum, and contact support if the database already contains inconsistent migration records.

Model Invocation Error Rate Increases After Upgrade

First remove the upgraded instance from the entry layer, then check Route Policy, channel health, upstream model service status, timeout configuration, and quota policy. After the error rate returns to normal, analyze logs and decide whether to continue the upgrade.

Console UI Shows Old Pages After Upgrade

Check browser cache, Nginx cache, CDN cache, Ingress static resource configuration, and static resource version. Refresh the cache and re-open the Console UI.