Skip to main content

Healthchecks and scaling

Available

Deployed and supported. The fields and commands described here are stable within version v1alpha1.

A healthcheck tells the platform when a container is ready to receive traffic and when it must be restarted. replicas, strategy and autoscale set how many instances run and how they are replaced during an update.

Prerequisites​

  • A service of kind deployment (these fields only apply to it).
  • No volumes on the service for replicas > 1 or autoscale (Storage).

Steps​

1. Add a probe and replicas​

api.yml
apiVersion: deploy.stackngo.io/v1alpha1
kind: App
metadata:
  name: api
spec:
  services:
    api:
      image: registry.gitlab.com/acme/api:1.2.0
      ports: ["8080"]
      healthcheck:
        http: /healthz          # GET /healthz on the first port
        interval: 15s
        timeout: 5s
        retries: 3
        startPeriod: 30s        # gives the application 30 s to start
      replicas: 2
      strategy: { type: rolling, maxSurge: 1, maxUnavailable: 0 }
      autoscale: { max: 6, cpu: 70 }
      ingress: { port: 8080 }
sng apply api.yml

2. Follow an update​

sng app patch api --image registry.gitlab.com/acme/api:1.3.0
sng app describe api
App      : api
Statut   : en cours
Message  : api : 1/2 à jour, 2 disponibles
SERVICE  KIND        IMAGE                               PHASE        URL                                          DETAIL
api      deployment  registry.gitlab.com/acme/api:1.3.0  Progressing  https://api-api.<tenant>.dev.stackngo.app/  RollingOut 1/2 à jour, 2 disponibles

With maxUnavailable: 0, there are always 2 instances available during the replacement.

Verification​

sng app describe api returns to Statut : prête, Reconciled 2/2 prêts. During a scale-up, the number of instances displayed exceeds replicas, up to autoscale.max.

healthcheck​

Exactly one of the three tests, plus common settings:

FieldDefaultRole
http: /path—GET request; healthy if the response is 2xx or 3xx
portfirst TCP port in portsport of the http test
tcp: 5432—healthy if the TCP connection opens
cmd: ["pg_isready"]—command run in the container; healthy if it exits with code 0
interval10sdelay between two tests (rounded to the second, 1 s to 1 h)
timeout5smaximum duration of a test
retries3consecutive failures before declaring the container unhealthy (1 to 30)
startPeriodnonedelay before the first liveness test, for slow startups

The same probe is used twice:

  • readiness: an instance that fails no longer receives traffic, without being restarted;
  • liveness: after retries failures, the container is restarted. startPeriod only applies to this test.

Without a healthcheck, an instance is considered ready as soon as its container has started.

Choose the right test

http for an API or a website, tcp for a database or a cache without an HTTP endpoint, cmd when the image provides a tool (pg_isready, redis-cli ping). Compose healthcheck CMD / CMD-SHELL translate to cmd.

replicas and strategy​

FieldDefaultRole
replicas1number of instances (0 to 100); 0 stops the service without deleting it
strategy.typerollingrolling: gradual replacement; recreate: full stop then restart
strategy.maxSurge1extra instances during the update (number or 25%); rolling only
strategy.maxUnavailable0unavailable instances tolerated during the update; rolling only

maxSurge: 0 and maxUnavailable: 0 together are rejected (the update could not progress). A service with volumes is always in recreate.

sng app scale api 3 changes replicas without touching the rest; it is refused if the service has an autoscale (modify autoscale.max) or a volume with a number greater than 1.

autoscale​

replicas: 2
autoscale: { max: 6, cpu: 70 }
FieldDefaultRole
max—maximum number of instances (1 to 100), ≥ replicas
cpu80target utilization, in % of the CPU reservation of the size

The minimum is replicas (1 if absent or 0). The number of instances goes up when average CPU utilization exceeds the target, then comes back down. The target is measured against the reservation of the size: an xs service (500m reserved) at 70% triggers at around 350m of CPU used.

With autoscale, the number of instances is no longer driven by the manifest; replicas only serves to set the floor.

Going further​

  • A deployment that does not complete (instances that never become ready) ends in Failed, reason RolloutFailed. The usual causes: image that starts as root, probe on the wrong port or path, missing Secret. sng app logs api gives the answer.
  • There is no rollback command for a service: reapply the old image.
  • A rolling update temporarily creates replicas + maxSurge instances: with a tight plan, it is this peak that can trigger QuotaExceeded (Sizes and resources).

See also​