What is deploy verification?#
With deploy verification on, the agent samples the checks you define for a while after a deploy finishes. When a check breaches its limit several times in a row, the deploy fails and, if chosen, rolls back to the image that was running before.
EJO runs this as a Verify deployment step added to the end of the pipeline. You do not add this step yourself.
Prerequisites#
- For Prometheus checks, a Prometheus the agent can reach. It can be an in-cluster address.
Setup#
- Open the Deploy verification section on the environment page.
- Turn on Verify deploys here.
- Enter the settings:
| Setting | Description |
|---|---|
| Prometheus URL | The Prometheus address the agent can reach. |
| Token variable (optional) | The name of a SECRET variable. Its value is sent as Authorization: Bearer and is never shown. |
| Duration | How long the checks are sampled. Default 5m; 30s to 60m. |
| Interval | Time between two samples. Default 30s; 10s to 5m. |
| Consecutive breaches | How many times in a row a check must breach its limit for the deploy to fail. Default 3; 1 to 10. |
| On failure | Rollback or Only fail the deploy. |
- Define up to 10 checks with Add check:
- Prometheus: a PromQL query, a comparison (Healthy when value is
<,<=,>,>=) and a Threshold. - HTTP: a URL and an Expected status.
- Prometheus: a PromQL query, a comparison (Healthy when value is
- Try the checks with Test.
Note: Test runs in EJO, not on the agent. If Prometheus is only reachable from the agent's network, the test can fail even though deploys work.
Queries and URLs can use these template variables; the agent fills them in for each deploy: {{.Service}}, {{.Environment}}, {{.Namespace}}, {{.Deployment}}, {{.CommitSHA}}, {{.ImageDigest}}.
sum(rate(http_requests_total{service="{{.Service}}",code=~"5.."}[5m]))
Service-specific checks#
Services can add their own checks with the verify: block of their .ejo file. These are added next to the environment checks; they never remove them. See The .ejo file.
When does the verify step run?#
- When verification is on for the environment and there is at least one check, either on the environment or in the service's
.ejofile. - Pull request and teardown runs are not verified.
The result shows on the deployment page: running, passed, failed or inconclusive.
Automatic rollback#
When On failure is Rollback and verification fails, EJO starts a rollback deployment to the image that ran before. That deployment shows auto-rollback as its trigger and a reason starting with auto: verification failed: .
The environment's When prod fail-rate is high setting decides what happens when the failure rate in production rises: Notify only, freeze or rollback.
Manual rollback#
- Use Roll back on a deployment page to go back to an earlier deploy.
- Use Rollback last succeeded deploy on the environment page to return to the last successful deploy.
If the environment requires approval, rollbacks wait for approval too. Freeze windows do not block rollbacks. With a supply chain policy on, the rollback image must pass the policy as well.
Run a failed run again#
| Option | What it does |
|---|---|
| Rerun failed steps | Starts a new deployment that reruns the steps that did not succeed and everything after them. Steps that succeeded are reused. |
| Rerun from here | Reruns the step you pick and every step after it. Earlier successful steps are reused. |
| Retry | Runs the whole pipeline again from the start. |
A rerun uses the pipeline version of the source deployment. Files from reused steps are not restored; if later steps need them, use Rerun from here on an earlier step or Retry.
Debug a failed step#
With Debug on failure on for the binding, when a step fails the agent keeps the step's container alive and you can open an interactive debug session into it from the console.
- The session only attaches to the failed step's container; there is no shell on the agent host.
- Only admins and the user who started the run can open one.
- At most 30 minutes in total per step.
- Older agents show "upgrade the agent".