1. Find the tests that failed
Choose the approach that fits how you test releases.From a pre-release workflow
If a workflow tests the release before you cut it, have the analytics CLI write a summary of the upload, and give that file to the agent.- In the workflow, upload the results with analytics CLI 1.0.0 or newer, adding
--summary-output-file trunk-summary.json. - Pass
trunk-summary.jsonto the agent, for example as a workflow artifact.
failuresholds the failed tests that weren’t quarantined, andquarantinedthe ones that were. Both are grouped by file.- Each test has its
id,name,file,failureMessage,failureTextandlink. idis the test’s ID in the API. Use it astestIdin the calls below. It isnullwhen Trunk couldn’t identify the test’s collection for the run; skip those tests, or find them by name withGET /v2/tests.
The summary reports each test’s final result in the report. A test that your runner retried and then passed counts as passed.
From a failure count monitor on release branches
Have Trunk label every test that fails on a release branch, then start from that label.- In the collection that holds the release branch’s results, create a failure count monitor:
- Branches: the release branch, or a pattern such as
release/*. - Failure count:
1. - Resolution timeout: long enough to cover your release cycle, so a test stays labeled after a single failure. The web app allows up to 14 days.
- Action: Apply labels, with a label such as
failed-on-release. See Automatic labeling from monitors.
- Branches: the release branch, or a pattern such as
- Have the agent list the labeled tests with
GET /v2/tests, usingtestCollectionIdandlabel=failed-on-release.
2. Enrich each test
Read quarantine from the test’s
quarantineState, not the run’s. A run’s quarantineState records what its upload reported.3. Report
Sort each test by its current state. Adjust these to your release policy:- Post a report grouped by severity. For each test, give its name,
codeowners, failure rate and a link to its latest failing run. - Track the tests that need attention with
POST /v2/tests/{testId}/add-labels, for examplerelease-risk, andPOST /v2/tests/{testId}/create-ticket. To fix them, see Fix Flaky Tests. - To hold the release on the result, have the agent exit non-zero when any test needs attention, before your tag or publish job.
- Don’t let the agent quarantine tests or mark them flaky to improve the result.
Related
- Fix Flaky Tests. Hand the tests that need attention to a coding agent.
- Explain a Failed Pull Request. Use the same upload summary to explain failures on a pull request.
- Failure count monitor. How the monitor labels tests that fail on a branch.