The Pipeline Was Green Because the Test Stage Never Ran

The build was green. Production had a null reference the tests cover. The test stage had not run. Azure DevOps treats a skipped stage as a successful pipeline unless you tell it otherwise. A condition that was true on the Tuesday you wrote it — "only test on main" — is false on the pull request, the stage skips, and the PR policy sees green.

This is a different failure from a flaky test. A flaky test is red sometimes. A skipped stage is never red. Nobody gets paged. The only signal is a stage in the timeline with a hollow circle, and people who look at the badge do not open the timeline.

How a stage skips without failing

A stage skips when its condition is false, or when a dependency was skipped and the default condition (succeeded()) is not met, or when a runtime parameter said so. The pipeline result stays succeeded.

- stage: Test
  dependsOn: Build
  condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
  jobs:
    - job: Unit
      steps:
        - script: dotnet test --no-build

On a pull request, Build.SourceBranch is refs/pull/.../merge, the condition is false, and Test does not run. The author thinks tests ran because the pipeline is required and it passed.

The condition you usually want on a PR validation pipeline is succeeded(). Branch filters belong on the trigger, or on a stage that is explicitly optional, such as a production deploy.

Make "did not run" a failure

For any stage that must run before merge:

1. Delete the branch condition, or move it to trigger / pr include lists so the pipeline does not start when it should not.
2. Keep condition: succeeded() so a failed build still skips tests for the right reason — there is nothing to test — and a successful build always tests.
3. Add a final stage that fails if Test was skipped.

- stage: Gate
  dependsOn: Test
  condition: always()
  jobs:
    - job: RequireTests
      variables:
        testResult: $[ stageDependencies.Test.result ]
      steps:
        - script: |
            if [ "$(testResult)" != "Succeeded" ]; then
              echo "Test stage result is $(testResult)"
              exit 1
            fi

always() is what makes this stage exist even when Test was skipped. Without it, Gate skips too, and you are back to a green pipeline. The dependency result for a skipped stage is Skipped. Succeeded is the only value that may pass.

What to click before you trust the badge

Open the run. Every required stage should be filled green, not outlined. A hollow stage is a stage that did not run. Add that glance to the PR checklist for one week and you will find the conditions that were copied from a deploy pipeline into a CI pipeline.

Slot swaps and smoke tests, covered in the slot-swap pipeline, have the same shape one stage later. A smoke test that is skipped is not a smoke test. Gate it the same way. The badge is allowed to be green only after the stages you care about have a result of Succeeded.