Automated Testing in Azure Pipelines

Q: How do I see test results in Azure Pipelines?

Publish test results using the PublishTestResults task. Results appear on the pipeline run summary page under the Tests tab. You can filter by test name, view failure details, and compare pass rates across runs. The task accepts JUnit, NUnit, VSTest, xUnit, and CTest XML formats.

Q: How do I fail a pipeline when tests fail?

The test runner itself should exit with a non-zero code on failure — most test frameworks do this by default. If you use the DotNetCoreCLI or pytest tasks, a test failure automatically fails the pipeline step. If running tests via a bash script, check the exit code and use "exit 1" to propagate the failure.

Q: What is test impact analysis in Azure Pipelines?

Test impact analysis (TIA) is a feature in the VSTest task that identifies which tests are likely affected by code changes and runs only those tests, skipping unaffected tests. This speeds up CI for large test suites. It works best with VSTest-based .NET projects and requires a baseline run to build the impact graph.

Tests without results in your pipeline are invisible. A test suite that runs but does not publish results, does not track coverage trends, and does not block on failure provides false confidence. This page shows how to wire unit, integration, and end-to-end tests into Azure Pipelines so failures are visible, coverage is tracked, and slow tests do not block fast feedback.

Where each test type runs in a pipeline

Not every test should run on every trigger. A thoughtful pipeline runs fast, targeted tests first and reserves slow tests for later stages.

Test type	When to run	Duration target	Blocks
Unit tests	Every commit, every PR	< 2 minutes	Build stage
Integration tests	Every PR merge to main	< 10 minutes	Deploy to Dev stage
End-to-end tests	After staging deployment	< 20 minutes	Staging → Prod promotion
Load tests	Scheduled or manual	Variable	Production gating (optional)

Unit tests with .NET and code coverage

# azure-pipelines.yml — .NET unit tests with coverage
- task: DotNetCoreCLI@2
  displayName: 'Run unit tests'
  inputs:
    command: 'test'
    projects: '**/*UnitTests.csproj'
    arguments: >
      --configuration Release
      --no-build
      --logger "trx;LogFileName=unit-test-results.trx"
      --collect:"XPlat Code Coverage"
      --results-directory $(Agent.TempDirectory)
    publishTestResults: false   # we'll publish manually for more control

# Publish test results to the pipeline Tests tab
- task: PublishTestResults@2
  displayName: 'Publish unit test results'
  inputs:
    testResultsFormat: 'VSTest'
    testResultsFiles: '$(Agent.TempDirectory)/**/*.trx'
    mergeTestResults: true
    failTaskOnFailedTests: true   # fail the pipeline if any test fails
    testRunTitle: 'Unit Tests - $(Build.BuildNumber)'

# Generate coverage report
- task: reportgenerator@5
  displayName: 'Generate coverage report'
  inputs:
    reports: '$(Agent.TempDirectory)/**/coverage.cobertura.xml'
    targetdir: '$(Build.SourcesDirectory)/CoverageReport'
    reporttypes: 'HtmlInline_AzurePipelines;Cobertura'

# Publish coverage results
- task: PublishCodeCoverageResults@2
  displayName: 'Publish code coverage'
  inputs:
    summaryFileLocation: '$(Build.SourcesDirectory)/CoverageReport/Cobertura.xml'
    reportDirectory: '$(Build.SourcesDirectory)/CoverageReport'
    failIfCoverageEmpty: true

Python tests with pytest and coverage

- task: UsePythonVersion@0
  inputs:
    versionSpec: '3.12'

- bash: |
    python -m pip install --upgrade pip
    pip install -r requirements.txt
    pip install pytest pytest-cov pytest-html
  displayName: 'Install dependencies'

- bash: |
    pytest tests/unit/ \
      --junitxml=$(Agent.TempDirectory)/junit.xml \
      --cov=src \
      --cov-report=xml:$(Agent.TempDirectory)/coverage.xml \
      --cov-report=html:$(Agent.TempDirectory)/htmlcov \
      --cov-fail-under=80 \
      -v
  displayName: 'Run pytest with coverage'

- task: PublishTestResults@2
  displayName: 'Publish pytest results'
  condition: succeededOrFailed()   # publish even if tests fail
  inputs:
    testResultsFormat: 'JUnit'
    testResultsFiles: '$(Agent.TempDirectory)/junit.xml'
    failTaskOnFailedTests: true
    testRunTitle: 'Python Unit Tests'

- task: PublishCodeCoverageResults@2
  displayName: 'Publish coverage'
  condition: succeededOrFailed()
  inputs:
    summaryFileLocation: '$(Agent.TempDirectory)/coverage.xml'

Integration tests against a deployed environment

Integration tests run after deployment to a dev environment. They test the actual HTTP endpoints, real database connections, and real Azure service interactions. The key challenge is passing the environment URL to the test runner.

- stage: Integration_Test
  displayName: 'Integration Tests'
  dependsOn: Deploy_Dev
  variables:
    # Pick up the app URL from the previous stage's output
    APP_URL: $[ stageDependencies.Deploy_Dev.DeployDev.outputs['deployStep.APP_URL'] ]
  jobs:
    - job: IntegrationTests
      steps:
        - checkout: self

        - task: UseDotNet@2
          inputs:
            version: '8.x'

        - task: DotNetCoreCLI@2
          displayName: 'Run integration tests'
          inputs:
            command: 'test'
            projects: '**/*IntegrationTests.csproj'
            arguments: >
              --configuration Release
              --logger "trx;LogFileName=integration-results.trx"
              --results-directory $(Agent.TempDirectory)
          env:
            API_BASE_URL: $(APP_URL)
            TEST_DB_CONNECTION: $(TEST_DB_CONNECTION)   # from variable group

        - task: PublishTestResults@2
          condition: succeededOrFailed()
          inputs:
            testResultsFormat: 'VSTest'
            testResultsFiles: '$(Agent.TempDirectory)/**/*.trx'
            testRunTitle: 'Integration Tests'
            failTaskOnFailedTests: true

End-to-end tests with Playwright

- stage: E2E_Tests
  displayName: 'End-to-End Tests'
  dependsOn: Deploy_Staging
  jobs:
    - job: PlaywrightTests
      pool:
        vmImage: 'ubuntu-latest'
      steps:
        - checkout: self

        - task: NodeTool@0
          inputs:
            versionSpec: '20.x'

        - bash: |
            npm ci
            npx playwright install --with-deps chromium
          displayName: 'Install Playwright'

        - bash: |
            npx playwright test \
              --reporter=junit \
              --output-folder=$(Agent.TempDirectory)/playwright-results
          displayName: 'Run Playwright E2E tests'
          env:
            BASE_URL: $(STAGING_APP_URL)
            CI: 'true'
          continueOnError: false   # fail the stage if E2E tests fail

        - task: PublishTestResults@2
          condition: succeededOrFailed()
          inputs:
            testResultsFormat: 'JUnit'
            testResultsFiles: '$(Agent.TempDirectory)/playwright-results/junit.xml'
            testRunTitle: 'E2E Tests (Playwright)'

        # Publish Playwright HTML report as artifact for debugging failures
        - task: PublishBuildArtifacts@1
          condition: failed()   # only publish on failure to save storage
          inputs:
            pathToPublish: 'playwright-report'
            artifactName: 'playwright-report-$(Build.BuildId)'

Running tests in parallel to speed up pipelines

Large test suites can be split across multiple agents using the parallelism feature. Each agent runs a subset of tests based on its shard index.

- job: UnitTests
  strategy:
    parallel: 4   # run 4 parallel agent jobs
  steps:
    - task: DotNetCoreCLI@2
      displayName: 'Run shard $(System.JobPositionInPhase) of $(System.TotalJobsInPhase)'
      inputs:
        command: 'test'
        projects: '**/*Tests.csproj'
        arguments: >
          --configuration Release
          --logger "trx"
          --results-directory $(Agent.TempDirectory)
          -- RunConfiguration.TestSessionTimeout=300000
             DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura
      env:
        # Some test frameworks support shard-based splitting natively
        TEST_TOTAL_SHARDS: $(System.TotalJobsInPhase)
        TEST_SHARD_INDEX: $(System.JobPositionInPhase)

    - task: PublishTestResults@2
      condition: succeededOrFailed()
      inputs:
        testResultsFormat: 'VSTest'
        testResultsFiles: '$(Agent.TempDirectory)/**/*.trx'
        mergeTestResults: true

Note

When running parallel test jobs, each job publishes its own test results. Azure Pipelines automatically aggregates them into a single combined result on the run summary. Set mergeTestResults: true in each PublishTestResults task so results from different agents are merged rather than showing as separate runs.

Common mistakes

Not publishing results when tests fail. The default behavior of many test commands is to exit with code 1 when tests fail, which causes the pipeline step to fail immediately without running the publish step. Add condition: succeededOrFailed() to your PublishTestResults task so results are always uploaded — failures are just as important to see as passes.
Running integration tests against production. Integration tests often create, modify, and delete data. Running them against your production database because it is convenient will cause data corruption. Always run integration tests against a dedicated test environment with isolated data that can be reset between runs.
Setting unrealistic coverage thresholds. A coverage threshold of 80% sounds reasonable but causes test paralysis — developers start writing trivial tests to hit the number rather than meaningful tests that catch bugs. Coverage is a signal, not a goal. Track it for trends, but do not gate deployments on coverage percentage alone.

Frequently asked questions

How do I see test results in Azure Pipelines?

Publish test results using the PublishTestResults task. Results appear on the pipeline run summary page under the Tests tab. You can filter by test name, view failure details, and compare pass rates across runs. The task accepts JUnit, NUnit, VSTest, xUnit, and CTest XML formats.

How do I fail a pipeline when tests fail?

The test runner itself should exit with a non-zero code on failure — most test frameworks do this by default. If you use the DotNetCoreCLI or pytest tasks, a test failure automatically fails the pipeline step. If running tests via a bash script, check the exit code and use "exit 1" to propagate the failure.

What is test impact analysis in Azure Pipelines?