5 Metrics For Automation Testing To Improve Your CI/CD Pipeline

You introduced automated testing to catch problems before they reach production. Six months later, your test suite takes 40 minutes to run, some tests fail on Tuesday and pass on Wednesday, and a bug still makes it into production.

At this point, nobody can tell whether that automation is actually making the release process better.

These five metrics can help you figure out where your automation is falling short.

Here’s what to track and when the numbers need your attention.

Why Test Automation Metrics Matter In CI/CD

A CI/CD pipeline depends on fast, reliable feedback. Automation metrics help teams track whether their tests are delivering it.

Metrics are only useful when they help the team make a decision. Using them to compare individual testers usually misses the point.

If a metric changes but nobody acts on it, there’s little reason to keep tracking it.

For broader QA indicators, including requirement coverage, see our guide to 6 KPIs for QA testers. Here, we’re focusing specifically on automation metrics within CI/CD.

The 5 Metrics At A Glance

Metric	Formula	CI/CD Signal	Action When It Worsens
Test Execution Time	Time from the first test starting to the last one finishing, per stage	Feedback time per commit	Parallelize, split suites
Flaky Test Rate	(Tests with inconsistent results on the same code / total tests) x 100	Unreliable results	Quarantine, fix the root cause
Pass Rate And Build Stability	(Passed tests / tests executed) x 100, plus builds failed by tests / total builds	Healthy or broken builds	Find the commit, separate infrastructure failures
Defect Escape Rate	(Defects found in production / total defects found) x 100, per release	Defects that passed the pipeline	Review the root cause, add the missing test
Critical-Flow Coverage	(Critical flows with automated tests / total critical flows) x 100	Gaps in what the pipeline protects	Automate high-risk gaps first

Set your targets around how your team actually tests and releases software.

The 5 Metrics For Automation Testing In CI/CD

1. Test Execution Time

• What It Measures: The total time for all tests to run – from the first test to begin to the last test finishing.
• Formula: Track the wall-clock time for each stage (unit, integration, and end-to-end). Record the median and 95th percentile to account for unusually slow runs.

In The Pipeline: Record execution time for each stage on every run. Run faster tests first so developers get feedback sooner.

Example Threshold: Agree on a limit per stage. Martin Fowler’s Continuous Integration article cites the Extreme Programming guideline of a ten-minute build as reasonable for many projects. Use it as a starting point.

• When It Worsens: Start with duplicate tests and unnecessary setup time. If the suite is still slow, look at parallel execution or moving less urgent checks to nightly runs.

2. Flaky Test Rate

• What It Measures: The percentage of tests that produce different results when run against unchanged code.
• Formula: (Tests with inconsistent results on the same code / total tests) x 100.

Say 12 out of your 400 tests fail inconsistently. That’s a 3% flaky test rate.

Industry Context: In 2016 Google Testing Blog post by John Micco reported that about 1.5% of Google’s test runs were flaky and almost 16% of its tests had some flakiness. These figures measure different things: one counts test runs, the other counts tests.

If a test fails and then passes on a rerun without any code changes, flag it. Keep an eye on tests that do it repeatedly.

Check whether the failure comes from timing, test data, or the environment. Quarantine the tests if it’s repeatedly blocking merges, but don’t leave it there indefinitely.

3. Test Pass Rate and Build Stability

• What It Measures: The share of tests that pass, and how often builds fail because of tests.
• Formula: (Passed tests / tests executed) x 100 for each run. Build stability is (builds failed by tests / total builds) x 100.

In The Pipeline: Report by suite, branch, and environment. Write a rule for infrastructure failures such as runner crashes and network timeouts, so they do not distort the number.

Example Threshold: Set it per suite. A very high pass rate can also point to weak tests, so read it beside coverage and escape rate.

If the pass rate drops suddenly, check the latest changes first. It could be a regression, or something as simple as an unstable test environment.

4. Defect Escape Rate

This metric is also known as defect leakage.

Defect escape rate shows how many bugs were found after release instead of during testing.

• Formula: (Defects found in production / total defects found for the release) x 100. State the time window, and weight by severity if you can.

Track where bugs were being caught. If the same type of issue keeps reaching production, it’s worth reviewing the tests that should have caught it.

Example Threshold: Set it by risk. Banking and healthcare teams often hold a stricter target than an internal tool team.

Look at the bugs that escape. Some might point to missing coverage, while others expose problems with test data or unreliable tests.

DORA’s change fail rate counts failed deployments, and defect escape rate counts defects. The two complement each other.

5. Critical-Flow Automation Coverage

This measures how much of your important user journeys are covered by automated tests. Think checkout, login, payments, or account creation.

• Formula: (Critical flows with automated tests / total critical flows) x 100. You can also report the automated share of all test cases.

Start with the workflows that would cause the most trouble if they broke. For a banking app, that might mean transfers and login before less-used features.

Example Threshold: aim for full coverage of high-risk flows first, then widen.

• When It Stalls: Automate high-risk gaps before adding tests to flows that already have coverage. Requirement traceability makes those gaps visible.

How These Metrics Connect To DORA

DORA measures software delivery performance through five metrics:

1. Change lead time
2. Deployment frequency
3. Failed deployment recovery time
4. Change failure rate
5. Deployment rework rate

The first three measure throughput, while the last two focus on instability.

Older DORA resources may list only four metrics, so check the publication date when comparing sources.

Automation metrics help explain what’s happening behind these delivery results.

Here’s how the automation metrics relate to DORA’s delivery measures:

Test Automation Metric	DORA Measure It Relates To
Test Execution Time	Change lead time
Flaky Test Rate And Pass Rate	Change lead time and deployment frequency, through reruns and blocked merges
Defect Escape Rate	Change fail rate and deployment rework rate

How To Choose Metrics Without Gaming Them

Before adding a metric to your dashboard, ask four questions:

1. What decision will this metric help your team make?
2. Who is responsible for monitoring it, and what should they do when it changes?
3. What is the agreed target, and when will you review it?
4. Could this metric be manipulated or used to rank individual team members?

A faster suite isn’t much of a win if more bugs are reaching production. The same goes for a high pass rate when important workflows aren’t covered.

Review these metrics in retrospectives, but don’t use them to rank individual team members.

How To Collect And Report These Metrics

• CI Tool: Execution time, build outcomes, and reruns from Jenkins, GitHub Actions, Git Lab CI, and other tools. Our top CI/CD tools guide compares how each reports test results.
• Test Reports: Individual test result output, exported as a standard format such as JUnit XML with each run.
• Test Management Tool: The links between tests, requirements, and defects. These feed coverage and escape rate.

There are four elements to a simple dashboard:

1. Speed
2. Reliability
3. Outcomes
4. Coverage

Review the trends weekly and note any significant changes, along with the actions your team plans to take.

We recommend a flaky test rate. Ignoring a red build makes every other metric unreliable.

How Kualitee Helps

When a test fails in Jenkins, its error can easily get lost in the console log. Kualitee lets teams import automation results and associate them with build numbers and test cases. Testers can then log defects and link them to the relevant requirements.

Teams can also review execution history across builds, which makes it easier to spot recurring failures and follow changes in test results over time.

Kualitee integrates with Jenkins, CircleCI, GitHub, Selenium, Playwright, and it can import automation reports.

See your Jenkins or CircleCI results next to your manual test runs. Try Kualitee free for 14 days. No credit card required.

The Bottom Line

You don’t need to track every metric from day one. Start with execution time and flaky tests. Once those numbers are reliable, add defect escape rate and coverage to see where your automation needs work.