Last Call for Quality

A look at why late testing fails, how to avoid this, and other common software testing anti-patterns.

Jul 03, 2024

Testing Too Late

One of the most common software testing anti-patterns is testing too late. In traditional quality assurance models, software is handed over from developers to testers at the end of the release cycle. This leads to QA as gatekeepers and/or a bottleneck, which is not ideal. Additionally, the later a bug is discovered, the more expensive it is to fix. By shifting testing left, bugs can be found earlier, making them less costly to fix.

When I talk about “shift-left,” this means moving things earlier in the process. Take a typical lifecycle process of design → code → test → ship → maintain. Shifting left means moving testing to the left.

Ideally, testing should not occur in isolation but as an integral part of each step.

You can read more about the differences between quality assistance and quality assurance in this article from 2023.

Quality Assistance, Not Assurance

Brie Ransom

October 18, 2023

Read full story

The Pitfalls of Late Software Testing

Testing is, ultimately, a feedback loop for developers, indicating whether their code performs as expected. When it doesn’t, this means the software has bugs. Viewed this way, testing too late means the feedback cycle is too long.

Why does testing happen too late? It’s often the culmination of multiple software testing anti-patterns and development practices, such as over-reliance on manual testing, separate development and test/QA teams, and a lack of quality ownership throughout the development process.

Consequences

Higher costs of fixing issues found late in the cycle.
Due to the higher cost, not everything found will be fixed, lowering the quality of the product.
Missed deadlines, as major bugs can delay software releases.
Stress on the team and potential burnout.
Impact on customer experience, satisfaction, and trust: persistent quality issues can damage the brand’s reputation and erode customer trust.

Alternatives and Best Practices

When we consider testing as a feedback loop, our goal should be to make that loop as short and meaningful as possible. We can do this by shifting testing left (earlier) and, therefore, shortening the feedback loops.

With shift-left testing, testing starts as early as possible in the software development lifecycle. There are many ways1 to add testing earlier, such as:

Requirements testing
Design reviews
Code reviews
QA kick-offs
Test Driven Development (TDD)
Early test planning
Unit testing
Paired testing
Continuous Integration and Continuous Deployment (CI/CD)
Automated testing
Risk analysis and risk-based testing
Increased communication and collaboration between testers and developers

Other Common Software Testing Anti-Patterns

Now that we’ve covered the pitfalls of testing too late, let’s look at some other testing anti-patterns.

Over-Reliance on Manual Testing

I want to start by stating that manual testing is a great tool and can be essential for a robust quality strategy. However, because it can be time-consuming and human-resource-intensive, we need to be intentional when we choose to use it.

Often, manual testing is done by a separate team after the code is “done.” Whether manual or automated, waiting and throwing it over the wall at the end of the release cycle like this is one of the primary drivers causing testing to happen too late.

Corrective actions:

Look at your code/test coverage metrics and compare them to the testing pyramid2 and testing trophy3. Where can you add more testing that is not manual? Where can you add more testing earlier in the lifecycle?

Ignoring Non-Functional Requirements

Overlooking non-functional aspects of the product, such as usability, security, and performance, leads to software that may meet requirements but doesn’t satisfy user expectations.

Corrective actions:

Include non-functional requirements explicitly in your product specs.
Add non-functional testing categories to your test planning.

Poor Test Data and Test Data Management

Poor test data and test data management can lead to the release of defective software, which can affect user experience and satisfaction.

Poor test data can lead to incorrect test results, causing defects to be missed (false negatives) or non-existent issues to be reported (false positives).

On the management side, inadequate test data handling can expose sensitive or confidential information. Using production data in testing environments can inadvertently create security vulnerabilities.

Corrective actions:

Create, maintain, and secure test data that accurately represents real-world scenarios. There are many tools4 to choose from that can help with test data management.

Lack of Test Environment and Configuration Management

A lack of proper test environment and/or configuration management can cause the "works on my machine" syndrome. This often leads to missed bugs because the pre-production environment does not match the production environment.

Corrective actions:

Tools such as docker images, Puppet, Chef, or Ansible can ensure pre-production environments match the production environment.

Conversation Starters

What strategies have you found effective in implementing shift-left testing in your development process?
What have you found to be the most effective ways to shorten feedback loops in software development and testing?
What methods have you used to prevent testing from becoming a bottleneck in your release cycle?

QUALITY BOSS

Quality Assistance, Not Assurance

Discussion about this post