Craig Atkinson

If something important is broken, will your build fail?

You and your team know you should write automated tests for your software. Maybe you already have some tests in place. Great!

But the tests you have aren’t helping - your users still run into issues in production:

  • It seems like after deploying a new feature or bug fix, something else breaks.

  • Bugs that were fixed weeks ago creep back in.

And your team’s productivity takes a nose dive once the software is in production:

  • Each code change requires a ton of throwaway manual testing to verify it works.

  • Code reviews are very stressful since reviewers have to focus on whether the functionality works.

First, let’s look more in depth into why your existing test suite may not be solving these issues. Then we’ll talk about a single testing principle you can apply today to catch issues before they reach production, lower your stress levels, and increase you and your team’s productivity.

Incomplete testing

You’ve seen it before - the test that calls a method but doesn’t actually verify any results. Or does some verification, but not enough to catch many potential issues.

For example, a unit test that mocks out a collaborator class. The test verifies the mock is called, but doesn’t verify the mock is called with the right arguments. The underlying code is broken and sets the arguments to the wrong values, but the test still passes.

Or the functional test that calls an API endpoint that returns JSON. But the test only verifies the HTTP status code returned by the API and doesn’t verify the body. The underlying code is wrong and doesn’t fill out required parts of the response body, but the tests still pass.

These are just two examples of incomplete tests - the underlying code is broken but the tests still pass. How can we fix this?

What to test

One debate I’ve repeatedly seen among teams is what types of tests to write. Often the answers center around the specific framework your app is built with. For example in a Spring Boot app your team may debate whether controller classes should have unit tests or functional tests, whether service classes should have unit or integration tests, and so on. This list grows and grows depending on what your application does. And each time you add a new type of thing to your application, you need to debate about how it needs to be tested. Or heaven help you if you change frameworks - time to start the debate all over again.

Or the debate is around the definition of test types - what makes a unit test, whether an API test is an integration or functional test, etc. While that debate can be an interesting academic exercise, it doesn’t help you deliver better software faster.

Guiding principle

You and your team can apply one single principle to guide which types of tests to write and what to cover in those tests:

If something important is broken, will your build fail?

Depending on what types of tests you currently have in your test suite and how thorough they are, getting to this point may take a decent amount of work. You may need to introduce higher-level types of tests like integration and functional tests for complete end-to-end verification that your software works as intended.

But the benefits are worth it.

Benefits

Why do this level of testing? I won’t pretend that writing tests this thorough is free - there is a cost (especially initially) to write and maintain a thorough test suite. But the benefits are so great that they easily outweigh the costs.

A sampling of some of the short and long term benefits of the safety net and thorough test suite provides:

Shorter-term benefits:

  • Know code changes will work correctly for your users before the changes go into production. Fewer users unhappy with broken features. Fewer midnight pages caused by new bugs.

  • As the reviewer, code reviews are easier and more thorough since the reviewer can look at the tests to quickly prove the code works correctly. Having the tests to prove the code works frees up the reviewer to review the code changes more deeply - is the code performant, is there a more concise way to write it, etc.

  • As the author, updating code based on code review comments is faster and less risky. If a reviewer asks you to change how a piece of code is written you don’t have to manually re-verify the code still works correctly. The tests automatically do that for you.

Longer-term benefits:

  • Have you worked on a new project where you started delivering features fast when the project first started, but that pace slowed over time? Especially after the project went into production. Changes become riskier - any change, no matter how small, could break some important existing functionality. With this thorough test suite you and your team can sustain the initial speed of development for years on a project.

  • Ever worked on a project that was stuck on a specific, older version of a framework because the upgrade is too risky? This safety net gives you the freedom to upgrade libraries and frameworks to get new features and bug fixes without fear of breaking your existing software.

  • Fixed a bug only to have it show up again months later? With this thorough test suite the build will fail before the bug reappears and makes it into production again.

  • Easier to change code to improve performance without risking accidentally breaking the functionality in the process.

  • Safer to refactor complex code, clean up duplication, etc. without risking introducing new bugs.

Conclusion

Moving your test suite to the point where if something important is broken, the build will fail won’t happen overnight. It will take time and dedication from you and your team to improve the testing coverage and add missing levels of test types. But the faster speed, lower stress, and other benefits of this safety net are more than worth it.