The Measurable Benefits of Unit Testing

What is unit testing?

In a nutshell, a unit test is a fast test of a small unit. What is a unit depends on the developer. It can be any small, testable piece of code, such as a method or a class.

Of course, that is just one definition. Some developers will define unit tests as “a waste of time,” to which I say: don’t hire them. Moving on...

Unit tests are low level tests. They force a small piece of application to perform some action and they check the final result. This action is very simple and the results are independent of other pieces of application. Failure in a unit test shows without a doubt which part of the code doesn’t work.

Moreover, executing unit tests doesn’t require the application to be running. And it can be done even before the whole application is built.

These are the facts. There are many other advantages of unit testing often cited around the web, such as:

unit testing leads to less bugs in the software;
developers feel more confident in deploying code that is covered by unit tests;
programmers following the Test-Driven Development (TDD) process claim that unit testing helps them achieve their goals faster, solving problems with less code and better code architecture.

But such arguments are either based on feelings or otherwise very hard to prove and for that reason I’m not going to focus on them.

Instead, I’d like to focus on a common element of every development process, with or without unit testing. Namely: bugs.

Bugs will always happen. The only difference is in how much time the developer will spend on finding those bugs, how soon he will notice them. This can be easily measured and compared, as I will show you next.

Can we measure if unit tests provide any benefits?

Let’s use an example to analyze what development looks like with and without unit tests. In our example, instead of an application we will be building a car. Instead of developers, we will work with electronics engineers.

We start with an electronics engineer who was asked to build an electronic controller for a car door window. He received all the documentation he needed. After some time, he built a controller, according to his knowledge and the documentation, and the time came to check if it works.

He decides to test the controller by installing it in the doors of the car. It may sound silly, but this exactly what it looks like when developers test their work without unit tests.

As could be expected, the controller in the car door doesn’t work on the first try. Perhaps it doesn’t even work on the second or third try, depending on the situation. Even when you work with specialists, mistakes will happen. They are only human.

Finally, on a subsequent try, the controller works.

How much time did our engineer spend on testing?

Let’s see. Installing the controller takes installation_time minutes. Manual testing takes testing_time minutes. He repeated this sequence n times—maybe once, maybe twice, maybe a few more times—until the controller worked. That gives us:

testing_time_without_unit_tests = n * (installation_time + testing_time)

Is there another solution to this problem?

Yes. The engineer could build a special circuit board which can do all the testing automatically. This board is his equivalent of unit testing.

Building a board will take him build_testing_board_time minutes. However, executing tests with this board is so fast that we can assume it is irrelevant. All in all, the formula changes to:

testing_time_with_unit_tests = build_testing_board_time

Is it worth to build this board? That depends. I will answer this question in a moment, but first we must add an important detail to our story.

We know that the car symbolizes a software product. And we know that IT projects have at least one thing in common: they are constantly changing.

So let’s go back to our engineer and electronic controller. At the beginning, the engineer was told that the controller should work with two windows. But a few weeks after his work is finished, it turns out that there has been a small change in the project and now our car will have four windows.

Sounds funny, but usually this is how IT projects work. A small change from the point of view of a client might have huge consequences from the engineer’s perspective.

Following the new documentation, our engineer does his best and updates the controller. He installs it in the door and... the new feature doesn’t work on the first try, doesn’t work on the second, and so on... Until finally it works again.

Here’s the part that separates the best from the rest. A good engineer will always test his work. But a top engineer will check if he didn’t break anything else.

And we work with the best, so our engineer checks if the previous features of the controller still work fine. Let’s assume that some of them don’t, and our engineer has to do some fixes and more testing.

All in all he spends this much of time on testing after creating the new feature:

testing_time_without_unit_tests = n * (installation_time + new_feature_testing_time) + previous_feature_testing_time

How would the situation change with a testing board? The engineer would spend extension_time minutes upgrading the testing board. And then he would test both the old and new features simultaneously in a so-short-it’s-irrelevant time, of course in n iterations because mistakes happen. That gives us:

testing_time_with_unit_tests = n * testing_time + extension_time

But since the testing_time is nearly instantaneous using the testing board, we can assume that value to be 0, ultimately resulting in:

testing_time_with_unit_tests = extension_time

Now ask yourself: what’s going to happen when we add a third, fourth, or n-th feature?

Without unit tests, each new feature extends the testing time because all of the previous features have to be tested by hand for compatibility. If you have unit tests in place, the previous features can be tested almost instantly.

Moreover, a lack of unit testing makes it difficult to predict the final testing time. The developer may estimate how much time he will spend on building a new feature, but no one knows how many iterations will be needed for testing during development, or how many conflicts with previous features will be created.

On the other hand, with unit tests life is much easier. An engineer can easily say how much time they need to create new unit tests.

In this case, the testing time during development is irrelevant. Time spent on fixing bugs is the same as without unit tests. But what we do get is more predictability in case bugs occur during development and less money spent on manual testing time.

With unit tests you can also improve parallel work. There is no need to wait for the car to be fully assembled to test the window controller. It can be tested using the testing board before the car is built.

Do I need unit tests?

As shown in the previous section, creating unit tests:

takes some time, but…
saves you time on testing during the development process.

Which of the two offers greater value—the time spent creating unit tests or the time saved on testing—depends on your project.

There are a few factors which can tell you that your project requires unit tests:

your project will grow in time,
your features are complex,
bugs in your project can cost you a lot of money,
predictable time to market is crucial for your project.

If your project matches any of the above, you will benefit from unit tests. The more factors fit, the more unit testing will help you.

But wait, I already use automated tests. They do the same thing, right?

Not exactly.

There are different types of tests for different goals:

component tests check how a whole component (comprised of several units) works;
integration tests check how a component works with other components;
automated tests check crucial paths in the whole application.

All of them are, strictly speaking, automated. They all save time in the same way unit tests do: you spend time building the tests to spend less time on manual testing.

So why should I create unit tests when I have those tests?

The answer is simple: to cover high and low levels of testing.

Automated tests are high-level tests. They use the available interface to force the application to do some action, and then they check the final result. Quite often this action can be complex. It can invoke many small steps before reaching the final state. Each of those steps can fail.

So the final result of each automated test depends on many smaller components and it is hard to say which of them failed. It only says that some scenario doesn’t work, or some element doesn’t work correctly with another element.

Unit tests are low-level tests. They point out exactly which unit and which part of that unit doesn’t work correctly. That information makes finding the source of bugs much faster and easier.

Conclusion

Unit testing has many advantages. I focused only on those which can be easily measured and compared. I pointed out the factors which can tell you if you will benefit from having unit tests.

Of course, projects are as diverse as the people working on them and each needs to be analyzed individually. But if you keep the values above in mind (numerical values, not lofty emotional ones), then you’ll have a compass to follow.

Thanks for taking the time to read this. If you have any questions I could answer, don’t hesitate to leave a comment.