Why Test?

As we start working with if statements, we need to consider how they affect the flow of our program and the different paths that can be taken. Unit testing is crucial to working with code that involves conditionals. The purpose of unit testing is to determine if a program behaves as expected. By writing tests for your code, you can catch errors early and ensure that your code is working correctly. However, it is impossible to test every possible input and output combination. Consider a function that takes a string as input and returns the number of vowels in that string; on a modern 64-bit computer, there are 127 * 2^64 possible strings, which is more than a thousand times the number of grains of sand on Earth. For this reason, it is usually impossible to decide if a program is 100% correct all the time, for any reasonably interesting function.

Fortunately, most functions have a relatively small number of distinctive inputs, for which the function behaves significantly differently. By carefully testing different cases of behavior, we can gain confidence that our function works correctly for all inputs.

Representative Cases of Input

In the previous unit, we learned about different types of input cases and how to test them effectively. We described how there are representative cases for each type of input. For example, for strings, we might test with empty strings, strings with whitespace, and strings with special characters. For numbers, we might test with positive numbers, negative numbers, and zero. Booleans are a simple case, as they only have two possible values: true and false. But for these other types, there are many other ways to categorize input cases that are often useful to consider as a starting point.

Specialized Cases of Input

from bakery import assert_equal

def has_won(score: int) -> str:
    if score > 10:
        return "You have won!"
    else:
        return "You have lost."

assert_equal(has_won(15), "You have won!")
assert_equal(has_won(5), "You have lost.")
assert_equal(has_won(10), "You have lost.")

Beyond the representative cases, there are often additional cases to consider that are specific to the function being tested. In the example shown here, the has_won function returns a string indicating whether the player has won or lost based on their score. The threshold for winning is a score greater than 10.

This leads us to consider at least three cases: when the score is greater than 10, less than 10, and equal to 10. Explicitly writing a test case for this last situation is useful to ensure that our function handles all possible inputs correctly, since it forces you to explicitly consider whether the game is won or lost when the score is exactly 10.

Kinds of Cases

When testing a function, it is important to consider a variety of input cases to ensure that the function behaves as expected. In general, we call these “normal cases” or “happy paths”. However, sometimes we need to consider very specialized cases that might be uncommon or unexpected, but can still happen nonetheless. We call these “special cases”, “edge cases”, or sometimes “unhappy paths”. The most valuable programmers are the ones who consider not only the happy paths through their programs, but also the unhappy paths.

Virtually always, you want to avoid writing invalid cases, since they can lead to false positives in your tests and make it harder to identify real issues. This means that you should not waste time testing inputs that are not valid for the function, like trying to pass an integer into a function that expects a string. Similarly, if you can absolutely guarantee that a function can never receive an empty string, then you wouldn’t want to test for that case either; of course, it is very rare that you can really guarantee that, and you should be careful about making such assumptions. It is possible that an unhappy path in another part of the application might lead to the function being called with an unexpected input, and you want to ensure that your function can handle that gracefully.

Code Coverage

Code coverage is a measure of how much of your code is executed when you run your tests. It helps you identify which parts of your code are being tested and which are not. However, the number is not magic. Instead, it is a simple percentage of the lines of code that were executed during testing, divided by the total number of lines of code. Many programmers treat code coverage as a primary indicator of code quality, but it is important to remember that high code coverage does not guarantee the absence of bugs.

You can fully test every line of code without testing all possible inputs, and therefore a bug could still be lurking. For example, in the example of has_won from before, we might have tested the cases where the score was greater and less than 10, but not 10 itself. That would result in 100% code coverage, but we would not have tested a very important case!

Example Coverage Calculation

from bakery import assert_equal

def combine_temps(first: str, second: str) -> str:
    if first == 'high' and second == 'high':
        return 'heat wave'
    return 'normal'

assert_equal(combine_temps('high', 'normal'), 'normal')
assert_equal(combine_temps('normal', 'high'), 'normal')
assert_equal(combine_temps('normal', 'normal'), 'normal')

Let’s look at a simple example of calculating coverage. The combine_temps function is supposed to consume two temperatures (either 'high' or 'normal') and determine if there is a heat wave, which happens when both temperatures are high. The program shown here has 8 actual lines of code, ignoring the whitespace. When the program is executed, the program starts executing from the beginning, evaluating the import and then defining the function (remember, the function body is not yet executed).

Then, each test case is executed. For each test case, we jump to the start of the function body and execute sequentially. The if statement causes us to skip over the body of the function unless both arguments are 'high'. Because the unit tests only cover the happy path (where both temperatures are not high), we do not have any tests that cover the case where both temperatures are high. The return 'heat wave' line is never reached during testing, so it is not covered by any tests. This means that we only reach 7 lines of code, resulting in 87.5% code coverage. If we added a new test case for combine_temps('high', 'high'), we would then reach all 8 lines of code, resulting in 100% code coverage.

Visualizing Code Coverage

Many programming environments can visualize code coverage directly in the editor. This helps you identify which parts of your code are being tested and which are not. Typically, any lines of code that were NOT executed are highlighted. If you see highlighted lines of code, you should add test cases to cover those lines.

Remember - such a feature is showing you the lines of code that were executed, not the lines of code that have been covered by test cases. If you define a function but never call the function, then the function’s body will be highlighted. On the other hand, if you just call a function without any tests, you will not see any uncovered lines. But that does not mean those lines are correct or tested, just that they have been executed. You still must look critically at your test cases.

Valid and Thorough Tests

Beyond thinking about code coverage, it is also helpful to think about whether your tests are valid and thorough. These terms are not about code coverage, but instead about the quality of your tests. Recall that there can be many possible implementations of a program - you and your friend might approach writing any given function very differently, but that does not mean that you are not both correct.

A valid set of tests should pass on any possible correct implementation of a program. Invalid tests are ones that fail or produce errors on some correct version of the program, even though they shouldn’t. Invalid tests mean that they are not correct, and are therefore not useful. This includes tests that attempt to test invalid inputs, such as testing a string being passed into a function that expects a number.

A thorough set of tests should either fail or produce an error on incorrect implementations. If your friends’ version of the program doesn’t handle a certain case correctly, then a thorough test will expose that mistake. Keep in mind that only one test case in a collection has to fail in order for the entire set of tests to be thorough. Expecting every test to individually fail for all possible incorrect implementations is unrealistic.

“Imposter” Programs

A judge weighing programs on a balance scale, evaluating them as either genuine (a smiling face with a halo) or imposter (an angry face), using assert_equal.
Judge and face icons: OpenMoji, the open-source emoji and icon project, CC BY-SA 4.0. Icons from Wikimedia Commons.

You might think of incorrect programs as being “imposters” that we are trying to catch. Unit tests help identify these imposters by checking if the actual output matches the expected output. A “genuine” program will always produce the expected output for all valid inputs, but an “imposter” program will produce incorrect output for at least one valid input. Valid test collections should help prove the innocence of genuine programs, and expose the guilt of the imposters.

Example Imposters

from bakery import assert_equal

def add_version_1(x: int, y: int) -> int:
    return x + y  # Genuine/Correct

def add_version_2(x: int, y: int) -> int:
    return x - -y # Genuine/Correct

def add_version_3(x: int, y: int) -> int:
    return x - y  # Imposter/Incorrect

assert_equal(add_version_1(0, 0), 0)
assert_equal(add_version_1(1, 0), 1)
assert_equal(add_version_2(0, 0), 0)
assert_equal(add_version_2(1, 0), 1)
assert_equal(add_version_3(0, 0), 0)
assert_equal(add_version_3(1, 0), 1)

Let’s look at an example of a test suite for a function that is supposed to simply add together two numbers. There are three versions of this function: add_version_1, add_version_2, and add_version_3. The first two approaches are both correct, because adding is the same as subtracting a negative. However, the third approach is incorrect, because it simply subtracts the two numbers.

The unit tests shown here are valid, because they pass on all correct implementations of the program (the first two versions). However, they are not thorough, because they do not catch the incorrect implementation (the third version). In order to make the tests thorough, we need to add a test case that specifically checks for the incorrect behavior. For example, we could add a new case that passes in 2 and 3, and expects the result to be 5. The third version of the program will fail this test, because it will return -1 instead of 5.

Critically Thinking about Imposters

As we said before, it’s impossible to create a perfect set of tests that catch all possible incorrect implementations. There are usually an infinite number of both correct and incorrect implementations. In particular, it can be easy to underestimate how many ways there are to write incorrect code. This is why it’s important to think critically about your tests and consider edge cases that you might not have initially thought of. Think about places where you were likely to have made a mistake, or where you might make a mistake in the future if you were to edit the program. It’s also helpful to get feedback from others, as they may spot issues that you missed. You can never have enough test cases, so keep adding more until you feel confident in your tests.

Summary