Previously, we discussed the importance of unit testing our code. We also talked about how to choose good inputs for testing.
For example, we talked about how a Point has two int components, which we can think of as saying that methods dealing with the point have a domain of $\mathbb{Z}\times\mathbb{Z}$.
Since this domain is (sort of) infinite, we can’t expect to write unit tests for all points. We must choose good boundary cases to reduce the effort required to create a representative set of test cases.
When testing for a move function, we mentioned that this could require testing points on axes, in each quadrant, moving in all directions, etc.
This is still a lot of work!
After all of that effort, we may still have errors, because there could be some corner case or boundary which we did not anticipate.
What we would like to be able to do is write a single test, which would exhaustively check the elements in the domain of the function in question.
This is ideal, because the domain might be infinite (or close enough to be a problem), and we have no way to specify all of the outputs to test against without performing the computation itself!
As a compromise, though, we could test the properties of a correct solution, instead of testing for a specific solution.
For instance, if we move a point, then either we moved by $(0,0)$, or the point is no longer in the same location.
This describes a property of a correct output without having to specify a specific output.
You can imagine how we could then do something like:
for(int i=0; i<Integer.MAX_VALUE; i++){
for(int j=0; j<Integer.MAX_VALUE; j++){
for(int k=0; k<Integer.MAX_VALUE; k++){
for(int l=0; l<Integer.MAX_VALUE; l++){
// test property for Point(i,j).move(k,l)
}
}
}
}
That, however, would still be too computationally intensive. Accordingly, we could generate random values and assume (hope?) that we generate a sufficiently thorough sample of inputs.
If any generated inputs lead to an output which does not have the given property, then we know we have a bug. We can report the failed inputs so we know what to investigate.
For some properties, getting a failing set of inputs may not be useful if the values are very large. For this reason, most property based testing libraries will take failing cases and attempt to “shrink” them to smaller inputs.
For example, consider the property:
the output of f should be a list of even numbers
And assume a bug in our code returns incorrect values when the input list has an odd number of elements containing a 3.
If we generate random lists, we will quickly encounter the bug, but the failing input might have thousands of elements. If we run the test over and over, we’re likely to always get big lists for our failing cases.
However, once we find a failing test case, we could slowly eliminate elements from it and retest it until we get a failing test input like $[3]$.
Should we stop doing traditional unit tests?
No! We will often still need to test that specific inputs give specific outputs, so we probably want to use a mixture of approaches.