Exceptional unit tests
Unexpected exceptions being thrown and caught inside application code, that weren't obvious until the unit test was run under a debugger.
I was working on a .NET application that had a nice suite of unit tests, and pleasingly the tests were all passing.
While making a change to the code, one of the tests failed (which is the whole point of having tests!). To better understand the reason for the failure I re-ran the test in the Visual Studio debugger.
I noticed something strange - the system under test was throwing a NullReferenceException, which was then being caught (and effectively swallowed) by an outer try/catch block.
The surprising thing was that this test should not have been doing that - a pretty straightforward test of some business logic. It was just a coincidence that in this case the exception wasn’t changing the observable behaviour of the code, which is why the test had previously been passing.
In this case, the underlying cause of the NullReferenceException turned out to be a missing mocked method on a dependency.
It did make me wonder though, are there other similar issues hidden elsewhere in the unit tests?
To find out, I opened up Visual Studio’s Exception Settings window (Debug | Windows | Exception Settings), searched for ‘NullReferenceException’ and ensured that it was set to ‘Break when thrown’
I then ran the entire test suite under the debugger (Test | Debug All Tests) and took note of each time the debugger stopped with a thrown exception. Some of these were other exception types that were expected (and I could turn off the ‘Break when thrown’ on those if they were too noisy).
I ended up finding a few other tests that had similar issues. There were also some paths in the application code where null handling could be made more robust.
The tests still pass, but they should now be a bit more reliable in the future for the next developer who is relying on them when making application code changes (which could be me!)