I ran into a bit of a quandary the other day. I had to modify a piece of legacy code in our system – one that has no tests written against it. My first task was to write as many JUnit tests as I could to document the behavior of the component to help ensure that I didn’t break existing functionality with my changes.

After a few tests, however, I began to notice something troubling. The component I was writing tests against had some bugs in it, and in order to make my unit tests pass, I had to code to those bugs. This in itself did not bother me too much, since I had no expectation that the component I was testing was error-free.

What bothered me is that unit tests are often the best documentation of how a system works, and I would be creating misleading documentation. So, I did what anyone in this situation would do: I emailed someone a lot smarter than me. 

The reply email contained several suggestions, all of which I plan to implement. The first suggestion is to carefully name the test methods to make it clear which ones work because of defects. The second suggestion is to create an annotation that can be added to a method to indicate that it works because of a defect, coupled with some way to report on all uses of the annotation.

After a few hours of coding, I’m happy to announce that I’ve created a Java annotation named KnownDefect as well as a detector plugin for FindBugs to collect all of the annotation instances.

In the parlance of “Working Effectively with Legacy Code“, I have been creating “characterization tests“. These tests look like normal unit tests, but they document the current state of the system, warts and all. Using the KnownDefect annotation helps make the warts stick out better.

Here’s an example:

public void testValidateNullResponse() {
   try {
      ProtocolParser.validateResponse(null);
      fail();
   } catch (InvalidRequestException expected) {
      // Caught expected exception
   }
}

This test documents the system as it currently exists, but the behavior is obviously not correct. The ProtocolParser class should be throwing an InvalidResponseException, not an InvalidRequestException. To document this defect, I change the method name and add the KnownDefect annotation to the test method, resulting in the following:

@KnownDefect("Should throw InvalidResponseException")
public void testValidateNullResponseShowsKnownDefect() {
   try {
      ProtocolParser.validateResponse(null);
      fail();
   } catch (InvalidRequestException expected) {
      // Caught expected exception
   }
}

The correct behavior is now documented, and the test still passes.

Instead of writing a new tool to manage the KnownDefect instances, I ended up writing a new plugin for FindBugs. The FindBugs detector will collect all instances of the annotation and group them under the “Correctness” heading so they’re all in one place. Bug reports can then be created and the defects fixed or not as business needs dictate. (Sadly, the FindBugs plugin for Eclipse does not find the annotation in Groovy code. The standalone version of FindBugs works fine, so I assume there’s a resource filtering issue with the Eclipse plugin.)

Instructions for using the annotation and the detector are included in the download file. Try it out – hopefully it will be useful to somebody other than myself.

(Download KnownDefects) (Subversion: http://subversion.megatome.com/projects/KnownDefects/trunk)

• Upgrade unit tests to JUnit 4. There are a lot of tests, especially the ones that mock a servlet container, that do a lot of repetitive work in setUp() . JUnit 4 has finally brought the ability to annotate methods to be run before and after all tests in the class with @BeforeClass and @AfterClass, respectively. Individual test setup activities can still be performed by annotating a method with @Before.
• Fix outstanding bugs. There’s only one right now, and it’s SOAP specific, so I may or may not fix it. It may take me longer to figure out how to reproduce the problem with unit tests than it’s worth.
• Add outstanding feature requests. Again, there’s only one, but it’s (in my opinion) a biggie – the ability to add parameters to the response when handling an event.
• Doc updates. The web services doc needs to be written, and I’m sure all of the other doc needs a good examination.

I’ll be testing the latest build with a web application that I’ve been working on for years that I’ve come to think of as an unofficial reference implementation of Frame2. I haven’t decided yet if the web app should become part of the project on Sourceforge or not. We’ll see.

Once the testing is done and I’ve released a new version of Frame2, I plan on working on the Eclipse plugin. The poor plugin has pretty much been ignored for several years now, and it needs some attention.

Naturally, this has led to unit test failures. Almost all of the failures have been easy to fix, but once in a while one pops up that makes me scratch my head. Here’s an example:

public void testGetIfEmpty() {
    assertTrue(this.errors.isEmpty());
    assertNull(this.errors.get(FOO));
    assertEquals(0, this.errors.get().length);
    assertEquals(0, this.errors.get(FOO).length);
}

In this instance, errors is a class that aggregates individual Error objects. The get(String) method returns an array of Error objects that have the specified key, while get() retrieves all Errors. Pretty simple, right? After my changes, the test fails on the assertNull() statement, which isn’t surprising.

The failing statement read assertNull(this.errors.iterator(FOO)) before I removed the iterator() method, which tells us a bit about how the API was originally coded. The iterator(String) returned null if there were no matching entries. However, as the unit test clearly shows, get(String) returns an empty array when it doesn’t find any matching Errors. Quite the difference, and potentially confusing – the developer has to remember which method may return null.

The API has now been changed to return an empty collection instead of null when no matching errors are found. Now, a user of the API simply has to call the method and loop over the result (zero times if it’s empty) without having to make null checks.

Whoever wrote this unit test obviously knew that one method returned null while the other returned an empty array for the same input, since the test passed. I only wish that person would have looked at those lines and noticed the inconsistency, instead of leaving it for me to find.