Strongly Typed

When I think about writing or modifying software without some way of knowing if my changes have had unintended consequences it makes me nervous - nervous in the way a pilot might feel if he were asked to fly a plane with no instrument cluster. Yet, on the other hand, when I spend more time fixing/maintaining/debugging tests than doing feature development it makes me think there has got to be a better way - the instrument panel exists for the benefit of the pilot, not vice versa.

At work we run a pretty wide variety of tests against our software - JUnit, JsUnit, and Selenium tests that span everything from short method-level tests to large functional tests that start up the server with a mock environment. The short JUnit tests have never given me a problem and they run very quickly, so they have a high utility/maintenance ratio. The Selenium tests, on the other hand, take somewhere in the range of an hour to run, seem to be somewhat flaky, and are difficult to troubleshoot. Most of the problem is not with the Selenium part, per-se, but with the way the tests are run as part of an automated build and the fact that server start-up takes a long time. The overall result, though, is that the utility/maintenance ratio is very low. It's frustrating to spend a lot of time maintaining something that isn't part of the core product and isn't actively improving the end-user's experience - especially when it keeps me from working on things that would.

As a result, I've come to the conclusion that when the utility/maintenance ratio for a test suite drops below some value N you should consider it broken and take steps to fix it. I'm not sure what the specific fix is, but you should either decrease the amount of time you have to spend in maintenance or increase the utility of the tests dramatically. Since it's easier to measure time spent in maintenance I'd start there. The utility of a test suite tends to be binary - it either catches a problem or it doesn't.