Inconsistency between NUnit and UnitRun

At Mainsoft we are using unit testing extensively as one of the means of assuring product quality. Typically we use NUnit, but I like JetBrains’ UnitRun because it allows me to run the test suite right from the development environment, along with making it easy to select which tests you want to execute. UnitRun also makes it more convenient to navigate to failed tests.

So far, JetBrains and NUnit have been playing nice together. That’s until recently, when I found out that one test that passed UnitRun failed on NUnit. After some trial & error it became obvious that there was a unwanted dependency between two of the tests in my TestFixture. If the test was being run isolated it succeeded under both systems, but under NUnit it failed when running the whole TestFixture.

xUnit (the general unit testing framework) doesn’t specify in which order to run the tests; it is up to each implementation to decide. NUnit sorts the methods in alphabetical order while UnitRun uses the order in which they occur in the TestFixture class. My test method A() was running before B() in NUnit while UnitRun had it the other way around. The inconsistency was created because A() was failing if B() had already been run.

In the end I fixed the inter-test dependency and moved around the methods inside the TestFixtures so that they now occur in alphabetical order.