Home Jonas Albert Rafael Noam Grasshopper Dekel Yaacov Roei Philippe Tal
 
  
« Service Pack 1 for Grasshopper is out!
Can you win the interop game? »

Inconsistency between NUnit and UnitRun

July 30th, 2007 by Jonas Martinsson

At Mainsoft we are using unit testing extensively as one of the means of assuring product quality. Typically we use NUnit, but I like JetBrains’ UnitRun because it allows me to run the test suite right from the development environment, along with making it easy to select which tests you want to execute. UnitRun also makes it more convenient to navigate to failed tests.

So far, JetBrains and NUnit have been playing nice together. That’s until recently, when I found out that one test that passed UnitRun failed on NUnit. After some trial & error it became obvious that there was a unwanted dependency between two of the tests in my TestFixture. If the test was being run isolated it succeeded under both systems, but under NUnit it failed when running the whole TestFixture.

xUnit (the general unit testing framework) doesn’t specify in which order to run the tests; it is up to each implementation to decide. NUnit sorts the methods in alphabetical order while UnitRun uses the order in which they occur in the TestFixture class. My test method A() was running before B() in NUnit while UnitRun had it the other way around. The inconsistency was created because A() was failing if B() had already been run.

In the end I fixed the inter-test dependency and moved around the methods inside the TestFixtures so that they now occur in alphabetical order.

Share This
Related Posts

This entry was posted on Monday, July 30th, 2007 at 8:33 am and is filed under Thinking Out Loud, Bits And Pieces. You can subscribe to the RSS feed of this blog and you can also follow any comments to this specific entry through the comment RSS feed. Both comments and pings are currently closed.

2 Responses to “Inconsistency between NUnit and UnitRun”

  1. Boris Kirzner Says:
    July 30th, 2007 at 10:50 pm

    It sounds like you’d better force your tests to run each time in different order to early discover the dependencies.
    It can be easily done by inheriting NUnit.Core.TestSuite, though I’m not sure how the IDE add-ins will handle this.
    There was a discussion about this issue once (http://discuss.develop.com/archives/wa.exe?A2=ind0308d&L=dotnet-cx&D=1&P=1382), and up to me the decision to hardcode the tests execution order was at least not too smart.

  2. Konstantin Triger Says:
    July 31st, 2007 at 10:36 pm

    I don’t think NUnit, UnitRun or any other unit testing framework should be compared in terms of integration with some IDE. Moreover, I think that it’s an IDE disadvantage if it lacks an integration with some popular tool! Fortunately for VS (and its users), there is a plugin which integrates it with NUnit - see http://www.testdriven.net/default.aspx.

   
 
Close
E-mail It