Table of Contents

Attributes

Note: any testing framework attributes that are not in this list have no corresponding attribute in xUnit.net.

NUnit 2.2 MSTest xUnit.net Comments
[Test] [TestMethod] [Fact] Marks a test method.
[TestFixture] [TestClass] n/a xUnit.net does not require an attribute for a test class; it looks for all test methods in all public (exported) classes in the assembly.
[ExpectedException] [ExpectedException] Assert.Throws or Record.Exception xUnit.net has done away with the ExpectedException attribute in favor of Assert.Throws. See Note 1.
[SetUp] [TestInitialize] Constructor We believe that use of [SetUp] is generally bad. However, you can implement a parameterless constructor as a direct replacement. See Note 2.
[TearDown] [TestCleanup] IDisposable.Dispose We believe that use of [TearDown] is generally bad. However, you can implement IDisposable.Dispose as a direct replacement. See Note 2.
[TestFixtureSetUp] [ClassInitialize] IUseFixture<T> To get per-fixture setup, implement IUseFixture<T> on your test class. See Note 3
[TestFixtureTearDown] [ClassCleanup] IUseFixture<T> To get per-fixture teardown, implement IUseFixture<T> on your test class. See Note 3
[Ignore] [Ignore] [Fact(Skip="reason")] Set the Skip parameter on the [Fact] attribute to temporarily skip a test.
n/a [Timeout] [Fact(Timeout=n)] Set the Timeout parameter on the [Fact] attribute to cause a test to fail if it takes too long to run. Note that the timeout value for xUnit.net is in milliseconds.
[Property] [TestProperty] [Trait] Set arbitrary metadata on a test
n/a [DataSource] [Theory], [XxxData] Theory (data-driven test). See Note 4


Note 1: Long-term use of [ExpectedException] has uncovered various problems with it. First, it doesn't specifically say which line of code should throw the exception, which allows subtle and difficult-to-track failures that show up as passing tests. Second, it doesn't offer the opportunity to fully inspect details of the exception itself, since the handling is outside the normal code flow of the test. Assert.Throws allows you to test a specific set of code for throwing an exception, and returns the exception during success so you can write further asserts against the exception instance itself.

Note 2: The xUnit.net team feels that per-test setup and teardown creates difficult-to-follow and debug testing code, often causing unnecessary code to run before every single test is run. For more information, see http://jamesnewkirk.typepad.com/posts/2007/09/why-you-should-.html and http://www.agileprogrammer.com/dotnetguy/articles/SetupTeardown.aspx.

Note 3: xUnit.net provides a new way to think about per-fixture data with the use of the IUseFixture<T> interface. By implementing this interface, you tell the test system that you wish to have certain fixture data provided to you before each test is run, the type of which is the T in the generic part of the interface. The runner will create a single instance of the fixture data and pass it through to your SetFixture() method before running each test. All the tests share the same instance of fixture data. After all the tests have run, the runner will dispose of the fixture data, if it implements IDisposable.

Note 4: The extensions library (xunit.extensions.dll) ships with support for data-driven tests call Theories. Mark your test with the [Theory] attribute (instead of [Fact]), then decorate it with one or more [XxxData] attributes, including [InlineData], [PropertyData], [ExcelData], [SqlServerData], and [OleDbData]. See the unit tests in the test.xunitext project for examples.

Assertions

xUnit.net has removed most instances of the words "Are" and "Is". The NUnit 2.2.10 Assert class is used for comparison purposes; note that NUnit 2.4 offers both NUnit 2.2.x style and new "natural language" assertions. Note: any testing framework assertions which are not in this list have no corresponding assertion in xUnit.net.

NUnit 2.2 MSTest xUnit.net Comments
AreEqual AreEqual Equal MSTest and xUnit.net support generic versions of this method
AreNotEqual AreNotEqual NotEqual MSTest and xUnit.net support generic versions of this method
AreNotSame AreNotSame NotSame
AreSame AreSame Same
Contains Contains (on CollectionAssert) Contains
DoAssert n/a n/a
n/a DoesNotContain (on CollectionAssert) DoesNotContain
n/a n/a DoesNotThrow Ensures that the code does not throw any exceptions
Fail Fail n/a xUnit.net alternative: Assert.True(false, "message")
Greater n/a n/a xUnit.net alternative: Assert.True(x > y)
Ignore Inconclusive n/a
n/a n/a InRange Ensures that a value is in a given inclusive range (note: NUnit and MSTest have limited support for InRange on their AreEqual methods)
IsAssignableFrom n/a IsAssignableFrom
IsEmpty n/a Empty
IsFalse IsFalse False
IsInstanceOfType IsInstanceOfType IsType
IsNaN n/a n/a xUnit.net alternative: Assert.True(double.IsNaN(x))
IsNotAssignableFrom n/a n/a xUnit.net alternative: Assert.False(obj is Type);
IsNotEmpty n/a NotEmpty
IsNotInstanceOfType IsNotInstanceOfType IsNotType
IsNotNull IsNotNull NotNull
IsNull IsNull Null
IsTrue IsTrue True
Less n/a n/a xUnit.net alternative: Assert.True(x < y)
n/a n/a NotInRange Ensures that a value is not in a given inclusive range
n/a n/a Throws Ensures that the code throws an exact exception

Last edited Nov 14, 2012 at 2:08 PM by BradWilson, version 30

Comments

jason_r Apr 10 at 5:37 AM 
Hi, sorry for the shame(less) plug, there is a chapter in my book Keeping Software Soft on "An Introduction to Unit Testing With xUnit.net". The book itself is aimed primarily at apprentice and journeyman software developers. KeepingSoftwareSoft.com

pejvan Mar 20 at 4:03 PM 
I have spent some time doing an exhaustive comparison of the 3 frameworks above, in their latest version. Available here: https://docs.google.com/spreadsheet/ccc?key=0AoNfKcTBEL4bdEZZb2JRYUwzSVZ6MXZvTTRPcDc3MGc&usp=sharing

jimnewkirk Mar 13 at 7:01 PM 
We do not track the latest version of NUnit or MSTest for that matter. The table was marked with NUnit 2.2 because that is when we did the comparison. Given that it specifically states that its NUnit 2.2 I do not think that it is unfair.

Nick60 Mar 13 at 6:00 PM 
Hi BradWilson, on Nov 14, 2012 NUnit was at version 2.6 (http://nunit.org/?p=releaseNotes&r=2.6.2). I'm really interessed on using xUnit, but if you are unfair on compare with other frameworks, this scare me!

jimnewkirk Nov 15, 2012 at 12:42 AM 
This is precisely why we will never implement Assert.That - Assert.That(() => myObj.TheFunc(1,2), Throws.Exception.InstanceOfType<InvalidOperationException>().With.Property("Message").Contains("Less Than Or Equal")); - this is horrifying.

MaziarTaheri Nov 16, 2011 at 6:37 AM 
So how about "Assert.That(someValue, Is.EqualTo(12))" in NUnit 2.5? which one is more readable? I vote a on NUnit.
And by the way, your table is outdated and has to be updated. for example in NUnit we have:
Assert.That(() => myObj.TheFunc(1,2), Throws.Exception.InstanceOfType<InvalidOperationException>().With.Property("Message").Contains("Less Than Or Equal")); and [ExpectedException] is not needed anymore.

jimmyzimms Jun 22, 2011 at 7:12 PM 
I have to concur that the fluent assertion interface in nunit is fantastic. Fantastic enough that we've even written wrapper libaries to MSTEST to model this pattern (for those that case occasionally we're stuck/cursed with using that POS and at least then it is palatable). This would be great to see it added to XUnit!

bartelink Jul 26, 2010 at 1:02 PM 
I disagree with most of your contentions, but then I am the converted - IMO they are all by design and have good justifications for them (i.e., anything that seems different for the sake of it). Bottom line for me is that there is no value in compatibility with an OTT batch of 70 attributes (half of which are abuses of attributes and taste and other ones which have been agreed to be bad practice such as ExpectedException) which nobody with a day job understands all of.

There are no books that cover xUnit.net directly that I've heard of.

I've personally read
1) Pragmatic Unit Testing in C# with NUnit. Good general NUnit book but no coverage of RhinoMocks or Moq. No major weaknesses other than being due an update
2) the Osherove book. Good general NUnit book but dated usage of RhinoMocks and not much (none IIRC) usage of Moq. Very opinionated and requires antidote of neutral advice (see 3). You wont regret buying it
3) the xUnit Test Patterns book. Its 800 pages and is not a beginner walkthrough. But its complete and you should have it - it really solidified my understanding.
4) The Newkirk/Vorontsov NUnit book. Gets too deep into webservices and doesnt cover important stuff Osherove and PragProgs books doo

If you have the base knowledge from one, two or three of the above, then reading all the blog posts referenced from the main wiki shouldnt leave any question unanswered

More detail at http://stackoverflow.com/questions/387326/unit-testing-videos-or-podcasts/388485#388485

NJKotze Jul 25, 2010 at 8:24 AM 
Hi Guys - I'm a tester and not a full time programmer. Looking at the ATTRIBUTES of xUnit diminished my excitement. They are not so easy to spot for the untrained eye or new users who does not code 8 hours a day. This makes proposing this framework much more difficult. I don’t understand the complete change of terms but here is my feedback on some of the ATTRIBUTES not appealing to me are:
1) [Fact] ( [Test] would have been much more clearer. )
2) Constructor ( Could have rather been [TestPrep] )
3) IDisposable.Dispose (complicated. [TestClosure] would have been easier to spot. )
4) [Trait] ( [Property] was nicer and clearer on what you are dealing with )
5) [Fact(Skip="reason")] & [Fact(Timeout=n)] ( This is very nice and descriptive, but the "Fact" attribute is still a thorn )
6) [Theory], [XxxData] ( This is odd. [XxxData] makes sense, but [Theory] not. [TestData] could have be nicer. )
7) IUseFixture<T> ( something like [FixtureIn]<T>/[FixtureOut]<T> or [FixtureUp]<T> / [FixtureDown]<T> would have read easier because you can see a start point and and end point. )
8) Assert.Throws or Record.Exception ( The idea behind it reading NOTE 1 makes sense. Not sure what "Record" does, but in testing we use the terms "Assert" to stop test if error occur, and "Validate" to log the error and continue testing. )

Can someone please recommend a book on xUnit for beginners / dummies.

Thanks guys

dschilling Jan 18, 2010 at 8:47 PM 
Since version 2.5, NUnit has the [TestCase] attribute for data-driven tests and also a [Timeout] attribute.

bartelink Sep 23, 2009 at 2:53 PM 
There is a Category in Extensions (it's derived from Trait). The GUI runner has trait-based filtering, see http://feedproxy.google.com/~r/BradWilson/~3/SNM_fROJjc8/xunitnet-15-shipped.html

Be careful with mixing 'slow' and 'fast' tests - integration tests should be called as such and probably separated into assemblies with that name

colinjack Sep 23, 2009 at 1:38 PM 
Do you think there is any chance you could add something like a category attribute in the future, I don't like having to split out slower tests out to a seperate project.

BradWilson Apr 17, 2009 at 7:16 PM 
@iSpeakGeek, to the best of my knowledge, all runners except Resharper will actually show you the values that were passed into the theory as parameters to the name of the test method. For example, this test:

public class MyTests {
[Theory, InlineData(27), InlineData(42)]
public void NumberMustBeAwesome(int value) {
Assert.Equal(42, value);
}
}

Shows the failure like this with the console runner:

1) MyTests.NumberMustBeAwesome(27) : Assert.Equal() Failure

iSpeakGeek Apr 17, 2009 at 12:13 PM 
I'd vote for being able to add context to my assertions - consider a test where we are verifying that changes are isolated e.g.

Assert.Equal(1, calculator.NumeracyLevel(qualifications));
Assert.Equal(0, calculator.LiteracyLevel(qualifications));
Assert.Equal(0, calculator.SomeOtherLevel(qualifications));

If either of the later two assertions fail, I can obviously find out what's wrong by looking at the line number. That's fine until I run this as a Theory test or if I have more than one or two numeric assertions. You might argue in the latter case that I should re-write my test to keep it to a small number of assertions, but in my situation that would make the individual tests less thorough or much harder to maintain.

With Theory tests it may not be obvious that I have multiple points of failure unless I carefully inspect my results to see whether the line numbers for my numeric assertions are different.

I'd also argue that comments can be useful in tracking down the failure. I've frequently found myself adding notes like:

// If this fails here, it's probably because of something like...

or

// If this fails here multiple times in the theory test then the test data may be incomplete - has the data changed?

Having these entries in the code is useful, but I could isolate which failing tests to concentrate on quicker if I could see these notes in the test output.

So, given that being able to optionally include messages in the assertions does no harm and can be useful in certain situations, I'd say it'd be better not to limit the users of the test framework wouldn't it?

Cheers,

John

kolorahl Feb 28, 2009 at 12:58 AM 
No, I'd have to agree with Brad. "The query did not return the expected number of results" is actually worse than "someValue does not equal 42." But not when you're specifically using the variable name "someValue." Let's say you got a query result and stored the count in 'queryResCount'. Now the generated exception message is more like "queryResCount does not equal 42." You see, the variable name clearly states what value you're comparing and the number 42 tells you exactly what your expected size was. Your example simply used "expected number of results." Well, what is the expected number?

The system was implemented more to prevent people from being lazy or accidentally skipping over data that should be displayed. I think that as long as you name your variables well the messages should be pretty good at telling you what went wrong for what reason.

DanielPuzey Jan 28, 2009 at 4:11 PM 
I'd have to disagree, Brad - surely any level of context you can add is useful? Why should someValue be 42? The example you gave may be useless, but an example like Assert.Equal(42, someValue, "The query did not return the expected number of results") is less so.

By forcefully excluding the reason, aren't you making it necessary to inspect the test code to work out why the test failed - instead of it being clear from the output?

I do like the use of pre-existing mechanisms - such as IDisposable for cleanup and constructors for initialisation. It makes the test classes much more portable and reusable outside of the test framework.

mknaup Nov 21, 2008 at 9:49 PM 
Are there any plans for a similar extension like NUnitForms? Or how do you test GUI applications with XUnit?

BradWilson Sep 8, 2008 at 3:12 PM 
@chilay, the reason we removed messages from everything except True/False is because the asserts should be descriptive enough. Assert.Equal(42, someValue, "someValue should've been 42") doesn't add a lot of value, and if you can't tell what an assert is testing when you write it, then it's probably broken anyway. Worst case, you can add a comment to the assert method. We left the messages on True/False, because these tend to be the catchall handlers used when a specific assertion is not present.

chaliy Sep 8, 2008 at 11:06 AM 
Guys, could you please explain reasons why you remove(not implemented) message for assertion methods?

BradWilson Aug 26, 2008 at 6:32 AM 
@cristianp, we have [AutoRollback] as an extension in xunitext.dll

cristianp May 26, 2008 at 5:03 PM 
Would you support [Rollback] Attributes in the test? (as mbunit and nunit extensions). And what about Assert.That as the nunit.framework.syntax in NUnit 2.4? I guess it would make the Tests Asserts a lot more clear.

BradWilson Apr 1, 2008 at 12:14 AM 
Gabriel, yes, in the sense that you can use [Trait], but no, in the sense that none of the runners make any use of traits for decide what should or should not be run. IMO, it's better to make separate test assemblies than it is to use Categories.

Gabriel Mar 17, 2008 at 1:12 PM 
Are there plans to add the equivalent of NUnits [Category] attribute?

rhwilburn Oct 16, 2007 at 7:10 AM 
This is a really great testing framework. I think its the best I have seen as it has scaled back all of the annotation bloat from setup and tear down methods which are not necessary as they are implied by the life of the object that hosts the test.