xUnit.net - Unit testing framework for C# and .NET (a successor to NUnit)

New Post: xunit2 and before-all-tests hook

kenegozi — Fri, 17 May 2013 23:13:05 GMT

thanks for the info

New Post: xunit2 and before-all-tests hook

bartelink — Fri, 17 May 2013 10:38:35 GMT

http://xunit.codeplex.com/discussions/440213
https://xunit.codeplex.com/wikipage?title=backlog&version=1

New Post: xunit2 and before-all-tests hook

kenegozi — Fri, 17 May 2013 00:40:14 GMT

In the following SO answer (http://stackoverflow.com/questions/13829737/xunit-run-code-before-and-after-all-tests), Brad hints about a before-all-tests hook to be implemented in xunit2.

is this in the works? (or otherwise - done?)

while at it - are there any details re xuni2 release?

Updated Wiki: backlog

BradWilson — Thu, 16 May 2013 00:09:01 GMT

Backlog

This page contains an informal backlog of tasks that Brad and Jim are working on. Items are roughly in priority order as determined by the team.

Target release: 2.0 Beta

Move DiaSession code out of VS runner into discoverer
- Unit testing for VS runner
Add support for IClassFixture<T>
Add support for test collections
Add support for ICollectionFixture<T>
Add support for test parallelization
- Move execution to background threads
- The main Run method in executor badly needs love
Add ability to run xUnit v1 tests with xUnit v2 runner utility library
Finish missing features in MSBuild runner
Console runner
Add support for running Windows Store tests in container (VS2012 runner only)
Review outstanding bugs / pull requests for v2 applicability

Commented Issue: Add overload of Assert.Equal(IEnumerable) with key comparer [9807]

bartelink — Mon, 06 May 2013 20:15:13 GMT

At present, I use the following helper which leverages the collection level diff report to be able to do an order-insensitive sequence comparison:

static class AssertExtensions
{
public static void SequenceEqual<T, T2>( IEnumerable<T> left, IEnumerable<T> right, Func<T, T2> keySelector, Func<T, T, bool> comparer )
{
Assert.Equal( left.OrderBy( keySelector ), right.OrderBy( keySelector ), new FuncEqualityComparer<T>( comparer ) );
}

public static void SequenceEqual<T>( IEnumerable<T> left, IEnumerable<T> right )
{
Assert.Equal( left.OrderBy( x => x ), right.OrderBy( x => x ) );
}

// http://stackoverflow.com/a/3719617
class FuncEqualityComparer<T> : IEqualityComparer<T>
{
readonly Func<T, T, bool> _comparer;
readonly Func<T, int> _hash;

public FuncEqualityComparer( Func<T, T, bool> comparer )
: this( comparer, t => 0 ) // NB Cannot assume anything about how e.g., t.GetHashCode() interacts with the comparer's behavior
{
}

public FuncEqualityComparer( Func<T, T, bool> comparer, Func<T, int> hash )
{
_comparer = comparer;
_hash = hash;
}

public bool Equals( T x, T y )
{
return _comparer( x, y );
}

public int GetHashCode( T obj )
{
return _hash( obj );
}
}
}
Comments: Consider making it as simple as possible by leaning on Tuple.Create as a way to implement a key extractor and skipping impl of equality comparers or Equals lambdas, see http://stackoverflow.com/a/1239337/11635

Closed Issue: Consider allowing [PropertyData] to be found on derived classes of abstract test classes [9738]

BradWilson — Sun, 05 May 2013 01:27:56 GMT

This works fine....

public interface IFoo
{
int AddsNumbers(int x, int y);
}

public class Foo : IFoo
{
public int AddsNumbers(int x, int y) { return x + y; }
}

public abstract class IFooTest<T>
where T: IFoo
{
private Func<T> factory;

public IFooTest(Func<T> factory)
{
this.factory = factory;
}

public static IEnumerable<object[]> Expectations
{
get
{
yield return new object[] { 1, 2, 3 };
}
}

[Theory]
[PropertyData("Expectations")]
public void Adds_ReturnsExpectedValues(int x, int y, int expected)
{
Assert.Equal(expected, factory().AddsNumbers(x, y));
}
}

Consider where IFoo was really IFoo<T> and the signature of AddNumbers becomes T AddsNumbers(T x, T y) -- trivial example, but sufficient for illustrative purposes I think. In this case, I'd like to see the abstract test class provides the reusable test code meat, and derived classes provide the data for the tests So, I would like to see something like this work:

public abstract class IFooTest<T>
where T: IFoo
{
protected Func<T> factory;

public IFooTest(Func<T> factory)
{
this.factory = factory;
}

[Theory]
[PropertyData("Expectations")]
public void Adds_ReturnsExpectedValues(int x, int y, int expected)
{
Assert.Equal(expected, factory().AddsNumbers(x, y));
}
}

public class FooTest : IFooTest<Foo>
{
public FooTest()
: base(() => new Foo())
{ }

public static IEnumerable<object[]> Expectations
{
get
{
yield return new object[] { 1, 2, 3 };
}
}
}

As it stands, this yields

Adds_ReturnsExpectedValues has failed

System.InvalidOperationException : An exception was thrown while getting data for theory xUnit.Theory.Test.FooTest.Adds_ReturnsExpectedValues:
System.ArgumentException: Could not find public static property Expectations on xUnit.Theory.Test.IFooTest`1[[xUnit.Theory.Test.Foo, xUnit.Theory.CodeRush.Bug, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]]
at Xunit.Extensions.PropertyDataAttribute.GetData(MethodInfo methodUnderTest, Type[] parameterTypes)
at Xunit.Extensions.TheoryAttribute.<GetData>d__7.MoveNext()
at Xunit.Extensions.TheoryAttribute.EnumerateTestCommands(IMethodInfo method)
Xunit.Extensions.TheoryAttribute.<>c__DisplayClass5.b__1()
Xunit.Extensions.TheoryAttribute.LambdaTestCommand.Execute(Object testClass)

I think this is a pretty straightforward tweak to the existing code base, but didn't know if you had any theoretical or design objections to enabling such functionality.
Comments: We have considered this and decided not to implement it. The inverse feature ("find property data on base classes") is coming in v2.

Edited Issue: Add assertion overloads for Func<> comparers [9808]

BradWilson — Sun, 05 May 2013 01:26:15 GMT

Now that we have access to general purpose lambdas, it would be nice to add overloads to the assertions which currently take things like IComparer<T> and IEqualityComparer<T> to include functions which do the comparisons (would likely be something like Func<T,T,int> and Func<T,T,bool>, respectively).

Closed Issue: Assert.NotNull should yield its argument [9809]

BradWilson — Sun, 05 May 2013 01:25:48 GMT

Within the Assert phase of a test, Assert.IsType can often lead to a very legible way of expressing the flow of one's observations. In this context, one is often also doing Null checks.
i.e., I want to be able to do an assert and an assign (0r follow-on Assert):

var result = Act();

var links = Assert.NotNull( result.Links);
Assert.NotEmpty( Assert.NotNull( links.Self));
Assert.NotEmpty( Assert.NotNull( links.SeeAlso));

Some reasons it should not be done:
- encourages overly constraining Assert sections in tests
- things like Assert.NotEmpty and Assert.IsType already do checks like this
- this sort of assertion is a bad idea in the Arrange and Act phases
Comments: The reason IsType and Single do this is because either the type changes in the former, and to save an extra line of code/re-enumeration in the latter. There is no value in this feature as specified.

Closed Issue: Install obfuscated at high DPI [9810]

BradWilson — Sun, 05 May 2013 01:23:41 GMT

Hi. If you run the xunit installer at high DPI, it looks bad, one can't read all the text http://i.imgur.com/W1rLi.png
Comments: The installer has been removed in v2.

Closed Issue: running tests in random order appears to be broken in 1.9.1 [9812]

BradWilson — Sun, 05 May 2013 01:22:51 GMT

imho, one of the very best features of xUnit.net is the running of unit tests in random order because this lessens substantially the chance of unit tests having dependency smells.

i'm using vs2012 RC to run some very simple tests ... i'm rerun them again and again and again but the order does not change.

xUnit version: 1.9.1.1600
runner: via NuGet: xunit.runners.1.9.1 tools xunit.gui.clr-4.exe
Microsoft Visual Studio Ultimate 2012 RC Version 11.0.50522.1 RCREL
win7; intel i3-350M processor (dual core, 4 logical CPUs)

they always run in this order:

FileFinder.xUnit.net.Tests FileFinderTDD thisTestAlwaysFails
FileFinder.xUnit.net.Tests ssssssssssssssssssssssss this_extra_xUnit_net_test_always_fails
FileFinder.xUnit.net.Tests ssssssssssssssssssssssss this_xUnit_net_test_always_fails

when i add more tests*, the lack of random behavior is consistent ...
-- classes are run in same order as in the source code
-- within classes, tests are run in reverse order to their position within their corresponding classes

g.

----------------------------------------------------------------

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace FileFinder.xUnit.net.Tests
{
public class FileFinderTDD
{
[Xunit.Fact]
public void thisTestAlwaysFails()
{
Xunit.Assert.True(false);
}
}
public class ssssssssssssssssssssssss
{
[Xunit.Fact]
public void this_xUnit_net_test_always_fails()
{
Xunit.Assert.True(false);
}
[Xunit.Fact]
public void this_extra_xUnit_net_test_always_fails()
{
Xunit.Assert.True(false);
}
}
}

--------------------------------------------------
* more tests version:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace FileFinder.xUnit.net.Tests
{
public class FileFinderTDD
{
[Xunit.Fact]
public void thisTestAlwaysFails()
{
Xunit.Assert.True(false);
}
[Xunit.Fact]
public void this_xUnit_net_test_always_fails()
{
Xunit.Assert.True(false);
}
[Xunit.Fact]
public void this_extra_xUnit_net_test_always_fails()
{
Xunit.Assert.True(false);
}

}
public class ssssssssssssssssssssssss
{
[Xunit.Fact]
public void this_xUnit_net_test_always_fails()
{
Xunit.Assert.True(false);
}
[Xunit.Fact]
public void this_extra_xUnit_net_test_always_fails()
{
Xunit.Assert.True(false);
}
[Xunit.Fact]
public void thisTestAlwaysFails()
{
Xunit.Assert.True(false);
}

}
}

--------------------------------------------------

Microsoft Visual Studio Ultimate 2012 RC Version 11.0.50522.1 RCREL
Microsoft .NET Framework Version 4.5.50501

Installed Version: Ultimate

Architecture and Modeling Tools 04166-004-0009005-02449
Architecture and Modeling Tools

UML® and Unified Modeling Language™ are trademarks or registered trademarks of the Object Management Group, Inc. in the United States and other countries.

LightSwitch for Microsoft Visual Studio 2012 RC 04166-004-0009005-02449
LightSwitch for Microsoft Visual Studio 2012 RC

Microsoft Office Developer Tools 04166-004-0009005-02449
Microsoft Office Developer Tools

Microsoft Visual Basic 2012 RC 04166-004-0009005-02449
Microsoft Visual Basic 2012 RC

Microsoft Visual C# 2012 RC 04166-004-0009005-02449
Microsoft Visual C# 2012 RC

Microsoft Visual C++ 2012 RC 04166-004-0009005-02449
Microsoft Visual C++ 2012 RC

Microsoft Visual F# 2012 04166-004-0009005-02449
Microsoft Visual F# 2012

Microsoft Visual Studio 2012 RC Code Analysis Spell Checker 04166-004-0009005-02449
Microsoft® Visual Studio® 2012 RC Code Analysis Spell Checker

Portions of International CorrectSpell™ spelling correction system © 1993 by Lernout & Hauspie Speech Products N.V. All rights reserved.

The American Heritage® Dictionary of the English Language, Third Edition Copyright © 1992 Houghton Mifflin Company. Electronic version licensed from Lernout & Hauspie Speech Products N.V. All rights reserved.

Microsoft Visual Studio 2012 SharePoint Developer Tools 04166-004-0009005-02449
Microsoft Visual Studio 2012 SharePoint Developer Tools

Microsoft Visual Studio 2012 Team Explorer RC 04166-004-0009005-02449
Microsoft Visual Studio 2012 Team Explorer RC

Microsoft Visual Web Developer 2012 RC 04166-004-0009005-02449
Microsoft Visual Web Developer 2012 RC

Microsoft SQL Server Data Tools 11.1.20425.00
Microsoft SQL Server Data Tools

NuGet Package Manager 1.8.30423.9026
NuGet Package Manager in Visual Studio. For more information about NuGet, visit http://docs.nuget.org/.

PreEmptive Analytics Visualizer 1.0
Visual Studio 11 extension to visualize aggregated summaries from the PreEmptive Analytics product.

Web Tooling Extensions RC 1.0
Page Inspector: Tool that offers an efficient way to decompose Web Applications and diagnose front-end issues.
Web Publishing: Extensions required for Web Publishing for both hosted servers as well as on premises.
Web Form Templates: Includes the default templates for Web Form Applications.
Editor Extensions: Includes HTML, CSS, and JS editor extensions that greatly enhance the development experience.
Comments: We have no plans to change this behavior at the moment. However, in v2, test collections will introduce parallelization which will introduce a much higher degree of randomness, including the behavior you're asking for.

Closed Issue: Allow tests in non-public classes [9801]

BradWilson — Sun, 05 May 2013 01:19:38 GMT

Remove the requirement that a class containing tests be marked `public`.

1. It's currently error-prone, as you might forget to mark your test class `public` and the tests will quietly never run, giving a false sense of security.

2. You never actually expect someone to consume a test class as a class, outside of the context of a unit testing framework. That is, it's not a public API in the normal sense.

3. There's a nice (although uncommon) pattern where the unit tests for a class live in a nested class called `Test`, like this:

class MyClass
{
class Tests
{
[Xunit.Fact]
void ATest()
{
//...
}
}
}

Comments: We have private tests in public classes but we not intend ti have tests in non-public classes.

Edited Issue: Exceptions from Dispose method should be aggregated with exceptions from test method [9817]

BradWilson — Sun, 05 May 2013 01:17:18 GMT

On occasion I like to use the Dispose() method on a test class to perform general asserts that should hold for every test case:

public class Tests : IDisposable {
public void Dispose() {
// general asserts
}

[Fact]
public void Test() {
// specific asserts
}
}

However, the current behavior is that an assert failure in the Dispose method will blow away any assert failures in the test method. This makes it tricky to debug regressions in tests because the UI presents incomplete information.
It would be nice if exceptions from the test method were aggregated with exceptions from the Dispose method, such that if something fails we see a full picture of what's going on (and make sure to preserver the order: test method first, Dispose second()).

Edited Issue: Add overload of Assert.Equal(IEnumerable) with key comparer [9807]

BradWilson — Sun, 05 May 2013 01:15:23 GMT

Commented Issue: Offer display options in VS2012 runner [9828]

BradWilson — Sun, 05 May 2013 01:12:01 GMT

The VS2012 runner today only shows tests by test method name. Options should allow the user to include class and namespace names, if desired.
Comments: For v2, we've shifted the default to be the fully qualified name. We'll see if people want the option for a switch after we ship the first beta releases.

Edited Issue: Offer display options in VS2012 runner [9828]

BradWilson — Sun, 05 May 2013 01:12:01 GMT

The VS2012 runner today only shows tests by test method name. Options should allow the user to include class and namespace names, if desired.

Edited Issue: Assert.ThrowsArgument(Null) needs Func

xUnit.net - Unit testing framework for C# and .NET (a successor to NUnit)

New Post: xunit2 and before-all-tests hook

New Post: xunit2 and before-all-tests hook

New Post: xunit2 and before-all-tests hook

Updated Wiki: backlog

Backlog

Commented Issue: Add overload of Assert.Equal(IEnumerable) with key comparer [9807]

Closed Issue: Consider allowing [PropertyData] to be found on derived classes of abstract test classes [9738]

Edited Issue: Add assertion overloads for Func<> comparers [9808]

Closed Issue: Assert.NotNull should yield its argument [9809]

Closed Issue: Install obfuscated at high DPI [9810]

Closed Issue: running tests in random order appears to be broken in 1.9.1 [9812]

Closed Issue: Allow tests in non-public classes [9801]

Edited Issue: Exceptions from Dispose method should be aggregated with exceptions from test method [9817]

Edited Issue: Add overload of Assert.Equal(IEnumerable) with key comparer [9807]

Commented Issue: Offer display options in VS2012 runner [9828]

Edited Issue: Offer display options in VS2012 runner [9828]

Edited Issue: Assert.ThrowsArgument(Null) needs Func

Closed Issue: Make it easier to create theory data [9839]

Commented Issue: Test runner - add a node for each Theory data item [9789]

Edited Issue: Test runner - add a node for each Theory data item [9789]

Edited Issue: Add FlowId support for TeamCity [9833]