Support for TestCaseData like in NUnit


NUnit supports defining test cases with the class TestCaseData (http://www.nunit.org/index.php?p=testCaseSource&r=2.5.9)

This class allows to define a name for the test case (instead of simply using the input values in the test name), which is great in cases when the pure values are hard to tell apart.

public class MyFactoryClass
public static IEnumerable TestCases
  yield return new TestCaseData( 12, 3 );
  yield return new TestCaseData( 12, 2 );
  yield return new TestCaseData( 0, 0 )

The TestCaseData also supports that a test/Fact may return a value that is compared against the expected value (Returns(4) in the sample). However, I don't like this because it is not obvious how the actual value is compared to the expected one.

For me it would be sufficient to support setting the name of the test.
Closed Apr 13, 2013 at 7:08 PM by BradWilson


bartelink wrote Mar 15, 2013 at 10:51 PM

Is this not a slippery slope? While the fact that the code for the test happens to be identical makes this tempting, I think I'd strongly consider splitting the other case out into another Fact with an Intention Revialign Name.

Kinda like having lots of messages in your assertions, I'm not sure if it's a good idea - all these feature ideas start with -100 for me http://blogs.msdn.com/b/ericgu/archive/2004/01/12/57985.aspx

Most of these devil's advocate arguments also apply to my http://xunit.codeplex.com/discussions/390946 (am brooding over a reply on that - would be interested in your thoughts too)

Impl idea: Perhaps the ToString of the parameters might have a role to play? (Obv ToString pollution isnt ideal though)

ursenzler wrote Mar 18, 2013 at 11:15 AM

I'm looking for a good way to give the individual test cases a meaningful name, without having to use a dedicated Fact, which leads to a lot of code duplication.

I don't see how using ToString could solve this.

A typical example in our code is that there are about 10 test cases, each slightly different, but resulting in the same behavior of the code. We use a Fact per dedicated behaviour, but multiple test cases per behaviour (normal and edge cases). And I'd like to set a test name like "upper boundary edge case".

BradWilson wrote Mar 18, 2013 at 11:28 PM

Your desired use is contrary to the purpose of theory.

A theory is a test for which the assertions are true only for the given set of input data. Having one set of data be "special" compared to the others is not keeping within that spirit.

As such, we would be unlikely to implement the feature as requested.

ursenzler wrote Mar 19, 2013 at 8:20 AM

I don't understand your point (or you mine?).

It is clear that all test cases per Theory are checked using the same assertion(s). If a test case results in a different behaviour then it belongs into a different theory.
But, I'd like to give an individual test case a better name than just the Theory name used for all test cases in that Theory. It just makes it easier to spot the problem when a single test case of a Theory fails.
Currently, we do this by adding a parameter to the test method that specifies an error reason. But using error messages is also against XUnit style. Therefore, I'd prefer a way to give an individual test case a more information revealing name (in the case that the pure values passed to the test method don't transport this information well).

BradWilson wrote Mar 19, 2013 at 2:53 PM

We are, of course, arguing about this without me actually knowing what you're trying to accomplish. I'm attempting to read your mind. :)

I believe, though, that you have two separate test cases here: that is, the code under test is failing for two different reasons (the first case, I'm not sure why, but the second case is because of division by zero). Those are two separate tests, even if they have identical assertions, because they are illustrating two different behaviors in the system under test.

ursenzler wrote Mar 19, 2013 at 3:39 PM

Ah I see. Sorry this is a really bad example. I just copied the sample from the NUnit Docu to show the syntax.

I completely agree with you that the "Division by zero" Fact belongs to its dedicated Fact method because the behaviour is different from normal division.

Let me give you a better example (sorry it's a bit bloated cause I copied it from a real project):

This Test checks whether zip file contents are extracted correctly. There are two test cases, either with or without a subfolder.

the test case source (NUnit)
    private IEnumerable<TestCaseData> ExtractFilesTestCases
            yield return new TestCaseData(
                    new FileWithContent(@"firstFile.txt", "first file with content"),
                    new FileWithContent(@"secondFile.txt", "second file with content"))
                    .SetName("Files in root directory");

            yield return new TestCaseData(
                    new FileWithContent(@"SubDirectory\firstFile.txt", "first file in subdirectory with content"),
                    new FileWithContent(@"OtherDirectory\secondFile.txt", "second file in subdirectory with content"))
                    .SetName("Files in sub directory");
the test method:
    [Test, TestCaseSource("ExtractFilesTestCases")]
    public void ExtractsFilesToDestinationPathIncludingRelativePathFromZipFile(FileWithContent firstFile, FileWithContent secondFile)
        var zipStream = CreateZipFile(firstFile, secondFile);

        this.testee.ExtractAllToDestinationDirectory(zipStream, DestinationPath);

        this.AssertFileExistsAndContentMatches(DestinationPath, firstFile);
        this.AssertFileExistsAndContentMatches(DestinationPath, secondFile);
Giving the test data a name simply helps to spot the problem faster in case only one of them fails.

bartelink wrote Mar 26, 2013 at 1:10 AM

@ursenzler not trying to be a smartarse, but I'd argue all the technotrickery in your example is a prime example of a case where you're better off just using plain old Facts with clear names.

Especially if you use something like AutoFixture to take away the noise of creating the Sut (and/or abstracting some of the setup).

Yes, I know its just an example, but I've often found myself arguing for the same sort of fancy contortions from the framework but they're 99 times out of a hundred simply not a generally applicable concept (see some of my wacko ideas around here). My devil's advocate question is thus: Do you have 3 different cases from real code where there's a clear expressiveness win ?

ursenzler wrote Mar 26, 2013 at 4:14 PM

Of course, splitting the single Test into several Facts is possible. The possibility to give a test case a name would only reduce code duplication. Instead of having the same arrange, act and assert in several Facts, I'd have to code them once and could give the individual test cases a good name.

But I can live with the fact that this nice-to-have does not outweigh it's implementation cost.

Thanks for the discussion.

EmperorXLII wrote Mar 27, 2013 at 12:24 AM

I think there are two separate concerns in the original question: 1) having a general-purpose "TestCaseData" type that can be used to give a test arbitrary values, and 2) applying a name to a particular test case instance to call it out as a special case.

For #2, the clear consensus is to call out special cases as separate Facts, rather than trying to munge them in with common cases:
public void F() { ... }  // divide by zero special case ...
For #1, you can use a Theory without any need to modify the test framework, and with very little modification to the original example:
public class MyTestCaseData { ... }
public static IEnumerable<object[]> TestCases {
  get {
    yield return new object[] { new MyTestCaseData( 12, 3 ) };
    yield return new object[] { new MyTestCaseData( 12, 2 ) };

[PropertyData( "TestCases" )]
public void T( MyTestCaseData testData ) { ... }  // common test cases
(Of course, there are simpler approaches to provide theory data in this example, but this most closely matches the format in the original question.)