Console runner test count is incorrect when using data theories


Version Used:

When using data-driven tests, the console runner reports its total as the total number of test methods, rather than test cases (context is the status message: "# of <Total>"). This is problematic because the current count is instead tied to test cases. The result is some strange output like:
"Tests complete: 11 of 6"
In a project where I have 6 methods decorated with TheoryAttribute.


BradWilson wrote Jul 13, 2010 at 9:26 PM

Worked as designed.

The second # is the estimated number of tests, based on the number of test methods. When doing theories, this number is usually too low, but we cannot enumerate the theory data to "correctly" count the # of tests, because (a) performance would be terrible, (b) there's no guarantee that a given data attribute will generate the same amount of data for each call.

** Closed by BradWilson 07/13/2010 2:26PM

cmrudolph wrote Jul 14, 2010 at 3:07 PM

Thanks for the comment -- your explanation makes sense. Would it make sense to format the output in a manner such as this:
Tests complete: <CompletedTestMethods> of <TotalTestMethods> -- Test cases executed: <CompletedTestCases>
So in my above example if I were on test method 2, I would see:
Tests complete: 2 of 6 -- Test cases executed: 11
I find it helpful to be able to quickly see (roughly) how far along a test run is. Something like the above would make it easier to gauge this at a glance.

BradWilson wrote Jul 15, 2010 at 12:18 AM

We'll examine this during bug fixes for v.Next and see if it's easy to achieve.

BradWilson wrote Dec 29, 2010 at 10:21 PM

We looked at this today, and it's not simple to change without breaking backward compatibility. We will consider doing this for version 2.0.

mariangemarcano wrote Sep 21, 2011 at 4:51 PM

Hello, I am using xunit 1.8 and we are having the same issue.

My team is seeing this information on CI server "Tests complete: 481 of 398 " which confuse them a lot.. and they ask me to take a look at fix it.

A quick fix will be completely removing the "of 398" part, at least you avoid displaying misleading/incorrect information on the console.

I'll submit a patch if that works best.

BradWilson wrote Nov 25, 2011 at 10:49 PM

After evaluating this, to perform the operation it will require us to extend (and therefore break) IAttributeInfo, in order to support invoking methods on attributes during enumeration. This would be an undue burden on the xUnit Contrib team to maintain two versions, so we will reconsider this issue for xUnit.net 2.0.