2

Resolved

Console runner does not emit proper test names in TeamCity mode

description

When using xUnit under TeamCity (or using the /TeamCity flag to repro), the multiple executions of the same test (with different parameters) all produce the same output.
##teamcity[testStarted name='MyParameterizedTest']
##teamcity[testFinished name='MyParameterizedTest' duration='6']
##teamcity[testStarted name='MyParameterizedTest']
##teamcity[testFinished name='MyParameterizedTest' duration='2']
etc...

This means that which invocation of a test was the one that actually failed cannot be determined from the TeamCity dashboard. For highly data-driven test suites(*), this significantly limits the usefulness of running the tests on the server at all.

Since TeamCity doesn't itself appear to provide any support for parameterized tests, the only fix appears to be that the xUnit runner should include the parameters as part of the recorded test 'name', eg:
##teamcity[testStarted name='MyParameterizedTest(name=duck)']
##teamcity[testFinished name='MyParameterizedTest(name=duck)' duration='6']
##teamcity[testStarted name='MyParameterizedTest(name=cow)']
##teamcity[testFinished name='MyParameterizedTest(name=cow)' duration='2']
This way the parameters used to create failure can be easily discerned (as they can in the ReSharper runner, for example).

(* we are using xUnit to verify a bunch of styling/structural rules over objects in a database schema, so most of the suite is dynamic and data-driven)

comments

BradWilson wrote Apr 15, 2013 at 3:55 PM

The behavior you're asking for is already the default behavior.

Are you overriding the display name of the test? If so, then the name is entirely of your choosing, and we don't embellish it at all. If you don't override the display name of the test, then we add the parameter values to the name.

Here is example output from the xUnit.net tests running inside of TeamCity:
StringAssertsTests+Equal.FailureCases(expected: null, actual: "", ignoreCase: False, ignoreLineEndingDifferences: False, ignoreWhiteSpaceDifferences: False, expectedIndex: -1, actualIndex: -1)
StringAssertsTests+Equal.FailureCases(expected: "", actual: null, ignoreCase: False, ignoreLineEndingDifferences: False, ignoreWhiteSpaceDifferences: False, expectedIndex: -1, actualIndex: -1)
StringAssertsTests+Equal.FailureCases(expected: "foo", actual: "foo!", ignoreCase: False, ignoreLineEndingDifferences: False, ignoreWhiteSpaceDifferences: False, expectedIndex: 3, actualIndex: 3)
StringAssertsTests+Equal.FailureCases(expected: "foo", actual: "foo ", ignoreCase: False, ignoreLineEndingDifferences: False, ignoreWhiteSpaceDifferences: False, expectedIndex: 3, actualIndex: 3)

piers7 wrote Apr 16, 2013 at 3:38 AM

Brad: that's not the behavior I'm seeing with the console runner and the /teamcity flag, and no - we're not overwriting the test name. I'm only seeing the 'stem' test name, and not the parameters.

Is this a versioning issue? We're using the console runner from the nuget package (1.9.1 from 4/10/2013) (apparently VS doesn't localize that date correctly, because that is presumably in US format)

BradWilson wrote Apr 26, 2013 at 3:08 PM

This appears to be a difference between Console and MSBuild runners. We will fix this in v2; in the meantime, please use the MSBuild runner as a work-around.

Thanks,
Brad