1

Closed

running tests in random order appears to be broken in 1.9.1

description

imho, one of the very best features of xUnit.net is the running of unit tests in random order because this lessens substantially the chance of unit tests having dependency smells.

i'm using vs2012 RC to run some very simple tests ... i'm rerun them again and again and again but the order does not change.

xUnit version: 1.9.1.1600
runner: via NuGet: xunit.runners.1.9.1 tools xunit.gui.clr-4.exe
Microsoft Visual Studio Ultimate 2012 RC Version 11.0.50522.1 RCREL
win7; intel i3-350M processor (dual core, 4 logical CPUs)

they always run in this order:

FileFinder.xUnit.net.Tests FileFinderTDD thisTestAlwaysFails
FileFinder.xUnit.net.Tests ssssssssssssssssssssssss this_extra_xUnit_net_test_always_fails
FileFinder.xUnit.net.Tests ssssssssssssssssssssssss this_xUnit_net_test_always_fails

when i add more tests*, the lack of random behavior is consistent ...
-- classes are run in same order as in the source code
-- within classes, tests are run in reverse order to their position within their corresponding classes

g.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace FileFinder.xUnit.net.Tests
{
public class FileFinderTDD
{
    [Xunit.Fact]
    public void thisTestAlwaysFails()
    {
        Xunit.Assert.True(false);
    }
}
public class ssssssssssssssssssssssss
{
    [Xunit.Fact]
    public void this_xUnit_net_test_always_fails()
    {
        Xunit.Assert.True(false);
    }
    [Xunit.Fact]
    public void this_extra_xUnit_net_test_always_fails()
    {
        Xunit.Assert.True(false);
    }
}
}

  • more tests version:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace FileFinder.xUnit.net.Tests
{
public class FileFinderTDD
{
    [Xunit.Fact]
    public void thisTestAlwaysFails()
    {
        Xunit.Assert.True(false);
    }
    [Xunit.Fact]
    public void this_xUnit_net_test_always_fails()
    {
        Xunit.Assert.True(false);
    }
    [Xunit.Fact]
    public void this_extra_xUnit_net_test_always_fails()
    {
        Xunit.Assert.True(false);
    }

}
public class ssssssssssssssssssssssss
{
    [Xunit.Fact]
    public void this_xUnit_net_test_always_fails()
    {
        Xunit.Assert.True(false);
    }
    [Xunit.Fact]
    public void this_extra_xUnit_net_test_always_fails()
    {
        Xunit.Assert.True(false);
    }
    [Xunit.Fact]
    public void thisTestAlwaysFails()
    {
        Xunit.Assert.True(false);
    }

}
}

Microsoft Visual Studio Ultimate 2012 RC Version 11.0.50522.1 RCREL
Microsoft .NET Framework Version 4.5.50501

Installed Version: Ultimate

Architecture and Modeling Tools 04166-004-0009005-02449
Architecture and Modeling Tools

UML® and Unified Modeling Language™ are trademarks or registered trademarks of the Object Management Group, Inc. in the United States and other countries.

LightSwitch for Microsoft Visual Studio 2012 RC 04166-004-0009005-02449
LightSwitch for Microsoft Visual Studio 2012 RC

Microsoft Office Developer Tools 04166-004-0009005-02449
Microsoft Office Developer Tools

Microsoft Visual Basic 2012 RC 04166-004-0009005-02449
Microsoft Visual Basic 2012 RC

Microsoft Visual C# 2012 RC 04166-004-0009005-02449
Microsoft Visual C# 2012 RC

Microsoft Visual C++ 2012 RC 04166-004-0009005-02449
Microsoft Visual C++ 2012 RC

Microsoft Visual F# 2012 04166-004-0009005-02449
Microsoft Visual F# 2012

Microsoft Visual Studio 2012 RC Code Analysis Spell Checker 04166-004-0009005-02449
Microsoft® Visual Studio® 2012 RC Code Analysis Spell Checker

Portions of International CorrectSpell™ spelling correction system © 1993 by Lernout & Hauspie Speech Products N.V. All rights reserved.

The American Heritage® Dictionary of the English Language, Third Edition Copyright © 1992 Houghton Mifflin Company. Electronic version licensed from Lernout & Hauspie Speech Products N.V. All rights reserved.

Microsoft Visual Studio 2012 SharePoint Developer Tools 04166-004-0009005-02449
Microsoft Visual Studio 2012 SharePoint Developer Tools

Microsoft Visual Studio 2012 Team Explorer RC 04166-004-0009005-02449
Microsoft Visual Studio 2012 Team Explorer RC

Microsoft Visual Web Developer 2012 RC 04166-004-0009005-02449
Microsoft Visual Web Developer 2012 RC

Microsoft SQL Server Data Tools 11.1.20425.00
Microsoft SQL Server Data Tools

NuGet Package Manager 1.8.30423.9026
NuGet Package Manager in Visual Studio. For more information about NuGet, visit http://docs.nuget.org/.

PreEmptive Analytics Visualizer 1.0
Visual Studio 11 extension to visualize aggregated summaries from the PreEmptive Analytics product.

Web Tooling Extensions RC 1.0
Page Inspector: Tool that offers an efficient way to decompose Web Applications and diagnose front-end issues.
Web Publishing: Extensions required for Web Publishing for both hosted servers as well as on premises.
Web Form Templates: Includes the default templates for Web Form Applications.
Editor Extensions: Includes HTML, CSS, and JS editor extensions that greatly enhance the development experience.
Closed May 5, 2013 at 2:22 AM by BradWilson
We have no plans to change this behavior at the moment.

However, in v2, test collections will introduce parallelization which will introduce a much higher degree of randomness, including the behavior you're asking for.

comments

gerrylowry wrote Aug 11, 2012 at 4:40 PM

my mistake ... the order of the classes does not change ...

within the classes, the order of the tests does change randomly


Suggestion: increase the randomness to also include class order, examples:

-- first run
FileFinder.xUnit.net.Tests FileFinderTDD thisTestAlwaysFails
FileFinder.xUnit.net.Tests ssssssssssssssssssssssss this_extra_xUnit_net_test_always_fails
FileFinder.xUnit.net.Tests ssssssssssssssssssssssss this_xUnit_net_test_always_fails

-- second run
FileFinder.xUnit.net.Tests ssssssssssssssssssssssss this_extra_xUnit_net_test_always_fails
FileFinder.xUnit.net.Tests FileFinderTDD thisTestAlwaysFails
FileFinder.xUnit.net.Tests ssssssssssssssssssssssss this_xUnit_net_test_always_fails

-- third run
FileFinder.xUnit.net.Tests ssssssssssssssssssssssss this_xUnit_net_test_always_fails
FileFinder.xUnit.net.Tests ssssssssssssssssssssssss this_extra_xUnit_net_test_always_fails
FileFinder.xUnit.net.Tests FileFinderTDD thisTestAlwaysFails
       et cetera
NOTE: by increasing the randomness, even more potential dependency smells will be removed.