4

Resolved

Enable configurable maximum degree of parallelism

description

Support configuration via an assembly-level attribute, as well as via argument to the runners (command line or otherwise).

Forum post: https://xunit.codeplex.com/discussions/474862

file attachments

comments

khorvat wrote Feb 13, 2014 at 10:27 AM

Hi,

we have an issue with too many parallel tests as we have some e2e tests that consume a lot of memory and after we have upgraded to v2 we can't run the tests at all as we get out of memory exceptions.

Is there any way to avoid that by some existing configuration or we need to wait until you add something like the proposed solution ?

Thanks

BradWilson wrote Feb 13, 2014 at 6:38 PM

A work-around until we can get a controlled level of parallelism is to disable parallelism by adding an assembly level attribute to disable it:
[assembly: Xunit.CollectionBehaviorAttribute(DisableTestParallelism = true)]

khorvat wrote Feb 17, 2014 at 1:30 PM

Thanks, this is working great. FYI right now we are having few issues with parallelism so it will be great to get the controlled parallelism soon.

Regards

BradWilson wrote Feb 17, 2014 at 6:23 PM

We started work on this this past weekend, and we have support plumbed through the execution pipeline and exposed in the MSBuild and Visual Studio runners now. We didn't quite get to a finished point to do another release, but you can sure it'll be in the next alpha drop. :)

https://xunit.codeplex.com/SourceControl/latest#src/xunit2/CollectionBehaviorAttribute.cs
https://xunit.codeplex.com/SourceControl/latest#src/xunit.runner.msbuild/xunit.cs
https://xunit.codeplex.com/SourceControl/latest#src/xunit.runner.visualstudio.testadapter/Settings/XunitVisualStudioSettings.cs

PawelPabich wrote Feb 17, 2014 at 11:41 PM

Same, here this is a must have when Selenium is used. I can't really open 150 Firefox browsers :)

PawelPabich wrote Feb 18, 2014 at 3:49 AM

Hi,

I wrote a simple test case with http://www.nuget.org/packages/xunit/2.0.0-alpha-build2576 and I was able to run 4 tests concurrently. Great. Then I upgraded to the latest version from the build server http://teamcity.tier3.com/viewLog.html?buildId=94&tab=buildResultsDiv&buildTypeId=xunit_ci and now the tests execute sequentially. I know it is still work in progress but I thought you might know about it.

PawelPabich wrote Feb 18, 2014 at 3:51 AM

For some reason the file has not been attached. Result

BradWilson wrote Feb 18, 2014 at 7:57 PM

Parallelization is done on a collection basis.

By default, each test class is its own test collection. As such, tests in the same class will not be run in parallel with each other, but they will be run in parallel with tests in other classes. To illustrate the parallelism, change your example to have one test per class instead of putting them all in the same class.

PawelPabich wrote Feb 19, 2014 at 1:19 AM

Ok, I did what you said and still does not work for me. I even added Collection attribute and no luck. I must be missing something simple here. My setup: https://dl.dropboxusercontent.com/u/7579883/Parallel.png

BradWilson wrote Feb 19, 2014 at 11:16 PM

You need to measure clock time (for example, with a watch). The total that Visual Studio is showing you is very likely the combined total run time, without regard to the fact that the tests are running in parallel.

PawelPabich wrote Feb 19, 2014 at 11:42 PM

I did, it does take 10 seconds. Not sure if it helps but this is what I found in the Tests output window: "System.AppDomainUnloadedException: Attempted to access an unloaded AppDomain. This can happen if the test(s) started a thread but did not stop it. Make sure that all the threads started by the test(s) are stopped before completion.
System.AppDomainUnloadedException: Attempted to access an unloaded AppDomain. This can happen if the test(s) started a thread but did not stop it. Make sure that all the threads started by the test(s) are stopped before completion.
System.AppDomainUnloadedException: Attempted to access an unloaded AppDomain. This can happen if the test(s) started a thread but did not stop it. Make sure that all the threads started by the test(s) are stopped before completion.
System.AppDomainUnloadedException: Attempted to access an unloaded AppDomain. This can happen if the test(s) started a thread but did not stop it. Make sure that all the threads started by the test(s) are stopped before completion."

BradWilson wrote Feb 20, 2014 at 12:49 AM

The app domain unload error is a known issue with the current HEAD in source control right now, and shouldn't affect your run times.

I'll see if I can reproduce this locally.

PawelPabich wrote Feb 21, 2014 at 3:37 AM

Thanks

PawelPabich wrote Feb 21, 2014 at 3:38 AM

Thanks

avidgamer123 wrote Mar 4, 2014 at 7:19 PM

It looks like with the current implementation this wouldn't be able to be set at runtime. Am I missing something? If not, could that be a part of this issue?

Thanks!

BradWilson wrote Mar 10, 2014 at 3:13 AM

This will be settable at runtime.

With build 2595, you can set this in both the MSBuild runner (via the MaxDegreeOfParallelism property on the <xunit> task) and the Visual Studio runner (via the configuration in Tools > Options > xUnit.net).

In future builds, we will also support setting this via .vssettings files (for the Visual Studio runner) as well as the console runner. Because our TD.NET runner is an installation-less runner, we will not be able to set this at runtime with TD.NET.

It will be up to third parties (Resharper, CodeRush, NCrunch, etc.) to support this somewhere in their UIs, once they start shipping support for v2 tests.

Assembly attribute:
https://github.com/xunit/xunit/blob/4175972098982540b9ce010adbfea7a060db12c4/src/xunit.core/CollectionBehaviorAttribute.cs#L46

MSBuild runner property:
https://github.com/xunit/xunit/blob/4175972098982540b9ce010adbfea7a060db12c4/src/xunit.runner.msbuild/xunit.cs#L39

Visual Studio runner settings:
https://github.com/xunit/xunit/blob/4175972098982540b9ce010adbfea7a060db12c4/src/xunit.runner.visualstudio.settings/XunitVisualStudioSettings.cs#L39-L52

PawelPabich wrote Mar 10, 2014 at 5:17 AM

Hi Brad,

This is great. Thanks for it. Just thing is not clear for me. When I run xunit tests using msbuild and I pass 5 as MaxDegreeOfParallelism but at the same time there is CollectionBehaviourAttribute that sets MaxDegreeOfParallelism to 10 then which value will be used by the test runner ?

Thanks

Pawel

BradWilson wrote Mar 13, 2014 at 5:15 PM

Any non-zero value specified in a runner will override the values specified by the assembly-level attribute.

PawelPabich wrote Mar 14, 2014 at 2:08 AM

Awesome! I will give it a try again.