|
Hi Ncage,
The xUnit Design Patterns book covers the myriad tadeoffs involved to a degree, as does the Feathers book.
I'm also interested in a proper in depth answer to this based on 2009 knowledge, especially in the context of C#3+ language features and xUnit.net.
You definitely don't want to get into a situation where you have Test Code In Production in any shape or form. This is what drives people toward the general policy of housing things in a separate binary.
There are a number of issues in play, many of which you have alluded to:
1) Dependecy on xunit.dll
2) Bigger code / Longer load times
3) Having to expose more IP by having to make stuff private or internal
4) Having to manage InternalsVisibleTo in the face of Strong Naming (dont do it)
5) Having contorted classes just to support testing and/or the risk of code that's only there to support tests getting into production
6) Having to suffer random build breaks because publicize.exe falls over (when using Test References/Private Accessors/publicize.exe)
7) Having dependencies on VSTT on build agent boxes because one used Private Accessors (when using Test References/Private Accessors/publicize.exe)
There is the notion of a test stripper (http://xunitpatterns.com/test%20stripper.html)
which would remove test related classes and the dependency on things like xunit.dll, but I'm not aware of one for .NET, and searching for anything involving Strippers on this here net be problmatic.
See the sample below for references to Approaches.
Summary of plusses/minuses of putting stuff in the same bin:
+ You can nest test classes into classes [Approach5] and/or use partial classes [Approach1,Approach2] to give access to privates instead of having to use stuff like:
* MsTest's 'Test References' stuff (using publicize.exe to generate a Private Accessor per class which uses reflection to access private stuff) [Approach3]
* Having a hand-written 'For Test Only' extension or derived class which exposes stuff needed for tests but would be redundant and/or potentially damaging to have in a final binary [Approach2A,Approach2B]
* Another idea is
http://stackoverflow.com/questions/1043899/non-code-generated-forwarding-shim-for-testing-private-methods
+ You dont need to use InternalsVisibleTo (which gets messy with strong naming] to grant access to non-public stuff [without having to pollute your interface by making it public) [Approach4]
- Potential for Test Code in production
- Need to use a Test Stripper or conditional compilation (I dont believe in using DEBUG for this - I want my tests to be runnable in both release and debug[1]
Based on the above, my considerations in your decision would be:
- InternalsVisibleTo is full of holes and there is a lot of consensus on not using it
- You dont want to have a dependency on xunit.dll -- But how are you going to remove that?
- If you use conditional compilation, it should be based on something other than just DEBUG
- You could use a Test Stripper if you had one, and it should a) remove the ref to the DLL b) remove classes with methods with FactAttribute-derived tags c) probably remove other Fto methods or classes
- You really dont want to get into using MSTest Test References/Private Accessors/publicize.exe) for so many reasons
I highly recommend the Meszaros book for helping one think through all of this stuff (I read Osherove's book too, which is worthy and an easier read but not as deep and neutral as xUnit Test Patterns).
Bottom line:
1. The Simplest Thing That Could Possibly Work is separate assemblies
2. Separate assemblies are not a perfect answer, so lots of people go down rabbit holes of hacks to make that work
3. Not many people use test code in the SUT, and of those, many people are leaving themselves open to a variety of Test Code In Production issues
I'd love to hear more on this, because I'm not happy with my answers and/or the currrent state of play on this.
--Ruben
using System;
using System.Runtime.CompilerServices;
using Xunit;
namespace
Approach1
{
namespace
SutAssembly
{
partial class
A
{
private
void F1()
{
}
internal
void F2()
{
}
}
#if !STRIP_TESTS
partial class
A // FTO stuff
{
public
void CallF1()
{
F1();
}
public
void CallF2()
{
F2();
}
}
#endif
}
namespace
TestAssembly
{
using SutAssembly;
public class
WhenAccessingPrivates
{
[
Fact ]
public
void CanAccessF1AndF2()
{
var
a = new
A();
a.CallF1();
a.CallF2();
}
}
}
}
namespace
Approach2A
{
namespace
SutAssembly
{
public partial
class
A
{
private
void F1()
{
}
private
void F2()
{
}
}
class FtoAttribute :
Attribute { }
public partial
class
A // FTO stuff
{
[
Fto ]
public
void CallF1()
{
F1();
}
[Fto]
public
void CallF2()
{
F2();
}
}
// Now we can look for FtoAttribute to either implement a Test Stripper or verify FTO code has been removed
}
namespace
TestAssembly
{
using SutAssembly;
public class
WhenAccessingPrivates
{
[Fact]
public
void CanAccessF1AndF2()
{
var
a = new
A();
a.CallF1();
a.CallF2();
}
}
}
}
namespace
Approach2B
{
namespace
SutAssembly
{
public class
A
{
private
void F1()
{
}
private
void F2()
{
}
class
FtoAttribute :
Attribute { }
[Fto]
public
class A2 :
A
{
[Fto]
public
void CallF1()
{
F1();
}
[Fto]
public
void CallF2()
{
F2();
}
}
// Now we can look for FtoAttribute to either implement a Test Stripper or verify FTO code has been removed
}
}
namespace
TestAssembly
{
using SutAssembly;
public class
WhenAccessingPrivates
{
[Fact]
public
void CanAccessF1AndF2()
{
var
a = new
A.A2();
a.CallF1();
a.CallF2();
}
}
}
}
namespace
Approach3
{
namespace
SutAssembly
{
public class
A
{
private
void F1()
{
}
private
void F2()
{
}
}
}
namespace
PrivateAccessorAssembly
{
using SutAssembly;
// IN ANOTHER ASSEMBLY
// Test References generate DLLs containing code like this via a Shadow MSBuild task that generates an accessor DLL using a tool called Publicise.exe
public class
A_Accessor
{
public
A_Accessor(A
a)
{
}
public
void F1()
{
// uses reflection to call F1
}
public
void F2()
{
// uses reflection to call F1
}
}
}
namespace
TestAssembly
{
using SutAssembly;
using PrivateAccessorAssembly;
public class
WhenAccessingPrivates
{
[Fact]
public
void
CanAccessF1AndF2()
{
var
a = new
A();
var
aAccessor =
new A_Accessor(a);
aAccessor.F1( );
aAccessor.F2( );
}
}
}
}
namespace
Approach4
{
namespace
SutAssembly
{
[
assembly:InternalsVisibleTo(
"TestAssembly" ) ]
public class
A
{
internal
void F1()
{
}
internal
void F2()
{
}
}
}
namespace
TestAssembly
{
using SutAssembly;
public class
WhenAccessingPrivates
{
[Fact]
public
void CanAccessF1AndF2()
{
var
a = new
A();
a.F1();
a.F2();
}
}
}
}
namespace
Approach5
{
namespace
SutAssembly
{
class A
{
public
void F1()
{
}
void
F2()
{
}
// Need to use either an FtoAttribute based Test Stripper, conditional compilation, a Test Stripper based on the presence of FactAttribute to remove this
public
class WhenAccessingPrivates
{
[Fact]
public
void CanAccessF1AndF2()
{
var
a = new
A();
a.F1();
a.F2();
}
}
}
}
}
|