MbUnit Combinatorial Tests

MbUnit rocks. Perhaps other unit test tools for .net rock as well, but right now I'm really happy with MbUnit and how easy it is to work with. Plus it has some killer features. I'm really impressed with the combinatorial test. I have no idea whether other xUnit frameworks have this feature, but in MbUnit it's really easy to impliment, and really powerful. Everyone doing unit testing should understand the possibilites of the combinatorial test, to improve the real-world coverage of thier test suites, regardless of the platform and toolsets you're working with.

The Problem: real-world test coverage

A common metric for automated testing is something called 'code coverage', which basically counts the number of lines of code that your tests excercise to tell you how much of your system you have under test. The problem with this (and any metric that relies on LOC metrics) is that it doesn't take into consideration the various VALUES that pass through your code, particularly data entered by users, and environmental factors (like current date/time). While an exceptional value might only cause your code to pass through one additional line of code (perhaps your error handler), there could be many opportunites for your code to experience that exceptional value. Even more troubling, it is often a specific combination of two or more values that cause problems and undesired consequences.

Most of the time, testers and developers will try to deal with this situation by running the same type of test with various inputs. Sometimes this can be done with the combinatorial test's 'little brother' called the RowTest. With a row test, the same test is run multiple times with values that the tester/developer will specify in a number of rows. Kind of like grabbing your inputs from an excel spreadsheet, but keeping your test values right in the test source code. While this is very useful, it still relies on someone to provide all of the combinations. Not only is this time consuming, it is likely that some combinations will be missed. This is where the combinatorial test comes in.

The Solution: Combinatorial Tests

With combinatorial tests, the testing framework determines all of the combinations from various sets of inputs. This isn't a particularly difficult thing for the framework to calculate - it's simply the cross-product of the sets of inputs. This is what computers are good at, so why should us mere humans try to do something our computers are so good at :). MbUnit provides a fairly simple way of supplying those sets of inputs and marking up your tests to use those inputs. MbUnit takes care of the rest. This saves a TREMENDOUS amout of time, and improves your real-world test coverage.

MbUnit supplies a few simple attributes to identify the sets of values and the tests which will consume them. They are: 'Factory', 'CombinatorialTest', and 'UsingFactories'. I won't get into the various flavors or specific features of each of these attributes - you can get that information from MbUnit documentation, or Google.

The "Factory" attribute marks methods which define iEnumerable sets of values. They could be simple native data types, but they can also be sets of complex objects. This can be particularly useful if you want to test sets of concrete types that impliment the same interface, or test subclassed instances of a base type. In the code example below, I'm only using simple string and decimal types, but the pattern is the same.

The "CombinatorialTest" attribute simply tells MbUnit to generate multiple tests from this single method, using the cross-product of the Factories specified.

The "UsingFactories" attribute is applied to parameters of the combinatorial test, indicating that this parameter's values will be supplied by the specified factory method.

The Sample

In the code sample below, I will only show you a very basic implimentation. This example tests a fake 'payment authorization' system, which simplisticly takes credit card types and values to authorize. The sample only enforces one business rule: amounts <= 0 are not authorized. The sample could be extended to cover additional business rules which actually rely on combinations of both sets of inputs (perhaps Amex really does allow $0 authorizations?), but I didn't want to make the sample any more complex; I'll leave that excercise for you. 🙂 I thought this sample would evoke more real-world ideas for you than the very generic 'x,y,z' type samples I've been able to find.

I've also shown a way to categorize your sets of inputs into groups which help organize your tests such that they all actually pass. In this case, I've grouped valid and invalid data seperately, allowing me to have two tests with different assertions. I used VB.net for my sample as most of the samples online I've been able to find are done with C#, so I thought this might add some marginal incremental value to the existing blog posts on this topic.

Imports MbUnit.Framework

<TestFixture()>;

Public Class SampleCombinatorialTests

      <CombinatorialTest()>;
      Public Sub CanAuthorizeValidAmounts(
          <UsingFactories("CCType")> ByVal CCType As String,
          <UsingFactories("ValidAmount")> ByVal amount As Decimal)

           Dim result = Payment.Authorize(CCType, amount)
           Assert.AreEqual("success", result)

       End Sub

       <CombinatorialTest()>
       Public Sub CannotAuthorizeInvalidAmounts(
           <UsingFactories("CCType")> ByVal CCType As String,
           <UsingFactories("InvalidAmount")> ByVal amount As Decimal)

           Dim result = Payment.Authorize(CCType, amount)
           Assert.AreEqual("fail", result)

       End Sub

       <Factory()>
       Public Function CCType() As List(Of String)

           Dim list = New List(Of String)
           list.Add("Visa")
           list.Add("MasterCard")
           list.Add("American Express")
           list.Add("DiscoverCard")
           Return list

       End Function

       <Factory()>
       Public Function ValidAmount() As List(Of Decimal)

           Dim list = New List(Of Decimal)
           list.Add(100.0)
           list.Add(Decimal.MaxValue)
           list.Add(0.01)
           Return list

       End Function

       <Factory()>
       Public Function InvalidAmount() As List(Of Decimal)

           Dim list = New List(Of Decimal)
           list.Add(0.0)
           list.Add(Decimal.MinValue)
           list.Add(-0.01)
           Return list

       End Function
   End Class

   Public Shared Function Authorize(ByVal CCType As String, ByVal amount As Decimal) As String

     Debug.Print(CCType & ", " & amount)
     If amount <= 0 Then Return "fail"
     Return "success"

   End Function
 End Class

The Results

And here's the resulting MbUnit test suite from these few lines of code.

Summary

I hope this post has helped you to see just how simple it is to use MbUnit to harness the power of Combinatorial testing to improve real-world test scenario coverage in your applications.