Unit Testing Suites

I was recently involved with an upgrade to the unit testing and automation suite that my company uses- I've mentioned it previously.  In developing and implementing the solution to our specific issues, I have had to comb the vast yet not very complete world of Unit Testing blogs, wiki's, and even at times, subject matter experts' personal emails.

One of those SME's, or at least one of the advocates and project leaders, Bruno Kinoshita is in the midst of developing a TAP (Test Anything Protocol) unit testing output display plug-in for Jenkins.  I asked if I could participate in his project to see if we could use the work he was doing for TAP and apply it more broadly to other unit testing suites.  One of the issues we had initially was that each of us use some similar (but not the compatible) version of unit testing, but that there is no single standard for displaying test results or constructing unit test cases.

In familiarizing Bruno to some of the protocols I had come into contact with, I put together a short primer about what I knew.  I recognize that my own knowledge is limited to just the basics (and I may have gotten some of it wrong) and just what I understand about the suites.  Bruno suggested that posting the info to a blog would be helpful to others who are struggling to understand and possibly even chose which suite to use.  Feel free to correct me when I'm wrong, or to add your own understanding/impressions about the suites I mention.

Original message follows:
-----
Here's my understanding of the nomenclature:

First, xUnit, is a generic term used to mean any language's unittest framework (they're all pretty similar in their implementation).  Here's the origin story: http://www.martinfowler.com/bliki/Xunit.html

Some of the unittest packages (for C++, .NET, Java, and Python) include CPPUnit/NUnit/JUnit/PyUnit and they all can produce output that is consistent with, and inspired by Smalltalk's unit testing format, SUnit (http://www.xprogramming.com/testfram.htm).  That's what is good about xUnit output is that no matter what language you use, there's probably a testing package that prints a generic xUnit output- all except JUnit produce output that looks like ...F...E...FF.., followed by lots of details on errors and failures.  JUnit has a formatter that is capable of producing XML unittest output, the kind that Jenkins reads.  (There's one for Perl too: http://search.cpan.org/~mcast/Test-Unit-0.25/lib/Test/Unit.pm)

There are two xUnit plugins we've been talking about- one for Jenkins (https://wiki.jenkins-ci.org/display/JENKINS/xUnit+Plugin) and one for Nose (http://packages.python.org/nose/plugins/xunit.html).  They both are designed to handle XML output, but in different ways- the xUnit plugin for Jenkins is designed to read xUnit style generic output- the ...F...E stuff- and produce XML that is consistent with the XML formatter that JUnit implements.  This is what Jenkins can read.  The xUnit plugin for Nose bypasses the need for the Jenkins plugin and natively produces XML-formatted (JUnit-consistent) test output that Jenkins can read.  Basically, everything has to end up in XML format for Jenkins to be happy.

When I mentioned our desire to have additional output including, possibly, data in the test case output, Kohsuke said he thought that might require a change to Jenkins' core, but that he was willing to do that to add the functionality.

One final note is that SubUnit includes a converter from TAP to Subunit.  It also includes a converter from Subunit to JUnit XML format.  This could be a helpful place to look (I'm sure that it's not a complete implementation of what TAP output can do, so you'd probably have to take the bits you want and then extend it). http://pypi.python.org/pypi/python-subunit/0.0.6


The G-Unit framework is a separate issue and should be handled with care.  http://en.wikipedia.org/wiki/G-unit ;-)
 
-Max