testresources

Have been working on an implementation of testresources , written by Robert Collins, for some time now, gotten it up and running, and now have good output in Jenkins...therefore, it seems about time to throw all that away in favor of a different implementation.

Although testresources appears much discussed in Python testing circles (see here, here and here), the documentation that exists on implementing it is...fairly sparse, to put it politely.  So over the last couple of days, I've waded hip deep into testresources package code.  I've found that the code itself is elegant and smart (I haven't tested the logic behind the various tools used in the implementation- digraph to graph, minimal spanning tree via Kruskals algorithm, etc.- but I do see that Mr. Collins put a great deal of effort into making it smart and powerful).

In summary, using the testresources package is a great way to allow access to expensive resources (in our case, a Selenium instance that fires up Firefox, imports a profile, and authenticates a user into our web interface) over multiple unit test cases in a suite that all require access to those resources.  The package even optimizes- <i>spelling it optimise to emfasise the British origin of the code, I guess</i>- the use of those resources by re-ordering test cases that call for the same resource to run all together so that, in theory, the resources persists only as long as there is a test case that is waiting to use it, then it is torn down via its native clean and tearDown methods (I say, "in theory," because if a test case changes the state of the resource and marks it <i>dirtied</i> the resource is destroyed and re-created- the order isn't smart enough to check and put any test cases that dirty the resource at the end of the queue so that the ones that can run happily one after the other don't get bogged down.)  To summarize, it does allow you to do some things that PyUnit doesn't allow, and it implements them very well.

Unfortunately, it also limits how you can structure your test cases (everything needs to be flat when you optimize it- one test suite consisting of all the test cases).  That means, you can't have nested suites (suites of suites), or sibling suites that contain cases that share resources.
This would work and be optimized with testresources:

<test suite>
     <test case 1>      - uses resource A
     <test case 2>
     <test case 3>      - uses resource A
     <test case 4>      - uses resource A
</test suite>

This doesn't:

<test suites>
     <test suite 1>
          <test case 1>      - uses resource A
          <test case 2>
          <test case 3>      - uses resource A
     </test suite 1>
     <test suite 2>
          <test case 4>      - uses resource A
     </test suite 2>

</test suites> 

So resource A would be created twice (at least), even though  it's essentially the same as the first structure (except that it uses a suite of test suites).

The next step, I think, is to try an implementation using the nose testing package.  More to follow.

Unit Testing

I'm currently working on a project to reorganize the output of our automated Selenium QA test cases to something that has a finer resolution (each functional test case should have its own output, and that output should be helpful/understandable).

There are a few challenges to this.  The continuous integration package that we currently use is Jenkins (until very recently known as Hudson).  Jenkins can read in XML output from test results, but we don't have any idea what standards Jenkins is looking for- can it handle nests of test cases (called a test suite)?  If it can handle test suite structures in XML, do we have any output generating test runners that can deliver that output?

Unfortunately, as is so often the case with open source code, the documentation to go along with these packages is spare to nonexistent (to be fair, Jenkins has a pretty good Wiki that is just missing coverage on the pieces that I care about now).  I may have to resort to emailing/posting a message to the developer Kohsuke (who is really responsive, it seems) to figure out what Jenkins is looking for.

We've been looking at zope.test and zope.testrunner as possible candidates to speed up execution of our test cases (they allow an environment layer to be set and the unit test cases to run within that environment).  Currently, we use python's integrated unittest package to implement test cases and suites and because of the way unittest works, each test case is run on the same level (so suites of suites of test cases all just become a list of test cases that run in the order they were assembled- so Jenkins displays their results that way- all on the same level).  This is non-optimal because we want to assemble suites to organize sets of test cases based on testing a specific function or page completely, and view the output in the corresponding organizational fashion.

Because of the dependency on the intermediate XML output, we've been playing with subunit2pyunit within subunit, and before that collective.xmltestreport as a possible translator- we switched to looking at subunit because of that message.

In addition, I just found out how to use Setup Tools' easy_install which is pretty fun on installs, but absolutely a pain to uninstall- might as well call it difficult_uninstall as a warning to potential future users.  (The advantage of easy_install for Python developers is that it sets PYTHONPATH references to packages, downloads the appropriate version much like apt-get on Linux, and does a lot of the dirty work of integrating it all into one location- helpful PYTHONPATH link.)  Also, just a note zopepackage is not the same as zope.testrunner (in fact, the zope package doesn't include testrunner anymore just test.)

Madison Marathon

Yesterday (Monday) was a pretty packed day.  Not only was it my first day at my new job, but my nephew Buddie Powell had a band concert, and I was supposed to be running 15 miles as part of my marathon training.

I tried my best to make all of that happen, but Buddie's concert and my new job were first on my list of priorities, so I was sure to make it to those.  The training didn't go very well- I had a nagging injury that was bothering me enough that I was very aware of it.  A tendon on the top of my left foot seemed to be irritated by the tongue of my shoe, and although I was able to run about 10 miles, that was my limit and I decided that I probably need more rest to help that injury heal.

The concert was really good for a high school band.  The freshmen band was not so great, but luckily, they only played a couple of songs and then the upper classmen took over.

Today I got the first portion of an overview of the InformaCast suite, a very neat and functional piece of software.  Another neat thing that happened today was that I got a call from Ken Bywaters, my company's Executive VP of Product Development.  We discussed the campus and culture of Texas A&M at some length, he introduced himself and welcomed me to Singlewire.  It was really a nice gesture- one of many ways my new co-workers have made me feel welcome.

New Job

As the title of this blog suggests, I've done a fair amount of computer programming in my day. I recently moved back to my hometown, Madison, WI, and was hired by a local software company to work on their flagship software package, InformaCast.

The company I work for is Singlewire Software. Their office is located on the West Beltline across from American TV near the Todd Drive exit. (For my seedier group of friends- it's near Selective Video on the Beltline).

View Larger Map