Structured Unit Testing

A Unit Testing Framework

• Unit testing follows a pattern
  • Setup and teardown
  • Lots of small, independent tests
  • Reporting
  • Aggregation (combine tests into a test suite, and test suites into larger suites)
• See a pattern, build a framework
  • Write shared code once
  • Encourage people to work a certain way
    • I.e., make it easy for them to do things right
• JUnit is a testing framework originally written by Kent Beck and Erich Gamma in 1997
  • Widely adopted
    • Now integrated into almost all Java IDEs
    • Made testing easy enough that programmers actually started doing it
  • Widely imitated:
    • Workalikes are available C++, Perl, .NET, etc.
    • Once you know one, you can easily learn and use the others
  • Widely extended
    • Add-ons for measuring test execution times, recording tests, testing web applications, etc.
• Comment on this slide

Mechanics

• Python's standard unit testing framework is the unittest module
• What you do:
  • import unittest
  • Derive a class (or several classes) from unittest.TestCase
  • Define one method for each test
    • Name must begin with "test"
    • Takes no argument (other than self)
    • Returns nothing
  • Call unittest.main()
• What the framework does:
  • Search the program to find classes derived from TestCase
    • See how this works in a later lecture
  • Search each class to find methods whose names begin with the word "test"
  • For each method:
    • Create a new instance of the class
    • Call the method
    • Record pass, fail, or error
• Actually check things inside test methods using methods provided by TestCase
  • Allows the framework to distinguish between test assertions, and normal assert statements
    • Since the code being tested might use the latter
  • Checking methods include:
    • assert_(condition): check that something is true (note the underscore)
    • assertEqual(a, b): check that two things are equal
    • assertNotEqual(a, b): the reverse of the above
    • assertRaises(exception, func, …args…): call func with arguments (if provided), and check that it raises the right exception
    • fail(): signal an unconditional failure
• Comment on this slide

Testing a Function

• You want to test a function that calculates a running sum of the values in the list
  • Replaces [a, b, c, …] with [a, a+b, a+b+c, …]
  • Test cases:
    • Empty list
    • Single value
    • Long list with mix of positive and negative values
  • Hm… Is it supposed to:
    • Return a new list?
    • Modify its argument in place and return that?
    • Modify its argument and return None?
  • Your tests can only ever be as good as (your understanding of) the spec
    • We'll assume it's supposed to return a new list
• First implementation of runningSum:

  • import sys, unittest
    
    def runningSum(seq):
        result = seq[0:1]
        for i in range(2, len(seq)):
            result.append(result[i-1] + seq[i])
        return result
    
    class SumTests(unittest.TestCase):
    
        def testEmpty(self):
            self.assertEqual(runningSum([]), [])
    
        def testSingle(self):
            self.assertEqual(runningSum([3]), [3])
    
        def testDouble(self):
            self.assertEqual(runningSum([2, 9]), [2, 11])
    
        def testLong(self):
            self.assertEqual(runningSum([-3, 0, 3, -2, 5]), [-3, -3, 0, -2, 3])
    
    if __name__ == '__main__':
        unittest.main()
    

    F.E.
    ======================================================================
    ERROR: testLong (__main__.SumTests)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "running_sum_wrong.py", line 21, in testLong
        self.assertEqual(runningSum([-3, 0, 3, -2, 5]), [-3, -3, 0, -2, 3])
      File "running_sum_wrong.py", line 6, in runningSum
        result.append(result[i-1] + seq[i])
    IndexError: list index out of range
    
    ======================================================================
    FAIL: testDouble (__main__.SumTests)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "running_sum_wrong.py", line 18, in testDouble
        self.assertEqual(runningSum([2, 9]), [2, 11])
      File "c:\Python23\lib\unittest.py", line 302, in failUnlessEqual
        raise self.failureException, \
    AssertionError: [2] != [2, 11]
    
    ----------------------------------------------------------------------
    Ran 4 tests in 0.000s
    
    FAILED (failures=1, errors=1)
    

  • Notice that test methods deserve to have meaningful names too
• Can you spot the error?
  • Default output gives the stack trace, to help you find the problem
    • You can also provide messages to the TestCase.assert family of methods
  • Tests are run in an arbitrary order
    • Classes store methods in a dictionary for fast lookup
    • Means that there's no guarantee they'll be run in the order in which you defined them
    • Yet another good reason to make each test independent
• Fix the method and re-run the tests

  • def runningSum(seq):
        result = seq[0:1]
        for i in range(1, len(seq)):
            result.append(result[i-1] + seq[i])
        return result
    

    ....
    ----------------------------------------------------------------------
    Ran 4 tests in 0.000s
    
    OK
    

  • Everything works
• Should you really go to this much effort to test a simple function?
  • Took less than a minute to write the four tests
  • Uncovered one gap in the requirements, and one error in the first implementation
  • Able to verify the fix almost instantly
  • Sounds pretty good to me…
• Comment on this slide

Eliminating Redundancy

• Setting up a fixture can often be more work than writing the test
  • Complex data structures, external files, etc.
• If the test class defines a method called setUp, the framework will call it before running each test
  • And if there's a method called tearDown, it will be run after each test
• Example: test whether a function can trace Make-like dependencies correctly
  • Several kinds of tests to run, and several fixtures to run them on
  • So create all the fixtures in setUp, and use them in the test methods
    • Alternatively, create a separate test class for each fixture
    • Worth doing if creating a fixture takes so long that you don't want any redundant setup

  • class TestDepends(unittest.TestCase):
    
        def setUp(self):
            self.f0 = {}
            self.f1 = {'a' : []}
            self.f2 = {'a' : ['a']}
            self.f3 = {'a' : ['b'], 'b' : []}
            self.f4 = {'a' : ['b', 'c'], 'b' : ['a'], 'c' : ['d'], 'd' : []}
    
        def testCanReachSelf(self):
            self.assert_(dependsOn(self.f1, 'a', 'a'))
            self.assert_(dependsOn(self.f2, 'a', 'a'))
            self.assert_(dependsOn(self.f3, 'b', 'b'))
            self.assert_(dependsOn(self.f4, 'c', 'c'))
    
        def testSingleStep(self):
            self.assert_(dependsOn(self.f3, 'a', 'b'))
            self.assert_(dependsOn(self.f4, 'a', 'b'))
            self.assert_(dependsOn(self.f4, 'c', 'd'))
    
        def testBackward(self):
            self.assert_(dependsOn(self.f4, 'b', 'a'))
    
        def testMultiStep(self):
            self.assert_(dependsOn(self.f4, 'a', 'd'))
            self.assert_(dependsOn(self.f4, 'b', 'd'))
    
        def testNegative(self):
            self.assert_(not dependsOn(self.f1, 'a', 'c'))
            self.assert_(not dependsOn(self.f3, 'b', 'a'))
    

• Comment on this slide

Testing for Failure

• Testing that code fails in the right way is just as important as testing that it does the right thing
  • Otherwise, someone will do something wrong some day, and the code won't report it
• Two ways to do it:
  • Use TestCase.assertRaises to check that a specific function raises a specific exception
  • Use try/except yourself
    • Have to do it this way in C++, Java, and most other sturdy languages
• Example: testing a function that finds all values inside a double-ended range
  • Raises ValueError if the range is empty, or if the set of values is empty

  • class TestInRange(unittest.TestCase):
    
        def testNoValues(self):
            try:
                inRange([], 0.0, 1.0)
            except ValueError:
                pass
            else:
                self.fail()
    
        def testBadRange(self):
            try:
                inRange([0.0], 3.0, 3.0)
                inRange([0.0], 4.0, -2.0)
            except ValueError:
                pass
            else:
                self.fail()
    

• Comment on this slide

Testing I/O

• Input and output often seem hard to test
  • Put a bunch of fixture files and expected results in a testing directory?
  • Create temporary files when and as tests execute?
• The best answer is to use fake I/O using strings
  • Python's StringIO and cStringIO modules can read and write strings instead of files
  • Similar packages exist for C++, Java, and other languages
• This only works if the function being tested takes streams as arguments, rather than filenames
  • If the function opens and closes the file, no way for you to substitute a fake file
  • An example of designing code to make it testable
• Example: find lines where two files differ
  • Nothing as fancy as diff, just a line-by-line comparison
  • Input: two streams (which might be open files or StringIO wrappers around strings)
  • Output: another stream (i.e., a file, or a StringIO)
  • As a side effect, we've made the function itself more useful
    • People can now use it to compare strings, or strings and files

  • class TestDiff(unittest.TestCase):
    
        def wrapAndRun(self, left, right, expected):
            left = StringIO(left)
            right = StringIO(right)
            actual = StringIO()
            diff(left, right, actual)
            self.assertEqual(actual.getvalue(), expected)
    
        def testEmpty(self):
            self.wrapAndRun('', '', '')
    
        def testLengthyMatch(self):
            str = 'a\nb\nc\n'
            self.wrapAndRun(str, str, '')
    
        def testSingleLineMismatch(self):
            self.wrapAndRun('a\n', 'b\n', '1\n')
    
        def testMiddleMismatch(self):
            self.wrapAndRun('a\nb\nc\n', 'a\nx\nc\n', '2\n')
    

• Comment on this slide

Testing With Classes

• Testing classes isn't much different from what we've already seen
  • Create objects, call their methods, check the results
• But how can we keep tests independent?
  • Can't test methods unless we can construct objects…
  • …but the only way to see if objects have been constructed properly is to invoke their methods
• The answer is to keep tests as independent as possible
  • The first tests you write use simple constructors, and simple inspector methods
  • Once those pass, start writing more complex tests
  • They probably won't be executed in this order once they're all written…
  • …but they still give you somewhere to set a breakpoint when simple things start to go wrong
• Much more interesting to think about how to use object-oriented ideas in constructing tests
  • Create one class that has a lot of tests
  • Derive other test classes that:
    • Create specific fixtures
    • Run specific tests
  • By default, the framework will run all the generic tests for the specific classes as well

    • Figure 13.1: Object-Oriented Testing

  • This only works if the tests in the base class are generalized
    • A little more setup work for a big payoff
• Example: billing application for a software consulting firm
  • Each consultant may be assigned to zero or more projects
  • Each project may have zero or more consultant currently assigned to it
  • The Assignment class keep track of who is currently assigned to what
    • Use strings for consultant and project IDs for now
  • Create a base class CommonTests to hold shared tests
    • Do not derive this from unittest.TestCase
    • If we did, the framework would automatically try to run it…
    • …and it's not designed to be run on its own
  • This class does not create a fixture
    • That's the job of the derived class
  • Add a test
    • Each test records the state of the fixture…
    • …does something…
    • …and then checks that the fixture is in the right final state
    • Remember, the fixture is recreated from scratch for each test

  • class CommonTests(object):
        '''Tests that can be applied to all fixtures.'''
    
        def testAddNew(self):
            '''Check that adding a single new assignment works.'''
    
            priorConsultants = self.fixture.getConsultants()
            priorProjects = self.fixture.getProjects()
    
            self.fixture.addAssignment('Alan', 'tables')
    
            self.assertEqual(self.fixture.getByConsultant('Alan'),
                             Set(['tables']))
            self.assertEqual(self.fixture.getByProject('tables'),
                             Set(['Alan']))
            self.assertEqual(self.fixture.getConsultants(),
                             Set(['Alan']) | priorConsultants)
            self.assertEqual(self.fixture.getProjects(),
                             Set(['tables']) | priorProjects)
    

  • Derive another class from both CommonTests and unittest.TestCase
    • The former ensures that it inherits all the common test cases
    • The latter ensures that the framework will find it, and know how to run it
  • This class creates a fixture, and adds a few fixture-specific tests

    • class TestEmpty(unittest.TestCase, CommonTests):
          '''Test an empty assignment table.'''
      
          def setUp(self):
              '''Create the empty assignment table.'''
              self.fixture = Assignment()
      
          def testNoConsultants(self):
              '''Make sure that nonexistent consultants aren't present.'''
              self.assertEqual(self.fixture.getByConsultant('nobody'), Set())
      
          def testNoProjects(self):
              '''Make sure that nonexistent projects aren't present.'''
              self.assertEqual(self.fixture.getByProject('nothing'), Set())
      

  • Run it
    • Framework finds TestEmpty
    • Goes through the test cycle (test, setup, run, teardown) for each testXYZ method in it, and in CommonTests
  • Now add another shared test to CommonTests:

    •     def testAddDelNew(self):
              '''Check that adding and then removing an assignment
              leaves things as they were.'''
      
              priorConsultants = self.fixture.getConsultants()
              priorProjects = self.fixture.getProjects()
      
              self.fixture.addAssignment('Alan', 'tables')
              self.fixture.delAssignment('Alan', 'tables')
      
              self.assertEqual(self.fixture.getConsultants(), priorConsultants)
              self.assertEqual(self.fixture.getProjects(), priorProjects)
      

  • And/or derive another test class:

    • class TestMulti(unittest.TestCase, CommonTests):
          '''Set up common tests with multiple assignments already present.'''
      
          def setUp(self):
              self.fixture = Assignment()
              self.fixture.addAssignment('Bhargan', 'dishes')
              self.fixture.addAssignment('Harald', 'dishes')
              self.fixture.addAssignment('Harald', 'floors')
              self.fixture.addAssignment('Rachel', 'floors')
              self.fixture.addAssignment('Rachel', 'counters')
              self.fixture.addAssignment('Sally', 'counters')
      

  • Multiplicative effect: (common tests × derived classes) + specific tests
• Comment on this slide

Test-Driven Development

• With test-driven development (TDD), you write your tests, then write the code
  • A key element of agile development methodologies like eXtreme Programming…
  • …but it was around long before they were…
  • …and pays off in almost every context
• Seems backward, but has very real advantages:
  • Gives programmers a definite goal
    • Coding is finished when all tests run
    • Particularly useful when trying to fix bugs in old code, as it forces you to figure out how to recreate the bug
    • Helps prevent the “one more feature” syndrome
  • Ensures that tests actually get written
    • People are often too tired, or too rushed, to test after coding
  • Helps clarify the API before it is set in stone
    • If something is awkward to test, it can be redesigned before it's written
  • A great way to communicate requirements
    • I write the tests
    • “All” you have to do is write code that passes those tests
• Example: I want you to write a function that:
  • A single rectangle (represented as an instance of class Rect) as its first argument
  • And a list of rectangles (each also an instance of Rect) as its second argument
  • And returns True if the rectangles in the list completely cover the first rectangle, or False if they don't
  • Best way for me to specify everything I want is to write some test cases

  • class TestOverlay(unittest.TestCase):
    
        def testNoRects(self):
            self.assert_(not overlay(Rect(0, 0, 1, 1), []))
    
        def testWithSelf(self):
            r = Rect(0, 0, 1, 1)
            self.assert_(overlay(r, [r]))
    
        def testHalfOnly(self):
            self.assert_(not overlay(Rect(0, 0, 2, 2),
                                     [Rect(0, 0, 1, 2)]))
    
        def testTwoHalves(self):
            self.assert_(overlay(Rect(0, 0, 2, 2),
                                 [Rect(0, 0, 1, 2), Rect(1, 0, 2, 2)]))
    

• Comment on this slide

Exercises

Exercise 13.1:

Python has another unit testing module called doctest. It searches files for sections of text that look like interactive Python sessions, then re-executes those sections and checks the results. A typical use is shown below.

def ave(values):
    '''Calculate an average value, or 0.0 if 'values' is empty.
    >>> ave([])
    0.0
    >>> ave([3])
    3.0
    >>> ave([15, -1.0])
    7.0
    '''

    sum = 0.0
    for v in values:
        sum += v
    return sum / float(max(1, len(values)))

if __name__ == '__main__':
    import doctest
    doctest.testmod()

Convert a handful of the tests you have written for other questions in this lecture to use doctest. Do you prefer it to unittest? Why or why not? Do you think doctest makes it easier to test small problems? Large ones? Would it be possible to write something similar for C, Java, Fortran, or Mathematica?