How to benchmark unit tests in Python without adding any code

Question

I have a Python project with a bunch of tests that have already been implemented, and I'd like to begin benchmarking them so I can compare performance of the code, servers, etc over time. Locating the files in a manner similar to Nose was no problem because I have "test" in the names of all my test files anyway. However, I'm running into some trouble in attempting to dynamically execute these tests. As of right now, I'm able to run a script that takes a directory path as an argument and returns a list of filepaths like this: ``` def getTestFiles(directory): fileList = [] print "Searching for 'test' in " + directory if not os.path.isdir(os.path.dirname(directory)): # throw error raise InputError(directory, "Not a valid directory") else: for root, dirs, files in os.walk(directory): #print files for f in files: if "test" in f and f.endswith(".py"): fileList.append(os.path.join(root, f)) return fileList # returns a list like this: # [ 'C:/Users/myName/Desktop/example1_test.py', # 'C:/User…

Accepted Answer

Using `nose` test runner would help to discover the tests, setup/teardown functions and methods.

`nose-timer` plugin would help with benchmarking:

A timer plugin for nosetests that answers the question: how much time does every test take?

Demo:

imagine you have a package named `test_nose` with the following scripts inside:

`test1.py`:

```
import time
import unittest

class TestExampleOne(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        cls.value = 1

def test_one(self):
        time.sleep(1)
        self.assertEqual(1, self.value)
```

`test2.py`:

```
import time

value = None

def setup():
    global value
    value = 1

def test_one():
    time.sleep(2)
    assert value == 1
```

`test3.py`:

```
import time
import unittest

value = None

def setup():
    global value
    value = 1

class TestExampleTwo(unittest.TestCase):
    def test_one(self):
        time.sleep(3)
        self.assertEqual(1, value)

class TestExampleThree(unittest.TestCase):
    def test_one(self):
        time.sleep(4)
        self.assertEqual(1, value)
```

install `nose` test runner:

```
pip install nose
```

install `nose-timer` plugin:

```
pip install nose-timer
```

run the tests:

```
$ nosetests test_nose --with-timer
....
test_nose.test3.TestExampleThree.test_one: 4.0003s
test_nose.test3.TestExampleTwo.test_one: 3.0010s
test_nose.test2.test_one: 2.0011s
test_nose.test1.TestExampleOne.test_one: 1.0005s
----------------------------------------------------------------------
Ran 4 tests in 10.006s

OK
```

The result is actually conveniently highlighted:

The coloring can be controlled by `--timer-ok` and `--timer-warning` arguments.

Note that `time.sleep(n)` calls were added for making the manual slowdowns to see the impact clearly. Also note that `value` variable is set to `1` in "setup" functions and methods, then in test function and methods the `value` is asserted to be `1` - this way you can see the work of setup functions.

UPD (running `nose` with `nose-timer` from script):

```
from pprint import pprint
import nose
from nosetimer import plugin

plugin = plugin.TimerPlugin()
plugin.enabled = True
plugin.timer_ok = 1000
plugin.timer_warning = 2000
plugin.timer_no_color = False

nose.run(plugins=[plugin])
result = plugin._timed_tests
pprint(result)
```

Save it into the `test.py` script and pass a target directory to it:

```
python test.py /home/example/dir/tests --with-timer
```

The `result` variable would contain:

```
{'test_nose.test1.TestExampleOne.test_one': 1.0009748935699463,
 'test_nose.test2.test_one': 2.0003929138183594,
 'test_nose.test3.TestExampleThree.test_one': 4.000233173370361,
 'test_nose.test3.TestExampleTwo.test_one': 3.001115083694458}
```

How to benchmark unit tests in Python without adding any code

Solution

Problem