Tutorial: The basics

This tutorial will walk you through the steps needed to install, configure, and run Cosmic Ray.

Installation

First you’ll need to install Cosmic Ray. The simplest (and generally best) way to do this is with pip:

pip install cosmic-ray

You’ll generally want to do this in a virtual environment, but it’s not required.

Source module and tests

Mutation testing works by making small mutations to the code under test (CUT) and then running a test suite over the mutated code. For this tutorial, then, we’ll need to create our CUT and a test suite for it.

You should create a new directory which will contain the CUT, the tests, and eventually the Cosmic Ray data. For the rest of this tutorial we’ll refer to this new directory as ROOT (or $ROOT if we’re showing shell code).

Now create the file ROOT/mod.py with these contents:

def func():
    return 1234

This file contains your code under test, i.e. the code that Cosmic Ray will mutate. It’s clearly very simple, and it has very few opportunities for mutation, but it’s sufficient for this tutorial. In fact, having simple code like this will make it easier to see what Cosmic Ray is doing without getting bogged down by scale.

Next create the file ROOT/test_mod.py with these contents:

import unittest
import mod


class Tests(unittest.TestCase):
    def test_func(self):
        self.assertEqual(mod.func(), 1234)

This contains the test suite for mod.py. Cosmic Ray will not mutate this code. Rather, it will run this test suite for every mutation that it creates.

Before moving on, let’s make sure that the test suite works correctly:

python -m unittest test_mod.py

This should show that all tests pass:

.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

If you see one test passing like this, then you’re ready to continue!

Creating a configuration

Before you do run any mutation tests, you need to create a configuration. A configuration is a TOML file that specifies the modules you want to mutate, the test scripts to use, and so forth. A configuration is used to create a session, something we’ll look at in the next section.

The new-config command

You can create a configuration by hand if you want. In fact, you’ll generally need to edit them by hand to get the exact configuration you need. But you can create an initial configuration using the new-config command. This will ask you a series of questions and construct a new configuration based on your answers.

To create your config for this tutorial, do this:

cd $ROOT
cosmic-ray new-config tutorial.toml

This will ask you a series of questions. Anwer them like this:

[?] Top-level module path: mod.py
[?] Python version (blank for auto detection):
[?] Test execution timeout (seconds): 10
[?] Test command: python -m unittest test_mod.py
-- MENU: Distributor --
  (0) http
  (1) local
[?] Enter menu selection: 1

This will create the file tutorial.toml with these contents:

[cosmic-ray]
module-path = "mod.py"
timeout = 10.0
excluded-modules = []
test-command = "python -m unittest test_mod.py"

[cosmic-ray.distributor]
name = "local"

Configuration contents

Let’s examine the contents of this file before moving on. On line 1 we define the ‘cosmic-ray’ key in the TOML structure; this key will contain all Cosmic Ray configuration information.

On line 2 we set the ‘module-path’ key to the string “mod.py”:

module-path = "mod.py"

This tells Cosmic Ray that we’re going to be mutating the module in the file mod.py. Every Cosmic Ray configuration refers to a single top-level module that will be mutated, and in this case we’re telling Cosmic Ray to mutate the mod module, contained in the file mod.py.

Note

The ‘module-path’ is a path to a file or directory, not the name of the module of package. If it’s a file then Cosmic Ray will treat it as a single module, but if it’s a directory then Cosmic Ray will treat it as a package.

When working on a package, Cosmic Ray will apply mutations to all submodules in the package.

Line 3 tells Cosmic Ray the maximium amount of time to let a test run before it’s considered a failure:

timeout = 10.0

In this case, we’re telling Cosmic Ray to kill a test if it runs longer than 10 seconds. This timeout is important because some mutations can cause the tests to go into an infinite loop. Without timeout we’d never exit the test! It’s important to set this timeout such that it’s long enough for all legitimate tests.

Next, line 4 tells Cosmic Ray which modules to exclude from mutation:

excluded-modules = []

In this case we’re not excluding any, but there may be times when you need to skip certain modules, e.g. because you know that you don’t have sufficient tests for them at the moment.

Line 5 is one of the most critical lines in the configuration. This tells Cosmic Ray how to run your test suite:

test-command = "python -m unittest test_mod.py"

In this case, our test suite uses the standard unittest testing framework, and the tests are in the file test_mod.py.

The last two lines tell Cosmic Ray which “distributor” to use:

[cosmic-ray.distributor]
name = "local"

A distributor controls how mutation jobs are assigned to one or more workers so that they can (potentially) run in parallel. In this case we’re using the default ‘local’ distributor which only runs one mutation at a time. There are other, more sophisticated distributors which we discuss elsewhere.

Create a session and baseline

Cosmic Ray uses a notion of sessions to encompass a full mutation testing suite. Since mutation testing runs can take a long time, and since you might need to stop and start them, sessions store data about the progress of a run.

Note

Most Cosmic Ray commands allow you to increase their “verbosity” via the command line. This will make them print out more information about what they’re doing.

Try adding “–verbosity INFO” to the command you run if you more details about what’s going on!

Initializing a session

The first step in a full testing run, then, is to initialize a session:

cosmic-ray init tutorial.toml tutorial.sqlite

Note

This command prepares all the mutations that will later be applied to code. As such, its execution time is proportional to the amount of code and the code complexitly. You can expect about 15-30s per 1kloc.

This will create a database file called tutorial.sqlite. There is a record in the database for each mutation that Cosmic Ray will perform, and Cosmic Ray will associate testing results with these records as it executes.

Baselining

Before running the full mutation suite, it’s important to make sure that the test suite passes in the absence of any mutations. If the test suite does not pass in the absence of mutations, then the results of the mutation testing are essentially useless.

You can use the baseline command to check that the test suite passes on unmutated code:

cosmic-ray baseline --report tutorial.toml tutorial.sqlite

This should report that the tests pass:

Execution with no mutation works fine.

You’ll also see that there is a new tutorial.baseline.sqlite database containing the results of the baselining.

Tip

Only one baseline can be stored in the baseline database. If the execution failed and you fixed the environment without changing the source code, you can re-baseline it with --force option without the need to run init again.

If this command succeeds, then you’re ready to start mutating code and testing it!

Examining the session with cr-report

Our session file, tutorial.sqlite, is essentially a list of mutations that Cosmic Ray will perform on the code under test. We haven’t actually tested any mutants, so none of our mutations have testing results yet. With that in mind, let’s examine the contents of our session with the cr-report program:

cr-report tutorial.sqlite --show-pending

This will produce output like this (though note that the test IDs will be different):

574ac31ac7d14169a8dc45d988803e69 mod.py core/NumberReplacer 1
8f0e988866f447d085ce9887e6e900e5 mod.py core/NumberReplacer 0
total jobs: 2
no jobs completed

This is telling us that Cosmic Ray detected two mutations that it can make to our code, both using the mutation operator “core/NumberReplacer”. Without going into details, this means that Cosmic Ray has found one or more numeric literals in our code, and it plans to make two mutations to those numbers. We can see in our code that there is only one numeric literal, the value returned from func() on line 2:

def func():
    return 1234

So Cosmic Ray is going to mutate that number in two ways, running the test suite each time.

The cr-report tool is useful for examining sessions, and it’s main purpose is to give you summary reports after an entire session has been executed, which we’ll do in the next step.

Execution

Now that you’ve initialized and baselined your session, it’s time to start making mutants and testing them. We do this with the exec command. exec looks in your session file, tutorial.sqlite, for any mutations which were detected in the init phase that don’t yet have results. For each of these, it performs the specified mutation and runs the test suite.

As we saw, we only have two mutations to make, and our test suite is very small. As a result the exec command will run quite quickly:

cosmic-ray exec tutorial.toml tutorial.sqlite

This should produce no output.

Note

The module and test suite for this tutorial are “toys” by design. As such, they run very quickly. Most real-world modules and test suites are much more substantial and require much longer to run. For example, if a test suite takes 10 seconds to run and Cosmic Ray finds 1000 mutations, a full exec will take 10 x 1000 = 10,000 seconds, or about 2.7 hours.

Reporting the results

Assuming it ran correctly, we can now use cr-report to see the updated state of our session:

cr-report tutorial.sqlite --show-pending

This time we see that both mutations were made, tests were run for each, and both were “killed”:

8f0e988866f447d085ce9887e6e900e5 mod.py core/NumberReplacer 0
worker outcome: normal, test outcome: killed
574ac31ac7d14169a8dc45d988803e69 mod.py core/NumberReplacer 1
worker outcome: normal, test outcome: killed
total jobs: 2
complete: 2 (100.00%)
surviving mutants: 0 (0.00%)

Tip

You don’t have to wait for exec to complete to generate a report. If you have a long-running session and want to see your progress, you can execute cr-report while cosmic-ray exec is running to view the progress the latter is making.

HTML reports

You can also generate a handy HTML report with cr-html:

cr-html tutorial.sqlite > report.html

You can then open report.html in your browser to see the details. One nice feature of these HTML reports is that they show the actual mutation that was used.