Cosmic Ray comprises a number of important and potentially confusing concepts. In this section we’ll look at each of these concepts, explaining their role in Cosmic Ray and how they relate to other concepts. We’ll also use this section to establish the terminology that we’ll use throughout the rest of the documentation.
An operator in Cosmic Ray is a class that represents a specific type of mutation. The first role of an operator is to identify points in the code where a specific mutation can be applied. The second role of an operator is to actually perform the mutation when requested.
An example of an operator is
cosmic_ray.operators.break_continue. As its name
implies, this operator mutates code by replacing
the initialization of a session, this operator identifies all of the locations
in the code where this mutation can be applied. Then, during execution of a
session, it actually mutates the code by replacing
break nodes with
Operators are exposed to Cosmic Ray via plugins, and users can choose to extend
the available operator set by providing their own operators. Operators are
implemented as subclasses of
Distributors determine the context in which tests are executed. The primary examples of distributors are
local distributor tests on the local machine, modifying an existing copy of the code in-place, running each test
serially with no concurrency.
The http distributor distributes tests to remote workers via HTTP. There can be any number of workers, and they can run the tests in parallel. Because of this concurrency, each HTTP worker will generally have its own copy of the code under test.
Distributors have broad control over how they execute tests. During the execution phase they are given a sequence of pending mutations to execute, and it’s their job to execute the tests in the appropriate context and return a result. Cosmic Ray doesn’t impose any real constraints on how distributors accomplish this.
Distributors can require arbitrarily complex infrastructure and configuration. For example, the HTTP distributor requires you to start the workers prior to starting execution, and it requires that you provide each worker with its own copy of the code under test.
Distributors are implemented as plugins to Cosmic Ray. They are dynamically discovered, and users can create their own distributors. Cosmic Ray includes two execution engines plugins, local and http.
A configuration is a TOML file that describes the work that Cosmic Ray will do. For example, it tells Cosmic Ray which modules to mutate, how to run tests, which tests to run, and so forth. You need to create a config before doing any real work with Cosmic Ray.
You can create a skeleton config by running
cosmic-ray new-config <config file>. This will ask you a series of
questions and create a config from the answers. Note that this config will generally be incomplete and require you to
edit it for completeness.
In many Cosmic Ray examples we’ll use the name “config.toml” for configurations. You are not required to use this name, however. You can use any file name you want for your configurations.
The full set of configuration options are not currently well documented. Each plugin can, in principle and often in
practice, use their own specialized configuration options. We need to work on making the documentation of these
options automatic and part of the plugin API. For detail on configuration options, the best place to check is
currently in the
Cosmic Ray has a notion of sessions which encompass an entire mutation testing run. Essentially, a session is a database which records the work that needs to be done for a run. Then as results are available from workers that do the actual testing, the database is updated with results. By having a database like this, Cosmic Ray can safely stop in the middle of a (potentially very long) session and be restarted. Since the session knows which work is already completed, it can continue where it left off.
Sessions also allow for arbitrary post-facto analysis and report generation.
Before you can do mutation testing with Cosmic Ray, you need to first initialize a session. You can do this using the
init command. With this command you tell Cosmic Ray a) the name of the session, b) which module(s) you wish to
mutate and c) the location of the test suite. For example, to mutate the package
allele, using the
run the tests in
allele_tests, and using the
local execution engine, you could first need to create a
configuration like this:
[cosmic-ray] module-path = "allele" timeout = 10 excluded-modules =  test-command = python -m unittest allele_tests distributor.name = "local"
You would run
cosmic-ray init like this:
cosmic-ray init config.toml session.sqlite
You’ll notice that this creates a new file called
allele_session.sqlite. This is the database for your session.
To be able to kill the mutants Cosmic Ray uses your test cases. But the mutants are not considered “more dead” when more test cases fail. Given that a single failing test case is sufficient to kill a mutant, it’s a good idea to configure the test runner to exit as soon as a failing test case is found.
nose that can be achieved with the
Once a session has been initialized, you can start executing tests by using the
exec command. This command
needs the config and the session you provided to
cosmic-ray exec config.toml session.sqlite
Normally this won’t produce any output unless there are errors.
Viewing the results¶
Once your tests have completed, you can view the results using the
This will give you detailed information about what work was done, followed by a summary of the entire session.
test-command field of a configuration tells Cosmic Ray how to run tests. Cosmic Ray runs this command from
whatever directory you run the
exec command (or, in the case of remote execution, in whatever directory the remote
command handler is running).
One difficulty mutation testing tools have to face is how to deal with mutations that result in infinite loops (or other pathological runtime effects). Cosmic Ray takes the simple approach of using a timeout to determine when to kill a test and consider it incompetent. That is, if a test of a mutant takes longer than the timeout, the test is killed, and the mutant is marked incompetent.
You specify a test time through the
timeout configuration key. This key specifies an absolute number of seconds that
a test will be allowed to run. After the timeout is up, the test is killed. For example, to specify that tests should
timeout after 10 seconds, use:
# config.toml [cosmic-ray] timeout = 10