Commands

TODO: This is pretty wildly out of date! Perhaps we can use value-add to do this.

Details of Common Commands

Most Cosmic Ray commands use a verb-options pattern, similar to how git does things.

Possible verbs are:

  • exec
  • help
  • init
  • load
  • new-config
  • operators
  • dump
  • run
  • worker
  • apply
  • baseline

Detailed information on each command can be found by running cosmic-ray help <command> in the terminal.

Cosmic Ray also installs a few other separate commands for producing various kinds of reports. These commands are:

  • cr-report: provides a report on the status of a session
  • cr-rate: prints the survival rate of a session
  • cr-html: prints an HTML report on a session

Verbosity: Getting more Feedback when Running

The base command, cosmic-ray, has a single option: --verbose. The --verbose option changes the internal logging level from WARN to INFO and thus prints more information to the terminal.

When used with init, --verbose will list how long it took to create the mutation list and will also list which modules were found:

(.venv-pyerf) ~/PyErf$ cosmic-ray --verbose init --baseline=2 test_session pyerf -- pyerf/tests
INFO:root:timeout = 0.259958 seconds
INFO:root:Modules discovered: ['pyerf.tests', 'pyerf.tests.test_pyerf', 'pyerf.pyerf', 'pyerf', 'pyerf.__about__']
(.venv-pyerf) C:\dev\PyErf>cosmic-ray --verbose init --baseline=2 test_session pyerf --exclude-modules=.*tests.* -- pyerf/tests
INFO:root:timeout = 0.239948 seconds
INFO:root:Modules discovered: ['pyerf.pyerf', 'pyerf', 'pyerf.__about__']

When used with exec, --verbose displays which mutation is currently being tested:

(.venv-pyerf) ~/PyErf$ cosmic-ray --verbose exec test_session
INFO:cosmic_ray.tasks.worker:executing: ['cosmic-ray', 'worker', 'pyerf.pyerf', 'number_replacer', '0', 'unittest', '--', 'pyerf/tests']
INFO:cosmic_ray.tasks.worker:executing: ['cosmic-ray', 'worker', 'pyerf.pyerf', 'number_replacer', '1', 'unittest', '--', 'pyerf/tests']
INFO:cosmic_ray.tasks.worker:executing: ['cosmic-ray', 'worker', 'pyerf.pyerf', 'number_replacer', '2', 'unittest', '--', 'pyerf/tests']
INFO:cosmic_ray.tasks.worker:executing: ['cosmic-ray', 'worker', 'pyerf.pyerf', 'number_replacer', '3', 'unittest', '--', 'pyerf/tests']
INFO:cosmic_ray.tasks.worker:executing: ['cosmic-ray', 'worker', 'pyerf.pyerf', 'number_replacer', '4', 'unittest', '--', 'pyerf/tests']
INFO:cosmic_ray.tasks.worker:executing: ['cosmic-ray', 'worker', 'pyerf.pyerf', 'number_replacer', '5', 'unittest', '--', 'pyerf/tests']
INFO:cosmic_ray.tasks.worker:executing: ['cosmic-ray', 'worker', 'pyerf.pyerf', 'number_replacer', '6', 'unittest', '--', 'pyerf/tests']

The --verbose option does not add any additional information to the dump verb.

Command: init

The init verb creates a list of mutations to apply to the source code. It has the following optional arguments:

  • --no-local-import: Allow importing module from the current directory.

The init verb use following entries from the configuration file:

  • [cosmic-ray] excluded-modules = []: Exclude modules matching those glob patterns from mutation. Use glob.glob syntax.

    Sample for django projects:

    excluded-modules = ["*/tests/*", "*/migrations/*"]

As mentioned in here, test directory can be handled via the excluded-modules option.

The list of files that will be mutate effectively can be show by running cosmic-ray init with INFO debug level:

cosmic-ray init -v INFO

Command: exec

The exec command is what actually runs the mutation testing.

Command: dump

The dump command writes a detailed JSON representation of a session to stdout.

$ cosmic-ray dump test_session
{"data": ["<TestReport 'test_project/tests/test_adam.py::Tests::test_bool_if' when='call' outcome='failed'>"], "test_outcome": "killed", "worker_outcome": "normal", "diff": ["--- mutation diff ---", "--- a/Users/sixtynorth/projects/sixty-north/cosmic-ray/test_project/adam.py", "+++ b/Users/sixtynorth/projects/sixty-north/cosmic-ray/test_project/adam.py", "@@ -20,7 +20,7 @@", "     return (not object())", " ", " def bool_if():", "-    if object():", "+    if (not object()):", "         return True", "     raise Exception('bool_if() failed')", " "], "module": "adam", "operator": "cosmic_ray.operators.boolean_replacer.AddNot", "occurrence": 0, "line_number": 32, "command_line": ["cosmic-ray", "worker", "adam", "add_not", "0", "pytest", "--", "-x", "tests"], "job_id": "c2bb71e6203d44f6af42a7ee35cb5df9"}
. . .

dump is designed to allow users to develop their own reports. To do this, you need a program which reads a series of JSON structures from stdin.

Concurrency

Note that most Cosmic Ray commands can be safely executed while exec is running. One exception is init since that will rewrite the work manifest.

For example, you can run cr-report on a session while that session is being executed. This will tell you what progress has been made.