The Major Mutation Framework

Easy and scalable mutation analysis for Java!

(version 2.0.0)


Download latest version

Mutator Plugin

Major’s mutator is a javac compiler plugin that traverses and translates the abstract syntax tree (AST). All mutants are embedded into the AST and compiled to byte-code. Individual mutants can be enabled at runtime without recompilation.

Configuration

The mutator plugin supports the following configuration options:

Logging levels: all, finest, finer, fine, warning, severe, and none

Logging formatters: xml, color, verbose-color, oneline, verbose, simple

Mml files define which mutation operators to enabled and what program elements to mutate. This allows for a fine-grained definition and a flexible application of mutation operators. Example mml files are provided in the release. Note that Major’s mutator interprets pre-compiled mml files. Use the mml compiler mmlc to validate and compile a mml file.

Mutation Operators

Major supports the following mutation operator groups, each of which groups multiple related mutation operators:

AOR: Arithmetic Operator Replacement

COR: Conditional Operator Replacement

LOR: Logical Operator Replacement

ROR: Relational Operator Replacement

SOR: Shift Operator Replacement

ORU: Operator Replacement Unary

LVR: Literal Value Replacement

EVR: Expression Value Replacement

STD: STatement Deletion

Logging of Generated Mutants

Major’s mutator generates a log file mutants.log, which provides detailed information about the generated mutants and uses a colon (‘:’) as separator. The log file contains one row per generated mutant; each row contains 7 columns with the following information:

  1. Mutants’ unique number (id)

  2. Name of the applied mutation operator

  3. Original operator symbol

  4. Replacement operator symbol

  5. Fully qualified name of the mutated method

  6. Line number in original source file

  7. Summary of the applied transformation (<from> |==> <to>)

Here is a log-entry example for an ROR mutation that has the mutant id 11 and was generated on line 18 in a method named classify of the class Triangle:

11:ROR:<=(int,int):<(int,int):Triangle@classify:18:a <= 0 |==> a < 0

Exporting Source Code Mutants

Mutants can be exported as individual source-code files. This feature is disabled by default; to enable it, set the export.mutants option. If enabled, Major duplicates the original source file for each mutant, injects the mutant into the copy, and exports the resulting faulty copy (to ./mutants by default). Use the mutants.directory:<DIR> option to control the output directory. Major automatically creates the export directory and parent directories if necessary.

Note that mutating a large code base and exporting all mutants to individual source files increases the mutation/compilation time and requires significantly more disk space than the log file.

Build System Integration

Major’s compiler plugin can be used standalone or in a build system.

Gradle

TODO

Apache Maven

TODO

Apache Ant

Consider the following compile target in an Apache Ant build.xml file:

<target name="compile" depends="init" description="Compile">       
        <javac srcdir="src" destdir="bin" debug="yes">
        </javac>                                                       
    </target>

Add the following options to enable Major’s mutator plugin:

<property name="major.jar" value="<path to major.jar>"/>
    <property name="mml" value="<path to compiled mml file>"/>

    <target name="compile" depends="init" description="Compile">
        <javac srcdir="src" destdir="bin" debug="yes">
          <classpath location="${major.jar}"/>
          <compilerarg value="-Xplugin:MajorPlugin mml:${mml}"/>
        </javac>
    </target>

Run-time Configuration

Mutated class files reference Major’s run-time configuration class to access mutant identifiers (_M_NO) and/or monitor mutation coverage (COVERED). The archive and source files of the default implementation are provided in the config directory. It can be extended to perform additional analyses. Note that the configuration class does not need to be available on the classpath during mutant generation (the mutator does not try to resolve this class at compile-time, but rather creates AST nodes for mutant identifiers and coverage invocations based on the interface defined by this class.

Major Mutation Language (MML)

Major’s domain-specific language mml supports fine-grained configuration of the mutant generation process. A mml file contains a sequence of statements, where each statement represents one of:

  1. Variable definition, e.g., for scopes or replacements.

  2. Replacement definition for mutation operators.

  3. Definition of statement types for the STD mutation operator.

  4. Definition of literal types for the LVR mutation operator.

  5. Invocation of a pre-defined or custom mutation operator (group).

  6. Definition of a custom mutation operator.

  7. Line comment.

The first five statement types are terminated by a semicolon, an operator group definition is encapsulated by curly braces, and a line comment is terminated by the end-of-line.

Statement scopes

Mml provides statement scopes for replacement definitions and operator invocations to support the mutation of a certain package, class, or method within a program. The following diagram shows the definition of a statement scope:

Scope corresponds to a package, class, or method. A scope is defined as a fully qualified name – referred to as flatname. A flatname can be either provided as a quoted String or as a variable. Note that a statement scope is optional. If no statement scope is provided, the corresponding definition or invocation is applied to the root package.

The following diagram shows the syntax rules for a flatname:

The naming conventions for valid identifiers (IDENT) follow those of the Java programming language, given that a flatname identifies a program element. The following four examples show valid flatnames for a package, a class, a set of overloaded methods, and a particular method:

The flatname syntax also supports the identification of innerclasses and constructors, consistent with the naming conventions of Java. For Example, the subsequent definitions address an inner class, a constructor, and a static class initializer:

Overriding and Extending Definitions

For a given scope, mutation operators can be enabled (+, which is the default if the flag is omitted, or disabled (-). In the following example, the AOR mutation operator is generally enabled for the package org but specifically disabled for the class Foo within that package:

    +AOR<"org">;
        -AOR<"org.Foo">;

Note that the flag (prefix) for enabling/disabling operators is optional. The default flag (+) for enabling operators improves readability but can be omitted. That is, +AOR<"org">; and AOR<"org">; are equivalent statements.

For replacement definitions, there are two possibilities: Individual replacements can be added (+) to an existing list or the entire replacement list can be overridden (!) – the latter is the default if this optional flag is omitted. In the following example, the general definition of replacements for the package org is extended for the class Foo but overriden for the class Bar. (The replacement lists that are effectively applied to the package and classes are given in comments.)

     BIN(*)<"org">     -> {+,/};    // * -> {+,/}
        +BIN(*)<"org.Foo"> -> {%};      // * -> {+,/,%}
        !BIN(*)<"org.Bar"> -> {-};      // * -> {-}

Custom Mutation Operator Groups

Mml allows the definition of custom operator groups to minimize code duplication (e.g., identical definitions for multiple scopes). A custom operator group may contain any statement that is valid in an mml file, except for a call of another custom operator group. A custom operator group has a unique identifier and its statements are enclosed by curly braces:

    myOp {
            BIN(*) -> {+,/};
            AOR;
        }

Example MML Files

The following example mml file performs three tasks:

  1. Define specific replacements for AOR and ROR.

  2. Invoke AOR and ROR, using the defined replacements.

  3. Invoke the LVR operator without restrictions.

// Define own replacement list for AOR
    BIN(*) -> {/,%};
    BIN(/) -> {*,%};
    BIN(%) -> {*,/};

    // Define own replacement list for ROR
    BIN(>)  -> {<=,!=,==};
    BIN(==) -> {<,!=,>};       

    // Define types of literals that should be mutated by the LVR operator.
    // Literal type is one of {BOOLEAN, NUMBER, STRING}.
    LIT(NUMBER);
    LIT(BOOLEAN);

    // Enable and invoke mutation operators
    AOR;
    ROR;
    LVR;

The next example uses the scoping feature in line 8 and 13-20, and defines a variable in line 11 to avoid code duplication in the subsequent scope declarations. Both features are useful if only a certain package, class, or method should be mutated.

// Definitions for the root node                                                           
    BIN(>=)->{TRUE,>,==};                                                                      
    BIN(<=)->{TRUE,<,==};                                                                      
    BIN(!=)->{TRUE,<,> };                                                                      
    LIT(NUMBER);
    LVR;                                                                                       
                                                                                               
    // Definition for the package org                                                          
    ROR<"org">;                                                                                
                                                                                               
    // Variable definition for the class Foo                                                           
    foo="org.x.y.z.Foo"; 

    // Scoping for replacement lists                                                                     
    BIN(&&)<foo>->{LHS,RHS,==,FALSE};                                                          
    BIN(||)<foo>->{LHS,RHS,!=,TRUE };                                                          

    // Scoping for mutation operators
    -LVR<foo>;                                                                                 
    ROR<foo>;                                                                                       
    COR<foo>;

The last example demonstrates the custom operator feature, which is useful if the same group of operations (definitions or invocations) should be applied to multiple scopes.

myOp{                                                                                      
        // Definitions for the operator group                                                  
        BIN(>=)->{TRUE,>,==};                                                                  
        BIN(<=)->{TRUE,<,==};                                                                  
        BIN(!=)->{TRUE,<,> };                                                                  
        BIN(&&)->{LHS,RHS,==,FALSE};                                                           
        BIN(||)->{LHS,RHS,!=,TRUE };                                                           
        // Mutation operators enabled in this group                                                                                       
        ROR;                                                                                   
        COR;                                                                                   
    }                                                                                          
                                                                                               
    // Calls of the defined operator group                                            
    myOp<"org">;                                                                               
    myOp<"de">;                                                                                
    myOp<"com">;

Default Analyzer

Major provides a default analyzer, which extends the Apache Ant junit task. This analyzer supports JUnit 3 and 4 tests. % Note that this analyzer does currently not support forking a JVM when executing JUnit tests, meaning that the fork option must be set to false.

Configuration Options

This table summarizes the configuration options for Major’s default analyzer. (Please refer to the official documentation of the junit task for all other options.)

Option Description Values Default
mutationAnalysis Enable mutation analysis. If set to false, all options are ignored. [true|false] false
analysisType Run preprocessing and mutation analysis or only one of the two.   [preproc_mutation|preproc|mutation] preproc_mutation
serializedMapsFile File name for serialized preprocessing results. <String> preprocessing.ser
mutantsLogFile The path for the mutants.log file produced my Major’s mutator. <String> mutants.log
testOrder Order and granularity of tests. [original|random|sort_classes|sort_methods] original
debug Enable debugging output. [true|false] false
timeoutFactor Multiplier for the (original) test runtime to compute the test timeout. <int> 8
timeoutOffset Offset in ms added to the computed test timeout. <int> 0
excludeFailingTests Exclude all failing tests (haltonfailure must be false). [true|false] true
excludeMutantsFile Exclude all mutants listed in this file (1 mutant id per row).  <String> null
includeMutantsFile Include only mutants listed in this file (1 mutant id per row). <String> null
excludeTestsFile Exclude all tests listed in this file (1 test id per row). <String> null
includeTestsFile Include only tests listed in this file (1 test id per row). <String> null
summaryFile Export summary of results to this file (csv). <String> summary.csv
executionDetailsFile Export detailed execution information to this file (csv). <String> null
mutantDetailsFile Export classification details for each mutant to this file (csv). <String> null
covMapFile File name for mutant-coverage matrix (csv). <String> covMap.csv
testMapFile File name for mapping of test id to test name (csv). <String> testMap.csv
exportKillMap Export mutant-kill matrix (executes every test on every covered mutant!). [true|false] false
killMapFile File name for mutant-kill matrix (csv). <String> killMap.csv

analysisType:

Allows running both steps of the analysis (preprocessing and mutation), which is the default, or only one of the two. For large code bases for which the mutation analysis itself should be parallelized, running preprocessing just once and caching the result (serializedMapsFile) will greatly speed up analysis time. If set to mutation, the serialized maps must be provided.

testOrder:

Setting Up an Analysis Target

Most Apache Ant build files provide a test target, which executes the corresponding unit tests. Even if no such target exists, it can be easily set up to execute a set of given unit tests. The following code snippet shows an example test target (See junit.html for a detailed description of this task):

<target name="test" description="Run all unit test cases">         
        <junit printsummary="true"                                    
                 showoutput="true"                                      
              haltonfailure="true">                              
            
            <formatter type="plain" usefile="true"/>                  
            <classpath path="bin"/>                                                                                                       
            <batchtest fork="no">                                      
                <fileset dir="test">                                   
                    <include name="**/*Test*.java"/>                   
                </fileset>                                             
            </batchtest>                                               
        </junit>                                                       
    </target>

To enable mutation analysis, set the option mutationAnalysis to true. For clarity, it is best to duplicate and then adapt an existing test target.

<target name="mutation.test" description="Run mutation analysis">         
        <junit printsummary="false"                                    
                 showoutput="false"                                      
              haltonfailure="true"

           mutationAnalysis="true"
                summaryFile="summary.csv" 
          mutantDetailsFile="details.csv">

            <classpath path="bin"/> 
            <batchtest fork="no">                                      
                <fileset dir="test">                                   
                    <include name="**/*Test*.java"/>                   
                </fileset>                                             
            </batchtest>                                               
        </junit>                                                       
    </target>

Performance Optimization

Mutation analysis repeatedly executes the tests on mutated code. For performance reasons, consider the following when setting up the mutation analysis target:

Due to frequent class loading and thread executions, the following JVM options are recommended, in particular for large projects: