The Major Mutation Framework

Easy and scalable mutation analysis for Java!

(version 3.0.1)


Download latest version

Tutorial

This step by step tutorial demonstrates how to use Major for:

All examples in this tutorial use the triangle program (example directory) and the provided mml files (mml directory).

Create an MML File (Optional)

Major’s domain specific language (mml) supports customizing mutant generation. Suppose only return statements, relational operators, and conditional operators should be mutated within the method classify of the class triangle.Triangle. The following mml file expresses these requirements:

targetOp{
        // Define the replacements for ROR
        BIN(>)->{>=,!=,FALSE};
        BIN(<)->{<=,!=,FALSE};
        BIN(>=)->{>,==,TRUE};
        BIN(<=)->{<,==,TRUE};
        BIN(==)->{<=,>=,FALSE,LHS,RHS};
        BIN(!=)->{<,>,TRUE,LHS,RHS};
        // Define the replacements for COR
        BIN(&&)->{==,LHS,RHS,FALSE};
        BIN(||)->{!=,LHS,RHS,TRUE};
        // Define the type of statement that STD should delete
        DEL(RETURN);

        // Enable the STD, COR, and ROR mutation operators
        STD;
        COR;
        ROR;
    }
    // Call the defined operator group for the target method
    targetOp<"triangle.Triangle::classify(int,int,int)">;

Customizing mml files is optional. Major is released with useful default mml files.

Major’s mml compiler (mmlc) validates an mml file and compiles it into a binary representation:

major$ mmlc tutorial.mml tutorial.mml.bin

Note that the second argument is optional – if omitted, the compiler will add .bin to the input file name.

Generate mutants

Major’s mutator uses the compiled mml file (tutorial.mml.bin) to configure the mutant-generation process.

Command line

The following command generates and compiles all mutants, using the tutorial.mml.bin file:

major$ major --mml tutorial.mml.bin -d bin src/triangle/Triangle.java
    Generated 86 mutants (128 ms)

Ant build file

To invoke Major’s mutator from an Apache Ant compile target, adapt the build.xml build file as follows:

<property name="major.jar" value="<path to major.jar>"/>
    <property name="mml" value="<path to compiled mml file>"/>

    <target name="compile" depends="init" description="Compile">
        <javac srcdir="src" destdir="bin" debug="yes">

          <classpath location="${major.jar}"/>

          <compilerarg value="-Xplugin:MajorPlugin mml:${mml}"/>

        </javac>
    </target>

The following command generates and compiles all mutants, using the tutorial.mml.bin file:

major$ ant -Dmml=tutorial.mml.bin compile

    compile:
        [javac] Compiling 1 source file to bin
        [javac] Generated 86 mutants (149 ms)

    BUILD SUCCESSFUL
    Total time: 0 seconds

Inspect generated mutants

Major’s mutator plugin reports the number of generated mutants and the total time (mutant generation and compilation). Additionally, it produces a log file mutants.log, which contains detailed information about the generated mutants. Here is an example for the mutants generated for the tutorial program:

major$ head -3 mutants.log
    1:ROR:<=(int,int):<(int,int):triangle.Triangle@classify(int,int,int):11:a <= 0 |==> a < 0
    2:ROR:<=(int,int):==(int,int):triangle.Triangle@classify(int,int,int):11:a <= 0 |==> a == 0
    3:ROR:<=(int,int):TRUE(int,int):triangle.Triangle@classify(int,int,int):11:a <= 0 |==> true

Export source-code mutants

Major supports exporting generated mutants to individual source-code files. This feature is disabled by default. See the documentation for more details.

Analyze mutants

The build.xml file has to provide a suitable test target that enables Major’s analyzer. The analyzer takes the set of generated mutants and executes a given test suite against them. The following mutation.test target enables the mutation analysis and exports the results to summary.csv, and details.csv (see the documentation for configuration details):

<target name="mutation.test" description="Run mutation analysis">
        <echo message="Running mutation analysis ..."/>                
        <junit printsummary="false" 
               showoutput="false"            
               mutationAnalysis="true" 
               summaryFile="summary.csv" 
               mutantDetailsFile="details.csv">                                                      

               <classpath path="bin"/>                                    
               <batchtest fork="false">                                   
                   <fileset dir="test">                                   
                       <include name="**/*Test*.java"/>                   
                   </fileset>                                             
               </batchtest>                                               
        </junit>                                                       
    </target>

Using Major’s default analyzer, the following command invokes the mutation.test target:

major$ ant mutation.test
    mutation.test:
         [echo] Running mutation analysis ...
        [junit] MAJOR: Mutation analysis enabled
        [junit] MAJOR: ------------------------------------------------------------
        [junit] MAJOR: Run 1 ordered test(s)to verify independence
        [junit] MAJOR: ------------------------------------------------------------
        [junit] MAJOR: Preprocessing time: 0.02 seconds
        [junit] MAJOR: ------------------------------------------------------------
        [junit] MAJOR: Serialize preprocessing results to preprocessing.ser
        [junit] MAJOR: ------------------------------------------------------------
        [junit] MAJOR: Mutants generated: 86
        [junit] MAJOR: Mutants covered:   86 (100.00%)
        [junit] MAJOR: ------------------------------------------------------------
        [junit] MAJOR: Export coverage map to covMap.csv
        [junit] MAJOR: Export test map to testMap.csv
        [junit] MAJOR: ------------------------------------------------------------
        [junit] MAJOR: Run mutation analysis with 86 mutant(s) and 1 test(s).
        [junit] MAJOR: ------------------------------------------------------------
        [junit] MAJOR: 1/1 - triangle.test.TestSuite (2ms / 86):
        [junit] MAJOR: 189 (76 / 86 / 86) -> AVG-RTPM: 2ms
        [junit] MAJOR: Mutants killed (FAIL-EXC-TIME) / live: 76 (76-0-0) / 10
        [junit] MAJOR: ------------------------------------------------------------
        [junit] MAJOR: Summary:
        [junit] MAJOR:
        [junit] MAJOR: Analysis time                 : 0.19 seconds
        [junit] MAJOR: Mutation score                : 88.37% (88.37%)
        [junit] MAJOR: Mutants killed (FAIL-EXC-TIME): 76 (76-0-0)
        [junit] MAJOR: Mutants live                  : 10
        [junit] MAJOR: Mutant executions             : 86
        [junit] MAJOR: ------------------------------------------------------------
        [junit] MAJOR: Export summary of results to summary.csv
        [junit] MAJOR: Export mutant details to details.csv
        [junit] MAJOR: ------------------------------------------------------------
        [junit] MAJOR: Done!
        [junit] MAJOR: ------------------------------------------------------------
    BUILD SUCCESSFUL
    Total time: 0 seconds

As configured in the build.xml file, the results of the mutation analysis are exported to the files summary.csv and details.csv, which provide the following information: