TestCocoon

Code Coverage Measurement for C/C++ and C#

Frequently Asked Questions

Code Coverage Instrumentation

Does TestCocoon support line coverage?

TestCocoon does not support line coverage because this kind of measurement and statistic is not accurate.

This metric depends on how you format the code. For example, take the following function:

   1 int main()
   2 {
   3   if (true) return 1;
   4     foo();
   5   return 0;
   6 }

Execute it and the line code coverage will produce:

   1 int main()
   2 {
   3 HIT   if (true) return 1;
   4 MIS   foo();
   5 MIS   return 0;
   6 }

Line Coverage: 33%
Reformat the code as follows:

   1 int main()
   2 {
   3   if (true)
   4     return 1;
   5   foo();
   6   return 0;
   7 }

Execute it and the line code coverage will produce:

   1 int main()
   2 {
   3 HIT   if (true)
   4 HIT     return 1;
   5 MIS   foo();
   6 MIS   return 0;
   7 }

Line Coverage: 50%
Reformat the code as follows:

   1 int main()
   2 {
   3   if (true)
   4     return 1;
   5   foo(); return 0;
   6 }

Execute it and the line code coverage will produce:

   1 int main()
   2 {
   3 HIT   if (true)
   4 HIT     return 1;
   5 MIS   foo(); return 0;
   6 }

Line Coverage: 66%
Reformat the code as follows:

   1 int main()
   2 {
   3   if (true) return 1; foo(); return 0;
   4 }

Execute it and the line code coverage will produce:

   1 int main()
   2 {
   3 HIT   if (true) return 1; foo(); return 0;
   4 }

Line Coverage: 100%
This small example shows that line coverage produces very different result depending on how the source code is formated. The decision coverage provided by TestCocoon is independent of the coding style.

Compiling

Microsoft® Visual Studio® rebuilds always my project even if it is not modified.

This problem occurs if a project is compiled with code coverage and precompiled headers support enabled. Simply deactivate the precompiled headers in the concerned build mode.

Instrumentation

How to exclude a source file from the code coverage analysis?

Excluding some source files from the code coverage analysis needs to be performed during the compilation. Two methods are possible:

  1. Compile using CoverageScanner with the command line option --cs-exclude-file-regex, --cs-exclude-file-wildcard and --cs-exclude-path.
  2. Use CoverageScanner pragmas to deactivate the instrumentation, by adding the following lines after all #include commands.
       1 #ifdef __COVERAGESCANNER__
       2 #pragma CoverageScanner (cov-off)
       3 #endif
    Adding this portion of code at the top of the source file (and so before all #include commands), will also deactivate the coverage analysis of the headers.

My source code contains inline functions, are they instrumented?

Inline functions are instrumented like other functions. To avoid instrumenting templates and inline functions of 3rd party library or the standard C++ library, CoverageScanner instrument all functions and procedures of the source file and the headers present in the current directory. If its header file is not in the current directory, it is necessary to tell CoverageScanner to instrument it too. The command line options --cs-include-file-regex, --cs-include-file-wildcard and --cs-include-path permits to force the instrumentation of additional files.

Is it normal that the .csmes file contains a copy of the source code?

Yes.
The .csmes file contains all information necessary for CoverageBrowser. That’s why the source code and the preprocessed source file is also included.

CoverageBrowser

Performance

Is there a way to minimize the CPU usage?

CoverageBrowser needs a lot of CPU resources and disk accesses to calculate the statistics for each execution. This calculation is performed in background, but can also be deactivated. Just proceed as follows: hide the column Coverage of the docking window "Executions" using the context menu "Show/Hide Columns->Coverage".

When I’m exporting the statistics per methods I can only see the 65536 first entries

This is a limitation of Microsoft Excel or OpenOffice. Excel 97 can only display 16384 rows. Excel 2000 and Excel 2003 can display up to 65536 rows and newer version of Excel can display up to 1048576 rows. OpenOffice Calc is limited to using 32000 rows.

Miscellaneous

Whre are stored the settings?

On Windows:%APPDATA%\testcocoon
On Linux$HOME/.config/testcocoon

Installation issue: CoverageBrowser does not work on my Linux™ distribution

Symptoms:
Calling CoverageBrowser brings the following error: 

symbol lookup error: /opt/TestCocoon/5.1.1/libQtCore.so.4: 
undefined symbol: _ZN13TestCocoon7QObject5eventEPNS_6QEventE

In this case, The Qt Library provided is not compatible with the linux distribution. Workaround:
The TestCocoon distribution provides a script buildqt.sh which permits to rebuild the Qt Library. Process as follows:

cd /opt/TestCocoon/5.1.1/
./buildqt.sh

Installation issue: PATH variable wiped after installation

Two sulutions:

  • The installer makes a backup of the PATH variable in the system variable PATH_TESTCOCOON_INSTALLER during the installation. Simply copy the contain of this variable in the PATH variable.
  • The installer create a system restore point before the installation and the uninstallation. Use the Microsoft® Windows System Restore program to go to the restore point called ’Uninstalled TestCocoon vx.x.x’ or ’Installed TestCocoon vx.x.x’.

Installation issue: CoverageBrowser windows are not redraw correctly

In this case, it is necessary to regenerate the Qt Library.(see chap. linux-symbols)

Installation issue: Microsoft® Visual Studio® Compiler warning D9002

Microsoft® Visual Studio® compiler produces following warning: 

cl : warning D9002: Unknown Option "--cs-on" ignored.

The path to CoverageScanner compiler wrapper is missing, add it as described in the documentation. (see chap. visual-studio-compiler)

Installation issue: Microsoft® Visual Studio® 2005 & 2008 Add-In is not installed

Verify that Microsoft® Visual Studio® 2005 & 2008 Add-In is loaded by Microsoft® Visual Studio®:

  1. Open "My Document" folder.
  2. Open the folder "Visual Studio 2005" (or "Visual Studio 2008")
  3. Open the folder "Addins"
  4. You should find the files TestCocoonVs2005AddIn.dll and TestCocoonVs2005AddIn.addin. If the files are missing, please reinstall TestCocoon.
  5. Start Microsoft® Visual Studio®.
  6. Click on "Tools->Options…". Select the entry Environment - Add-in/Macro Security. Verify that %VSMYDOCUMENTS%\addins is present in the path, and that the macros are enabled.
    pictures/addinsettings.png pictures/manual.tmp034.png
    Microsoft® Visual Studio® Option Dialog
    Microsoft® Visual Studio® Option Dialog
  7. Click on "Tools->Add-In Manager…". The list of addins loaded are displayed. Verify that TestCocoon is checked and also that the startup flag is set.
    pictures/addinmanager.png pictures/manual.tmp035.png
    Add-In Manager
    Add-In Manager
  8. Open a C/C++ project.
  9. Click on "Tools->Code Coverage Build Mode…". Microsoft® Visual Studio® 2005 & 2008 Add-In window should appear.

How to report an issue concerning CoverageScanner?

It is possible to generate a log file which permits to investigate on issues concerning CoverageScanner. Proceed as follows on Microsoft® Windows:

  1. In a console window: 
    cd %TESTCOCOON%
testcocoon_debug.bat

    This will install a debug version of CoverageScanner which will produce a set of log files.

  2. Reproduce your issue.
  3. In the console window, press enter to install the normal version of CoverageScanner.
  4. The log files are located in %TESTCOCOON%\logfiles. Zip it together and send it to TestCocoon support (pictures/email.gif support@testcocoon.org).

Proceed as follows on Linux™:

  1. In a console window: 
    cd /opt/TestCocoon/{version_of_testcocoon}
mv coveragescanner coveragescanner.back
cp coveragescannerdbg coveragescanner

    This will install a debug version of CoverageScanner which will produce a set of log files.

  2. Reproduce your issue.
  3. Restore the release version of CoverageScanner: 
    cd /opt/TestCocoon/{version_of_testcocoon}
mv coveragescanner.back coveragescanner
  4. The log files are located in /tmp/TestCocoon. Zip it together and send it to TestCocoon support (pictures/email.gif support@testcocoon.org).
The log file may contains part of your source code. If this is an issue, simply remove the concerned lines with your favorite editor.

How to debug Microsoft® Visual Studio® 2005 & 2008 Add-In?

  • Deactivate Microsoft® Visual Studio® 2005 & 2008 Add-In
    This is necessary in order to avoid issues with files which cannot be overwritten.
    • Start Microsoft® Visual Studio®.
    • Start the Addin manager: click on "Tools->Add-in Manager".
    • Uncheck all items in the line in which TestCocoon is listed.
    • Close Microsoft® Visual Studio®.
  • Load Microsoft® Visual Studio® 2005 & 2008 Add-In project
    • Unpack the source code of TestCocoon.
    • Open Microsoft® Visual Studio® 2005 & 2008 Add-In with the corresponding version of Microsoft® Visual Studio®.
    • Edit the property settings:
      • Debugging option: select external application and enter the path of the Microsoft® Visual Studio® IDE executable (devenv.exe).
      • Build option: enter the path of the Microsoft® Visual Studio® 2005 & 2008 Add-In DLL: %USERPROFILE%\Documents\Visual Sudio 20??\Addins\TestCocoonVS20??AddIn.dll
    • Start the Addin manager and reenable the Microsoft® Visual Studio® 2005 & 2008 Add-In.
  • Start debugging: Microsoft® Visual Studio® should start and setting breakpoints into the addin should work.

How can I get informed about new releases?

New versions of TestCocoon are published using the RSS feed http://www.testcocoon.org/testcocoon_rss.xml.