Wednesday 18 May 2011

Static Analysis for .NET Builds

Back to the fifth installment of JenkinsHeaven....

So you have successfully compiled and tested your application. The little ball is blue (or green) and the console log says "BUILD SUCCESSFUL" That's good, but it is far from the end of the story.

This week I intend to talk about the next part of the build process - Reporting. More specifically: static analysis.

The dashboard nature of Jenkins and its ability to bring all your reports together is one of its many strengths, the job's "homepage" can be enriched with lots of useful metrics with the addition of post build steps to the build process. These post build steps should be placed after all your testing steps (if part of the same build) or in a downstream build.

The key purpose of static code analysis is to improve the quality of your code base (improving the maintainability and manageability aspects). The ideas of technical debt and DRY are key here.

Static analysis highlights where your code base needs further work - especially refactoring (DRY). These results can highlight to the project manager where

the quick wins are and be an input into the development of a roadmap for future releases - both bug and feature.

Reporting and the graphing capabilities of Jenkins makes exposing and communicating these metrics to the wider stakeholder community easy.

There are a myriad of static analysis tools available, so I will not attempt to cover them all.

Here are the ones I currently use:

Open Tasks Trend

This scans the files matching the mask you specify (e.g. **/*.cs) for incidents of "task tags" By default when you check the "Scan workspace for open tasks"

checkbox High priority is set to FIXME, Normal priority is set to TODO, Low priority is set to TBD.

The result is a task histogram on the project page.

FXCop

You will need to have FXCop installed on your build machine.

Microsoft documentation on the FXCop tool here.

Version 10.0 download.

Version 1.36 download.

Windows 7 SDK download

Configuring FXCop for Jenkins is a two step process:

  1. Add a Windows Batch command post build step to execute FXCop passing in the workspace relative path the application dll and outputting a .xml result file.

  2. Configure the Violations plugin to produce a report using the xml file output from FXCop as the input. Here you can also set the number of violations that you want to represent "sunny", "thundercloud" and "unstable" (yellow ball).

CCM

You will need to have the command line version of CCM installed on your build machine. You will need to tell Jenkins where it is installed in the Manage Jenkins page.

Also, install the CCM plugin from the updates center. This is not part of the violations stable of tools.

  1. Add an Invoke CCM post build step to execute CCM passing in the list of workspace relative paths of folders that contain your source. (separated with spaces works for me).

The result is a CCM graph on the project page.

There appears to be a bug with CCM (that or there is something I don't understand): When you tell Jenkins where your CCM installed you also specify a number of how hard you want it to look at your code. If you increase this number (and don't change your code) you'll get a higher complexity result. Doesn't seem right. Give it a try yourself. Am I wrong? Is there something I'm missing?

Simian

I love this plugin. Simian assists you with the DRY principle by highlighting similar pieces of code which should encourage you to refactor. I was banging my head with this plugin for a while until the developer kindly released version 2.3.31 on 3rd February. (My friend Bruno suggested that the release be called the "Gray" release after I reported my issue. I see that version 2.3.32 was released at the end of April.)

Being part of the Violations stable of plugins, configuring simian to run as part of your build is the very similar to FXCop

  1. Add a Windows Batch command post build step to execute Simian passing a file mask and outputting a .xml result file. I also tell Simian to exclude all designer.cs, Reference.cs and Test.cs files. like so:
  2. simian-2.3.35 **/*.cs -formatter=xml:simian-output.xml -failOnDuplication- -reportDuplicateText+ -excludes="*/*.designer.cs" -excludes="*/*Reference.cs" -excludes="*/*Tests.cs"
  3. Configure the Violations plugin to produce a report using the xml file output from simian as the input. Here you can also set the number of violations that you want to represent "sunny", "thundercloud" and "unstable" (yellow ball).

All these static analysis graphs appear above my test results graph in the RHS pane on the job home page.

Hopefully you found this post useful. What else would you like to hear about?

I would be interested to hear what metrics others are collecting to help improve the quality of their code base.

Come visit my other site at fullcirclesolutions.com.au. All my best ideas are distilled into commercial tools and made available for purchase.

Till next time....

11 comments:

  1. Thanks for the article, very insightful!

    Maybe if would be better if you provided links to the tools you are using, just to avoid an additional step of Googling. I've heard about some tools for the first time thanks to your article.

    Some links:

    FXCop: Included with the Windows SDK 7.1.
    CCM: http://www.blunck.se/ccm.html
    Simian: http://www.harukizaemon.com/simian/

    Hope that helps...

    ReplyDelete
  2. Thanks David.

    I have updated the post and added the above links to the headings (CCM and Simian) and just below FXCop.

    ReplyDelete
  3. First off great Set of Posts.

    I am having some trouble configuring Simian. At the moment I am doing the following.

    I have a batch file that performs these commands:
    SET absolutePath=%WORKSPACE%
    echo 'begin simian run'
    echo %WORKSPACE%
    C:\BuildTools\Simian\bin\simian-2.3.32.exe %absolutePath%\Code\**\*.cs -formatter=xml:%absolutePath%\simian-results.xml
    echo %ERRORLEVEL%
    exit %ERRORLEVEL%

    Something about the above causes the build to fail. The message is:

    Copyright (c) 2003-2011 Simon Harris. All rights reserved.
    Simian is not free unless used solely for non-commercial or evaluation purposes.

    C:\Jenkins\Service\jobs\Trunk-Isis\workspace>echo 1
    1

    C:\Jenkins\Service\jobs\Trunk-Isis\workspace>exit 1
    Recording NUnit tests results
    Finished: FAILURE



    I know it has nothing to do with the Violations plugin because it is disabled at the moment. If I remove the batch the build is sunny again. The result file does get created.

    Once I do enable the plugin will it simply be a matter of pointing it at the simian.xml result file?

    ReplyDelete
  4. Hi J. Barrett,

    Thank you for reading and thank you for the positive feedback.

    I'll start with your last question first: Yes, once the simian-results.xml file is created you just need to specify it in the violations config (workspace relative to the workspace root). Nothing special. It should "just work" and produce the graph on the job home page.

    The help comment in Jenkins for "Execute Windows batch command" says:
    "...the script will be run with the workspace as the current directory...If you already have a batch file in the SCM, you can just type in the path of that batch file (again relative to the workspace directory)..."

    I think therefore the issue with your script is the presence of the %absolutePath%. I would suggest the following changes:

    1. The first parameter to simian should be a workspace relative path (Code\**\*.cs).
    2. I just write the xml file out to the root of the workspace, like so: -formatter=xml:simian-results.xml

    I'll post another comment soon with the actual command I am using in my build.

    FWIW, I currently do not have my Windows Batch commands in SCM. I am just typing them straight into Jenkins. It works fine, but...Rereading that last line of the help makes me think that I can improve my setup of I put all my windows batch commands under SCM (VisualSVN in my case currently).

    ReplyDelete
  5. Further to my last post my Simian batch command is:

    "C:\Program Files\simian-2.3.31\bin\simian-2.3.31" **/*.cs -formatter=xml:simian-output.xml -failOnDuplication- -reportDuplicateText+ -excludes="**/*.designer.cs" -excludes="**/*Tests.cs" -excludes="**/*Reference.cs"
    exit %%ERRORLEVEL%%

    And it is working

    ReplyDelete
  6. Have you tried NDepend? http://www.NDepend.com
    With NDepend you can readily create code rules and check them during each build process. More than 150 default code rules are provided.

    ReplyDelete
    Replies
    1. Hi Patrick, I did take a look at NDepends website. There are many great commercial products out there doing .NET code coverage and static analysis. I chose Simian as I was try to find all open source solutions as much as possible.

      Delete
  7. Hi Andrew, thank for your post which is a very good synthesis about code analysis tools.
    Just one question: did you manage to use the FxCop 10.0 with the violation plugin? Because I cannot. I tested with FxCop 1.36 and all is ok. When I try with FxCop 10.0, I do have a graph showing the violations trends, then when I want to see violations details, I obtain a blank page. Did you have any issue like that?

    Thank you

    ReplyDelete
  8. It's been a while since I looked into this: I have not upgraded to fxcop 10 myself yet.

    ReplyDelete
  9. [CCM] Parsing of file C:\Users\IBM_ADMIN\Desktop\Macys SCM\CodeCoverageFinal\CodeCoverageToXMLConverter\CodeCoverageToXMLConverter\Properties\Settings.Designer.cs failed due to an exception:

    org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)
    at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
    at org.apache.commons.digester.Digester.parse(Digester.java:1916)
    at hudson.plugins.ccm.parser.CcmParser.parseCCMXmlFile(CcmParser.java:106)
    at hudson.plugins.ccm.parser.CcmParser.parse(CcmParser.java:71)
    at hudson.plugins.analysis.core.AbstractAnnotationParser.parse(AbstractAnnotationParser.java:54)
    at hudson.plugins.analysis.core.FilesParser.parseFile(FilesParser.java:324)
    at hudson.plugins.analysis.core.FilesParser.parseFiles(FilesParser.java:282)
    at hudson.plugins.analysis.core.FilesParser.parserCollectionOfFiles(FilesParser.java:233)
    at hudson.plugins.analysis.core.FilesParser.invoke(FilesParser.java:202)
    at hudson.plugins.analysis.core.FilesParser.invoke(FilesParser.java:32)
    at hudson.FilePath.act(FilePath.java:991)
    at hudson.FilePath.act(FilePath.java:969)
    at hudson.plugins.ccm.CcmPublisher.perform(CcmPublisher.java:101)
    at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:139)
    at hudson.plugins.analysis.core.HealthAwarePublisher.perform(HealthAwarePublisher.java:68)
    at hudson.plugins.analysis.core.HealthAwareRecorder.perform(HealthAwareRecorder.java:259)
    at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:75)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
    at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:726)
    at hudson.model.Build$BuildExecution.post2(Build.java:185)
    at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:671)
    at hudson.model.Run.execute(Run.java:1766)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:381)

    ReplyDelete
  10. CCM is failing in JENKINS build

    ReplyDelete