Update all your git repos in one go

I have a root directory where I clone all my git repos to.  Something like C:\GitRepositories.

Making sure I have the latest of each repository used to be a bit mechanical.

So I wrote a script.  It is sitting in my C:\GitRepositories directory.

I call it updateAll.sh

Here it is:

#!/bin/sh

ls -l -d */ > temp.txt

while read line          
do          
    gawk '{print $NF}' | sed s/.$//
done < temp.txt > out.txt
rm temp.txt
echo "Updating the following repos:"
less out.txt

while read line
do
    cd $line
    git pull
    cd ..
done < out.txt
rm out.txt

My new home for these kinds of scripts are here.


Till next time

Using Jenkins with Git on BitBucket

Welcome to the seventh instalment of the JenkinsHeaven blog.

Apologies for the wait (life has kind of thrown some curveballs...)

Recently Bitbucket.org started offering private GIT repositories for free. :)

This a great option if you aren't ready to open source your code but you want to share your development effort with a small group (up to 5 including yourself) of friends.

So my migration problem was to migrate from a local VisualSVN installation to a remote BitBucket GIT repository and then get my local Jenkins master connecting to Bitbucket and successfully checking out the code.

I thought this would be a piece of cake....em....not quite.  But then again everything is easy when you know how.

Before we get into the nitty gritty detail of how to configure Jenkins correctly I must first thank Bruno Kinoshita from TupiLabs for putting up with me sending docx files full of screenshots as we did the "its-working-on-my-machine-why-not-yours" thing.

Tools you need to install
To work with GIT repositories all you need to install is Git Extensions (version 2.29 is what I have installed, currently).  this will install git itself (msysgit, which is GIT for Windows), a VisualStudio plugin and PuTTY  PuTTY is needed for the SSH certificates which is how Jenkins will talk to Bitbucket. (but we'll get to that)  Right now just download and install Git Extensions.  Follow the instructions here

Environment Variables

All I have and all you need is the following to make Jenkins work with GIT:

GIT_HOME=C:\Program Files\Git    --> default location from Git Extensions installation
HOME=C:\SSHKeys                      --> best to have a directory with no spaces
Path contains C:\Program Files\GitExtensions\PuTTY

That's it for environment variables.

Create an account and Repository on Bitbucket

1. Go to bitbucket.org and register an account with an email address and password

2. Click Repositories dropdown and click "create repository" at the bottom of the menu.

3. Fill in the form and submit

(At this stage it is an empty repository...thats ok, we'll fill it up later)

Creating certificates

Now to get Jenkins and Git Extensions talking to Bitbucket you need SSH certificates

I have two sets of SSH Keys (a "set" being a pair made up of a public and private key) one for Jenkins and one for Git Extensions. 

I would argue that this is necessary because the private key you get from the key creation process is called id_rsa with no ppk extension and as such will not be read by Git Extensions.  Git Extensions can only be configured with file with the .ppk extension.  And yes, before you ask if you rename id_rsa to id_rsa.ppk it doesn't work.

Set 1 (for Jenkins)

Follow the instructions here.  From heading "Next: Set Up SSH Keys" down.

As a result you should have 3 files in the C:\SSHKeys\.ssh directory (id_rsa, id_rsa.pub and known_hosts)

Set 2 (for Git Extensions launched via Visual Studio)
Simply use puttygen.exe which was installed with Git Extensions in the PuTTY subdirectory. 

This process will create a *.ppk private key and associated public key.

I have called mine pushonlykey_private.ppk and pushonlykey_public.  These two files live in the C:\SSHKeys directory (NOT C:\SSHKeys\.ssh) 

Populating your Bitbucket repository
In the below C:\GITRepositories is where I keep my local  working copy of my projects

> Export your source code from subversion with tortoiseSVN or command line to c:\temp
> open GITBash

> Execute the following command in GITBash
cd to c/GITRepositories
mkdir myrepo
cd myrepo
git init  -->  this creates a .git folder in c:\GITRepositories\myrepo.  A bare git repo

> copy all source code files in c:\temp to c:\GITRepositories\myrepo with Windows File Explorer

> Execute the following command in GITBash
git status   --> lots of unversioned files
git add *
git add *.*
git status   --> lots of files ready to be committed
git commit -m "initial commit"
git status   --> nothing to be committed
git remote add origin git@bitbucket:<your username>/<your repo name>.git
git push origin master

> Navigate your browser to https://bitbucket.org/<your username>/<your repo name>/src and you will see your source code. :)

Configuring Jenkins to build from your Bitbucket repo

Now after all that preparation we are getting to the really fun stuff.

GIT SETUP

First you need to install the git plugin via the update center in Jenkins.

I have the following configuration for GIT in Manage Jenkins > Configure System (search the page for "Git installations"):

Name: Git (can be anything)

Path to Git executable: C:\Program Files\Git\cmd\git.cmd

Install automatically is turned off.

JOB CONFIG

Almost there!!!

In your Jenkins job:

1. Under Source Code Management, select the Git radio button

2. Set repository URL to: git@bitbucket.org:<your username>/<your repo name>.git

3. Set Branches to build to **

4. Select bitbucketweb from the Repository browser dropdown

5. Set URL to http://bitbucket.org/<your username>/<your repo name>/

6. Hit save and start drinking!

Setting Up Git Extensions
1. In an open repository select from the menu Remotes > Manage remote repositories.
2. In the details box set Name to what you want and Url to git@bitbucket.org:<your username>/<your repo name>.git
3. In the PuTTY SSH box set Private key file to "C:/SSHKeys/pushonlykey_private.ppk"  (without the quotes) 

Access from Visual Studio
As you installed the Visual Studio plugin when you installed Git Extensions you will now have the following additions to the menus and toolbars to interact with Git Extensions and do at that Git goodness like staging your commits:

I'm pretty sure the above instructions will work correctly.  I specifically retested the populating BitBucket steps from scratch with another test repo.  Let me know in the comments if anything else does not make everything "just work"

Till next time.

Static Analysis for .NET Builds

Back to the fifth installment of JenkinsHeaven....


So you have successfully compiled and tested your application.  The little ball is blue (or green) and the console log says "BUILD SUCCESSFUL"  That's good, but it is far from the end of the story.


This week I intend to talk about the next part of the build process - Reporting.  more specifically static analysis.


The dashboard nature of Jenkins and its ability to bring all your reports together is one of its many strengths, the job's "homepage" can be enriched with lots of useful metrics with the addition of post build steps to the build process.  These post build steps should be placed after all your testing steps (if part of the same build) or in a downstream build.


The key purpose of static code analysis is to improving the quality of your code based (improving the maintainability and manageability aspects).  The idea
of technical debt and DRY are also key here
Static analysis highlights where your code base needs further work - especially refactoring (DRY).  These results can highlight to the project manager where
the quick wins are and be an input into the development of a roadmap for future releases - bug and feature.
Reporting and the graphing capabilities of Jenkins makes exposing and communicating these metrics to the wider stakeholder community easy.

There are a myriad of static analysis tools available, so I will not attempt to cover them all.

Here are the ones I currently use:


Open Tasks Trend

This scans the files matching the mask you specify (e.g. **/*.cs) for incidents of "task tags" By default when you check the "Scan workspace for open tasks"

checkbox High priority is set to FIXME, Normal priority is set to TODO, Low priority is set to TBD.


The result is a task histogram on the project page.


FXCop
You will need to have FXCop installed on your build machine.
Microsoft documentation on the FXCop tool here
Version 10.0 download.
Version 1.36 download. 
Windows 7 SDK download

Configuring FXCop for Jenkins is a two step process:

1. Add a Windows Batch command post build step to execute FXCop passing in the workspace relative path the application dll and outputting a .xml result file.
2. Configure the Violations plugin to produce a report using the xml file output from FXCop as the input.  Here you can also set  the number of violations that you want to represent "sunny",  "thundercloud" and unstable ("yellow ball").

CCM
You will need to have the command line version of CCM installed on your build machine. You will need to tell Jenkins where it is installed in the Manage Jenkins page.
Also, install the CCM plugin from the updates center.  This is not part of the violations stable of tools.

1. Add an Invoke CCM post build step to execute CCM passing in the list of workspace relative paths of folders that contain your source. (separated with spaces works for me).

The result is a CCM graph on the project page.


There appears to be a bug with CCM (that or there is something I don't understand):  When you tell Jenkins where your CCM installed you also specify a number of how hard you want it to look at your code.  If you increase this number (and don't change your code) you'll get a higher complexity result.  Doesn't seem right.  Give it a try yourself.  Am I wrong?  Is there something I'm missing?

Simian

I love this plugin.  Simian assists you with the DRY principle by highlighting similar pieces of code which should encourage you to refactor.  I was banging my head with this plugin for a while until the developer kindly released version 2.3.31 on 3rd February. (My friend Bruno suggested that that release be called the "Gray" release after I reported my issue.  I see that version 2.3.32 was released at the end of April.)

Being part of the Violations stable of plugins, configuring simian to run as part of your build is the very similar to FXCop

1. Add a Windows Batch command post build step to execute Simian passing a file mask (**/*.cs) and outputting a .xml result file.  I also tell Simian to exclude all designer.cs, Reference.cs and Test.cs files.
2. Configure the Violations plugin to produce a report using the xml file output from simian as the input.  Here you can also set the number of violations that you want to represent "sunny",  "thundercloud" and unstable ("yellow ball").

All these static analysis graphs appear above my test results graph in the RHS pane on the job home page.

Hopefully you found this post useful.  What else would you like to hear about?


I would be interested to hear what metrics others are collecting to help improve the quality of their code base.

Till next time....

Building the Build (first .NET Build)

Apologies for the delay this week…(I’ll blame the Easter holiday mode I was in). Getting back to it:

Creating and configuring a build in Jenkins is known (at least in my neck-of-the-woods) as “Building the Build”

When you first look at that configuration page (especially as you start adding more and more plugins, which, trust me, you will) it can seem daunting.  When I think back to when I started playing with H^%#$& three plus years ago it would take me hours to get even the simplest bits working.

Many a long night was spent (entire weekends in fact) trying to first, simply understand the mechanisms but then; to actually get an execution environment that would actually work.

(When I refer to the “simplest bits” I am talking about the unit and selenium tests – it was a bit of a pain to figure it out).

It might sound cruel but I would not recommend short cutting this “understanding process”.  I feel personally that I am able to “run” given that I have already “crawled” and “walked” first.

That said I will try and cover some useful things.

Configuring a Jenkins job is very much a process of constructing a build process (I would encourage you to view it in this way).  The goal should be to automate ALL the steps that need to happen to create a production-ready system, with the starting point being the core source code.  The process may or may not be a simply linear sequence (Step 1, Step 2, Step 3, complete), there may be some steps that depend on more than one previous step (like a stateflow diagram), the build pipeline can start getting complicated (but I digress, and that’s a topic for another post).

Very quickly you should start to see what you can do with Jenkins and before long your build process has a large number of steps.  These steps need to be “built up” step-by-step, layer-upon-layer hence the term “Building the Build”.

The temptation is to dive in and start filling out fields.  That’s what I did.  But don’t

 

Take a second to consider the high level goals of Jenkins:

1. Check out the source code
2. Do builds and/or tests
3. Record results
4. Notify people

 

Have a look at that list again.  EVERYTHING, ALL the plugins fit into these 4 categories.  That’s right, only 4.  It’s worth keeping these in mind as you progress to maintain perspective.

Hopefully by now I have whetted your appetite to some examples.  Let’s get started.

Alas, .NET does not currently have an analogous tool for Maven (a build lifecycle out of the box).

Microsoft has MSBuild, which is basically just ANT (essentially just batch scripting where start with a proverbial blank piece of paper and have to tell the computer what to do step by step).

This means that for the time being we have to choose “Build a free-style software project” when creating a new job to build a .NET project.

(To all those who are using Java, is anyone still using ANT?  Seriously?  When you have Maven why would you consider ANT anymore? Surely ANT is dead.  In a previous life we mavenised all our legacy ANT application whenever we could.)

I already have a menagerie of plugin installed so my config may look a bit different in the screenshots but I will focus on the plugins and settings that we need to concern ourselves with.  Once we have created the job first basic setting:

• I like to keep the last 5 jobs as general rule.  I think it keeps a nice balance between managing disk space and being able to see trends in the graphs that we will set up to be produced.

First thing the job needs to do as per number 1 above is check out the source code. (Note: a job does not always have to checkout source code, in fact in some circumstance you do not want to do this at all.  We might get to this in later posts).

As I am using VisualSVN I am yet to bother with repository browsing (that and I just spent an hour trying to get any of the Jenkins-supported ones to work and couldn’t on my Windows box.)

• Once the code is checked out we need to start defining what this freestyle build will do.  Do that by Selecting the Add Build Step button and select ‘Build a Visual Studio project or solution using MSBuild’
• Select the MSBuild version defined in Manage Jenkins > Configure System page.
• Define the target you want MSBuild to run.

I run /t:Clean and /t:Rebuild

Nearly all the remaining Build Steps are achieved using ‘Execute Windows batch command’

Running Tests

Assuming the code compiles the next thing you are going to need to do is run Unit Tests, and then functional web tests.

To do this you need to install NUnit and write some tests in C#.

Executing a Windows Batch command, you specify the command:

"<Installed Path to NUnit>\nunit-console.exe" trunk\<TestProjectName>\bin\Debug\<TestProjectName>.dll /xml=nunit-testresult.xml

exit %%ERRORLEVEL%%

This should run your Unit tests.  Same approach is taken with Selenium tests:

"C:\Program Files\NUnit 2.5.9\bin\net-2.0\nunit-console.exe" trunk\<SeleniumTestProjectName>\bin\Debug\<SeleniumTestProjectName>.dll /xml=selenium-testresult.xml

exit %%ERRORLEVEL%%

This should run your Selenium tests.

Currently I have most of my steps in one Job.  One improvement I could make is to separate out the Selenium Step (as they take a long time to run on the one machine I have at home) into its own job, according to the principle that CI should fail fast to speed up feedback.

Next Week: Further along the build process: Static Analysis for .NET projects