Thursday, 25 February 2016

Improving Deployments to Test Environments

Welcome to the latest JenkinsHeaven post!

This is a follow up to the last post about giving team members 1-button deployments to test environments

Generally speaking the deployments have been working extremely well. Previously what was a 30-minute manual (and therefore error-prone) deployment that the development team had to do (which in and of itself reduced iteration capacity) has been reduced to a 2 minute automated process that just works.

Since the last post there have been some major and a minor (incremental) improvements. I'll talk about the incremental one first.

We decided to move the step that runs any database updates that have been checked in since the last build to the front. We found that it is the most likely to fail and therefore don't want to deploy the web application if it does. By making this simple ordering change the whole build is more transactional in nature.

Is it ok to do a deployment now?

So we are running static analysis, unit tests and code coverage as part of the job that runs on every commit. Everyone thought that was great. Also, we could deploy to any nominated environment. Everyone thought that was awesome.

One small wrinkle: The unit tests were running as part of the run-on-every-commit job but were not a gate keeper to the deployment jobs. As a result the Test team had to keep asking: "Is it ok to deployment now?"

This was an issue and needed to be resolved.

Come with me on the journey of how I spent the last two days (thankfully the start of the iteration) solving this issue before we got back into coding and required a deployment "service". Take heart, it is possible with the right mix of plugins.

Firstly some key points about the environment we are operating in:

  1. Using Web Deploy for deployments means that all the knowledge of the remote target IIS is kept with the solution in publish profiles.
  2. The Test team are required to be able to press a button to execute a deployment.
  3. Due to item 1, an artifact repository and the promoted builds plugin are not much help because Web Deploy wants the workspace that has been tested as input, not the compiled binaries.

The Web Deploy mechanism (which I execute through the /deploy parameter to msbuild) works so well, I wanted to keep this in place unchanged. The problem was therefore reduced to: "How do I hand a successfully unit-tested workspace to a deploy job?"

Job 1 Overview

This is the job that I granted users in the DEV and TEST team read and build permissions. Remember to grant yourself all permissions. Its main purpose is to act as gatekeeper of job 2, which actually does the deploy. Job 1 does this by running the unit tests.

Job 1 Configuration

Block build if certain jobs are running: On (Build Blocker Plugin)

As this job is running the XUnit tests, if it is kicked off while the main build job is running (due to a developer commit) it will fail to execute the XUnit tests and fail due to an empty results file. Blocking this job until the main job finishes. Deployers just have to be a little more patient and wait the extra (up to) 10 minutes for the main job to complete.

Discard old builds

We're going to be archiving the workspace. No matter how much disk space you have your going to want limit how much you keep. I currently have this set this to 10, although I am thinking of reducing it to 5.

Permission to copy artifacts (Copy Artifacts Plugin)

Specify the name of job 2 as a project allowed to copy artifacts

Check out the source code from TFS

Use Update: off. This is really important for our angular typescript application. If one of the developers moves or otherwise alters a .ts file in the solution. You really don't want any old .js and files hanging around on the filesystem. Get a fresh copy of the entire workspace every time.

Clean the solution

Command line arguments: /t:Clean /p:Configuration=Release;Username=<domain\userId>;Password=<password>

Username and password needs to be of an account of a domain that has access to the internet. In this corporate environment this means that the build can restore NuGet packages.

Rebuild the solution

Command line parameters: /t:Rebuild /p:Configuration=Release;Username=<domain\userId>;Password=<password>

Execute the (XUnit) unit tests (XUnit Plugin)

Windows Batch Command: JenkinsScripts\RunUnitTests.bat

Publish XUnit test result report

Add NUnit-Version N/A (Default) and set Pattern to TestReports\xunit-*output-as-nunit.xml Leave checkbox as is. For both Failed Tests and Skipped Tests set both Total and New to 0

Archive the artifacts

Files to archive: **/*.* (Remember: you want to archive the entire workspace)

Excludes (under Advanced): **/bin/**, **/obj/** (We don't want the binaries)

Archive artifacts only if build is successful: On

Execute other projects

Once we have got to this point we a good to go. Execute Job 2 by specifying its name. I call mine DoDeploy__ (I actually have 3 flavours of deploy DoDeploy_TEST_RELEASE, DoDeploy_TEST_DEBUG, DoDeploy_PILOT_RELEASE)

Job 2 Overview

This is where the rubber hits the road, where we have a green light to deploy.

Job 2 Configuration

No Source Code checkout

We want to get the archived workspace that Job 1 has deemed ok to be deployed. Its the workspace that just got unit tested successfully.

Delete workspace before build starts (Workspace Cleanup Plugin): On

To be sure, to be sure

Copy artifacts from another project

Project Name: Name of Job 1. I call mine: DeployTPS__ because this is what the user wants it to do.

Which build: Upstream job that triggered this build

Artifacts to copy: blank (We want everything baby)

Artifacts not to copy: blank

Target directory: blank (workspace is default)

Parameter filters: blank

Ready to deploy

At this point we have secured for ourselves a workspace that has been successfully unit tested and ready for deployment. QED.

It would make life easier if Jenkins has a plugin that allowed me to more easily archive the workspace, because that's what Web Deploy prefers. It would be easier to do all these steps in one job rather having to split over two.

Some more observations

If you find yourself with the same environment limitations, the above set up does work. However it is not perfect. Be aware of the following things you can and can't do.

You can:

  • Deploy the latest successfully unit tested code to a given environment.

You can't:

  • Deploy artifacts to an artifact repository (because you don't have one) and deploy particular versions of your application to a given environment. This is less than ideal and will be where I'll be working next to enhance our capability.

Thanks for reading and hopefully you have found this helpful. As always, if you have any questions, feedback or comments leave them in the comments section below. Let me know if there is a better way to skin this cat.

Till next time...


  1. Great write up mate.

    i) What are you doing about storing passwords? Is it stored in the job configuration? Is it being printed in the console log? Or is it in your SCM?

    ii) I found that with our Subversion server, with the Always check out a fresh copy in SCM settings was super slow. But the emulate a clean check out worked way better and faster.

  2. Thanks Bruno.
    i) As I am Mr Jenkins on the project, I am the only one that has access to see the config pages. So I have my password in plain text (KISS principle). On our relatively small project this is adequate but probably not for a larger installation.
    ii) Checking code out of TFS with the new (v4+ Aug 2015) Jenkins TFS plugin is pretty fast. No complaints there. It does an excellent job of getting the code out of TFS.

    We had our first developer-requested deployment yesterday to the TEST environment. It completed successfully and took 14min 10sec.

    See above for some updates I have made to this post