On our current project we are aiming to reach that goal of continuos delivery. There are a lot of things you need to get right to achieve that but one of the more important ones is a functional deployment build pipeline where a defined version of the source code is being built and pushed through various stages. The later the stage the more confidence you should have in your software and you should feel more sage to actually deploy it to production if this is requested by the business. The whole process should be automatic, except for some stages where one might want to trigger that stage manually.
A really nice build server which supports build pipelines out of the box is the commercial Go build server from ThoughtWorks. It was built with all the ideas of CruiseControl (an open source build server partially developed by ThoughtWorks as well) in mind but going a step further and applying all the latest real life experiences from projects on how to build high quality software.
Today I want to talk about how to build a deployment pipeline in Jenkins as this is a commonly used open source build server. It’s possible to simulate a deployment pipeline with Jenkins although it’s not as optimal as if it’s really supported out of the box from the build server. I’ll try to provide a set of steps which are required to reach our goal:
Define your pipeline stages
Before you can start building your pipeline you need to define the stages that every build should pass in order to be promoted to a release finally which can then be deployed as a final step maybe. Which stages you need will totally depend on your project but a general rule of thumb is that you want to have a quick commit build stage which checks out (or clones) the version of the software you want to build from your favorite (D)VCS, compiles the production code and tests and then runs all the fast unit tests (including stuff like bdd frameworks, functional in memory tests which might use headless browers, jasmine tests, etc. whatever is fast enough to execute) to get a first solid confidence in the quality of the software you build.
A next step is to run more high level tests on your software. This might be a set of slower running graphical UI tests for a desktop application which might test some user journeys or if you are developing a web application like many of us are doing these days you probably want to deploy your binary to a webserver in order to run some user acceptance tests against these to actually validate that you meet the requirements of your stories. As these tests usually run relatively slow, I would like to take the opportunity here and highlight the fact that it makes a lot of sense to not write loads of them which are more or less on a unit test level but instead write a user journey which covers many acceptance criteria at once. You can then also later extend these journeys to cover more requirements. This greatly improves performance and maintainability.
So given the stages explained above, your pipeline might look like the one below, where =A=> means that the following stage is triggered automatically, and =M=> means it’s triggered manually:
commit_build =A=> deploy_to_jetty =A=> run_selenium_user_journey_uats =A=> tag_release_and_publish_to_nexus =M=> deploy_to_qa_env =M=> deploy_to_production
Major principles
We only build the binaries for the software once. You use the compiled binary which is than passed to all the different stages in the pipeline. if you are using a scripting language which doesn’t necessarily compile to some byte code you still want to be sure that you only use on produce set of files or maybe use a tool which can package your script files, even if it’s just a simple zip which you can then pass around through the stages.
Another important thing, at least for me, is to have a clear separation between production and test code. I really can’t emphasize this enough. Things are really getting messy if you deploy test fixtures to production which are suddenly visible in production for whatever reason. The tests should use production code but not the other way around. This becomes somewhat harder when you have to deploy your testcode to let’s say some kind of servlet container in order to run tests there. If you have to do stuff like that I’d strongly recommend to build a separate deployment with the test sources or have a different binary/jar which is deployed as well. There might be other opinions how strict one should go but I’d clearly vote for a clean separation between tests and production code.
Implement the pipeline
To actually implement the pipeline use the latest version of Jenkins and install the Build Pipeline Plugin. Please follow then the steps given on the page of the plugin to actually implement the pipeline we defined above. That page has good explanations for it. Another good detailed explanation with an example can be found here.