Tuesday, July 16, 2013

Continuous Integration - Hybrid Cloud with Eucalyptus, AWS & Jenkins

Recently I forked the Jenkins EC2 plugin to address bugs with its support of Eucalyptus and to demonstrate use of both public and private clouds for continuous integration. This was neither an academic exercise or a chance to use "hybrid cloud" in a sentence;  Eucalyptus users are already running CI across Eucalyptus and AWS. 

These users benefit from lowered development costs and greater control / consistency of the dev and test environment. It's the kind of stuff lots of development teams would benefit from. Unfortunately their efforts are locked up because their organizations inhibits contributing back. (*cough* legal department *cough*).....So Vic Iglesias and I built a version anyone can use and in this post I'll explain how you, too, can take advantage of it. 

By the way, if you're just here for the show check out this short video (3:30) demonstrating the updated plugin in action.

Jenkins Primer:
Jenkins is possibly the most popular continuous integration solutions available as open source today. It basically uses a master-slave model by which the Jenkins server (master) may dynamically scale up/down worker nodes (slaves). The EC2 plugin allows the Jenkins server to launch and slaves on the AWS cloud, giving a development team seemingly unlimited capacity.

Using the Jenkins EC2 plugin is very straight forward. Essentially it lets you tell Jenkins about the clouds you want to use. Then you describe one or more images (ami's) Jenkins may launch as needed. There's considerably more to the plugin and I encourage you to visit the Jenkins EC2 Plugin page for more info. Or hit up the cool folks at CloudBees; they offer commercial support with Jenkins Enterprise. The subscription nets you enhanced features, increased scalability and support of major plugins.


Preparation

If you already use Jenkins and have access to a Eucalyptus cloud then this section will be a breeze. On the other hand, if you're not using Jenkins or Eucalyptus then you'll need these resources to play along:
  • Eucalyptus Get Started - This page describes different options for getting your own Eucalyptus cloud (or account) going.
  • Eucalyptus Documentation - Exactly as advertised.
  • Meet Jenkins  - This page describes Jenkins and provides installation options.
  • Jenkins: The Definitive Guide - An excellent O'Reilly book covering all things Jenkins. Download the pdf, or purchase online.
I assume you already use AWS and have your credentials handy.

Installing the updated EC2 Plugin
This is pretty straight forward stuff. Remember, if you have a version of the EC2 plugin installed then be sure to back it up. The updated plugin should work as well, if not better than version you're running. But no need to take chances :-)

1) Get the updated plugin
     Download the version used in the demo
2) Manually install it
      Just copy it into your [JENKINS_HOME]/plugins directory
3) Restart Jenkins

Add a Eucalyptus Cloud
If you already use Jenkins and AWS then these next two sections will get you setup for a hybrid cloud. If you aren't already using AWS in your Jenkins environment then just know that the setup is largely the same for Eucalyptus and AWS. Here's how to add a Eucalyptus Cloud to your Jenkins environment:

1) Navigate to Jenkins -> Manage Jenkins -> Configure System
2) In the Cloud section, click "Add a new cloud"
3) Select "Eucalyptus"

4) Fill in the fields for your Eucalyptus cloud / account

    CAUTION: You must include a trailing "/" to the Eucalyptus URLs, or you will receive the dreaded "Login Failure: all modules ignored" message

 
5) Click "Test Connection" to validate your settings

Add an EMI
Jenkins now needs to know which image(s) to launch when using this cloud. You can use just about any image in your Eucalyptus cloud. Some organizations choose to use a base operating system image that gets software injected upon startup using init scripts or configuration automation tools like Ansible, Puppet and Chef. Others like to bake an image with all the necessary software and tools. Both models work just fine for this example, so figure out which image you want to use and follow these steps:

1) In the AMI section of your Eucalyptus cloud definition click "Add"
2) Fill in the appropriate fields - as shown in this example:

3) Click Save to save the Cloud and EMI settings.

In the demo video you may notice the "Usage" setting is "Utilize this slave as much as possible" and the "Idle Termination" was disabled. One of the plusses to running instances on Eucalyptus is that it costs the same whether you run it for a minute, an hour or days at a time ($0.00). Leaving it up means never taking a hit on slave startup time.

Testing Your Hybrid CI

With Amazon and Eucalyptus clouds added to Jenkins you can easily configure jobs to build in either environment, or in a specific cloud if required. One of the easiest ways to ensure jobs run in the correct environment is to apply labels to the images (EMI / AMI) defined in Jenkins. For example let's say you setup both an AWS and Eucalyptus cloud, each with one image. Then you assigned labels indicating what instance size the image will launch as and a label for which cloud. The labels you'd set for the AWS AMI might be "M1.Medium" and "AWS". The labels for your Eucalyptus EMI would be "M1.XLarge" and "EUCA". You can, of course, define many images and many labels.

Now to tie a job to one cloud or the other you'd simply enable the "Restrict where this project can run" setting, then set the appropriate labels. Using the implied labeling scheme above, here are example restrictions you might set:
   Label Expression          Meaning
AWS                               Only run this job on an AWS instance. No instance size restrictions
EUCA                             Only run this job on an Eucalyptus instance. No instance size restrictions 
M1.Medium                    Only run this job on medium instance size. Any cloud will do.
M1.Medium -> EUCA    Only run this on a medium instance, running on your Eucalyptus Cloud.

Now you multiple clouds, a labeling structure and some rules for directing jobs. Fire off a build and watch the magic happen.

What's next?
It's our intention (Vic and me) to contribute back the Eucalyptus specific fixes. We've already reached out to the EC2 plugin maintainer and hope to have a pull request ready to go shortly. Once that's complete I'll update this blog post and you'll be able to use the built-in mechanism to update the plugin.

Hope you found this post helpful. Feel free to join me on the #eucalyptus IRC channel or Eucalyptus mailing list with any questions.

Edit: Added Caution notice regarding trailing "/".