Technistas

Matthew D. Laudato writes about software and technology

Comparing Continuous Integration Tools, Part 1

with 6 comments

One of the more enjoyable parts of my job at OpenMake Software is getting to examine and analyze the various build tools on the market. This is partly to see what the competition is up to, and partly to make sure that I can effectively communicate the technical bits with our customers, many of whom have multiple build tools in their environments.

To that end, I recently embarked on a continuous integration tool evaluation. I chose to look at Hudson, an opensource tool commercially supported by Sun Microsystems;  TeamCity, a commercial tool from JetBrains;  Bamboo, a commercial tool from Atlassian; and Mojo, a freeware and commercially supported tool from OpenMake Software. My goal was to compare the tools along several vectors:

  • Installation
  • Configuration
  • Running a simple job
  • Viewing logs
  • Interacting with source control
  • Performing complex distributed build workflows

I decided to break the effort into two parts. The first part, covered in this post, is the ‘getting my feet wet’ portion of the evaluation. I tackled the first four bullets above to get a sense of how the tools were installed and configured, and to see if I could get them each to do something useful. The useful thing was to run a job that spits out the current environment, the equivalent of running the ‘set’ command from a DOS prompt in Windows.

The table below summarizes my findings, and below that, I give some general impressions about the tools and the evaluation process.

PRODUCT: Mojo Bamboo Hudson Team City
FEATURE:        
Version 7.31 2.5.5 1.332 5.1.1
Installation
method
Windows installer Windows installer Executable war file Windows installer
Download
size
50M 84M 27M 268M
Need
License?
Free, unlimited
single-server license
30 day trial Free, unlimited
single-server license
Free, unlimited
single-server license
Installation
notes
Does not ask for default
port as part of install. That is configured once you have started the client.
Server starts as part of installation and gets installed as Windows service.
Start Menu group and icons installed for access to thick and web client.
Asks for default port as
part of install. Does not start server as part of startup. When you do start
the server, does not recognize your port choice.
No issues. Hard to figure
out how to change the default ports.
No issues. Asked for
default port in install wizard. Starts server and build agent as windows
service as part of install and then runs web interface.
Initial
setup
None. If you like the
defaults, then you can create a workflow immediately through the thick client
Asks you to ‘Create a Plan’
as the first activity. Did not like this as it forces me to digest their
meaning of the generic word ‘Plan’
None. If you like the
defaults, then you can create a workflow immediately.
Wants you to create
projects and build configurations but does not define exactly what these are.
Configuring
a simple job (ENVPEEK – prints build server environment vars to build log)
Easy. Create a workflow,
add a ‘Mojo | Execute shell command’ activity, and type in the command
(‘set’).
Difficult. In order to
‘Create a Plan’ you need to go through an 8 step wizard. The second wizard
screen requires you to select an SCM system and a repository location. I had
to give it a repository location from my Subversion server to get past this
screen. Annoying, since for this job I don’t care about SCM. Rest of the
wizard was OK, but way too many steps just to set up a simple job.
Easy. Create a new build
job. Use the ‘Execute Windows batch command’ option and type in the command
(‘set’).
Moderate. Team City asks
you to create a project, which is pretty easy. You then have to create at
least one Build Configuration. There is a web-based wizard that like Bamboo
has an SCM screen, but you can choose to ignore it. You can then choose a
command line Build Runner, in which you specify the ‘set’ command.
Running
a simple job
Easy. Open the workflow,
either in the thick client or in the web interface, and press the run button.
Runs successfully
Moderate. From the Bamboo
home, select the Plan, then select ‘Run Build’ from the Plan Actions menu on
the right. Because of the SCM choice, even jobs that don’t require SCM will
check out from Subversion. Tool is geared for building code projects – does not appear to be a general workflow tool.
Easy. Select the job and
select the ‘Schedule a build’ button.
Easy. From the Project tab,
find the project that you want to run and then click on the Run… button.
Viewing
job logs
Easy. In the thick client,
open the workflow, and go to the History/Trends tab. Select the run that you
want to see and double-click. In the web interface, select the workflow and
submit a query to retrieve the run information. Select the specific run you
want to view.
Moderate. You have to click
on the plan, then the Completed Builds tab, then click on the build you want,
then click on its Logs tab. Lots of drilling down required.
Easy. Click on the Job name
and then select any link from the Build History link.
Easy. From the Projects
tab, click on the Project that you want to view. Select the link for the run
that you want to view.

 

Overall, Hudson and Mojo were the easiest tools to install and use. Hudson definitely takes the cake when it comes to installation, since you don’t have to install it – you just run the executable war file from the command line. Mojo, TeamCity and Bamboo have more traditional installers, of which the Mojo install was the most straight-forward, asking the fewest questions before proceeding with the install. Atlassian’s Bamboo has the most restrictive trial license, but Mojo, Hudson and TeamCity all have a more open approach – you can use them in very useful forms without any cost or special licensing.

Once the tools were installed, I next looked to do any initial configuration, which I define loosely as ‘stuff the tool requires me to do before it lets me do what I really want to do’. On this measure, I again put Mojo and Hudson in the lead, as I didn’t have to do anything – I just went straight to thinking about the job I wanted to run. TeamCity wanted me to create a project and a build configuration, which was fairly easy – but I had to figure out what they meant by ‘project’ and ‘build configuration’. Bamboo was by far the most difficult tool to configure. Any time I see an 8-step wizard just to turn the engine over and get the motor running, my initial response is ‘who wrote this thing’?

Getting the actual job configured was again easy in Mojo and Hudson. The Mojo interface is very straight-forward – you select a machine to run on, and then start adding workflow steps (called activities). There is a large built-in list of activities (around 50) for interacting with commercial and opensource tools. I used the ‘Execute shell script’ activity type to run the set command, and that constituted the entirely of my ‘ENVPEEK’ job. Hudson was also easy to set up. TeamCity and Bamboo were the most painful to set up for actual jobs – you are forced into their concepts, instead of just being able to think about the job at hand. The other comment on both TeamCity and Bamboo is that they are both very ‘source code biased’. By that I mean that they have an implicit assumption that your jobs require interaction with source control. In both tools I was required to specify a location in a source control tool (I used Subversion from Collabnet). Since my initial job was a codeless one, this was annoying.

Running jobs in all tools is fairly easy, as is reviewing the logs – though in Bamboo I did have drill down quite a bit to get to my logs. Going back to my ‘source control bias’ comment, Bamboo needed to check out code from a repository location that I specified – and then ignored it since my inital job was just to run ‘set’.

Next installment: doing actual code builds with each of the tools, and then putting together complex build processes.

Happy Building!

- Matt

About these ads

Written by Matthew D. Laudato

June 7, 2010 at 4:09 pm

6 Responses

Subscribe to comments with RSS.

  1. […]  Check out the full results on technistas.com. […]

    • I’m curious about the products included/excluded from the comparison group. Would you consider Hudson, Bamboo and TeamCity to be technical leaders who you wanted to measure up against? Would the lack of a product in this list , such as Anthill Pro, therefore indicate that you did not view it to be in the top 3 leaders.

      Tom Canova

      July 5, 2010 at 7:29 pm

      • Tom – my focus was on products that are readily available for trials, rather than on products that require interaction with a company sales force to trial. AHP is certainly an important product in the CI market.

        Matthew D. Laudato

        August 7, 2010 at 11:48 am

  2. Good article. Where’s part 2?

    I highly recommend dating articles. 10 years from now this link may be still around, and may either be totally irrelevant, or very useful. W/o a date, one can’t even guess. Metadata, metadata!

    Charles Roth

    February 4, 2011 at 4:34 pm

  3. Good point on the dating of the article. As you might have guessed, other obligations (like new jobs) have derailled my plan to do part 2. Expect to see me post on other topics of general tech interest – including web services, APIs and integrations.

    – Matt

    Matthew D. Laudato

    May 6, 2011 at 6:42 pm

  4. hi matt,

    very good article! I’m going to compare Bamboo with TeamCity. I’m actually running Apache Continuum but this software seems not to be maintained anymore. There isn’t support for any other build type (rake, .net,…), either. So I installed JetBrains TeamCity and I’m very impressed by the features (we are already running other development tools from JetBrains: PHPStorm and RubyMine).

    Do you think it’s worth to install JIRA’s Bamboo? We’re using JIRA for Issue Management. Would you recommend Bamboo in this case?

    The current Version of Bamboo is 5.X
    The current version of Teamcity is 7.0.X. I found some interesting stuff here: http://blogs.jetbrains.com/teamcity/2012/04/24/ (scroll to the paragraph “build chain visibility”).

    thx
    golbie

    golbie

    August 1, 2012 at 5:38 pm


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: