Posts Tagged ‘TeamCity’
Comparing Continuous Integration Tools, Part 1
One of the more enjoyable parts of my job at OpenMake Software is getting to examine and analyze the various build tools on the market. This is partly to see what the competition is up to, and partly to make sure that I can effectively communicate the technical bits with our customers, many of whom have multiple build tools in their environments.
To that end, I recently embarked on a continuous integration tool evaluation. I chose to look at Hudson, an opensource tool commercially supported by Sun Microsystems; TeamCity, a commercial tool from JetBrains; Bamboo, a commercial tool from Atlassian; and Mojo, a freeware and commercially supported tool from OpenMake Software. My goal was to compare the tools along several vectors:
- Installation
- Configuration
- Running a simple job
- Viewing logs
- Interacting with source control
- Performing complex distributed build workflows
I decided to break the effort into two parts. The first part, covered in this post, is the ‘getting my feet wet’ portion of the evaluation. I tackled the first four bullets above to get a sense of how the tools were installed and configured, and to see if I could get them each to do something useful. The useful thing was to run a job that spits out the current environment, the equivalent of running the ‘set’ command from a DOS prompt in Windows.
The table below summarizes my findings, and below that, I give some general impressions about the tools and the evaluation process.
| PRODUCT: | Mojo | Bamboo | Hudson | Team City |
| FEATURE: | ||||
| Version | 7.31 | 2.5.5 | 1.332 | 5.1.1 |
| Installation method |
Windows installer | Windows installer | Executable war file | Windows installer |
| Download size |
50M | 84M | 27M | 268M |
| Need License? |
Free, unlimited single-server license |
30 day trial | Free, unlimited single-server license |
Free, unlimited single-server license |
| Installation notes |
Does not ask for default port as part of install. That is configured once you have started the client. Server starts as part of installation and gets installed as Windows service. Start Menu group and icons installed for access to thick and web client. |
Asks for default port as part of install. Does not start server as part of startup. When you do start the server, does not recognize your port choice. |
No issues. Hard to figure out how to change the default ports. |
No issues. Asked for default port in install wizard. Starts server and build agent as windows service as part of install and then runs web interface. |
| Initial setup |
None. If you like the defaults, then you can create a workflow immediately through the thick client |
Asks you to ‘Create a Plan’ as the first activity. Did not like this as it forces me to digest their meaning of the generic word ‘Plan’ |
None. If you like the defaults, then you can create a workflow immediately. |
Wants you to create projects and build configurations but does not define exactly what these are. |
| Configuring a simple job (ENVPEEK – prints build server environment vars to build log) |
Easy. Create a workflow, add a ‘Mojo | Execute shell command’ activity, and type in the command (‘set’). |
Difficult. In order to ‘Create a Plan’ you need to go through an 8 step wizard. The second wizard screen requires you to select an SCM system and a repository location. I had to give it a repository location from my Subversion server to get past this screen. Annoying, since for this job I don’t care about SCM. Rest of the wizard was OK, but way too many steps just to set up a simple job. |
Easy. Create a new build job. Use the ‘Execute Windows batch command’ option and type in the command (‘set’). |
Moderate. Team City asks you to create a project, which is pretty easy. You then have to create at least one Build Configuration. There is a web-based wizard that like Bamboo has an SCM screen, but you can choose to ignore it. You can then choose a command line Build Runner, in which you specify the ‘set’ command. |
| Running a simple job |
Easy. Open the workflow, either in the thick client or in the web interface, and press the run button. Runs successfully |
Moderate. From the Bamboo home, select the Plan, then select ‘Run Build’ from the Plan Actions menu on the right. Because of the SCM choice, even jobs that don’t require SCM will check out from Subversion. Tool is geared for building code projects – does not appear to be a general workflow tool. |
Easy. Select the job and select the ‘Schedule a build’ button. |
Easy. From the Project tab, find the project that you want to run and then click on the Run… button. |
| Viewing job logs |
Easy. In the thick client, open the workflow, and go to the History/Trends tab. Select the run that you want to see and double-click. In the web interface, select the workflow and submit a query to retrieve the run information. Select the specific run you want to view. |
Moderate. You have to click on the plan, then the Completed Builds tab, then click on the build you want, then click on its Logs tab. Lots of drilling down required. |
Easy. Click on the Job name and then select any link from the Build History link. |
Easy. From the Projects tab, click on the Project that you want to view. Select the link for the run that you want to view. |
Overall, Hudson and Mojo were the easiest tools to install and use. Hudson definitely takes the cake when it comes to installation, since you don’t have to install it – you just run the executable war file from the command line. Mojo, TeamCity and Bamboo have more traditional installers, of which the Mojo install was the most straight-forward, asking the fewest questions before proceeding with the install. Atlassian’s Bamboo has the most restrictive trial license, but Mojo, Hudson and TeamCity all have a more open approach – you can use them in very useful forms without any cost or special licensing.
Once the tools were installed, I next looked to do any initial configuration, which I define loosely as ‘stuff the tool requires me to do before it lets me do what I really want to do’. On this measure, I again put Mojo and Hudson in the lead, as I didn’t have to do anything – I just went straight to thinking about the job I wanted to run. TeamCity wanted me to create a project and a build configuration, which was fairly easy – but I had to figure out what they meant by ‘project’ and ‘build configuration’. Bamboo was by far the most difficult tool to configure. Any time I see an 8-step wizard just to turn the engine over and get the motor running, my initial response is ‘who wrote this thing’?
Getting the actual job configured was again easy in Mojo and Hudson. The Mojo interface is very straight-forward – you select a machine to run on, and then start adding workflow steps (called activities). There is a large built-in list of activities (around 50) for interacting with commercial and opensource tools. I used the ‘Execute shell script’ activity type to run the set command, and that constituted the entirely of my ‘ENVPEEK’ job. Hudson was also easy to set up. TeamCity and Bamboo were the most painful to set up for actual jobs – you are forced into their concepts, instead of just being able to think about the job at hand. The other comment on both TeamCity and Bamboo is that they are both very ‘source code biased’. By that I mean that they have an implicit assumption that your jobs require interaction with source control. In both tools I was required to specify a location in a source control tool (I used Subversion from Collabnet). Since my initial job was a codeless one, this was annoying.
Running jobs in all tools is fairly easy, as is reviewing the logs – though in Bamboo I did have drill down quite a bit to get to my logs. Going back to my ‘source control bias’ comment, Bamboo needed to check out code from a repository location that I specified – and then ignored it since my inital job was just to run ‘set’.
Next installment: doing actual code builds with each of the tools, and then putting together complex build processes.
Happy Building!
– Matt
Technistas 