February 23, 2016
In practical terms, continuous integration, or CI, is the process of taking completed code (feature, bug fix, or otherwise) and integrating them into the main repository on a regular basis.
The concept isn’t foreign to most experienced developers – once a feature is completed, the code is pushed back into the repository – either the main development branch, or a feature branch. The next step involves work that many haven’t experienced. Upon integrating/merging the code into the repository, a build server pulls the changes and builds the project, testing it for functionality, often through unit tests. Depending on the project, the process of building or compiling on the server may also identify newly introduced issues not otherwise noticed by the developer (in theory testing “works on my machine” syndrome).
This process allows teams to quickly identify problems that might otherwise go unnoticed for some amount of time. Quickly identifying issues means they can be resolved before a system of dependencies become built around them – something that becomes much more difficult to unwind as more code is added.
Because CI is often coupled with proper unit and functional testing, the build server is able to verify that the tested functionality hasn’t changed since the last successful integration. Furthermore, CI can be used to test code in numerous build and runtime environments. Some practical examples could include testing builds against various system libraries and configurations, an array of browsers and versions, or a suite of test phones and tablets spanning sizes and operating systems. Here the developer makes large efficiency gains as repetitive tasks are handled by the server.
While investigating various CI services, the mobile team here at Valtech consider several factors that influenced our final decision. Among them,
Platform support
We needed a system capable of building for both iOS and Android (implicitly, Mac OS X ).
Ease of use
Is it relatively easy to configure the system and keep it running with minimal intervention?
Customization
Can we customize the build process outside of modifying the application project files?
Availability
Some services are cloud-based, while others are self-hosted (possibly only available on the local network).
Cost
Depending on the company size, number of repositories, or concurrent builds, price may become prohibitive
Based on this list we evaluated the following CI tools and services:
- Jenkins (http://jenkins-ci.org/)
- CircleCI (https://circleci.com/)
- Bitrise (https://www.bitrise.io/)
- Buddybuild (https://buddybuild.com/)
- and many others along the way
Jenkins
Jenkins is an open source CI server that has been one of the go-tos for a few years (more if you count its roots with Hudson). It finds favour in the development community because of its versatility and platform support (Mac, Windows, Linux, BSD). With myriad publicly-contributed plugins, Jenkins is quite customizable.
Let’s see how it stacks up against our needs:
Platforms
As mentioned above, Jenkins runs on Mac OS X, Windows, Linux, and BSD. This meets our first requirement – Mac OS X.
Ease of Use
Once installed and with jobs successfully running, Jenkins is quite maintainable. Once running, it just keeps going. Coupled with tools like Fastlane, (and soon Android) it makes the build process very easy to configure.
There’s a caveat: Getting Jenkins running in the first place can be difficult. On Macs, at least for the purpose of building iOS projects, there are a few special steps that must be taken in order to allow Xcode to operate properly. Kunstmaan Labs has a good step-by-step guide to installing Jenkins and configuring Fastlane.
Besides setup issues, one important factor is your team’s ability to maintain a server, should anything happen. Some developers may feel completely comfortable caring for a server, while others completely lost. This is an important factor to consider when deciding whether or not a self-hosted solution like Jenkins is right. Generally, our experience with Jenkins running on Macs has been positive, and we have enough expertise to handle administration should an issue arise.
Customization
Jenkins is one of the most customizable CI tools available – certainly more so than popular cloud-hosted tools. Out of the box, it provides a simple build template: Provide the repository’s URL, branch(es) to build, frequency the build should run, and a short shell script used to actually build the project. A number of plugins are available for both Xcode and Android projects to ease build configuration. Since we use Fastlane (https://fastlane.tools/), the command line is sufficient, but it doesn’t end there. If you need more functionality, Jenkins can do things like send status emails, create build reports, upload binaries for testing, perform specific unit and functional tests (including software simulators and actual hardware), post notifications, deploy to staging and production, and even turn on a Hue light.
Some of these features are bundled with hosted CI services, but that’s where the extensibility ends with them. Because we value being able to tweak our build configurations and perform extra tasks, Jenkins gets a bit of extra credit while others fail miserably.
Availability
This is rather important – are you able to access the build machine when you need to? Self-hosting a Jenkins server could mean it sits behind a firewall, blocking your access while out of the office. The typical work-around involves port forwarding to the Jenkins server, or requiring the team to VPN into the network. Another less common option is leasing a Mac server in a datacenter such as MacMiniColo.net. Because we already have a spare Mac Mini in the office, that will be our Jenkins box, so the colocation idea is out. Deciding on how to connect from the outside is yet to be determined.
Cost
Jenkins is free. When you look at the available features, you get a lot more than what many paid services offer. Don’t be fooled that the free price tag means it’s completely free. The hidden cost of maintenance, upgrades, and downtime are extremely important. Though Jenkins is easy to use once running, there are times when an engineer must spend time doing routine work, which comes at a cost of their time. Hosted services don’t incur these costs (or presumably, are baked into the monthly or per-user fees). Given a twelve-month timespan, it may be a wash.
Conclusion
As we’ve seen, Jenkins offers us many benefits and a few drawbacks. On one hand, we have a very powerful, extensible system. On the other, the burden of maintenance is on our team and potentially limited access is a concern.
CircleCI
CircleCI bills itself as an all-in-one build management, testing, and deployment solution for mobile and web. A quick overview of important points:
- You’ll have to decide if the pricing scheme is sustainable as your project/org scales.
- It is completely built around GitHub integration; there is no support for BitBucket, self-hosted Perforce, or anything else currently.
- Build inference and configuration are not as smart as we had hoped.
Let’s take a look at the feature breakdown.
Platforms
Supports web, Android and iOS. That said, the support for mobile seems immature, especially for iOS as of this writing.
Ease of Use
We wanted to set it up to build a complete, extant project. In our line of business, a lot of what we do is taking over work on someone else’s codebase. We often don’t have the luxury or freedom to radically restructure an app or its build process and requirements. So to realistically test this kind of scenario, we decided to fork a project on GitHub.
The process of setting up a project on CircleCI was actually very smooth and easy. It lets you pick any repo you have access to from your GitHub account, you hit the “Build project” button, and it imports everything and tries to detect reasonable build settings.
First we tried Zebra Crossing, the famous barcode scanning software from Google. After a couple hours of fiddling, we decided to give up and try another project, so as not to waste too much time. Next we tried the DuckDuckGo searchapp for Android. That ended up not working out either. Finally, we tried the 2015 Google IO schedule app. This we ultimately got working, though we had to lean on a few hacks and workarounds that we wouldn’t feel comfortable leaving in production code.
First, the unit tests for the server component were failing for some reason. Most likely it’s related to the fact that they’re running on a container, or a slight difference in some library. We were mostly interested in testing its ability to build a mobile app, so we just disabled those tests. Second, CircleCI’s build settings inference didn’t set it up so that the app was actually built. We had to create a circle.yml build config file in the repo, and add a manual gradle command to actually generate the APK. Third, apparently Android builds under Gradle are a little cavalier with threads and memory, which was causing the process to be killed on the container. We found a StackOverflow answer dealing with this, and found that limiting the number of threads fixed the dying builds. Fourth, getting it to actually collect and link the build artifact was a little bit fiddly, and required us to spam a bunch of commits to our branch as we searched for the correct incantation in the circle.yml file. In the end we were unable to get it to move or rename the apk, so anyone looking to download it later would have to drill down through several layers of sub directories and manually rename the file to something other than “android-debug.apk”.
Customization
CircleCI is intimately tied to GitHub. You create an account by logging in with GitHub, and builds are all tied to your GitHub commits. If your workflow is built around GitHub, then this is all incredibly simple and convenient. It also supports an array of secondary technology and service integrations, to hook into your toolchain and workflow, including chat and email, analytics, and deployment services.
CircleCI analyses your code and project config files, and tries to figure out how to make it all just work for you, automagically. They also offer you fairly comprehensive manual configuration options: there is a way to enter custom, private, encrypted environment variables which will be available to the build scripts; there is a simple webform where you can enter manual commands for different build steps; and there is a highly customizable YML file you can include in the root of your repo to override and specify almost any aspect of the build apparatus. Builds can be customized based on regex matching against your git tags.
If you have the requisite configuration files set up correctly in your repo, it will automatically download build dependencies for many of the most common systems package/dependency management systems:
- CocoaPods
- Homebrew
- Ruby Gems
- Maven
- Android SDK/tools
- …and others.
CircleCI will run any tests it detects (or that you specify) for each build. Because you are sequestered in your own lightweight container, you can install any libraries or software you need. If any test fails, it considers this a failed build, and will notify you of your stupidity. Beyond standard unit tests, CircleCI supports integration with e.g. Appium to do UI testing.
Cost
CircleCI is based on the idea of containers, which are basically lightweight Ubuntu GNU/Linux (or OSX) virtual machines. Builds can be parallelized across containers, allowing multiple simultaneous builds, or a single build conducted in parallel, or some compromise of the two. For example, with four containers, you can do four simultaneous builds, we build split across four containers, or two simultaneous builds, each split across two containers.
Each time you push code to one of the github repos you’ve added to your CircleCI account, a build is triggered. So if you or your team push multiple commits in quick succession, builds will be queued up when you run out of available containers. With the basic service, which allows you to have unlimited users and projects/repositories, you get a single container for free. Additional containers are $50 per month.
If you’re an individual or on a small team, and your builds are fairly quick, this is probably very reasonable. As your needs scale up, this could start getting expensive quickly, but this may be offset by the larger engineering and devops budgets typical of larger organizations. You’ll have to decide where that sweet spot is for you, and assess accordingly.
Conclusion
We’re sure a lot of our difficulties are due to ignorance on our part; with the knowledge we gained getting the iosched project working, we could probably go back and get zxing and dukgo working. On the other hand, we were hoping CircleCI’s build settings inference technology would have been a little smarter. CircleCI really feels like it’s got a lot of potential, but it’s maybe a bit young. Or at least that’s true of the mobile development experience. We, of course, didn’t put it through its paces, but the support for web deployment and CI seems much more mature and complete.
In the end, the fact that it’s so intimately tied to GitHub probably means we won’t go with CircleCI. Not that there’s anything wrong with GitHub (it’s our preference, in general). But even if we decided to move all our current repositories over from BitBucket, what do we do when we inherit a large code base from a customer that lives on their internal SVN servers? Migration and interop between git and SVN is not the strong suit of everyone on our team, and for a small company that doesn’t have a dedicated build engineer or devops, the overhead and headache of trying to make it work seems prohibitive right now.
Bitrise
Overview
Bitrise prides itself as a “mobile continuous integration and delivery service for your whole team, with dozens of integrations for your favourite services”. Their goal is to allow users to craft powerful workflows for every development stage including testing, deployment, signing, packaging and delivery.
Platforms
Bitrise has support for iOS, Android, and Xamarin builds. There is a web portal with a UI that allows you to manage your workflows in a visual manner. There are also command line tools that let you accomplish the same tasks. The tools for the iOS build seem to be fairly mature; however, the Android environments are in Beta release at the moment.
Ease of Use
Bitrise provides out of the box integration with Github and Bitbucket. It takes about 15 minutes to get up and running once you select the repository and branch, set up the SSH keys, and project build configuration. We were able to get the iOS project we built for the Grammys up and running fairly quickly. However, there were some build errors on the Grammys Android project that had to do with the Gradle specific permissions. We contacted their support team, but were unable to get these issues resolved so we did not get a working Android build with Bitrise.
Customization
The platform offers all sorts of different integrations which are really useful including:
- Fastlane
- Slack notifications
- Deployment to iTunes and Google Play
- Fabric/Crashlytics deployer
- Unit Testing services
- Scripting support
Availability
The entire service resides in the cloud and is accessible from any browser or command line.
Cost
There are three different tiers for the product:
Voyager – Free
- 2 team members
- 200 builds / month
- 1 concurrent build
- 10 min build duration
Galileo – $29 per month
- 5 team members
- Unlimited builds
- 1 concurrent build
- 30 min build duration
Atlantis – $129 per month
- 15 team members
- Unlimited builds
- 2 concurrent builds
- 45 min build duration
Conclusion
While it appears Bitrise has a promising future in the continuous integration and delivery market, the current product is lacking and therefore not a good candidate for our purposes.
Buddybuild
Buddybuild is a cloud-hosted CI for both iOS and Android platforms. Though it’s a relative newcomer to the game, Buddybuild has started off strong – offering a number of features and third party integrations developers are bound to find useful.
Platforms
Buddybuild supports only Android and iOS projects, but keep in mind this also includes React Native, PhoneGap, and Ionic development frameworks. The service also supports common dependencies such as CocoaPods, Carthage, Maven, and npm.
Ease of Use
Configuring Buddybuild is relatively straightforward – provide a git repository URL, perform a few configurations as needed, invite testers, sync provisioning profiles (iOS), and be on your way. Whenever a new commit is pushed to the repository, Buddybuild will kick off a new integration build and send the results to your testers. Of particular interest to iOS developers is Buddybuild’s ability to automatically manage devices and provisioning profiles. Their web interface is easy to understand and works well.
Customization
Buddybuild offers a significant number of configurations to support building a variety of project configurations. Out of the box, Buddybuild offers build notifications (email, Slack, and HipChat), unit testing, environment variables, and binary versioning. Among iOS project customizations, you are able to select a version of Xcode (including betas), build scheme & configuration, and signing identities.
While those features are great, they are limiting if your needs fall outside a basic build and deploy strategy. Because Buddybuild is running integrations on their servers, your access to the system is limited. You don’t have access to installing specialized software tools, building for other platforms, running custom shell scripts, or interacting with unsupported third party integrations. Depending on your project’s needs, this could be a dealbreaker.
Availability
Buddybuild is available anywhere you have an internet connection.
Cost
At the time of this writing, Buddybuild is free, but the service is expected to be paid.
Conclusion
Buddybuild makes continuous integration and deployment a breeze for iOS and Android developers. It offers enough customization to be useful where it matters – notifications, builds, testing, and binary distribution. Where it lacks is the ability to do anything outside of typical mobile build cycle. While this is to be expected with hosted solutions, it’s not the choice for every team.
And the Winner Is…
We’ve had the opportunity to take a look at a variety of CI options. While hosted CI solutions offer tremendous ease of use, yet lack customizability. On the other hand, powerful tools such as Jenkins offer immense flexibility, but aren’t always easy to use or administer. The route that is best for your team will depend on the tradeoffs you’re willing to take. If you’re looking for something that requires the least amount of configuration and administration, Buddybuild is the clear winner. If you need flexibility and advanced control such as running Fastlane or shell scripts during the build process, Jenkins is the tool for the job.
The decision wasn’t easy, but ultimately we chose Buddybuild. We appreciate how easy it is to set up new projects and invite new testers. Our CI build needs are not complex and Buddybuild gets the job done with the least amount of work and maintenance. It just works. While we strongly considered Jenkins, we decided the power it offers isn’t required, nor are we interested in administering our own build server. We’re not ruling it out entirely should our needs change.
