iOS Continous integration: Xcode Server, Jenkins, Travis and fastlane

Twice in the last year my team was looking at the tremendous task of choosing and sticking with a continuous integration(from now on CI) server to meet our needs.

With some of my previous CI experience and wishes expressed by our iOS developers, here is what we needed such a server to do for us:

  • Build and sign all flavors of our app.
  • Upload our app-store flavor to iTunes Connect
  • Upload ipa's,dSYM's and changelogs to HockeyApp
  • Run Unit and UI tests continuously on release and develop branches
  • Build each Merge Request (MR) and report test results back.
  • Continuously attempt building and signing to make sure we didn't introduce issues.

As far as command line tooling goes there are a few contenders:
xcodebuild - developed by apple, used under the hood by Xcode to build and test. Difficult to remember sometimes but very configurable.

fastlane - in fact not a single tool but a few tools which allow building, testing, uploading to iTunes Connect, provisioning profile management, screenshot taking, dsym upload/download to major crash reporting platforms.

xctool and others - Among others I mean tools like the nomad tools which have been deprecated or have dwindling support, or become abandoned in time. Just because Facebook is behind a tool, it should bring no comfort that it is going to be maintained.

The test of time would later reveal the winner to us(Keep reading).

At the same time tooling is nothing without a server for it to run on continuously. There were a few CI servers being suggested and a choice had to be made in order to decide where to invest time.

The usual contenders are:

  • TravisCI/CircleCI - Hosted severs, free for open source projects, accessible from anywhere, robust. Less configurable than Jenkins, only allow integration with Github. Pricey for private repositories.
  • Xcode Server - Highly integrated with Xcode, in fact can only work with Xcode, developed by apple, hopefully would need very little tinkering.
  • Jenkins - The old king of CI servers, a lot of plugins, a lot of integrations, requires some tinkering and maintenance but very robust

The rumble:

1.TravisCI

A simple CI server, built around Github. Important: If your code is not on Github, you're out of luck. Many companies use alternatives to Github like Bitbucket or Gitlab due to pricing or the need to host their own code for security reasons. But say your code is on Github and you are willing to pay the monthly fee for Travis/Circle. Here is the Travis page of popular iOS library, AFNetworking.
Travis's UI is focused around making builds and Github Pull Requests. It provides out of the box simulators and Xcode images(you can specify which Xcode version you want to use for building). Travis CI is still in beta with running jobs periodically which is surprising as Jenkins had this for a very long time.

Pros:

  1. Out the box, easy to setup.
  2. Configurable, Xcode and tooling is maintained for you.
  3. Great integration with Github
  4. Well documented and widely adopted in open source community
  5. Nice user interface.

Cons:

  1. Doesn't work if your code isn't on Github.
  2. Pricey or limited build capacity for paid plans.
  3. A lot less configurations and plugins compared to Jenkins.

Verdict: In the end, Travis's and for that matter CircleCI's lack of integration with Bitbucket and Gitlab were a deal breaker. Our code was not on Github. Even if it were, our necessity to run builds and tests continuously would simply require too much build time(and time is money, literally, CircleCI charges for build minutes). They also lacked the ecosystem of plugins that made Jenkins so desirable(be it the plugin that allows you to create a dashboard, or the one that retries a build 3 times if it failed).

2. Xcode Server

A CI server created by Apple in order to assist iOS (and macOS?) developers with their CI needs. Xcode Server(XCS) calls them bots, Jenkins calls them jobs but they are the same thing, scheduled tasks. Xcode Server bots show in our project navigator like this:

And you can decide if you want your bot to run unit or ui tests, analyze or even generate a ipa ready to be installed(archive action):

After an integration has been executed you get this overview which is the best part of XCS:
And this test overview.

There is roughly one single guide on the whole world wide web about how to work with XCS and that is Honza dworzky's. If you'd like to know more about how to setup XCS, read it. And not by accident, Honza is now working at Apple on developer tooling.

Xcode server might know a lot about Xcode, but it knows very little about continuous integration. It can't run two jobs(bots) or more in parallel, it fails to pick up git submodule changes(need a custom script just for this). When you have a continuous running task that is not necessarily related to Xcode(for example downloading the localizable.strings translations every day) XCS can't do it without first building or testing something. When XCS fails to build, Apple's simplicity philosophy plays against it and XCS's errors are simply too vague (/usr/bin codesign failed)

The really bad part is that everything outside building and testing in XCS you have to create yourself, using the limited api that XCS has. No integrations with Gitlab/Bitbucket/Github, no plugins to communicate with apis no job pipelines. In fact the only thing that XCS can do is run unit+ui tests on devices easily. Even then, when it fails, it doesn't say why, leaving you with the burden of figuring it out.

Pros:

  1. Integrates very well with Xcode
  2. Everything works out the box
  3. Has a web dashboard for monitoring your bots.
  4. Can create installable ipa's and you can install them in the web dashboard.

Cons:

  1. Has issues with the most basic tasks like checking out source code when you have submodules.
  2. Cant add multiple builder nodes and bots are slow to run, impossible to scale up.
  3. Too focused around XCode specific tasks, doesn't allow for other kinds of jobs.
  4. Almost non-existant third-party integrations.
  5. Impossible to debug build issues.
  6. Only good for unit and ui testing.

Verdict:
Apple clearly didn't set out to build a fully fledged CI server. Maybe it was just targeting small teams and indie developers. After our extensive experiments with XCS it looks like even small teams would have tremendous difficulties in maintaining, keeping it alive, debugging it and getting feedback from XCS in reasonable time.

When you don't encounter such problems, XCS is simply not able to offer more than building and testing. Uploading to iTC, provisioning profile download and installation, uploading to hockey app(or fabric), reporting test results back to our git repository manager,other CI tasks are out of reach for XCS and would be much more difficult with XCS than our final alternative.

3. Jenkins + fastlane

This one is a little bit different because an introduction is needed as to what is fastlane and what it does.

Fastlane is a set of command line tools, that allows a developer to build his app:
fastlane gym --scheme YourSchemeName

Or to run unit tests:
fastlane scan --scheme YourSchemeName

To run UI Tests on a specific device
fastlane scan --scheme UITestsScheme --devices 'iPhone 5s'

Or even to upload his app test flight:
fastlane pilot upload --ipa PathToIpa

These four simple actions allow us to achieve the goals set out in the introduction of this article. Fastlane is able to do much more (provisioning profile download and installation for example) but these commands will be enough for now.

Due to the simplicity and portability of fastlane commands as well as the ability to put them under source control in a Fastfile, you're able to migrate to any CI server you like with little to no effort because your Fastfile is a recipe for building, testing and distributing your app while your server, is a way to host them.

Now back to Jenkins. Jenkins is a webserver that can run on any machine(include the macbook this post is written on). When installed it looks like this:

Using the fastlane commands previously described a developer will be able to easily create Jobs like running unit and UI Tests by invoking appropriate fastlane scan commands. In order to create the build and upload job, calls to fastlane gym and fastlane pilot should be enough. By making this job parametrized with a GIT parameter(Git parameter plugin) you will be able to trigger a build from any branch:

The Merge Request builder is an interesting case. It needs to be triggered for every Merge Request, execute unit tests(fastlane scan) and report back to gitlab. Using the Gitlab plugin (Something that previous servers don't have) it becomes a matter of clicking a few buttons to build when a change is pushed:

And report the results back as green checkmarks (or red X's) next to the commit in Gitlab,Bitbucket.

With the power of the Build Monitor View plugin, a developer can add the jobs that run unit and UI tests in a nice view like this:

If more granularity is needed then by using fastlane's ability to easily indicate a specific simulator to run UI Tests on (fastlane scan --scheme UITestsSchema --devices 'iPhone 4S') a more granular UI Testing dashboard can be created by combining six jobs that invoke fastlane scan for different simulators:

Another job might use fastlane sigh to renew and download provisioning profiles every day. Another job might create builds and upload them to hockeyApp after each Merge Request for your testers. Sending slack notifications with the changelog and download link of the app when a new build is available? All possible.

Jenkins's ability to scale up by adding more builder nodes is also worthy as is the fact that it is free. The only bad part about Jenkins is that because it is free, you have to host and maintain it. It's UI is also historically not so good. Other than that, it's plugins, robustness, reliability and integration with all kinds of third party services is world-class.

Verdict:
Even small teams will enjoy using fastlane to automate their iOS app processes (builds, uploads, tests). Configuring a Jenkins server to run those fastlane commands is not difficult and saves a team member from having to look at his terminal a few hours every while these commands run. Great and transparent error handling is a primary goal of fastlane so combined with a Jenkins server you will have a robust solution for your team to use. The great documentation of fastlane and the already proven robust mister Jenkins with it's myriad of plugins will allow you to automate even your wildest processes. All at a click's distance. Xcode Server trails at the end, other CI servers are less powerful (and only for Github) and so Jenkins wins the battle for our preferred CI Server.

PS: I am still giving a chance to Xcode Server to become a part of our CI setup by running UI tests on physical devices. It remains to be seen, if XCS will be able to at least handle that task, or we will have to uninstall it completely.

The Code Bug

A passionate iOS developer. Looking to radically improve the way we all develop software.

Amsterdam