iOS Automated Testing with Calabash, Cucumber, and Ruby

This article was written a long time ago. I’ve only updated the parts related to installing Ruby. It’s possible that Calabash might still work, but if you’re trying to set up Ruby on your Mac, your best option in 2024 is to use the Ruby on Mac script.

While researching automated testing tools for mobile applications earlier this year, the only one that met all of my criteria — well-documented, easy to set up and maintain, updated regularly, supports iOS and Android, runs on devices untethered, and allows you to write tests that are readable by everyone — was Calabash.

The other tools I evaluated were Anteater, FoneMonkey (before it became MonkeyTalk), Frank, KIF, TouchTest, UI Automation, and Zucchini. Karl Krukow (one of the developers of Calabash) already wrote a great blog post describing and comparing many of the above tools, so I won’t be discussing any of them here.

This post includes a tutorial for setting up Calabash on a Mac, examples of Cucumber features and Ruby scripts, and demos of automated tests running on the iOS Simulator. This tutorial assumes you already have access to the source code for the iOS app you’ll be testing. If not, you can still follow along right away by using any of these open source iPhone apps. For this tutorial, I’ll be using Grant Paul’s newsyc, an open-source iOS Hacker News app.

Step 1: Install Ruby on Mac

You can do that by either buying my Ruby on Mac script, which will save you a lot of time, or by following my step-by-step tutorial for installing Ruby on Mac.

Step 2: Install the calabash-cucumber gem

$ gem install calabash-cucumber

Step 3: Generate the features folder and irb shell scripts

$ calabash-ios gen      #After you run this command, press return when prompted

This will create two irb shell scripts: irb_ios4.sh and irb_ios5.sh, and a features folder that contains a sample Cucumber test (my_first.feature) and two folders: step_definitions (which contains calabash_steps.rb and my_first_steps.rb) and support (which contains env.rb, hooks.rb, and launch.rb). We’ll go over most of these files later on.

Step 4: Set up your Xcode project manually

This is the safest way to integrate Calabash with your iOS app. As mentioned in the official Calabash installation guide, the Fast track automatic setup is still experimental and is not guaranteed to work with all iOS projects.

If you have experience with Xcode, then the instructions in the Setting up Xcode project section of the installation guide should suffice. If not, read on for a detailed step-by-step tutorial.

Launch Xcode, select File->Open (⌘O), and open the newsyc folder:

Click on the newsyc project in the leftmost pane, then right-click (or two-finger tap) on newsyc under TARGETS, and select Duplicate. Or, simply click on newsyc and press ⌘D. If you get a Duplicate iPhone Target prompt, click on the Duplicate Only button.

This should result in a new target called newsyc copy, as shown in the screenshot below:

Double-click on newsyc copy to put it in edit mode, and rename it to newsyc-calabash, or newsyc-cal for short. The name you choose doesn’t really matter, as long as it makes sense to you.

Click on newsyc in the dropdown at the top left, to the right of the “Stop” button, and select Manage Schemes…:

Click once on newsyc copy, then wait, then click once again, making sure to click on the name itself, not anywhere else on that row. This will put you in edit mode, and you can rename the scheme to newsyc-cal.

Click OK, then click on Build Settings in the center pane, search for “product name”, then double-click on newsyc copy and rename it to newsyc-cal:

Download the latest version of calabash-ios from https://github.com/calabash/calabash-ios/downloads.

Unzip the file (you will end up with a folder called calabash.framework), then position the Finder window in one half of the screen, and the Xcode window in the other half. I use the awesome Moom app anytime I need to reposition and zoom windows. This will make it easier to drag and drop the calabash.framework folder from the Finder to the Frameworks folder in Xcode. Another app which would make dragging and dropping in this situation very easy (I have read great things about it but haven’t tried it yet) is DragonDrop.

Once you drop the calabash.framework folder in Xcode, you will be prompted to “choose options for adding these files”. Make sure the following are true: Copy items into destination group's folder (if needed) is checked, Create groups for any added folders is selected, and only your newly-created target (in our case newsyc-cal) is checked, as shown in this screenshot:

Click Finish, then click on the newsyc-cal target, click on Build Phases, expand Link Binary with Libraries, click on +, click on CFNetwork.framework, and click on Add. You should end up with something very similar to this:

With the newsyc-cal target still selected, click on Build Settings, click on All (if it’s not already selected), then search for other linker, click on the Other Linker Flags row, then click once under Yes to enable edit mode, and copy and paste the following: -force_load "$(SRCROOT)/calabash.framework/calabash" -lstdc++. Click anywhere outside of the text field to save your changes. You should end up with something like this (note that what appears after -force_load will be different for you since that is the path to your project on your computer):

Step 5: Test your setup

In the Scheme dropdown in the top left (to the right of the “Stop” button), select the newsyc-cal target on the left side of the dropdown, select iPhone 5.1 Simulator on the right side of the dropdown, then click the Run button (or press ⌘R).

While the iOS Simulator is launching, click on the middle button in the View section in the top right of Xcode to display the console output at the bottom of the center pane. Once the app is ready to launch in the Simulator, you will receive a dialog asking if you want the application to accept incoming network connections. Click Allow, and when you see something like this in the console output, you’re good to go:

2012-06-24 11:08:34.010 newsyc-cal[4668:15203] Creating the server: <LPHTTPServer: 0x8076530>
2012-06-24 11:08:34.019 newsyc-cal[4668:15203] Started LPHTTP server on port 37265
2012-06-24 11:08:34.703 newsyc-cal[4668:17203] Bonjour Service Published: domain(local.) type(_http._tcp.) name(Calabash Server)

Step 6: Write your first test

With Calabash, automated tests are run using Cucumber, and are written in Gherkin, a domain-specific language that’s readable by everyone involved in your app’s development cycle. Tests are made up of two components: feature files and step definitions.

Feature files are text files written using Cucumber jargon and saved with a .feature extension. Their purpose is to describe a feature and provide examples of expected outcomes depending on certain conditions. The first part of a feature file is the feature description, which can be written using the following template:

Feature: <name of feature>
  In order to <meet some goal>
  As a <type of stakeholder>
  I want <a feature>

For example:

Feature: Comments
  In order to contribute to the discussion
  As a Hacker News reader
  I want to be able to add a comment

Examples of expected outcomes are captured in Scenarios with Steps, also known as Givens, Whens, and Thens:

Scenario: User logged in
  Given I am logged in
  When I go to comment on a submission
  Then I should see a comment form

Scenario: User not logged in
  Given I am not logged in
  When I go to comment on a submission
  Then I should be prompted to log in

Step definitions are reusable Ruby scripts that execute the individual steps in the scenarios. Calabash comes with a bunch of predefined steps that allow you to start testing your app right away without having to write any Ruby scripts. For example, if we wanted to automate canceling the Comment form in the newsyc app, we could create the following feature file (saved as reply_to_submission.feature in the features folder that was created in Step 3). Note that the scenario below is just an example of what you can do with Calabash out of the box. We will rewrite the scenario in Step 8 to follow best practice.

Feature: Reply to a submission
  As a Hacker News reader
  I want to be able to comment on a submission
  So that I can contribute to the discussion

Scenario: User logged in, but cancels comment form
  Given the app is running
  When I touch "Profile"
  Then I wait to see "Login"
  Then I enter "my_username" into text field number 1
  Then I touch list item number 2
  Then I enter "my_password" into text field number 2
  Then I touch done
  Then I wait to see "Logout"
  Then I touch "Home"
  Then I touch list item number 1
  Then I touch "reply"
  Then I touch "Cancel"
  Then I should see "Submission"

Calabash interacts with the UI elements in your app via accessibility labels. That’s how it knows which button to tap when you issue a command like Then I touch "reply". As the author of the feature file, you can determine the labels of the various UI elements using the nifty Accessibility Inspector in the iOS Simulator. To turn it on, follow these steps:

1. Run your app in the iOS Simulator via Xcode
2. Stop the app
3. Swipe from left to right to go to the first screen
4. Tap on Settings
5. Tap on General
6. Tap on Accessibility
7. Turn the Accessibility Inspector toggle ON

Step 7: Run your test

Once you’ve written a scenario, you can verify that it works by running the app in the Simulator, then running the test from the command line:

$ cd /path/to/your/iOS/project
$ DEVICE=iphone OS=ios5 NO_LAUNCH=1 cucumber features/reply_to_submission.feature

Calabash supports the iPhone and iPad, iOS 4, and iOS 5. So, if you wanted to test on an iPad with iOS 4, you would use the following command:

$ DEVICE=ipad OS=ios4 NO_LAUNCH=1 cucumber features/reply_to_submission.feature

If you don’t specify a specific feature file, then running DEVICE=iphone OS=ios5 NO_LAUNCH=1 cucumber will execute all the scenarios in all the .feature files in your features folder. Whether you’re still experimenting with Calabash or you’ve already written hundreds of scenarios, you will often only want to run a subset at a time. Luckily, Cucumber allows you to do that using tags. To assign a tag to a scenario, simply add the tag name, preceded by the @ sign, above the Scenario header, such as:

@negative_test
Scenario: User logged in, but cancels comment form
...

You can also assign a tag to all the scenarios in a feature by adding the tag above Feature:, at the very top of the .feature file.

To only run scenarios with a particular tag, you add the --tags parameter to the cucumber command. For example:

$ DEVICE=iphone OS=ios5 NO_LAUNCH=1 cucumber --tags @negative_test

You can also skip scenarios with a certain tag by adding a tilde (~) before the tag in the command line:

$ DEVICE=iphone OS=ios5 NO_LAUNCH=1 cucumber --tags ~@slow_test

Calabash also allows you to test on actual iOS devices, without having to keep them plugged in to your computer. In order to be able to install an app on your device from Xcode, you need to be enrolled in the iOS Developer Program. Once your computer is properly configured for pushing builds to your device, follow these steps to run the automated test on your device:

1. Plug in your iOS Device
2. Launch Xcode
3. In the Scheme dropdown in the top left (to the right of the “Stop” button), select the newsyc-cal target on the left side of the dropdown, and select your device on the right side of the dropdown, then click the Run button (or press ⌘R)
4. If you get a prompt to Enable 'Developer Mode' on this Mac?, click Enable. If all went well, you should see “Build Succeeded” in Xcode and the app should launch on your device.
5. Click on the “Stop” button in Xcode
6. Unplug your device (this is optional)
7. In your iOS device, go to the “Settings” app
8. Tap on “Wi-Fi” and turn it ON if it’s not already on
9. Tap on the blue disclosure arrow at the far right of the cell that corresponds to your Wi-Fi network
10. Make a note of the “IP Address”
11. Launch the app manually on your device
12. From the Terminal app on your computer, run the following command:

$ DEVICE=iphone OS=ios5 NO_LAUNCH=1 cucumber features/reply_to_submission.feature DEVICE_ENDPOINT=http://your.device.ip.address:37265

You should now see the test running on your iPhone. How cool is that?

Step 8: Refactor (custom step definitions, DRY)

The User logged in, but cancels comment form scenario I wrote in Step 6 was just an example of a quick test you can run to try out the predefined steps in Calabash. In practice, you’ll want to make the scenario more readable and easier to maintain. Let’s refactor it by writing some custom step definitions and applying the DRY principle.

A scenario is an example of how the app should behave in a particular situation, so it’s best to keep it as short and readable as possible, and to avoid including steps that involve touching specific UI elements. The actual path to get to each screen in the app should be captured in a step definition. That way, if a redesign of the app causes the flow to change, or a UI label to change, you will only need to modify one step definition, as opposed to multiple scenarios that might be using the same UI flow.

Here’s how I would rewrite the Scenario:

Scenario: User logged in, but cancels comment form
  Given I am logged in
  When I go to comment on a submission
  But I cancel the comment form
  Then I should see the submission
  And a comment from "username" should not appear

Since these steps describe actions specific to the app, we need to write custom step definitions in Ruby to support the steps. These custom step definitions can go in calabash_steps.rb, my_first_steps.rb, or any file in the step_definitions folder (created in Step 3) that ends with _steps.rb. It’s up to you how you want to organize your steps, but please read the Step Organisation section of the Cucumber Wiki for some recommendations.

If you’re not familiar with the Ruby programming language, I recommend you go through Code School’s excellent 15-minute interactive introduction to Ruby, called Try Ruby. Go ahead, I’ll wait.

All set? Alright, let’s write the first step definition for the step Given I am logged in. Open my_first_steps.rb in your favorite text editor (I recommend Sublime Text) and add the following code:

Given /^I am logged in$/ do

end

All step definitions must start with one of the keywords Given, When, Then, And, or But, followed by a string or regular expression that matches the step in the scenario. Then, between do and end (the code block), is where you define what the step is supposed to do.

In this case, we want to make sure we’re logged in. One way to verify that state is by checking for the presence of the Logoutbutton on the Profile tab. If the Logout button exists, then we know we are logged in. Otherwise, we know we have to log in.

Now that we know what to do, we have to read the documentation and dig into the code to figure out how to look for a particular UI element. We can start by going through the predefined steps where we can find the following predefined step in the Waiting section, which sounds like what we’re looking for:

Then I wait for the "login" button to appear

Next, we need to look for the Ruby code that defines that step. At the top of the predefined steps page, they mention that the step definitions can be found in the calabash_steps.rb file. If you open that file and look for button to appear, you will find on lines 130-132 the following step definition:

Then /^I wait for the "([^\"]*)" button to appear$/ do |name|
  wait_for(WAIT_TIMEOUT) { element_exists( "button marked:'#{name}'" ) }
end

Line 2 above tells us that we can use the element_exists function to check for a button’s label. Next, we need to figure out how to incorporate predefined steps in our custom step definition. Thankfully, Calabash makes this very easy with the macro function, as explained in the Writing custom steps page of the wiki. Basically, you can use any of the predefined steps or your own custom steps inside a step definition.

Armed with this information, we can complete our step definition for Given I am logged in:

Given /^I am logged in$/ do
  macro 'I touch "Profile"'
  if element_exists("button marked:'Logout'")
    sleep(STEP_PAUSE)
  else
    macro 'I enter "my_username" into text field number 1'
    macro 'I touch list item number 2'
    macro 'I enter "my_password" into text field number 2'
    macro 'I touch done'
    macro 'I wait to see "Logout"'
  end
end

For more functions you can use in your custom step definitions, I recommend reading the Calabash iOS Ruby API page that has recently been added to the wiki. In fact, there is one we can use right now as an alternative to element_exists: view_with_mark_exists. If you were specifically looking for a button labeled “Logout”, then it would be best to keep using the element_exists function. Otherwise, you can replace

if element_exists("button marked:'Logout'")

with

if view_with_mark_exists("Logout")

Now we can move on to the next step, When I go to comment on a submission, which should be easy to implement using macros:

When /^I go to comment on a submission$/ do
  macro 'I touch "Home"'
  macro 'I touch list item number 1'
  macro 'I touch "reply"'
end

Same with the But I cancel the comment form step:

But /^I cancel the comment form$/ do
  macro 'I touch "Cancel"'
end

Next is Then I should see the submission:

Then /^I should see the submission$/ do
  macro 'I should see "Submission"'
end

Finally, And a comment from "username" should not appear. This one is tricky, and requires some Objective-C knowledge, as well as reading through the Calabash documentation. Let’s break down the problem into manageable steps.

First, we need to figure out what we need from the app to verify that a comment from the logged in user does not appear. After tapping on a submission, the app displays all the comments for that submission. In order to verify that the user is not the author of any of the comments, we need the ability to do two things:

1. Search for the username string on the screen
2. Scroll through all the comments

As mentioned in the Getting started guide, one of the ways Calabash finds UI elements is via their accessibility labels. Let’s run the newsyc-cal target in the iOS Simulator (as explained in Step 5) with the Accessibility Inspector turned on and collapsed. Tap on any entry on the Home tab, then expand the Inspector. Now tap any of the comments. If the Inspector doesn’t highlight the entire comment cell, and if it thinks each comment is a Button with the Label “reply”, it means that the cell does not have an accessibility label, which means Calabash can’t read any of the text inside the comment cells. We can solve that problem by setting an accessibility label programmatically in Xcode.¹

If you didn’t write the app’s code, or if you’re a curious QA Engineer who doesn’t want to interrupt the developer(s) unless you really have to, you can use Calabash to find out which [Class](https://en.wikipedia.org/wiki/Class(computerscience) you need to modify to add the accessibility label. With the app still running in the Simulator, tap on any entry on the Home tab. Notice that the resulting screen is made up of rows, or cells, that contain one comment each.

This type of view is known as a Table View. We want to know which Class each Table View cell belongs to. We can find out thanks to the Calabash console, which you can launch via the command line:

$ cd path/to/newsyc
$ calabash-ios console

This will open an IRB shell with Calabash loaded, so you can interact with the app. By reading through the Getting started guide, Query syntax, and Calabash iOS Ruby API, we know we can use the following command to get more info on the first Table View cell:

irb(main):001:0> query "tableViewCell index:0"
=> [{"class"=>"CommentTableCell", "frame"=>{"y"=>4820, "width"=>320, "x"=>0, "height"=>118}, "UIType"=>"UIView", "description"=>"<CommentTableCell: 0xb8cd370; baseClass = UITableViewCell; frame = (0 4820; 320 118); clipsToBounds = YES; autoresize = W; layer = <CALayer: 0xb8e0600>>"}]

As you can see, running query "tableViewCell index:0" returns various properties for the first cell (first row that contains a comment), including the Class, CommentTableCell. Keep in mind that you can only query what’s visible on the screen. So, if there are 10 comments, but only the first 4 are visible when you’ve scrolled to the top of the screen, then querying for the 5th cell will return an empty array, []. In some cases, if the first comment is very long, querying for it will return []. If you run into that situation, scroll the screen down until the submission title is no longer visible, then try the query again.

As an alternative to using the Accessibility Inspector, we can use the Calabash console to check if the first cell has an accessibility label:

irb(main):002:0> label "tableViewCell index:0"
=> [nil]

Now that we know which Class we need to modify, let’s go find it in Xcode. Click on the folder icon in the top left (under the Run button), type comment in the search field at the bottom left, then click on CommentTableCell.m in the file list. Type in the following code on lines 231 and 232 of CommentTableCell.m:

self.accessibilityLabel = user;
self.isAccessibilityElement = YES;

In practice, you’ll want to set the accessibility label to something more meaningful than just the comment’s author, but to keep things simple, we’ll just set it to user. Why this code works, how it works, where it needs to go, and what accessibility labels should consist of are all beyond the scope of this tutorial.

Let’s test our code changes. If the app is running in the Simulator, stop it, save your changes in CommentTableCell.m, run the app, tap on any entry in the Home tab, then launch the Calabash console in Terminal. If the Console was already running, you might have to stop it by running quit, then start it up again with calabash-ios console. If everything went well, when you run label "tableViewCell index:0" you should now see an array that contains the comment’s author’s username.

irb(main):002:0> label "tableViewCell index:0"
=> [spicyj]

Now that we’ve figured out how to search for the username on the screen, we can move on to our next objective: how to scroll through all the comments. Once again, the documentation is our best friend. The API provides two functions that we can use. The first will give us the total number of rows in the Table View:

irb(main):003:0> query("tableView",numberOfRowsInSection:0)
=> [22]

The second will allow us to scroll to a specific row:

irb(main):003:0> scroll_to_row "tableView", 3

We are now finally ready to write our last custom step definition for And a comment from "username" should not appear. Below is the completed step definition, which we’ll go over.

And /^a comment from "([^\"]*)" should not appear$/ do |username|
  users = []
  total_rows = query("tableView",numberOfRowsInSection:0)
  end_of_range = total_rows[0] - 1
  (0..end_of_range).each do |row|
    scroll_to_row "tableView", row
    sleep(STEP_PAUSE)
    users.push(label "TableViewCell")
  end
  result = users.select {|user| user.include? "#{username}"}
  if result.empty?
    sleep(STEP_PAUSE)
  else
    raise "'#{username}' was found but wasn't expected!"
  end
end

This code will make more sense if you have experience with Ruby, but I’ll walk you through it in basic programming terms. The "([^\"]*)" in the first line is a regular expression that matches whatever string you type in between the quotes in the last step of the Scenario in Step 8. This string will then be assigned to a variable called username.

As for the code block, our goal is to go through each comment and make sure the username of the logged in user does not appear. We need a way to keep track of each username as we scroll down the list. One way to accomplish that is to store all the usernames in an array so we can look in the final array and make sure the username we’re looking for isn’t present.

First, we create an array called users, which starts out empty. Then, we assign the total number of rows to a variable called total_rows. We will then use the Ruby Range Class and the Each method to iterate through each row. This is similar to using a for loop. Since the first row is actually row 0, our range will be from 0 to the total number of rows minus 1. We’ll assign the total number of rows minus 1 to a variable called end_of_range.

Note that query("tableView",numberOfRowsInSection:0) returns an array, but end_of_range needs to be a number. The number we’re looking for is the first item in the total_rows array. The location of an item in an array is referred to as the index. In Ruby (and many other languages), the first index is 0. So, to get the number we want, we use total_rows[0].

Next, going from 0 to end_of_range, we’re going to assign the row number to a variable called row, then scroll to each row, wait a bit, then add the authors of the visible comments to the users array using the push method.²

Once we’ve gone through all the rows, we’re going to look for the logged in user’s username in the final users array using interpolation, and the select and include? methods, then assign that resulting array to a variable called result.

Finally, if the username is not found, result will be an empty array, so we can check for that using the empty? method. If it’s empty, we’ll just pause³ for a bit. Otherwise, our test has failed, so we’ll display an error message in Terminal using the raise method.

Aaaaand we’re done! I will leave you with some exercises and reading material. I hope you found this tutorial informative. I would love to hear your feedback via email or Twitter.

Extra Credit

1) To make the Given I am logged in step definition easier to read and maintain, we’d like to replace these three lines:

macro 'I enter "my_username" into text field number 1'
macro 'I touch list item number 2'
macro 'I enter "my_password" into text field number 2'

with these lines:

macro 'I fill in "Username" with "my_username"'
macro 'I touch "Password"'
macro 'I fill in "Password" with "my_password"'

Can you figure out how to make these new lines work? Hint: read about placeholders in the Predefined steps Wiki page.

2) Once we start writing more scenarios that require logging in, we will end up with multiple scenarios containing the step Given I am logged in, which goes against the DRY principle. Which Cucumber features can we use to write the step once, and have it automatically run before the first step of certain scenarios?

Further Reading

The Cucumber Book: Behaviour-Driven Development for Testers and Developers

calabash-ios Google Group

iOS App Programming Guide

why’s (poignant) guide to Ruby