While researching automated testing tools for mobile applications earlier this year, the only one that met all of my criteria — well-documented, easy to set up and maintain, updated regularly, supports iOS and Android, runs on devices untethered, and allows you to write tests that are readable by everyone — was Calabash.
The other tools I evaluated were Anteater, FoneMonkey (before it became MonkeyTalk), Frank, KIF, TouchTest, UI Automation, and Zucchini. Karl Krukow (one of the developers of Calabash) already wrote a great blog post describing and comparing many of the above tools, so I won’t be discussing any of them here.
This post includes a tutorial for setting up Calabash on a Mac running Lion and Xcode 4.3, examples of Cucumber features and Ruby scripts, and demos of automated tests running on the iOS Simulator. This tutorial assumes you already have access to the source code for the iOS app you’ll be testing. If not, you can still follow along right away by using any of these open source iPhone apps. For this tutorial, I’ll be using Grant Paul’s newsyc, an open-source iOS Hacker News app.
Step 1: Create an RVM gemset
If you are on Lion and using Xcode 4.2+, I recommend that you install Ruby 1.9.3 with RVM. You can do that by following my very thorough tutorial for setting up a Mac for development.
Once you have RVM and Ruby 1.9.3 installed, create a new gemset for automated testing:
RVM allows you to create independent ruby and gem setups. By creating a separate gemset for playing with calabash, you will have a clean environment that only includes the calabash-cucumber gem (to be installed in Step 2) and any of its dependencies that it installs.
Read more on my post about RVM gemsets and other RVM tips and tricks.
Step 2: Install the calabash-cucumber gem
This will create two irb shell scripts:
irb_ios5.sh, and a
features folder that contains a sample Cucumber test (
my_first.feature) and two folders:
step_definitions (which contains
support (which contains
launch.rb). We’ll go over most of these files later on.
Step 4: Set up your Xcode project manually
This is the safest way to integrate Calabash with your iOS app. As mentioned in the official Calabash installation guide, the
Fast track automatic setup is still experimental and is not guaranteed to work with all iOS projects.
If you have experience with Xcode, then the instructions in the Setting up Xcode project section of the installation guide should suffice. If not, read on for a detailed step-by-step tutorial.
Launch Xcode, select File->Open (⌘O), and open the
Click on the
newsyc project in the leftmost pane, then right-click (or two-finger tap) on
TARGETS, and select
Duplicate. Or, simply click on
newsyc and press ⌘D. If you get a
Duplicate iPhone Target prompt, click on the
Duplicate Only button.
This should result in a new target called
newsyc copy, as shown in the screenshot below:
newsyc copy to put it in edit mode, and rename it to
newsyc-cal for short. The name you choose doesn’t really matter, as long as it makes sense to you.
newsyc in the dropdown at the top left, to the right of the “Stop” button, and select
Click once on
newsyc copy, then wait, then click once again, making sure to click on the name itself, not anywhere else on that row. This will put you in edit mode, and you can rename the scheme to
OK, then click on
Build Settings in the center pane, search for “product name”, then double-click on
newsyc copy and rename it to
Download the latest version of calabash-ios from https://github.com/calabash/calabash-ios/downloads.
Unzip the file (you will end up with a folder called
calabash.framework), then position the Finder window in one half of the screen, and the Xcode window in the other half. I use the awesome Moom app anytime I need to reposition and zoom windows. This will make it easier to drag and drop the
calabash.framework folder from the Finder to the
Frameworks folder in Xcode. Another app which would make dragging and dropping in this situation very easy (I have read great things about it but haven’t tried it yet) is DragonDrop.
Once you drop the
calabash.framework folder in Xcode, you will be prompted to “choose options for adding these files”. Make sure the following are true:
Copy items into destination group's folder (if needed) is checked,
Create groups for any added folders is selected, and only your newly-created target (in our case
newsyc-cal) is checked, as shown in this screenshot:
Finish, then click on the
newsyc-cal target, click on
Build Phases, expand
Link Binary with Libraries, click on
+, click on
CFNetwork.framework, and click on
Add. You should end up with something very similar to this:
newsyc-cal target still selected, click on
Build Settings, click on
All (if it’s not already selected), then search for
other linker, click on the
Other Linker Flags row, then click once under
Yes to enable edit mode, and copy and paste the following:
-force_load "$(SRCROOT)/calabash.framework/calabash" -lstdc++. Click anywhere outside of the text field to save your changes. You should end up with something like this (note that what appears after
-force_load will be different for you since that is the path to your project on your computer):
Scheme dropdown in the top left (to the right of the “Stop” button), select the
newsyc-cal target on the left side of the dropdown, select
iPhone 5.1 Simulator on the right side of the dropdown, then click the
Run button (or press ⌘R).
While the iOS Simulator is launching, click on the middle button in the
View section in the top right of Xcode to display the console output at the bottom of the center pane. Once the app is ready to launch in the Simulator, you will receive a dialog asking if you want the application to accept incoming network connections. Click
Allow, and when you see something like this in the console output, you’re good to go:
1 2 3
Step 6: Write your first test
With Calabash, automated tests are run using Cucumber, and are written in Gherkin, a domain-specific language that’s readable by everyone involved in your app’s development cycle. Tests are made up of two components: feature files and step definitions.
Feature files are text files written using Cucumber jargon and saved with a
.feature extension. Their purpose is to describe a feature and provide examples of expected outcomes depending on certain conditions. The first part of a feature file is the feature description, which can be written using the following template:
1 2 3 4
1 2 3 4
Examples of expected outcomes are captured in Scenarios with Steps, also known as Givens, Whens, and Thens:
1 2 3 4 5 6 7 8 9
Step definitions are reusable Ruby scripts that execute the individual steps in the scenarios. Calabash comes with a bunch of predefined steps that allow you to start testing your app right away without having to write any Ruby scripts. For example, if we wanted to automate canceling the Comment form in the
newsyc app, we could create the following feature file (saved as
reply_to_submission.feature in the
features folder that was created in Step 3). Note that the scenario below is just an example of what you can do with Calabash out of the box. We will rewrite the scenario in Step 8 to follow best practice.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Calabash interacts with the UI elements in your app via accessibility labels. That’s how it knows which button to tap when you issue a command like
Then I touch "reply". As the author of the feature file, you can determine the labels of the various UI elements using the nifty Accessibility Inspector in the iOS Simulator. To turn it on, follow these steps:
- Run your app in the iOS Simulator via Xcode
- Stop the app
- Swipe from left to right to go to the first screen
- Tap on
- Tap on
- Tap on
- Turn the
I’ve also recorded a quick Accessibility Inspector screencast showing you how to turn it on and how to use it.
Step 7: Run your test
Once you’ve written a scenario, you can verify that it works by running the app in the Simulator, then running the test from the command line:
Calabash supports the iPhone and iPad, iOS 4, and iOS 5. So, if you wanted to test on an iPad with iOS 4, you would use the following command:
If you don’t specify a specific feature file, then running
DEVICE=iphone OS=ios5 NO_LAUNCH=1 cucumber will execute all the scenarios in all the
.feature files in your
features folder. Whether you’re still experimenting with Calabash or you’ve already written hundreds of scenarios, you will often only want to run a subset at a time. Luckily, Cucumber allows you to do that using tags. To assign a tag to a scenario, simply add the tag name, preceded by the
@ sign, above the Scenario header, such as:
1 2 3
You can also assign a tag to all the scenarios in a feature by adding the tag above
Feature:, at the very top of the
To only run scenarios with a particular tag, you add the
--tags parameter to the
cucumber command. For example:
You can also skip scenarios with a certain tag by adding a tilde (
~) before the tag in the command line:
Calabash also allows you to test on actual iOS devices, without having to keep them plugged in to your computer. In order to be able to install an app on your device from Xcode, you need to be enrolled in the iOS Developer Program. Once your computer is properly configured for pushing builds to your device, follow these steps to run the automated test on your device:
- Plug in your iOS Device
- Launch Xcode
- In the
Schemedropdown in the top left (to the right of the “Stop” button), select the
newsyc-caltarget on the left side of the dropdown, and select your device on the right side of the dropdown, then click the
Runbutton (or press ⌘R)
- If you get a prompt to
Enable 'Developer Mode' on this Mac?, click
Enable. If all went well, you should see “Build Succeeded” in Xcode and the app should launch on your device.
- Click on the “Stop” button in Xcode
- Unplug your device (this is optional)
- In your iOS device, go to the “Settings” app
- Tap on “Wi-Fi” and turn it ON if it’s not already on
- Tap on the blue disclosure arrow at the far right of the cell that corresponds to your Wi-Fi network
- Make a note of the “IP Address”
- Launch the app manually on your device
- From the
Terminalapp on your computer, run the following command:
You should now see the test running on your iPhone. How cool is that?
User logged in, but cancels comment form scenario I wrote in Step 6 was just an example of a quick test you can run to try out the predefined steps in Calabash. In practice, you’ll want to make the scenario more readable and easier to maintain. Let’s refactor it by writing some custom step definitions and applying the DRY principle.
A scenario is an example of how the app should behave in a particular situation, so it’s best to keep it as short and readable as possible, and to avoid including steps that involve touching specific UI elements. The actual path to get to each screen in the app should be captured in a step definition. That way, if a redesign of the app causes the flow to change, or a UI label to change, you will only need to modify one step definition, as opposed to multiple scenarios that might be using the same UI flow.
Here’s how I would rewrite the Scenario:
1 2 3 4 5 6
Since these steps describe actions specific to the app, we need to write custom step definitions in Ruby to support the steps. These custom step definitions can go in
my_first_steps.rb, or any file in the
step_definitions folder (created in Step 3) that ends with
_steps.rb. It’s up to you how you want to organize your steps, but please read the Step Organisation section of the Cucumber Wiki for some recommendations.
All set? Alright, let’s write the first step definition for the step
Given I am logged in. Open
my_first_steps.rb in your favorite text editor (I recommend Sublime Text 2) and add the following code:
1 2 3
All step definitions must start with one of the keywords
But, followed by a string or regular expression that matches the step in the scenario. Then, between
end (the code block), is where you define what the step is supposed to do.
In this case, we want to make sure we’re logged in. One way to verify that state is by checking for the presence of the
Logoutbutton on the
Profile tab. If the
Logout button exists, then we know we are logged in. Otherwise, we know we have to log in.
Now that we know what to do, we have to read the documentation and dig into the code to figure out how to look for a particular UI element. We can start by going through the predefined steps where we can find the following predefined step in the
Waiting section, which sounds like what we’re looking for:
Next, we need to look for the Ruby code that defines that step. At the top of the predefined steps page, they mention that the step definitions can be found in the calabash_steps.rb file. If you open that file and look for
button to appear, you will find on lines 130-132 the following step definition:
1 2 3
Line 2 above tells us that we can use the
element_exists function to check for a button’s label. Next, we need to figure out how to incorporate predefined steps in our custom step definition. Thankfully, Calabash makes this very easy with the
macro function, as explained in the Writing custom steps page of the wiki. Basically, you can use any of the predefined steps or your own custom steps inside a step definition.
Armed with this information, we can complete our step definition for
Given I am logged in:
1 2 3 4 5 6 7 8 9 10 11 12
For more functions you can use in your custom step definitions, I recommend reading the Calabash iOS Ruby API page that has recently been added to the wiki. In fact, there is one we can use right now as an alternative to
view_with_mark_exists. If you were specifically looking for a button labeled “Logout”, then it would be best to keep using the
element_exists function. Otherwise, you can replace
Now we can move on to the next step,
When I go to comment on a submission, which should be easy to implement using macros:
1 2 3 4 5
Same with the
But I cancel the comment form step:
1 2 3
Then I should see the submission:
1 2 3
And a comment from "username" should not appear. This one is tricky, and requires some Objective-C knowledge, as well as reading through the Calabash documentation. Let’s break down the problem into manageable steps.
First, we need to figure out what we need from the app to verify that a comment from the logged in user does not appear. After tapping on a submission, the app displays all the comments for that submission. In order to verify that the user is not the author of any of the comments, we need the ability to do two things:
- Search for the username string on the screen
- Scroll through all the comments
As mentioned in the Getting started guide, one of the ways Calabash finds UI elements is via their accessibility labels. Let’s run the
newsyc-cal target in the iOS Simulator (as explained in Step 5) with the Accessibility Inspector turned on and collapsed. Tap on any entry on the
Home tab, then expand the Inspector. Now tap any of the comments. If the Inspector doesn’t highlight the entire comment cell, and if it thinks each comment is a
Button with the
Label “reply”, it means that the cell does not have an accessibility label, which means Calabash can’t read any of the text inside the comment cells. We can solve that problem by setting an accessibility label programmatically in Xcode.1
If you didn’t write the app’s code, or if you’re a curious QA Engineer who doesn’t want to interrupt the developer(s) unless you really have to, you can use Calabash to find out which Class you need to modify to add the accessibility label. With the app still running in the Simulator, tap on any entry on the
Home tab. Notice that the resulting screen is made up of rows, or cells, that contain one comment each.
This type of view is known as a Table View. We want to know which Class each Table View cell belongs to. We can find out thanks to the Calabash console, which you can launch via the command line:
This will open an IRB shell with Calabash loaded, so you can interact with the app. By reading through the Getting started guide, Query syntax, and Calabash iOS Ruby API, we know we can use the following command to get more info on the first Table View cell:
As you can see, running
query "tableViewCell index:0" returns various properties for the first cell (first row that contains a comment), including the Class,
CommentTableCell. Keep in mind that you can only query what’s visible on the screen. So, if there are 10 comments, but only the first 4 are visible when you’ve scrolled to the top of the screen, then querying for the 5th cell will return an empty array,
. In some cases, if the first comment is very long, querying for it will return
. If you run into that situation, scroll the screen down until the submission title is no longer visible, then try the query again.
As an alternative to using the Accessibility Inspector, we can use the Calabash console to check if the first cell has an accessibility label:
Now that we know which Class we need to modify, let’s go find it in Xcode. Click on the folder icon in the top left (under the
Run button), type
comment in the search field at the bottom left, then click on
CommentTableCell.m in the file list. Type in the following code on lines 231 and 232 of
In practice, you’ll want to set the accessibility label to something more meaningful than just the comment’s author, but to keep things simple, we’ll just set it to
user. Why this code works, how it works, where it needs to go, and what accessibility labels should consist of are all beyond the scope of this tutorial.
Let’s test our code changes. If the app is running in the Simulator, stop it, save your changes in
CommentTableCell.m, run the app, tap on any entry in the
Home tab, then launch the Calabash console in Terminal. If the Console was already running, you might have to stop it by running
quit, then start it up again with
calabash-ios console. If everything went well, when you run
label "tableViewCell index:0" you should now see an array that contains the comment’s author’s username.
Now that we’ve figured out how to search for the username on the screen, we can move on to our next objective: how to scroll through all the comments. Once again, the documentation is our best friend. The API provides two functions that we can use. The first will give us the total number of rows in the Table View:
The second will allow us to scroll to a specific row:
We are now finally ready to write our last custom step definition for
And a comment from "username" should not appear. Below is the completed step definition, which we’ll go over.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
This code will make more sense if you have experience with Ruby, but I’ll walk you through it in basic programming terms. The
"([^\"]*)" in the first line is a regular expression that matches whatever string you type in between the quotes in the last step of the Scenario in Step 8. This string will then be assigned to a variable called
As for the code block, our goal is to go through each comment and make sure the username of the logged in user does not appear. We need a way to keep track of each username as we scroll down the list. One way to accomplish that is to store all the usernames in an array so we can look in the final array and make sure the username we’re looking for isn’t present.
First, we create an array called
users, which starts out empty. Then, we assign the total number of rows to a variable called
total_rows. We will then use the Ruby Range Class and the
Each method to iterate through each row. This is similar to using a for loop. Since the first row is actually row
0, our range will be from
0 to the total number of rows minus 1. We’ll assign the total number of rows minus 1 to a variable called
query("tableView",numberOfRowsInSection:0) returns an array, but
end_of_range needs to be a number. The number we’re looking for is the first item in the
total_rows array. The location of an item in an array is referred to as the
index. In Ruby (and many other languages), the first index is
0. So, to get the number we want, we use
Next, going from
end_of_range, we’re going to assign the row number to a variable called
row, then scroll to each row, wait a bit, then add the authors of the visible comments to the
users array using the push method.2
Once we’ve gone through all the rows, we’re going to look for the logged in user’s username in the final
users array using interpolation, and the select and include? methods, then assign that resulting array to a variable called
Finally, if the username is not found,
result will be an empty array, so we can check for that using the empty? method. If it’s empty, we’ll just pause3 for a bit. Otherwise, our test has failed, so we’ll display an error message in Terminal using the raise method.
Aaaaand we’re done! I will leave you with a screencast showing the test running in the Simulator and in Terminal, as well as some exercises and reading material. I hope you found this tutorial informative. I would love to hear your feedback via email or Twitter.
1) To make the
Given I am logged in step definition easier to read and maintain, we’d like to replace these three lines:
1 2 3
with these lines:
1 2 3
Can you figure out how to make these new lines work? Hint: read about placeholders in the Predefined steps Wiki page.
2) Once we start writing more scenarios that require logging in, we will end up with multiple scenarios containing the step
Given I am logged in, which goes against the DRY principle. Which Cucumber features can we use to write the step once, and have it automatically run before the first step of certain scenarios?
- Even if you won’t be using Calabash to test your app, it’s still a good idea to make your app as accessible as possible. ↩
- Note that this time, we’re not specifying an index when looking up the label of the TableViewCell. Without an index, this will return all the labels in all the rows that are visible at that moment. This is necessary because when it reaches the bottom of the screen, we don’t know how many rows will be visible, and scrolling to the last row could still display several rows, and there is no way to know in advance what the index of that last row will be. ↩
- The sleep method allows you to suspend the current process for any amount of time you want, specified in seconds. Here, we’re using the Calabash variable STEP_PAUSE, which is set to 0.5 seconds. If you don’t want to use a variable, you can specify a sleep time of, say, 2 seconds with “sleep(2)”. If you want to change the value of the STEP_PAUSE variable globally, you can edit “features/step_definitions/calabash_steps.rb”, found inside your local copy of Calabash (not your project’s directory). To find out where the calabash-cucumber gem is installed on your computer, run “gem environment” in Terminal while in your project’s directory, then look for the path to your calabash gemset in the “GEM PATHS” section (assuming you created it in Step 1). To that path, add “/gems/calabash_cucumber-0.9.x” where “x” is the last version number you installed. For example, the full path to “calabash_steps.rb” on my machine is “/Users/moncef/.rvm/gems/ruby-1.9.3-p194@calabash/gems/calabash-cucumber-0.9.74/features/step_definitions/calabash_steps.rb”. ↩