Getting Started with Xamarin Test Cloud

There are tens of thousands of unique mobile devices being used every day. Ensuring your mobile app not only runs, but runs well on all of these devices can be a daunting, if not an impossible task. Purchasing more than a few devices is not fiscally feasible. A great service that closes the gap on the number of devices to test on is Xamarin Test Cloud. Xamarin allows you to submit UI tests (written by you) to their test cloud service where they will run on thousands of real devices (not emulators). This might not cover every single device, but it will cover the vast majority of the most popular devices for a fraction of the cost and effort otherwise.

To get started, log into Test Cloud and create a New Test Run. This button will not be available until after you subscribe (or start a 30 day free trial). 

Once you have a New Test Run, select the type of app that you want to create (iOS or Android) and the team that the app should be a part of and then click Next. (Teams are good for large organizations where people are only working on specific apps)

Teams are used to group apps and people together. Each team is given an API key (will be discussed later) used for each app submission. If you submit multiple apps with the same key, then they will be grouped into the same team. The user's permissions are also managed at the team level with the current available permissions being: see test runs, upload apps, manage apps, and manage team.

Next, you can select which devices you want to run the tests on. There are lots of filters allowing you to quickly find the device(s) that you need. In addition to showing you the most important information about each unique device, it also shows the number in inventory and an estimated wait for that device to become available.

The next step in the Test Run configuration allows you to choose the "Test Series", which allows you to group similar types of runs together, and the "System Language". Depending on the size of your app and organization, you may never change these. 

The last screen will give you instructions on how to submit your app via the command-line (on OS X and on Windows). It includes your app's unique API key, the current user's email address, and the device hash of the devices that you selected on a previous step. This is useful to copy and paste when running your tests the first time (or just occasionally) but becomes a hassle when repeatedly submitting tests frequently (since you have to pull full file paths to everything).

In Xamarin's document site, they have a sample bash script for submitting a test via the commandline. I've taken it and customized a little bit and also ported it to powershell so that it can be run on Windows too.

Bash (OS X)

#!/bin/sh

export SLN_DIR=/Volumes/HD/Projects/<projectpath>

export APP_NAME=<projectname>
export IOS_APP_NAME=<ios_executable_name>
export TEST_ASSEMBLIES=$SLN_DIR/<projectname>.Tests.UI/bin/Debug
export DROID=$SLN_DIR/<projectpath>/<projectname>.UI.Droid
export IOS=$SLN_DIR/<projectpath>/<projectname>.UI.iOS
export APK=$DROID/bin/Debug/com.<projectname>.droid.apk
export IPA=$IOS/bin/iPhone/Debug/<ios_executable_name>.ipa

### Optional used for building before submitting
export SLN=$SLN_DIR/<solutionname>.sln
export DROID_PROJ=$DROID/<projectname>.UI.Droid.csproj

### This will have to be updated when Xamarin.UITest is updated via NuGet.
export TESTCLOUD=$SLN_DIR/packages/Xamarin.UITest.2.0.5/tools/test-cloud.exe

### Tescloud credentials
export USER_EMAIL=<your_testcloud_email>
export XTC_API_KEY=<apikey>
### 1 device
#export IOS_DEVICE_ID=a972be73

### 7 & 7+ 10.2
export IOS_DEVICE_ID=98906359
### 6+ devices
#export IOS_DEVICE_ID=659a654e
### 9 Devices
#export IOS_DEVICE_ID=4a33c645
### 1 Device
export ANDROID_DEVICE_ID=436ca70f
### 12 Devices
# export ANDROID_DEVICE_ID=a4a8cffb

### Uploading the dSYM files is optional - but it can help with troubleshooting
export DSYM=$IOS/bin/iPhone/Debug/$IOS_APP_NAME.app.dSYM

### iOS : build and submit the iOS app for testing
#/Applications/Xamarin\ Studio.app/Contents/MacOS/mdtool build "--configuration:Debug|iPhone" $SLN
# Submit the IPA to Xamarin Test Cloud
mono $TESTCLOUD submit $IPA $XTC_API_KEY --devices=$IOS_DEVICE_ID --user=$USER_EMAIL --assembly-dir=$TEST_ASSEMBLIES --dsym=$DSYM --series "master" --locale "en_US" --app-name=$APP_NAME

### Android: Build and submit the Android app for testing using the default keystore
#/Library/Frameworks/Mono.framework/Commands/xbuild /t:Package /p:Configuration=Debug /p:AndroidUseSharedRuntime=false /p:EmbedAssembliesIntoApk=true $DROID_PROJ
# Submit the APK to Xamarin Test Cloud
#mono $TESTCLOUD submit $APK $XTC_API_KEY --devices $ANDROID_DEVICE_ID --assembly-dir $TEST_ASSEMBLIES --user $USER_EMAIL

Powershell (Windows)

$SLN_DIR="D:\Projects\<projectpath>"

$APP_NAME="<projectname>"
$IOS_APP_NAME="<ios_executable_name>"
$TEST_ASSEMBLIES="$SLN_DIR\<projectname>.Tests.UI\bin\Debug"
$DROID="$SLN_DIR\<projectpath>\<projectname>.UI.Droid"
$IOS="$SLN_DIR\<projectpath>\<projectname>.UI.iOS"
$APK="$DROID\bin\Release\com.<projectname>.apk"
$IPA="$IOS\bin\iPhone\Debug\<ios_executable_name>.ipa"

### Optional used for building before submitting
$SLN="$SLN_DIR\<solutionname>.sln"
$DROID_PROJ="$DROID\<projectname>.UI.Droid.csproj"

### This will have to be updated when Xamarin.UITest is updated via NuGet.
$TESTCLOUD="$SLN_DIR\packages\Xamarin.UITest.2.0.5\tools\test-cloud.exe"

### Tescloud credentials
$USER_EMAIL="<your_testcloud_email>"
$XTC_API_KEY="<apikey>"
### 1 device
#$IOS_DEVICE_ID="a972be73""

### 7 & 7+ 10.2
$IOS_DEVICE_ID="98906359"
### 6+ devices
#$IOS_DEVICE_ID="659a654e""
### 9 Devices
#$IOS_DEVICE_ID="4a33c645""
### 1 Device
#$ANDROID_DEVICE_ID="436ca70f""
#$ANDROID_DEVICE_ID="3e97901e"
### 7 devices
$ANDROID_DEVICE_ID="93d2d2a8"
### 12 Devices
#$ANDROID_DEVICE_ID="a4a8cffb"

### Uploading the dSYM files is optional - but it can help with troubleshooting
$DSYM="$IOS\bin\iPhone\Debug\$IOS_APP_NAME.app.dSYM"

### iOS : build and submit the iOS app for testing
#/Applications/Xamarin\ Studio.app/Contents/MacOS/mdtool build "--configuration:Debug|iPhone" $SLN
# Submit the IPA to Xamarin Test Cloud
#& $TESTCLOUD submit $IPA $XTC_API_KEY --devices=$IOS_DEVICE_ID --user=$USER_EMAIL --assembly-dir=$TEST_ASSEMBLIES --dsym=$DSYM --series "master" --locale "en_US" --app-name=$APP_NAME

### Android: Build and submit the Android app for testing using the default keystore
#/Library/Frameworks/Mono.framework/Commands/xbuild /t:Package /p:Configuration=Debug /p:AndroidUseSharedRuntime=false /p:EmbedAssembliesIntoApk=true $DROID_PROJ
# Submit the APK to Xamarin Test Cloud
& $TESTCLOUD submit $APK $XTC_API_KEY --devices $ANDROID_DEVICE_ID --user $USER_EMAIL --assembly-dir $TEST_ASSEMBLIES --series "master" --locale "en_US" --app-name=$APP_NAME

These have variables at the top of the files for you to easily tweak the submission - perhaps changing the devices, the user, or the nuget version and then resubmitting easily. 

Tests can also be submitted directly from Xamarin Studio or Visual Studio. Right-clicking the test or the project (depending on the IDE) and selecting "Run in Test Cloud" from the context menu will allow you to submit the test to Test Cloud. This will launch the website and start you from the "New Test Run" dialog that was mentioned previously. That means that submitting from the IDE will cause you to go through the dialogs each time. The only difference is the last step doesn't show the command line submission since it's already been submitted.

Once the tests complete, you can log in to Test Cloud to view the results. Test Cloud provides lots of information and details about the tests and the devices themselves.

Selecting a device will give you a deep dive into that device and all of the details.

The following options are available per device:

• Device Log* - shows the console output of the device during the entire execution.

• Test Log* - provides all of the commands and metadata about each step in the test.

• Details - show all of the specifics about that particular device. 

• Full Size - shows the screenshot at full resolution. This is a great feature for compiling images needed for the app stores.

*Both of the log options are helpful when trying to determine why a test might have failed.

The overview gives a nice summary for the entire test run. It is a great way to see a quick roll-up the results.

If you, unfortunately, have a failure, it presents it very clearly for you. Depending on the combination of devices you ran the tests on, it will help you pinpoint where the failures is. 

As you can see, Xamarin's Test Cloud provides a great way to test your mobile application on numerous devices. The rich interface provides an easy way to navigate the site and view exactly what you need.

Is this interesting, want to learn more, or are having the same discussions in your own department? Sparkhound's certified mobile developers work with Xamarin and many other technologies daily! Get in touch with us so we can discuss how we can help your business achieve its goals through leadership and technology.

Information and material in our blog posts are provided "as is" with no warranties either expressed or implied. Each post is an individual expression of our Sparkies. Should you identify any such content that is harmful, malicious, sensitive or unnecessary, please contact  marketing@sparkhound.com