Mobile App Testing Without a Device Farm

A device farm is the part of mobile testing that nobody wants to own. Whether it is a rack of phones in a closet, a rented cloud of real devices, or a shelf of Mac minis running simulators for CI, it needs feeding, and the person who feeds it is rarely the person who wanted the tests. We build TesterArmy (YC 26), so the second half describes our product, and one boundary comes first: "without a device farm" here means without maintaining your own. Our runs happen on iOS Simulators and Android emulators, never on physical devices, and the last section covers what that leaves on real hardware.

What a device farm is for

A device farm does two jobs that get lumped together. The first is hardware truth: does the camera flow work on this sensor, does Face ID enrolment behave, how does the list scroll on a phone from four years ago, what does this manufacturer's Android skin do to your permissions dialog. Only a real device answers those questions.

The second job is flow regression: after this pull request, can a user still sign up, get through onboarding, find the thing your app exists for, and pay. That is where most day-to-day breakage lives, because it is where the code changes most, and none of it requires silicon. A simulator running your build shows the same screens, the same navigation, and the same broken button a device would.

Teams usually stand up a farm for the first job and then depend on it for the second, which is how a hardware question turns into an infrastructure bill.

What running your own rig costs a team

The cost is rarely the machines. For iOS it is macOS runners kept on a supported Xcode version, simulator runtimes downloaded and pinned, a build step that produces a simulator artifact rather than a device archive, and someone who understands why the run is hanging on a boot screen this morning. On top of the rig sits the test framework: native integration in the app, identifiers sprinkled through the UI, and test code that changes when a screen is redesigned. We compared that model with the agent path for the most common React Native framework in why teams are replacing Detox for React Native testing. The framework and the rig share an owner, and when that owner is busy, the mobile tests are the first thing that quietly stops running.

What the agent path needs from you: a build

TesterArmy takes one input you already produce, a build, and the docs are exact about which kind. For iOS it needs an iOS Simulator build: a .app bundle built with -sdk iphonesimulator, zipped so the .app directory sits at the archive root (MyApp.app.zip containing MyApp.app/, never Payload/MyApp.app/). An .ipa is a device build and is rejected. For Android it needs a single .apk or a split APK archive (.apks); .aab and .xapk are not supported, and release or debug APKs both work.

From an existing Xcode project the simulator build is one command, quoted from the app uploads docs:

xcodebuild -workspace ios/MyApp.xcworkspace \
  -scheme MyApp \
  -configuration Release \
  -sdk iphonesimulator \
  -destination 'generic/platform=iOS Simulator' \
  -derivedDataPath ios/build \
  build

Expo and React Native projects run npx expo prebuild --platform ios first when native files need generating. Uploading goes through the dashboard's Mobile tab, the presigned API flow for large artifacts, or the CLI, which accepts the raw .app directory and zips it for you:

testerarmy upload-app \
  --app-path ios/build/Build/Products/Release-iphonesimulator/MyApp.app \
  --project <projectId>

Add --remove-after 3600 for temporary CI uploads. Each project has a 2 GB storage limit, and "Auto-delete Oldest App" is on by default, so a new upload that would exceed the limit removes the oldest build while never touching one used by a queued or running test. There is one build type that will not run: a React Native or Expo development build that expects a Metro dev server. The run ends with MOBILE_RELEASE_BUILD_REQUIRED and asks for a release build with the JavaScript bundle embedded.

Writing the flow in plain English

There is no test code. A mobile test is a list of steps written the way you would brief a colleague, and the docs give the contrast directly: "Open settings" is a weak step, while "Tap the profile icon in the top right, open Settings, and verify the notifications toggle is visible" is a strong one. The agent looks at the simulator screen and works out what to tap, so there are no test IDs to add to the app and nothing that binds a step to a component name.

Two mobile-specific capabilities come from the same docs. Flows that need media, such as an avatar upload or a receipt scan, use photos (.png, .jpg, .jpeg) or videos (.mp4, .mov) uploaded to the project's Files tab and attached to a files step; TesterArmy preloads them into the device photo library before the run, and the agent picks them through your app's own media picker. On Android, a step like "turn off the network, check the offline banner, then go back online" makes the agent switch the emulator's connectivity through Android itself, so your app receives real ConnectivityManager callbacks; the agent waits for Android to report the new state and restores connectivity afterwards. That one is Android-only, because iOS Simulators expose no equivalent switch the app can observe.

Runs target the most recent uploaded build for the platform automatically, and you pin a specific app ID only when you need to. The supported targets, as documented:

iPhone runs are portrait. iPad runs are chosen in the run menu or with --device-model ipad on the CLI, execute in landscape for the whole run, and require a universal build with native iPad support; an iPhone-only build launched on iPad fails during setup with a clear error rather than producing misleading results.

Running it on every build

The recommended sequence in the mobile overview is to build, upload, create one test and run it manually, then wire it into CI so the same tests run on every change. Two paths are documented: Expo EAS, where EAS already produces the simulator app and a workflow hands it over, and GitHub Actions, where the TesterArmy action accepts the .app directory directly so you skip the zip step. The EAS pipeline end to end, including PR wiring, is in e2e testing for Expo apps, and the framework-free path for plain React Native is in React Native e2e testing without Detox or a device farm.

Every completed mobile run includes a downloadable simulator or emulator video alongside the same PASSED, FAILED or BLOCKED verdict and step results as web runs. The docs also include a guide for exercising Apple Pay checkout in an iOS app on the simulator, from payment sheet to success screen. The product page for this stack is mobile testing.

What stays on real hardware

Simulators and emulators cover UI, navigation and flow regressions, which is where apps break between releases. They do not cover the first job from the top of this post. The docs are direct about it: camera access and biometrics such as Face ID and Touch ID are not supported, and teams that need real devices are asked to contact support. Device performance, manufacturer skins and anything that depends on a physical sensor stay on hardware, with TesterArmy or without it.

Login gates deserve a word as well, because they are the second reason mobile tests stall. One-time codes and CAPTCHA challenges have their own guides, written for the web agent: how to test OTP and 2FA flows automatically and how to test flows behind a CAPTCHA. Run a mobile login once manually before wiring it into CI, so you know what your test configuration needs to bypass.

FAQ

Can you do mobile e2e testing without real devices? Yes, for UI and flow regressions. TesterArmy runs uploaded iOS Simulator and Android emulator builds in the cloud and returns a video of every run. Hardware-dependent behaviour such as camera, Face ID and Touch ID is not supported on simulators and needs a real-device pass.

What build do I upload for iOS? An iOS Simulator .app bundle, zipped with the .app directory at the archive root, or the raw .app directory through the CLI or GitHub Action. An .ipa is a device build and is rejected. Expo and React Native dev builds that need Metro are rejected too; upload a release build with the JavaScript bundle embedded.

Does this work with Expo? Yes. Expo EAS can produce the simulator build, and a workflow uploads it and runs your saved tests; the docs and our Expo guide cover the full pipeline. Run npx expo prebuild for the platform first when native files need to be generated.