Tuist Your Circle — Putting It To The Test — Part 1

In the previous article, I provided some insights on how to build a Tuist-based iOS app on CircleCI efficiently. In this one, I will look into 2 different approaches to automated testing, and explain how Tuist can help improve your CI workflow.

Testing with Xcodebuild

If you are familiar with running automated tests from the command line then you will know at least some of the 70+ options available for xcodebuild.

For example, a basic command to run all the tests defined in the Pokedex project with a scheme of the same name on an iPhone 14 simulator running iOS 16.4 looks like this:

xcodebuild -workspace Pokedex.xcworkspace -scheme Pokedex -sdk iphonesimulator -destination "platform=iOS Simulator,name=iPhone 14,OS=16.4" test

If you need to run the tests for all the different modules you would have to call each one individually or create a custom scheme that includes all the testing targets of the app. When you use this combined scheme, you are resigned to always running all the tests regardless of changes. It clearly isn’t very efficient. Apart from that, it can be tricky to get all the parameters correct and occasionally be quite frustrating.

Tuist Test

Tuist improves on this with the test command. By knowing how the project is structured, Tuist can generate a project and scheme for all the test targets, run a build, and execute the tests — all in the same command. Tuist provides sensible overridable defaults so that commands are zero-flags.

This simpler syntax is nice but there is something else far more valuable — reduced running time by using optimising what tests need to be executed. Tuist stores hashes for source targets in ~/.tuist/Cache/TestsCache Each time it is called Tuist compares the hashes, and determines which test targets can be skipped. That can look like this:

HomeTests has not changed from last successful run, skipping...
NetworkKitTests has not changed from last successful run, skipping...
BackpackTests has not changed from last successful run, skipping...
CommonTests has not changed from last successful run, skipping...
BackpackSnapshotTests has not changed from last successful run, skipping...
HomeSnapshotTests has not changed from last successful run, skipping...
HanekeTests has not changed from last successful run, skipping...

If you have a highly modular project, you will potentially see much shorter CI run times — with two caveats: automated tests consist of mostly unit tests, and the project has more than a few thousand tests.

Consider this sample project that although it only has 4 scenes, contains 8 modules. Each scene is defined as a micro-feature module, with test targets for unit and snapshot tests, an example app that launches the module, and UI tests to validate it. Although technically snapshot testing can be regarded as the same genre as unit testing, for the sake of clarity, I have separated them out into dedicated test targets in the project. There are also modules for networking, common shared code, UI components (one shared view), and an Objective-C image cache library (Haneke). The dependency graph looks like this:

Pokedex modular application with example apps

Incremental Test Execution

When it comes to running test targets, Tuist can skip a target if the hash has not changed. For example: When a change is pushed in the Home module, the tests for Home and Pokedex are run. All the others are skipped. When a change in Common is pushed, then it will run the testing targets for all the modules that depend on it as well as the module itself, as one would expect. Depending on the number of unit tests, this can save a lot of time.

In the example project I didn’t see any real difference in the testing run time because of the small number of tests. The impact is just not there. Others in the Tuist community have reported an impressive reduction in the CI time for their respective projects in production.

Caching In On The CI Cache

When it comes to a Tuist project, caching is something that is fundamental to achieving reductions in the execution time and improvements in overall performance. In the previous article, I explained how to define local caches in CircleCI for the .tuistcache, and the project dependencies.

The updated project has an improved approach where the YAML generates a Tuist.lock file based on the project manifest and the Tuist folder. A checksum is generated to use as a key. So long as the manifest doesn’t change, then this same key can be generated and the Tuist cache can be restored.

When it comes to running tests, the hashes need to be stored after each successful run of a test target. The above caching doesn’t update those hashes correctly, so I added a separate cache for them with a lock file based on the sources.

With the 2 caches in place, the unit testing optimisation is working as expected.

Reducing Build Time

Now, I want to go back again to the beginning, where I was discussing xcodebuild . It is necessary to build the project prior to running all the tests. And for that, there is another trick that can make a huge difference: Tuist Cloud.

Although this feature merits an article by itself, I will just mention a few commands here. When configured (you need an account on Tuist Cloud and an Amazon S3 bucket for storing compiled artefacts) tuist cache warm -x will compile the modules and dependencies into XCFrameworks and push them to S3. tuist generate -x can pull the compiled artefacts and generate a project that links the artefacts instead of the sources. Running a build with compiled XCFrameworks will complete up to an order of magnitude faster. See the docs for more info.

But what about testing? Can you just run tuist test with the cloud configuration and it magically optimises it all? Alas no, that isn’t working just yet. If you use the build-for-testing option of xcodebuild, and then test-without-building at least the build will be much faster, even if the tests running are not optimised.

Parallel Testing on CircleCI

One of the key benefits of CircleCI is that it can run tests in parallel. It leverages the build-for-testing / test-without-building feature for the parallel instances. The build-for-testing option in xcodebuild will build the workspace and generate an XCTestRun XML file. (More on xcodebuild here.) The test splitting feature depends on a configuration fastlane for the scan command. More on that here.

CircleCI internally keeps a record of the running time of each test and can use this data to divide them up over the number of parallel instances. There is a certain spin-up time for an instance, so it can take a little trial and error to work out the optimum number of instances that make sense to have the most efficient scale-out. For example, in eDreams, the iOS team uses 11 instances for UI testing (yes, we turned our testing up to 11), and another 2 for each of the unit tests, and snapshot tests, giving a total of 13, for the pull request workflow. This results in a sub-30 minute run time of the workflow, of which 15 minutes is for running all the 8000+ automated tests.

If you are curious to know more about XCTestRun all the properties are detailed here.

Optimising Persist-to-Workspace

There is another optimisation to consider that can remove minutes from the CI time: The build command generates an enormous set of intermediate files which are not required for testing. Therefore selecting only the specific files and folders that are essential to running testing to be persisted to the Circle CI workspace can significantly reduce the execution time of this step. What we (the iOS team at eDreams) found was the following set:

      - persist_to_workspace:
          root: .
          paths:
            - "Gemfile"
            - "Gemfile.lock"
            - "<appName>.xcodeproj"
            - "<appName>.xcworkspace"
            - "fastlane/*"
            - "<mocks>"
            - "path/to/__Snapshots__/*"
            - "<appName>/Supporting Files/*"
            - "derivedData/Build/Products/*/*.app"
            - "derivedData/Build/Products/*/*.xctest"
            - "derivedData/Build/Products/*.xctestrun"

By selecting the app and tests, the amount of data is less than 400MB, down from over 5GB. This is a huge saving and reduces the time for the CI.

Xcode Parallel Testing

Note that the schemes are configured to not run tests in parallel. I have seen that using the xcodebuild option for parallel tests actually makes it slower because cloning the simulator seems to take forever, and for CI at least, it is best avoided (Xcode 14.x). But, by all means, take the time to check this for yourself.

UI Testing Issues

In the example project, it has 7 example apps, each with UI tests. Although unit tests can be skipped by the optimisations, this does not apply to UI tests on example apps. All the UI test targets will run regardless of changes for every run of the CI workflow.

Also note that the test command has a parameter skip UI tests but this only works with UI tests defined in the main application, not for the example apps UI tests. There is plenty of room for improvement here.

Snapshot Testing Is Unit Testing

When you look at the logs for snapshot tests, it is clear these are optimised in the same way as ordinary unit tests. In the project manifest, the target product type is .unitTests , so this should not be surprising. If your project only consists of unit and snapshot tests, then this can really benefit from the optimisations of tuist test.

Round-Up

• If your project automated testing consists of unit tests then you can potentially see huge reductions in the CI workflow time.
• If you configure the project to use Tuist Cloud, you will see huge reductions in build times (both locally and on CI).
• If you have a large number of UI tests, consider using parallel instances.
• Avoid using the Xcode feature for parallel testing on CI.

Future Plans

I have been discussing the issues I have pointed out above with some of the core team of Tuist, and many plans are in motion to address them along with many more improvements. Binary caching will be included in the test workflows, such that the hack I added for CircleCI caches won’t be needed. Expect some very big updates in the months ahead.

If you have taken the time to look at the CircleCI yaml here, you may think that it could be made simpler — I totally agree. It would be far nicer to have an orb that wraps up all this infrastructure code and reduces what you need to know about it — this is a core part of the Tuist philosophy.

Conclusion

I have shown the potential of how Tuist can optimise your CI pipelines if you are using unit or snapshot tests. If you have a lot of UI tests, those benefits don’t work at the moment but are known issues and will be fixed in the coming months. Overall, you are better off not relying on UI tests if at all possible and PointFree’s Snapshot testing library has a lot of potential to cover some of the UI actions and integration testing, and will be far cheaper, and much faster to run — it’s a win-win.

In the next article, I will go through the options for gathering test coverage data in a Tuist project, parsing and downloading it, and more.