Tuist Your Circle — Putting It To The Test — Part 2
In the last article I outlined some ways on how to improve the build time on CI, optimising running of tests using Tuist, and parallel testing. This time I take testing further and look into analysing the generated test output to enable basic quality gates that are essential to maintain code coding standards of a software project (not limited to Tuist projects). Then I propose how to optimise UI testing. And finally I show a way to optimise CI pipelines and the execution of SonarQube code analysis.
This article (and the whole series) is based on a simple project here. It consists of 4 screens loading Pokemons, and stores them locally. Each screen is in a separate feature module with unit, snapshot and UI tests covering all the features.
Enabling Test Coverage
If you regularly write automated tests to your code, you will be aware you need to keep an eye on code coverage. In the same way that Xcode doesn’t enable this by default, as it introduces a performance hit — you have to configure it explicitly in Tuist. There are 3 ways it can be done:
- Custom Scheme
- Project
- Workspace
A custom scheme (an example is provided here — lines 442–485) can define TestActionOptions , that has a coverage flag and an array of testing targets. The project has an options parameter where the automatic scheme options has a flag for coverage (see here lines 140–150). But by far the simplest way, and in fact the most useful, is to enable it for the default scheme of the workspace. The workspace autogenerated scheme will include all the testing targets defined in all the projects (I only have one project for this example). Of particular interest here is the codeCoverageMode. This is an enumerated value (see docs here) and although the simplest option is .all this isn’t the best idea as it will generate a coverage report that will include everything — the test targets and external dependencies. It’s better to have it focused on specific modules, like the feature modules listed here:
let workspace = Workspace(
name: "Pokedex",
projects: ["./**"],
generationOptions: .options(
autogeneratedWorkspaceSchemes:
.enabled(codeCoverageMode: .targets([
"Pokedex",
"Home",
"Catch",
"Backpack",
"Detail"]),
testingOptions: [.randomExecutionOrdering])
)
)Note once these targets are defined here,
tuist generaterequires them, so for example generating a project focussed only on sayHomewill fail with an error:Target ‘Pokedex’ at ‘/root/Projects/Tuist-Pokedex’ doesn’t exist. Fatal linting issues found. It also breakstuist cache warm.A work-around is to rename the file so that Tuist doesn’t pick it up for generating focused targets, and then move it back again for CI.
Code Coverage Data: Profdata and XCResult Bundles
When xcodebuild runs tests it generates binary output files. Which files it generates, and where they can be found, depends on the parameters passed to the command.
With the release of Xcode 7 in 2015, Apple simplified generating and analysing code coverage, and moved over from gcov format to generating profile data (Coverage.profdata). This is a large binary file usually located under derived data for the application in a folder /Build/ProfileData/<UUID>/Coverage.profdata
The llvm-cov command is provided to extract out the code coverage data from this binary file. It needs the application binary (and sources as an optional filter) to produce readable text, JSON or HTML output and is highly efficient.
Xcode 9.3 (2017) added support for viewing test code coverage directly in the application, and generates an tests.xcresult bundle for this purpose. Xcode can open these bundles directly, and display the coverage in the IDE. The xccov command can also convert that output to JSON so as to be “human readable” and parseable in other tools such as CI/CD workflows.
Xcode 11 (2019) refined the formatting of the output and this is the version (3) that is available today in Xcode 15 (2024).
The current defaults for xcodebuild will generate both Coverage.profdata and the xcresult bundle in the derived data folder. Note that when xcodebuild is called with the resultBundlePath <path>option, it will not generate the Coverage.profdata binary. A new .xcresult bundle is generated for each test run and a time and date is used in the name. For example:
Test-AppWithPartialCoverage-Workspace-2023.11.05_18–01–19-+0100.xcresult
Extracting Code Coverage Data
Generating JSON from the XCResult bundle can be done in a like this:
xcrun xccov view --report --json tests.xcresult > coverage.json
The JSON data model has this structure (I omitted some properties for clarity):
XCResult
lineCoverage (the total coverage, normalised)
targets array
Target
name
lineCoverage (the module coverage, normalised)
files array
File
name, path
lineCoverage (normalised)
functiony array
Function
name
lineNumber
lineCoverage
A Swift script to parse this is JSON focusing only on the targets is here. This produces an output like this:
Backpack.framework coverage: 0.0
Catch.framework coverage: 0.0
Detail.framework coverage: 96.63
Home.framework coverage: 85.71
Pokedex.app coverage: 18.7Note that the coverage % here is only of the tests that were run. tuist test optimises the running of tests (selective testing) and skips testing targets when the code has not changed in that module. That then results in very different coverage data and could give the impression that something is not right. For example I know for sure that the Catch.framework has over 80% coverage.
What’s The Difference?
One simple way to check what has changed in the branch compared to say main is to run git diff main --name-only to get a list of the changed files. In the sample project I follow a naming convention for all feature modules of Features/<ModuleName>/Sources/Scenes/<files>.swift. A grep Sourcesgives the files of interest.
A Simple Quality Gate
In the sample repo I added a file with the code coverage from main branch. Then in scripts folder compareCoverage.sh uses this reference to compare against the JSON output and checks that the coverage % has not decreased. For example when I add a new function without a test it will fail:
Feature modules that have changed: ["Home"]
Error: Home.framework Coverage 91.3 is lower than reference: 93.33
Coverage check: FailThis check can flag when code coverage is dropping relative to the main branch on a per module basis.
Optimising Coverage Checks
Let’s imagine you have a large project where there is patchy code coverage (most large projects, let’s be honest), and the team agrees they would like to improve this by adding a new rule to have a threshold test coverage for all new code. The data required for this check is available in the JSON, and some simple parsing can extract the numbers. An example is here. In this example by adding 2 source files to Git, and then running tuist test , generating the JSON coverage output, then parsing out the data, the Quality Gate will output this to the terminal.
As a little bonus, I added basic ASCII colours to the script output to enhance the visual aspect of it.
The full command to run is swift scripts/runTestsForQualityGate.swift as it combines extra steps to clean out previous runs, and ensure that all the steps are executed in the correct order.
Optimising UI Testing
Tuist enables efficient running of unit test targets, but this doesn’t extend to UI tests. Even though they are rapidly being replaced by snapshot testing, they probably won’t go away altogether. Using the filter script for changed modules, tuist generate HomeUITests PokedexUITests for example.
Let’s say that team decides that the UI tests on the main app are required to run always, and the others can be run selectively based on the changes. For this to work, I take the previous script to find the modules that have changes, generate the full Xcode workspace, list the targets in JSON, filter out all the UI test targets, and generate the Xcode project with the UITests target using Tuist. Then finally I use xcodebuild to run all the tests on this optimised project. The code is under scripts here.
SonarCloud
SonarCloud is a feature rich code analysis platform that can give a wide range of insights into code quality from complexity to flagging code smells, or duplication. It is recommended for commercial projects to have a set of quality metrics to ensure coding standards are adhered to.
In this project I have integrated a Docker instance of a parser that runs on Linux. It requires an XML data export from XCResult bundle in their proprietary format. There are several options to run this parsing but the optimum solution is the shell script provided by SonarCloud.
When I first looked into how to generate the XML for SonarCloud, I found multiple options, and didn’t understand the difference. So I dug deep into the code and pushed this repo with the analysis. The end result to that over it is better to use the SonarCloud shell script over any binary, which I didn’t expect.
There is one drawback from Tuist’s optimising of tests however — if there are no tests to run, then there is no XCResult bundle, and in turn no XML file to parse. This causes the workflow to fail. The problem lies in the fact that the Sonar analysis step is non-optional.
A possible solution to this is to disable the automatic trigger from the repo to start a workflow, and then to use a local script to determine the changes, and if no sources have changed to launch a different workflow where Sonar analysis isn’t used.
The script for this approach is here.
Note that SonarCloud needs to be configured for only flagging coverage for changes, or the selective testing with fail, as I mentioned above.
Summary
In this and the previous post, I covered how to optimise
- Automated testing
- Test coverage generation
- Extraction of coverage data
- Quality gate checks
- Dynamic workflows
In the filtering of modules I only considered features, but this could be easily extended to all modules in the dependency graph using Tuist. I build CLI exactly for this purpose here to query the graph. (It can also be done by parsingtuist graph --format json but I like the CLI more.)
Putting all of this together saves considerable time, and of course money. The difference can be enormous. Although what I have described is for a simple project, these are the basic steps that, when extended to a specific project, will scale to massive projects with 10s of developers, hundreds of modules, and millions of lines of code.
Conclusion
When configuring the project for code coverage, explicitly list all the targets of interest. Leverage local scripting to enable simple quality gates before pushing to CI so that at least if it fails, it fails faster. On the CI optimise not only unit and snapshot tests, but also UI testing to only run the tests that change by determining the targets from the changed sources. But also it is possible to optimise the CI even further by dynamically choosing the workflow that needs to be run.
Swift is a very powerful language for scripting when there is a lot of complexity, and where bash would be much more difficult to read, such as paring JSON coverage and extracting out the value for the files changed. Stating the obvious, but use the best tools available for the job at hand.
This brings to an end a series I started almost 2 years ago about how to migrate a simple project to Tuist, modularise it, and build tooling around it for optimising continuous integration workflows on Circle CI.
I started this series as a way to document work that I had done in a previous company, and as a platform to experiment with Tuist and see how far I could go with optimising the flow. During this time Tuist evolved and forced me to keep this project updated from version 1.x to the just released 4.0 last week. That meant that it took me far longer at times to resolve issues as they came up.
What’s next? To date I still haven’t fully evaluated Tuist Cloud, but with this latest release this is now top of the agenda. TCA and the phenomenal work that Brandon Williams and Stephen Celis output on a weekly basis on PointFree.co is also what I put time into. Let’s see.
All the code mentioned above is in the Tuist-Pokedex repo in the scripts folder. I hope you find this useful in your day-to-day work and can improve testability and optimise those CI pipelines while working with Tuist, or just Swift.
References
Repos
Test Coverage Fixtures example
Find Dependencies and Dependents
Apple links
Xcode Help from Apple
Tools
Sonar Source XCResult Conversion Script
Another tool that I am aware of, but did not evaluate is XCParse.
Blog posts
Code coverage analysis with llvm-cov / Xcode 7 and Slather.
Another post (on PSPDFKit) on code coverage analysis with llvm-cov and Slather
A post about XCParse, another tool that extracts data from tests but takes a different approach by using xcresulttool , but isn’t relevant to what I was investigating.
Other References
