Tuist Your Circle — Building A Project Faster With CircleCI — Part 2

Old cogs used long time ago on cableways Sugarloaf Mountain. Photo by Isis França

Getting started with building a project on CircleCI is simple, doing it efficiently requires deeper insights. In this article I build on part 1 where I introduced the syntax basics, and go through the steps of installing and caching Tuist. Then I optimise the workflow and reduce the running time by more than 85%.

The project I will use as a reference is Tuist-Pokedex, the same one I have been writing about in the previous article series here on Medium. I will now take it a step further.

The Starting Blocks

The set of commands that a workflow needs to step through are the following:

• Checkout
• Install tuist
• Fetch dependencies
• Cache warm
• Generate the Xcode project file
• Build the project
• Run all or some of the tests

The first step is a pre-defined command in CircleCI and is simplycheckout. It is possible to define your own commands using this syntax:

Example of a custom command

Note that the pipe symbol can be used with the run-command syntax to execute terminal commands directly in a separate line (and multiple lines) as if it were a shell script. For this workflow I group install, fetch, cache warm and generate together as one command. In this project I am not using a signing certificate, thus the tuist build command will not work. The alternative is to fall back to using xcodebuild with parameters that define the simulator with a specific OS version.

Thus the first working workflow is the following:

First working build and test YML

Caching In The Cache

When tuist is installed on a system, it uses its own caching for the command binary. If a workflow caches these files on CircleCI, then tuist won’t have to be installed every time, and it can reduce the running time a little.

There are 2 commands that come in useful here: save_cache and restore_cache . The save command requires a unique key that the restore command uses to restore the files. The standard way to generate a key is to use a combination of branch and revision like so:

source-tuist-{{ .Branch }}-{{ .Revision }}

The save command can be as follows:

- save_cache:
    name: Save ~/.tuist
    key: source-tuist-{{ .Branch }}-{{ .Revision }}
    paths:
      - ~/.tuist/Cache
      - ~/.tuist/Versions

The restore command is quite similar:

- restore_cache:
    name: Restore ~/.tuist
    keys:
      - source-tuist-{{ .Branch }}-{{ .Revision }}
      - source-tuist-{{ .Branch }}-
      - source-tuist-

Thus if the restore_cache is working then the workflow can skip the install command. If you want to use a specific version of Tuist, you can store that number in a .tuist-version file. Note that the curl command to install Tuist will always take the latest version. When a workflow runs any of the Tuist commands, tuist will install the version defined in the version file before executing that command if it is different. This means that tuist can be installed twice. A work-around is to download the compiled binaries as a zip file from GitHub and use that instead. Another option is to use the tuist bundle option to add the tuist binary to the repo directly. The tuist install command stores the binary in the tuist cache folder and can be checked by listing ~/.tuist/Versions. The conditional command for the install can check for the absence of a version using the shell command [ ! -f ~/.tuist/Versions/3.2.0/tuist ] .

The install command with checks and version looks like this:

One thing to note with a CircleCI cache is that once it is created, it cannot be modified or deleted (CircleCI takes care of this deletion process). The save_cache command should be run only at the end of a workflow as several of the steps using tuist commands generate data that should be in stored the cache. For example the tuist test step generates hashes for each test target and it uses them the next time it runs to know which targets it can skip.

Another thing to note is that once a specific verison of tuist is installed the tuist command must use the full path from Versions like so:

~/.tuist/Versions/3.2.0/tuist

At this point I ran a quick test to see how is the CI performing. I created a branch called basic-circleci-with-cache and added a small code change in the HomeUI module to trigger a CI build. Below is a generated graph (using tuist graph) of the modules to give a better idea of how it fits in the hierarchy. HomeUI is the orange cylinder on the left in the middle row.

Generated graph of all the targets in the Pokedex project

In previous runs I only made changes in the YML file so the tuist cache would hold the previous cached values for the tests. By making a code change, this triggers tuist to compile the source code again. Tuist is smart enough to only run the test targets that have changed. It won’t run the test targets (in green) of the other modules — it will only run HomeUITests and PokedexTests. The only thing is that in making such a change, Tuist will compile all the targets in the workspace. So although it is smart enough to run a subset of the testing targets, there is plenty of room for improvement, and it is on the Tuist roadmap. This made me question why would I have an explicit build step in the workflow when it isn’t required? It isn’t needed. tuist test will also generate a project, so there is no need for an explicit step for it.

Optimising The Configuration

With the changes in place, if there are no code changes then the most significant time taken is tuist fetch . Can a workflow cache this as well? It turns out that it can.

The fetch command has most significant impact in overall run time when caching is implemented.

When a workflow runs the fetch command, Tuist parsesDependencies.swift and downloads external dependencies to theTuist/Dependencies folder. This is what needs to be cached. The checksum of the Swift source file can be used as the key. Thus the commands look like this:

When I test this out I get some really surprising results in that it doesn’t work as expected.

Fetch downloads JGProgressHUD and takes 25s

For reference, here is a screen shot of the fetch process downloading the single dependency that takes 25 seconds. To check the impact of the cache, I add a second call to fetch in the workflow to use as a reference to compare times.

As you can see the log for both cases is the same, therefore the time taken should be the same, but it clearly isn’t. The second run takes a mere 2 seconds whereas the first takes 24, which is only slightly different from the case without a cache being restored. The fetch command takes advantage of the Package.resolved file under Lockfiles and can skip downloading anything that it has done before. But for some reason it isn’t doing that on the first run when the dependencies cache was restored. I concluded that I need to refine the fetch command further. It must check for the existence of the .build folder under SwiftPackageManager, and skip the command if it exists.

[ -d Tuist/Dependencies/SwiftPackageManager/.build ] && echo "Skipping fetch" || ~/.tuist/Versions/3.2.0/tuist fetch

Now we’re talking! This works really well.

The final list of command are as follows:

- restore_tuist_cache      
- checkout        
- restore_dependencies_cache      
- install_swiftlint      
- install_tuist       
- setup_tuist      
- run_tests      
- save_dependencies_cache      
- save_tuist_cache

The one catch with this action is that the restore_dependencies_cache must be after the checkout command because it relies on the source to generate the key.

Good (Build) Times

Since running a build on a CI is mostly related to ensuring that code changes have not broken any of the functionality, the compile time of the code will still be the most significant factor in the workflow.

Workflow timing diagram without caching where the blue indicates tuist test

Workflow timing diagram with caching

By enabling caching the build time can be reduced from about 5 minutes to 43 seconds. When I committed a code change to trigger the source code building the run time is about 4 minutes 45 seconds. This is only slightly faster than without caching. The difference is small because being a sample project, there are very few tests added, and so optimising testing has minimal effect. In a larger project I would expect the difference to be very noticeable.

Credits And Running Cost

When it comes to build times, time = money. A faster build time means it will cost you less. Although I have been running this demo project on the free tier, that isn’t enough for a large development project.

There are other factors to consider such as resource class or the power of the vCPUs being available and the number of GB of storage, among others. You can see the consumption of each workflow in the Insights tab and it’s a good idea to keep a regular check on usage to prevent surprises happening (like it did with us).

For comparison using the medium resource class:

1 minute = 48 credits
5 minutes 54 seconds = 293 credits

Recap

In this article I started with the basic build and test of a Tuist-based project. I then introduced 2 caches that can reduce the run time of the workflow by over 85%. Having a cache for the Tuist cache folders, it is possible to avoid the install and cache warm with the latter being more significant. Caching the external dependencies means avoiding having to run the fetch command. Since tuist test will generate the project and compile the code, there is no need to have a separate step for these.

In the next article I will take a deeper look into testing and more.

The CircleCI project that has been the test case is here.

This is the final configuration:

Update

The cache warm command can be omitted as it doesn’t do anything for the CI. It is only useful for running on a local machine.