In my current project at my place of work we had a serious problem: Long build times and slow unit tests.
What do I mean with long? On our Jenkins a full build with unit tests needs approx. 1:20 hours. On our developer notebooks, the build without any tests took the same amount of time.
For me, coming from the nice and beautiful Java world at the university where builds execute within seconds or at least minutes, this was incredible. Hence nobody in our team has executed all unit tests locally for a while. Even worse!
Since I was new to QT projects and hadn’t any experience with qmake, the makefile generator that comes with QT, I decided to invest a rainy weekend to see what is possible. Luckily my first successes look so promising that my team and I decided to spend even more time on build time reduction in the following sprint.
In this post I will summarize what helped us and what didn’t.
Main Achievements First!
You may want to know what we have achieved. Right?
So all in all we were able to reduce our build and test time from 1:20 hours to 24 minutes. Not perfect in my opinion, but much better. From these 24 minutes, approximately 16 minutes account for a full build and only 8 minutes go on the unit test time.
Before our optimizations, the unit tests alone took more than 20 minutes.
Main Reasons for Long Build Times
In our project, there were three main reasons for the long build times:
- Many Dependencies
- Many Executables / Targets
- Odd Build Files
Lets look on these reasons in detail.
Dependencies are the root of all evil I guess. Whenever you compile one class, you need also to compile the classes it depends on and their dependencies and so on and so forth.
Newer parts of the code base were written using TDD or at least while keeping testability in mind. Therefore, the number of dependencies is low and almost each class can be tested on its own.
However, older parts of the code are a bit like what Robert C. Martin calls a “big ball of mud” and the number of dependencies in these areas is very high. And so, the tests include many many headers and sources that need to be compiled in order to test a single class.
The best approach to shorter build times would be to break these dependencies to obtain test suites that only compile the test cases and the class under test. Sadly, this requires understanding and rewriting most of the tests and the productive code of course which would take much more time and is better done incrementally. Therefore I decided to try other, less invasive, possibilities first. More about that in a minute…
Executables and Targets
Luckily, our application consist of different libraries and plugins that can be compiled (and deployed) independently. So far so good. And also our unit test suites are independent targets and every test suite yields its own executable. This is a good thing in general since we can compile and execute every test independently.
However, this has some severe drawbacks when compiling more than one target, if multiply targets are based on the same source and header files. In this case, these files are compiled over and over again.
In the schematic example above, you can see three targets, Test 1, 2 and 3 which depend on different classes. The following table shows you, which classes need to be compiled for each test.
|Test 1||Test 2||Test 3|
Easy to see, that this awkward looking Resources class needs to be compiled for every single test. And even class C needs to be compiled in Test 3 which tests class C of course, but also in Test 2, which tests Class B (that depends on C). Sometimes, there are ten or more test targets that depend on the same classes. And every test compiles every source file again.
In combination with many dependencies, this is really really bad and has a huge impact on the build time.
Odd Build Files
Since Qmake (at least for Qt 4.8.6) is not that good documented and there exist many undocumented features almost nobody knows about, every developer fiddles around in the Qmake files until finally the build seems to do what she expected.
Needless to say that the first working solution is not invariably the best. Many of our *.pro and *.pri files contained needless lines or lines that harm the build time.
One example for this is the statement “CONFIG += ordered” in a subdirs-project which causes all subdir projects in the current directory to be compiled in order no matter if there are dependencies between these projects or not.
Our Approaches to Shorter Build Times
Our main issue was, that sources were compiled over and over again. Some sources were compiled once for the productive lib or plugin itself and then again for every unit test.
But not only compilation was done very often for these files. Also, moc objects were generated very often. Moc objects are an intermediate step of the qmake makefile generator that extends classes with the “magic” that is needed for QT signals and slots and some other things. Before compilation, every class that inherits from QObject is translated into a moc class using the preprocessor. Afterwards, these moc classes were compiled like other classes.
Modifying MOC_DIR and OBJECTS_DIR (~ -37,5%)
Therefore, our first approach was to reduce the number of compilations per class. For this purpose, qmake allows us to modify the OBJECTS_DIR and the MOC_DIR.
These qmake variables define the place were the intermediate files and the o-files are stored. By default, every target has its own build folder were these files are stored.
I decided to redefine these variables so that there only exists one such folder per plugin or library. As conclusion, if the productive code was already compiled, it is not recompiled while the unit tests subprojects are compiled. Also the moc files have not to be regenerated.
Another advantage of this redefinition is an increased TDD-ability. Before, when changing the productive code, we had to switch to the test class, recompile the test project and execute it. Now, wie just need to recompile the class under test and it will automatically replace the old object file which directly affects the test case. This speeds up the Red-Green-Refactor-Cycle a bit.
Though this approach has decreased our build time by approximately 30 minutes, it has some drawbacks and pitfalls I will discuss later on.
Reduce Usage of CONFIG += ordered (~ -18,75%)
When using qmakes subdirs project template, you define subprojects as follows:
SUBDIRS += subdir-a subdir-b subdir-c
Furthermore, you can define that all subdir projects are compiled in order which is necessary if there are compile time dependencies in between:
CONFIG += ordered
However, in our project, this configuration option was set in every single project. On the highest project level, this option was defined and on every level below too. Everything was build in order which makes life easy because you do not have to think about dependencies, but it also makes compilation slow.
Luckily, we had a subproject that only contains all our plugins. By definition, there should never by compile time dependencies between plugins and so, I removed the ordered option here. Furthermore, many of our test projects have not to be compiled in order. I removed this option there too.
However this caused some problems with dependencies between subdir projects in the tests directory. At some points this was a hint for bad project structure sind tests depend on each other for the reason of common used mocks or fake objects. In these cases, we just extracted these mocks into a mock subdir project.
But this hasn’t solved the issue. Our tests subdir project structure looked as follows:
SUBDIRS += mocks test-a test-b test-c
No problem here with CONFIG += ordered since the mocks were compiled before the tests. Without CONFIG += ordered however, this is not guaranteed.
In order to get the best out of both worlds and keeping the modifications small, we just pulled the mocks a lever higher. Therefore, the pro files looked as follows:
CONFIG += ordered SUBDIRS += tests/mocks tests
And the tests subdir itself:
# no CONFIG += ordered here SUBDIRS += test-a test-b test-c
Now, the mocks are build first and then all the tests, unordered.
This gave us 15 minutes of build time reduction.
Precompiled Qt Headers (~ -6,25%)
We also tried precompiled headers for often used Qt classes in the core and gui module. Therefore, for every plugin or lib, the Qt headers were only compiled once.
Only a small improvement, but approximately 5 minutes in total.
Updated jom to 1.1.0 under Windows
Removing CONFIG += ordered worked fine on Linux machines but just had no effect on our local build machines that run Windows.
I found the reason in our jom version, which was 1.0.14 at that time. Jom is a replacement for mingw-make which is able to use multiple cores and therefore builds much faster than the default make from Mingw.
However, in version 1.0.14, it silently builds all subdir projects in order, wether the ordered option was defined or not. We stumbled about this since our Linux build crashes due to ordering issues, after removing the ordered option from the pro files while our local builds worked fine.
By upgrading jom to version 1.1.0, we got support for the ordered option (or its absence) also on Windows machines.
Bonus Tip: Do not use += in pri files.
We only had this issue in very rare cases. In qmake files such as pro and pri files, you can add a configuration to a variable using += or *=.
The only difference is, that it is not added again if it is already contained, when you use *=. If however += is used, the option is added again.
In pro files, both variants are equal since pro files are only called once per compilation. Pri files, in contrast, are meant to be included in pro files and other pri files as well. Hence, when resolving all includes in a pro file transitively, a pri file might be included more than once.
If this pri file adds an option to a variable using +=, this option is contained multiple times which slows down the qmake phase and therefore the generation of make files.
What has not helped
Removal of duplicate includes in pro and pri files
I wrote a script to clean redundante include directives in pro and pri files since I thought, that maybe unneeded and duplicate includes may slow down the qmake phase.
This had exactly no effect. Obviously, this is not an issue or it is already handled by qmake itself. So don’t waste your time on this!
Removal of CONFIG += ordered under jom 1.0.14
As I told you: Removing CONFIG += ordered while using an outdated jom also had no effect on Windows machines.
Problems and Drawbacks
As I mentioned before, the optimizations, discussed above, came not without any price. In this section, I will explain some disadvantages of these changes and how to cope with them.
The main reason for the issues we have seen with our improvements were due to putting all moc- and o-files in the same directory. Having one directory for all objects in a plugin as well as the objects in unit test projects for that plugin can give you several problems:
Potential Name Clashes
You will get name clashes if you have files with the same name within a plugin or one of its tests.
Normally I would say that this should not happen in productive code. (Why have two components with the same name within the same project / plugin / lib?)
However, we faced these issues mainly in the unit tests since developers sometimes had copied mocks from one test to another if they were needed in both tests.
So the good thing about this drawback is, that it reveals some smells in the structure of your test projects. We extracted these mocks as mentioned earlier and the problem was solved.
Truncated Object Files
This one cost me a lot of time. Since we build multiple subprojects (test projects) unordered and give them the same output directory (the same directory for all o-files), the build breaks if two subdir projects are compiled and relay on the same (yet uncompiled) source files.
In this case, the file is compiled twice in parallel while the output is written to the same file. This leads to truncated object files and a red build of course.
Since when doing a full build we compile the productive code first and the test code after, this issue mainly arose from tests having cross includes to other tests. Again: Bad test structure.
Tests should be independent from each other and therefore not have any dependencies to other tests. Luckily, these dependencies mainly were due to mocks that reside in one test but were also needed by another test. These mocks need to be extracted into a separate subdir project for mocks that is compiled in order before every test for the specific lib or plugin.
Again I would say that this problem is self made and a strong hint to smells in our test structure. Thanks to the build improvements we had to solve some of these awkward structures in our unit test, which at the end is a good thing.
After all these improvements our build and test time is only 30% of what it was before. This is a great deal for us since our development was speeded up a lot and we save an immense amount of time day by day.
This speedup is not due to the build improvements alone but also due to unit test optimizations.
The Role of Unit Tests
While improving the build we have also started to measure the execution time of every single unit test and to optimize tests if possible. In total, of the 70% speedup, approximately 60% come through build optimizations and 10% (12 minutes) through unit test optimizations.
This reveals one of the issues of slow builds: One minute more or less does not matter and therefore bad and slow unit test can sneak in and increase the build time more and more. As conclusion to this, we started to track the unit test time on our Jenkins to get a quick feedback when introducing slow unit tests.
What is Next?
For me, 24 minutes seem still too long but it is a lot better than 1:20 hours. Further improvements could be possible but would mainly require changes in our code base and will cost us a lot of time. Here the Pareto principle strikes, I guess.
Another thing to try out could be distributed builds. Since we do not rely on ordered builds as much as we have done before, it might be possible to build some parts on different nodes (or on heavier build machines).
If you have ideas for further improvements feel free to contact me or leave a comment below this post. I am always interested in new approaches to this build time problem!