Long story short, it's about parallelization and the use of cache.

The context is that a year ago I made many CI jobs to test and verify the quality of the code and one of them uses CodeChecker, a powerful but RAM/CPU-greedy tool. This was because my machine isn’t powerful enough to run it locally (even though the project’s codebase is small). This way, I could push to GitHub and the CI job would verify code quality and give me results as an HTML file.

It was good -GitHub gives a lot of CPU/RAM power for free- but over time I added more code to check (I was thinking: why not statically analyze unit tests too? It's weird but why not :). And I didn’t know when it happened (build was always failing due to linter's findings). Recently I saw that the build was failing because it was killed by GitHub due to RAM/CPU usage!! There were no results either, so the whole idea of crunching the data remotely failed.

I tried to optimize and finally I’m happy with the results: 1h -> 4min. But there were many steps to make it work. They are more or less in chronological order.

Optimize Dockerfile

I removed what I no longer need (for me it was include-what-you-use, because well, I will no longer include anything - I will use C++20 modules anyway, but that’s a different story). Using multi-stage builds where the most frequently modified parts are at the end can help too. Good to cleanup but this didn’t give me a lot.

Split GitHub's CI .yaml files into multiple ones

The syntax of the .yaml GitHub uses is a huge boilerplate. The indent-based nature of .yaml doesn’t help either. I was just not capable of fixing one big yaml. It's like that now. No gains, but very useful for the next step. 

Build caching

Saved me 10 minutes—very useful!

The idea is that if the Dockerfile was not modified from a previous build, it will not be rebuilt - a cached copy of what was built will be used.

It works in a way that the principal build does the building if needed; other jobs wait for it. If no changes for me it takes ~20 sec. If there were changes (very rare), full build, me me its ~10min. Because I almost never update Docker file I almost always finish in the first case so I'm now saving 10min (previously it was always 10 minutes).

One negative is that each of those parent jobs will have their build artifacts with them. There is no single place to get all build artifacts from a commit. I would prefer to have a tree-like structure with a node as a build name and subnodes of build artifacts.

Matrix jobs

This doesn’t speed up things but is helpful in the next steps. This avoids boilerplate code in .yaml, as a simplified example:

strategy:
  fail-fast: false
  matrix:
    include:
      - name: Business Logic
        make_target: test_biz
      
      - name: Device
        make_target: test_dev
        
steps:

  - name: Run Unit Tests
    run: |
      "cd /workspace/build && cmake -G Ninja -DCMAKE_BUILD_TYPE=Debug .. && ninja ${{ matrix.make_target }}"

Parallelize the static analysis

This is the game changer with execution down to 4 minutes!!

The thing is CodeChecker uses compile_commands.json, generated by CMake. I have a "main" CMake that includes parent CMakes, and CMake will just put every cpp/hpp used by any projects he is able to build inside compile_commands.json.

We need separate compile_commands.json for every executable and run parallel builds for each of those executables. It's not eassy to generate compile_commands.json, but I've found a trick gow to make it:

We can run CMake like that (note that I use ${{}} trick from previous step):

cmake -G Ninja -DEXPORT_SINGLE_JSON=${{ matrix.target }} -DCMAKE_BUILD_TYPE=Debug .. && ninja CMakeFiles/cstatic

 Then in the CMake I have:

  if(DEFINED EXPORT_SINGLE_JSON AND EXPORT_SINGLE_JSON STREQUAL "${TARGET_NAME}")
        message(STATUS "Enabling compile_commands.json export for: ${TARGET_NAME}")
        set_target_properties(${TARGET_NAME} PROPERTIES EXPORT_COMPILE_COMMANDS ON)
    endif()

This works and it's the main point of this post :) 

Swap for Docker image in case we run out of RAM

I added it just in case in yaml file (note that it can't be more than 4G or GitHub will fail to run the job):- name: Enable Swap Space

  run: |
    sudo fallocate -l 4G /swapfile
    sudo chmod 600 /swapfile
    sudo mkswap /swapfile
    sudo swapon /swapfile
    free -h

Limit CodeChecker jobs

Use --jobs 1, TBH, I don’t know if, with all the modifications above, higher values couldn’t be used for some jobs.

    COMMAND CodeChecker analyze compile_commands.json
        --output ${CODECHECKER_ANALYZE_DIR}
        --file ${CODECHECKER_SOURCE_DIRS}
        --skip ${CODECHECKER_SKIP_FILE}
        --jobs 1

That's it all is on GitHub, CI is failing due to ongoing other refactor.