Build docker images
- If an MR changes anything in docker containers, or needs changes in the docker images (i.e bump some version), someone has to manually trigger the updates on all machines we use as runners. This isn't very scalable and rather fragile.
- If for any particular MR or thing the docker images are changed, they are changed for all branches and pipeline at that point in time.
- Building docker images takes forever, as all images are rebuild using the tests. Docker use caches, but the tests are still run for all images.
@lasser and I discussed some idea, which are just loosely presented here. In some areas, I have to dig deeper and I'm not really sure how much work that is. So any input is appreciated!
Spin up a docker registry. This would be run on a VM from RBG, then the docker images would only be build on a single machine and be pushed to the registry regularly. The GitLab runners would use the registry instead of local docker images. Most likely there would then be a new pipeline stage at the front, which would check for changes in the docker files and if necessary rebuild the docker images.
To make that work properly, and also reduce the necessary amount of work, a small "dependency tree" is most likely necessary. What I mean by that, is that the build script should check what docker files changed, and know what images depend on the changes ones and build them. A change in the a leaf image, should only need the change in that particular image. Changes in images further up will require more images to be build. So at some point I'd port the build script to python and in some way represent that tree there.
For the second point, I think some form of versioning in the docker images will be necessary. My first idea was to just add a last '.xx' at the end of all images, which would be the internal versioning. An update, which is not reflected in the versioning (e.g. the gcc-10 image doesn't version its CMake version), would increment this last digits. Then an MR could bump this number, update the CI images and be independent of other branches and pipelines. But it's a little hacky, so I'm very open to suggestions!