CI with Multigig-sized Repo with Jenkins

Background: The company I work for focuses on digital industry solutions, such as integrating the virtual and physical, hardware and software, design and manufacturing worlds for our customers. That's why we needed the best CI/CD solution.

We had three main challenges:

1. The ability to checkout reliably and then embed custom JUnit Reports and QualityGates Reports – not the one which Jenkins plugin provides – like CPD, Inspection.
2. To make sure that the CI builds run for all commits/tags to release branches.
3. Ensure error handling and reporting happen promptly.

Goals: Implementing continuous integration and continuous testing for a repo as big as 100GB.

Solution & Results: To tackle our first challenge – Reliable Checkout – we used the Jenkins Git plugin, which provides several options to alter the way checkout happens. Its advanced options and pipeline code gave us the capability to handle checkout smoothly. It was straightforward to use, and the APIs and code are intuitive. To limit the size of checkout the job has to do, we used refspecs, alternate repos settings, and Sparse Checkout options. Using Scripted Pipeline and Shared Pipeline, we were able to make sure that checkout happens reliably by retrying on its first failure.

To handle continuous builds, Jenkins provides the ability to accept Generic Webhook from any client. Using that, we made Bitbucket push webhooks to Jenkins whenever a PR is merged. Since PR destination is a release branch – with a bit of processing and filtering – we could control the CI builds precisely as we wanted. Again, this all was more controlled as we were using scripted pipelines for our webhook receiver job.

As to the need for better error handling and reporting, the Jenkins extended-email plugin helped us in this regard. Using pipeline, try-catch block enabled us to exit at any stage and report the error to desired user groups.

The Parameterized Trigger Remote Job plugin helped us trigger jobs across the Jenkins server, enabling us to handle them better. This plugin simplified our use case of triggering various quality gates spread across different Jenkins. The JUnit plugin helped us create reports embedded in the Jenkins build, giving us an idea of how each delivery affected the release branch from a Unit Testing perspective. Also, it tells the commit/tag where failures/errors were first introduced.

One thing that I really missed in this plugin is the ability to get the counts and list of tests. We need to use Jenkins APIs for this purpose. It would be great if this plugin had a way to get these details upfront.

My favorite plugin is the Generic Webhook which is an excellent way to trigger builds on Jenkins from BitBucket/GitHub. Although there are plugins specific for each vendor, I think this plugin works great and is easier to use than others.

Because of the large set of mature plugins and the support of the Open Source community, Jenkins is the first choice for any CI/CD solution. We are incredibly pleased with our results:

• continuous testing of release branches
• faster feedback on the quality of code
• improved quality of the product because of CI and CT
• 10x faster integration of new QualityGates