CI/CD at Scale

Background: The CI/CD of the company was hard to maintain and used anoldish stack'. The challenge was to create a reliable, versatile and fast pipeline for different kinds of technologies in order to help teams deploy faster and with autonomy on the on-premise infrastructure. Many challenges have had to be solved :

• how to create this pipeline
• how to maintain it in a reliable way with tests
• how to convince every project team of its benefits

Moreover, unit and smoke tests have

become mandatory in order to help project teams maintain a minimum confidence in frequent delivery.

Goals: Easy CI/CD for everyone.

Solution & Results: We used a scripted pipeline with a shared library. This library is based on the abstract "Builder" which defines the contract of building an application, releasing it with version increment in order to promote it to different environments. Many "Builders" have been created: Maven, Gradle, npm, INMI, PHP and more are coming!

Next, we define a "Deployer," which is in charge of deploying a version of an application on an environment. Due to the legacy infrastructure, this feature enables deploying on the on-prem infrastructure with dedicated VM on customized Linux distribution and also on a new Kubernetes Cluster on an hybrid infrastructure. But it can also deploy applications on proprietary software when a CLI is available.

Those two key components are tight together with some DSL instructions used by the project in order to have a Jenkinsfile with less than 20 lines. Then, some metrics were added to enable visualization thanks to an Elastic Stack with some Kibana diagrams available to everyone in the company. Every stage in the pipeline adds a new metric in Elastic with the name of the stage, the technology, the name of the application, the environment, and so on. Thus, the management can follow the adoption and the delivery rate of projects. Teams also inspect metrics to track a high rate of failure of deployment and anticipate corrections. Finally, the state of the pipeline is published on Slack to help teams to follow their build and deployment.

The key capabilities we employed were:

• The Jenkins shared library: It is the key feature of this project. It enables the sharing of the same code for every project with ease and allows teams to concentrate their effort on the code of the application with little worry about the pipeline.
• One of the benefits of Jenkins is the "build with parameters" which has been heavily customized depending on the type of pipeline the project wants to run. It differs based on the git branch, type of technology and configuration written in the Jenkinsfile.
• Groovy scripts are also very easy to handle for java people but pipeline tests can also be time consuming.
• Around 60 DSL instructions have been added to help projects to use the pipeline and 100 groovy classes have been written for the pipeline.
• The Docker plugin is the most used in this pipeline as every single script or external program is run in Docker containers with custom images dedicated to different tech.

The development started 4 years ago with the very first version of the scripted pipeline and it has been rock solid since. Results included:

• development release shortened from weeks to days
• no more human approval for UAT deployment
• Docker and Kubernetes deployment without fear for teams