Background: Amarula Solutions places a strong emphasis on Continuous Delivery to enhance their development process and foster improved team cooperation. Their fundamental approach involves integrating various software tools to achieve greater efficiency in development and build processes, alongside the rapid identification of potential issues.

Goals: Involve seamlessly integrating diverse software tools to significantly boost efficiency in development and build processes, while simultaneously enabling the rapid identification and resolution of potential issues.

Solution & Results: Primary solution was to strong embrace of Continuous Delivery, which involves the strategic integration of multiple software tools. This concerted effort yields significant results, including a demonstrably enhanced development process, improved cooperation among their teams, and a notable increase in efficiency across their development and build operations. Furthermore, this approach enables them to swiftly identify and address potential issues, leading to a more robust and streamlined workflow.

Here's how Jenkins is an integral part of Amarula Solutions' story:

• Automated Verification: Whenever a developer submits a commit to Gerrit, Jenkins springs into action. It automatically fetches the commit and runs a verification build. This crucial step ensures that the new changes haven't broken the existing codebase. If a build fails, Jenkins automatically notifies the team (via Mattermost) and retains logs for troubleshooting.
• Development Flow and Quality Gates: Jenkins is deeply integrated into Amarula's development flow:
  • New changes or patchsets in Gerrit automatically trigger Jenkins verification jobs.
  • These jobs can build the product, run tests, and perform static code analysis (using tools like Sonarqube and Codechecker).
  • Jenkins posts verification scores back to Gerrit, ensuring that changes are merged only after meeting specific quality standards (e.g., +2 review score and +1 verification score).
  • Static code analysis results, including new violations, are provided directly to Gerrit, allowing developers to address issues proactively.
• Flexible Build Triggers: Beyond automatic triggers from Gerrit, builds can also be initiated manually by users through the Jenkins UI or scheduled to run at specific times.
• Distributed Builds with Slave Nodes: Amarula Solutions utilizes Jenkins slave nodes, which can be configured with specific operating systems or hardware to handle different types of jobs. This allows for powerful machines to build Android OS, while basic nodes can handle Android app builds, and dedicated nodes with connected test devices run automated tests. This distributed architecture enhances efficiency and scalability.
• Optimized Downloads with Proxy Cache: To reduce the time spent downloading dependencies for builds, Jenkins uses a dedicated Docker container that acts as a proxy cache. This container caches all artifacts that have been requested at least once, significantly speeding up subsequent builds.
• Shared Libraries for Reusability: Amarula Solutions extracts common pipeline code into shared Jenkins libraries. This approach reduces redundancy, keeps code clean, and ensures consistency across various projects. These libraries handle tasks like checking out source code, running builds in specific Docker environments, and providing build results to communication channels.
• Release Automation: Jenkins pipelines are used to automate the entire release process for Android applications, including building and uploading app bundles to the Google Play Store. This includes signing the bundles with secure keys and persisting uploaded bundles and configurations for tracking.
• Multi-Repository Projects: For complex projects composed of multiple Git repositories (like AOSP), Jenkins can handle cross-dependent code changes. It can check out all related changes across all repositories under the same Gerrit topic, ensuring that verification builds are performed on a consistent and complete codebase.
• Security Integration (Mend): Jenkins pipelines are extended to automatically run Mend scans, which detect and remediate security and compliance issues in open-source components. The results are sent to communication channels, tagging project owners to ensure timely action on vulnerabilities.
• Parametrized Builds: Jenkins allows users to configure builds conveniently via a graphical interface using parameters. This enables setting options such as specific software flavors, build types (production vs. debug), source code revisions, and deployment targets.
• Changelog Generation: A dedicated Jenkins changelog library is used to automatically generate changelogs between specified start and end references, providing clear documentation of changes for each build.
• Beyond Software Development: Jenkins' capabilities extend beyond traditional software development at Amarula Solutions. They utilize Jenkins jobs and graphical interfaces to automate processes not directly related to software, such as generating employee reports by accessing APIs from YouTrack and Gerrit, filtering data, and producing XML reports for invoicing. This demonstrates Jenkins' versatility as an automation platform.

Our results include:

• Enhanced Development Process
• Improved Cooperation Among Teams
• Better Efficiency in Development and Build Process
• Quicker Identification of Potential Issues
• Automated User Account Commissioning and Decommissioning
• Automatic Onboarding Process (including welcome emails and documentation)
• Full Overview of Task Completion and Commits (via Gerrit and YouTrack integration)
• Automatic Testing Execution on Each YouTrack Task
• Automatic Verification Builds for Each Patch
• Ability to Quickly Run Builds on Any Commit
• Detection and Remediation of Security and Compliance Issues (via Mend)
• Generation of Vulnerability Reports (PDF and tabled)
• Automated Changelog Generation
• Automation of Non-SW Development Processes (e.g., employee reports, user account creation)

In today's business world where the digital wave is surging, Alauda Company has always been oriented towards customers' needs and actively explored innovative methods to create more value for them. Among these, Jenkins has become a crucial basic platform for us to achieve this goal. By adding plugins, we have endowed Jenkins with more powerful capabilities, providing strong support for customers' digital transformation journey and significantly enhancing user satisfaction.

Facilitating Customers' Digital Transformation

In the case of a financial services customer, we utilized Jenkins plugins to realize the automated testing and deployment of the financial trading system. The real-time and accuracy requirements of financial trading are extremely high, and any slight error could lead to huge losses. Through our solution, whenever the system is updated, the Jenkins plugins will simulate various complex trading scenarios for rigorous testing to ensure the stability and accuracy of the system. Meanwhile, the rapid automated deployment capability enables new features to be launched in a timely manner, meeting the rapidly changing demands of the financial market and helping the customer gain an edge in the digital financial services field.

Enhancing User Satisfaction

By providing customers with powerful functions based on Jenkins plugins, we have significantly enhanced user satisfaction. During the digital transformation process, customers have deeply felt the improvement in efficiency and the reduction in risks. They no longer need to worry about cumbersome manual operations, frequent errors, and long waiting times. Customers have been full of praise for our solution. Their businesses have flourished after digital transformation, and the user satisfaction has reached an unprecedented level.

Summary

In conclusion, by adding plugins on the basis of Jenkins, Alauda Company has unearthed more potential value for customers, effectively promoted the process of customer digital transformations, and simultaneously won high satisfaction and trust from customers, laying a solid foundation for future cooperation and development.

The Network UPS Tools project will soon be a quarter of a century old and, being the first open multi-vendor power hardware monitoring solution for UPS, ePDU, Solar and such, it became the de-facto standard in this area (de-jure with RFC 9271 recently published). Over the years it was integrated and included in all sorts of server and desktop operating systems, from embedded to mainframe and NAS, and built with a wide array of toolkits and third-party dependencies. As evolution marches on, warnings are eradicated and features are added, the codebase is still expected to run on any platform released over the past couple of decades. If the machines and their OSes still run, modern NUT should work there too.

With one of my first goals as an active community member and eventually project maintainer, I set out to address the hundreds of compiler warnings emitted by different implementations and versions of toolkits during builds -- since they did raise valid concerns, and presence of those reports obscured visibility of new bugs introduced by contributions. In fact, several large and useful changes sat in the PR queue for years because nobody including authors had a good grasp on their reliability. This "fightwarn" effort took a couple of years with an in-house Jenkins farm for a project using NUT, and Travis CI (free for FOSS at that time) as well as one team member's BuildBot instance handling several versions of Linux for a few CPU architectures to cover different bitness and endianness, as well as a mix of GCC and LLVM CLANG. Quite a few issues were only relevant to certain generations of toolkits, C standard revisions, Make or Shell implementations, Autotools versions... and some fixes for one platform caused bugs for another. So to keep NUT working everywhere, its iterations had to be regularly built everywhere.

Eventually the reign of free Travis CI ended, and NUT got VMs for a CI farm sponsored by Fosshost.org and later by DigitalOcean to proceed with the multi-platform testing.

Using a custom Jenkins instance to take care of the project's codebase builds and make use of numerous operating systems in other VMs (as SSH Build Agents), as well as build agents contributed by community (e.g. Swarm agents), was a natural choice: by that time work had started on the https://github.com/networkupstools/jenkins-dynamatrix project. It is a Jenkins shared library which constructs a huge Map of parallel stages similar to a standard matrix build.

However in this case, it is not the static matrix definition fully dictating what should be built, but rather the build agents are expected to report capabilities with their labels -- such as which platform they run and what versions and implementations of toolkits are available to test with, and whether "everything" can be built or just certain profiles (not all OS distributions offer all of the dependencies that NUT can build against). The library also helps determine which C or C++ language revisions can be built according to GCC or CLANG version installed. Whenever a build starts, the NUT Jenkinsfile (consumer of jenkins-dynamatrix library) can evaluate what it can build today -- based on currently known population of build agents, and so constructs the matrix for testing.

A sibling https://github.com/networkupstools/jenkins-swarm-nutci project is used on many of the build agents which can dial in to the NUT CI farm and provide their services and capabilities (conveyed by the labels which are part of their configuration), and includes some scripting for a common configuration approach as well as for automated start-up on numerous platforms. It might be usable "as is" or easily adapted by other projects.

This arrangement allows NUT to be built on a multitude of platforms such that common free CI platforms do not offer. It is possible to find ways to build FOSS on recent Linux, MacOS and Windows, and sometimes even on non-x86 CPUs. It is nearly impossible to find builders for FreeBSD, OpenBSD or Solaris/illumos, to name a few, or for Linux distributions older than half a dozen years.

Every iteration of NUT is now tested with GNU, BSD and Sun Make implementations, with Shell helpers tested by Bash, Dash, Ksh, Busybox and others, with GCC starting from early 4.x releases and Clang from 3.x up to the dozenth releases out recently, across a dozen hardware platforms (some in QEMU), and several revisions of C/C++ standard with and without GNU extensions. All in all, with recently regularly connected builders about 250 stages are prepared (some running several build scenarios as scripted inside, so roughly a thousand builds happen) for each code iteration. Sometimes a nuance warning causes one of these scenarios to complain during PR builds -- on this or that OS, this or that compiler. It keeps us confident both that such mechanism remains useful, and that any OS the community cares about enough to contribute a builder for remains fully supported (or in the worst case, that limitations are known, documented, and perhaps worked or fenced around).

With the hundreds of builds succeeding without warnings -- reliably, repeatably, every time -- the floodgates for ingestion of long-standing PRs were open. Largest beasts included the dual-citizenship with libusb-0.1 and libusb-1.x support (queued for 5 years) and revival of the NUT for Windows effort (queued for 9 years), confidently merged without introducing warnings and regressions. The overwhelming majority of these builds happen on the custom "NUT CI" farm with Jenkins driving them. A few scenarios run on CircleCI and Appveyor to take advantage of some free FOSS builds on MacOS and Windows respectively.

Of notable interactions with the Jenkins community, this and related efforts led to improvements in the git-client-plugin to extend the use of reference repositories, instant-messaging-plugin and ircbot-plugin for querying Jenkins server status via IRC servers, lockable-resources plugin for many itches scratched here and there, and pipeline-groovy-lib-plugin for easier use of test library versions along with test pipeline branches. The inadvertent stress tests of a Jenkins controller and/or its remote agents caused by juggling several simultaneous builds each with hundreds of parallel stages, casually with a thousand items in the build queue, have also exposed certain deficiencies and led to fixes elsewhere in the Jenkins ecosystem. And very many lessons learned and friends made :)

Background: For our railway software, we needed to improve the quality, stability and delivery time of a legacy project, while giving our team a way of working smarter and faster.

Goals: Monitoring and operation of embedded railway traffic management systems.

Solution & Results: We migrated almost all components from a Subversion repository where, most of the time, people are working on a trunk branch to a Bitbucket Repository and Jenkins Pipeline.

Now every change is a node on a dedicated branch and merged after a reviewed pull request and successful Jenkins build.

We achieved superior results with Jenkins Pipelines, including:

• Fewer failed unit tests
• Improved code quality, by doing code reviews for every pull request
• Increased transparency and awareness of code changes
• Easier testing
• Build stability; Our master build is now much more stable

Background: I work for a company that creates intelligent and insight driven software that

enables manufacturers to build value chains that sense, learn, and adapt. Because of that we were experiencing the number of integrations increasing every month. This was making it hard to manage the deployments and rollbacks in an effective way.

Goals: Integrate different business units.

Solution & Results: We introduced Jenkins for our build and release process. Initially, some extra time had to be spent setting up our integrations on Jenkins. But once that was set up, our releases were straight forward and had no manual errors. Redeployment became easy. Managing and documenting releases was even easier, too. There was no more dependency on project people for the release process. Simple steps could be followed by any team member to do the release.

We relied on the following key capabilities:

• Jenkins integration with multiple source code repositories
• Automation in build and release processes

Results using Jenkins included:

• Problem free releases
• Less dependency on project team members
• Faster releases

Background: We needed to solve a Ruby review application. One in which we could build and deploy for any instance. We also wanted it to be automatically verified and push it to stage and into production.

Goals: To create a review application that will only merge once code changes are deployed and verified. We wanted Heroku-like functionality.

Solution & Results: Jenkins played a great role for integrating all tools.

We created a pipeline that will allow any code changes to the VCS software system and let it run by itself. Any developer who has done code changes and wants to see how the build will look can create a pull request. The pull request will be open through the Jenkins-pipeline, which will be a trigger to GitHub Payload to respond.

There was no plugin available to perform this kind of stuff, so we did some R&D and created conditions according to Json of payload. Once a developer is satisfied with changes, (s)he will merge the code in the main/master branch. Hence the pull request will be closed and at the backend and the pipeline will do its task.

We are deploying it on our virtual machine with the help of multiple docker containers, and later on we'll deploy on Kubernetes.

We achieved the results we were hoping for, including:

• Heroku-like review feature created
• Solving the issue to act as per pull requests
• Build logs and deployment of URL exposed features to GitHub

Background: The main challenge was to incorporate -- and have in one single place --a seamless validation and deployment of our Salesforce project, with a third-party front-end framework. That implied minimizing human input and decreasing the release time by integrating some of the necessary pre- and post-deployment steps.

Initially, just one of our sandboxes was set up for CI and there was a freestyle project for each step. Although the builds were sequential and had a scheduled pull, we felt that it was limited. After validating and integrating the changes, the development team would have to locally pull the latest and do an Ant deploy. Things had to change.

Goals: Simplifying and creating a reliable, innovative ecosystem for students and universities around the globe.

Solution & Results: We moved to Jenkins Pipelines and covered all three steps, specifically declarative pipelines which were easy to set up and Blue Ocean made it easy to visualize. Having a Stage View, the failures were easier to debug.

We then proceeded to configure the projects for UAT and production. That meant running Apex tests, as well. Doing so added to the build time and we experienced timeouts issues. Being able to monitor the situation came in handy; it turned out that we needed to increase the heap memory. Once that was done, the pipelines ran smoothly.

We made use of all these great plugins and features:

• Blue Ocean
• JavaMelody
• Git
• Ant
• SAML
• Pipeline Plugin

We had great results using Jenkins, including:

• Shorter build times
• Release times which also decreased from more than half a day to around 2 hours
• A better, more comprehensive feedback for the development team

Background: At our customer services software company, we need to be able to automate as much as we can as possible to make customer onboarding a pleasant process. Our platform needs to be fast and reliable. For that, we decided to use pipelines to:

• Automate the new environment
• Set user accounts
• Automatically communicate to our customers
• Accelerate the processes for new user/customer onboarding

Goals: Create a chatbot platform.

Solution & Results: The solution we came up with includes:

• A repository
• User accounts creation
• Deployment of 3 microservices
• Customer-generated API keys
• Customer notification of all the information they need to access and use the platform

We implemented this using lots of plugins that we were able to connect to our other systems. We built very flexible pipelines using Jenkins, and we developed groovy functions to encapsulate business logic and many other functionalities.

Results using Jenkins included:

• A faster process to onboard new customer
• Low-cost process
• Easy-to-change workflows
• Operational efficiency

Background: We needed a CI platform that would enable faster building and automatic deployment to our dev environments. The tools for this finance company are heavily restricted, so the tech team was spending an inordinate amount of time trying to figure out what to use and how to execute necessary tasks. The security constraints imposed on them limited them significantly – particularly when it came to the use of plugins.

Because of this, they were forced to use extensive bash scripts to supplement the tools provided. Another huge challenge was the need to trial all possible solutions for each of the necessary steps to be confident they designed things correctly. Finally, they had to figure out how to incorporate BladeLogic into the CI/CD picture.

Goals: Implement CI/CD to our development process.

Solution & Results: We used Jenkins to provide the orchestration needed for each build and deployment. With Jenkins, we were able to make the connection to Nexus repository, pull source code from BitBucket, and create the BladeLogic packages. Unfortunately, due to all of the internal security restrictions, we could only rely on basic Jenkins capabilities. Our team wanted to use Groovy scripts from the onset, but we ended up filling pre- and post-build steps, replacing several plugins with bash scripts (as the Jenkins agent is on RHEL).

With Jenkins, we had great results, including:

• A simple-to-use pipeline for CI/CD
• Automation of deployment
• Faster development process

Background: I wanted to ensure a successful development of the continuous integration platform. I chose Jenkins because it is the best tool for this. My objective is to improve the quality of the source code and allow the managers to check the work of the developers.

Goals: Checking the quality of the source code of the company's applications.

Solution & Results: I solved these challenges thanks to some assistance from others as well as relying on the documentation of a Jenkins server. With the diversity of plugins in Jenkins, almost anything is possible. Ant is also a perfect tool to manage the linters or other tools.

I relied on Jenkins to help me with these key capabilities:

• Warnings Next Generations is a very good plugin for visualizing warnings or errors in code for different languages
• Perfpublisher provides beautiful graphics for the strength of an application
• Publish HTML report provides a good web interface for tools which need it

I was pleased with the results, which included:

• improved product quality
• managers finally being able to understand the code
• a fully-functional continuous integration platform