Daily Life as a Jenkins Administrator
The operational lesson here is not simply that Jenkins can support both self-managed and Kubernetes-based agents, but that each model creates a different support boundary. Once an organisation allows user-managed nodes alongside standardised ephemeral pods, the CI platform stops being just a build system and becomes a multi-tier service with uneven guarantees. That has consequences for incident triage, change control and policy enforcement.
The strongest architectural pattern in this setup is standardisation around a single curated agent image and Java 21 baseline. That reduces environmental drift, makes network and package-repository behaviour more predictable, and turns many “Jenkins problems” into image-governance or pipeline-compatibility problems that can actually be tested. The trade-off is image sprawl pressure: as more teams request tools, the base image becomes heavier, slower to evolve and harder to validate. Pre-pulling and internal registries help performance, but they do not remove the long-term maintainability risk of turning the default image into a monolith.
For platform teams, the practical question is where to draw the line between flexibility and operability. A useful next step is to classify workloads explicitly: default ephemeral Linux builds, exception-based persistent or hardware-bound builds, and legacy jobs needing remediation. That classification can drive SLOs, support expectations and security controls. It also helps identify which failures should trigger platform fixes versus which should be pushed back to pipeline owners.
For Jenkins users, the article also implies a governance principle: if builds depend on ad hoc internet examples, bespoke package sources or untracked local agent changes, reproducibility will degrade faster than build success metrics reveal. Stable CI at scale depends less on adding agents than on reducing uncontrolled variation between them.
Agent ecosystem overview
The second blog post focused on one specific part of your Kubernetes environment. I felt the urge to write more about it, so I thought a post about agents and managing them would be a good follow-up.
Our agents include:
-
200 user-managed agents running on Windows and Linux
-
100,000 Kubernetes pods created each month
| Every Jenkins agent runs on Java 21. |
The agents
User-managed agents
Users are allowed to add agents themselves. We provide guides for the most common operating systems (Windows, Linux, and macOS) to help them set up and configure their agents correctly, which also reduces our support effort. Users are responsible for maintaining and managing their agents. We offer support and guidance for their questions and issues. They can run builds directly on their agents, use any Docker image they choose, or use our agent image.
Kubernetes-managed agents
The agents are created as pods within our Kubernetes cluster. These agents are ephemeral and are automatically provisioned and deprovisioned based on the runtime of the build. We maintain the cluster environment. The pods run in a CinC setup (e.g. DinD). Kubernetes agents use only our agent image.
Which agent should a user choose?
When deciding which agent to use, consider the following points. Use our Kubernetes agent with our image if:
-
You are getting started with Jenkins
-
You have no special requirements
-
You don’t have the time to maintain your own agent
Use a user-managed agent if:
-
You have to use Windows OS
-
You need a connection to a hardware device (e.g. a dongle)
-
You have jobs which run longer than a few hours
-
You need full control over the environment
| Users can make a request to us if they need a new tool or package which can be added to the agent image. |
Our agent image
The image is based on a Linux distribution and includes a lot of different tools, configurations, environment variables, languages, and packages. This gives users a ready-to-use environment for their builds and makes things more convenient for us, too. Users can request changes or additions to the image. We also handle version bumps for existing tools and packages. For bigger changes, we announce them to users in advance. The image is regularly built to keep databases with security-relevant information up to date. As you can imagine, the image is rather large, which is why it is pre-pulled onto each node in the cluster and available from an internally hosted repository. And since all use the same image, we actually have less to put on the nodes.
Maintenance changes
Every change to the image triggers a series of Jenkins jobs to ensure that it remains stable and compatible with our build environment. This includes unit tests for individual tools and configurations, integration tests for common build scenarios, and performance tests to measure build times and resource usage.
Troubleshooting
Network
Most network issues are related to users not using our image (e.g. network proxy vars are missing or incorrectly configured). Another common cause is not using our internal package repositories. The remaining issues are usually temporary network problems.
Build issues
The most common issues:
-
The latest changes to the Jenkinsfile were not that good as expected (e.g. syntax or logic error, or relied on something from the internet that does not apply to our environment)
-
Missing the latest maintenance or shared library announcements
-
Building a job created in 2019 without updating it
-
Try to use an agent with Java 8
More about issues and some guidance on how to resolve them will be discussed in a future blog post.
| The last two points would of course never happen. 😉 |
Third blog post & my ideas for the future
I have a few more topics in mind:
-
Issues, tips and tricks to make your life easier / user support
-
Plugin management
-
Backup and restore / disaster recovery
-
Team collaboration / shared libraries
-
Your choice
If you have feedback or questions, or would like one of these topics covered in the next blog post, please let me know in the comments below.
Enjoyed this article? Sign up for our newsletter to receive regular insights and stay connected.