Making Kubernetes platform upgrades predictable

Across more than 25 Kubernetes clusters, component upgrades usually happened when something stopped working or a new requirement made them necessary. Each cluster had its own configuration, and the reasons for many differences were no longer clear.

A new platform team inherited this setup after years of ad hoc changes. Other engineering teams used the clusters for their projects, while the platform team maintained around 15 shared components for networking, ingress, monitoring and logging.

The company had a quarterly maintenance window, but preparing platform changes remained difficult. An upgrade that worked on one cluster could behave differently on another. Application teams had limited visibility into upcoming changes and little opportunity to check compatibility beforehand.

With that uncertainty, leaving a working component alone was often the safest immediate choice.

Understanding the differences between clusters

Standardisation required a clear picture of the existing configurations. Some differences served a specific purpose. Others had accumulated over time without a documented reason.

Work began with components that had limited impact on application teams, including tools used only by the platform team. This allowed the approach to develop before changes reached more widely used services.

Comparing configurations across clusters helped establish a common definition for each component. Environment settings and necessary cluster-specific differences remained in separate override files.

The shared definition covered the settings that could be consistent. Overrides preserved the flexibility needed for particular environments or clusters, while making those exceptions easy to find and review.

Component versions were also defined by environment. This gave the platform team a clear view of what should be running and which differences to expect.

Making changes visible before deployment

The configuration lived in a GitOps repository, with separate groups for Development, Test and Production. Each component had its own Helm configuration and Argo CD ApplicationSet.

Before a deployment, Argo CD showed the difference between the proposed configuration and the running application. The team could inspect the effect of a change before synchronising it.

After deployment, Git remained the record of the intended configuration. Argo CD made drift visible and used reconciliation and self-healing to keep managed resources aligned with it.

This made troubleshooting more straightforward. The expected settings, cluster-specific exceptions and change history were available in one place.

The transition happened gradually across components and clusters, allowing the same approach to extend through the platform.

Giving teams time to prepare

A shared configuration made upgrades easier to understand. A defined release path gave application teams time to respond to them.

Development was where incoming platform changes could be evaluated. Both the platform team and application teams could test there against upcoming component versions.

Changes accepted in Development moved to Test in the following cycle. Test represented the next intended production state, giving application teams a place to check compatibility before the production upgrade.

Those changes moved to Production in the next cycle.

By the maintenance window, the proposed configuration had already been available in earlier environments. Teams had an opportunity to identify problems and adjust their deployments while production still ran the previous configuration.

A more predictable maintenance window

Dedicated monitoring, alerting and logging gave the platform team visibility into its own components. These tools ran in separate instances and namespaces on the shared infrastructure, following the same approach used by application teams.

Within six months, the platform had a shared configuration structure and a predictable release procedure.

Maintenance windows could follow an upgrade plan that had already been tested in earlier environments. Configuration differences were visible, and application teams had time to prepare for the changes.

Maintenance weekends became shorter and more productive. Complaints from application teams decreased, and troubleshooting became easier.

The platform still supported individual cluster requirements. Those differences now had a defined place within a common configuration and release process, making the next upgrade easier to plan.