Bringing infrastructure access under control

Getting access to a Kubernetes cluster used to mean receiving a copy of an administrator kubeconfig file. Anyone who had it could pass it on. Access depended on trust, and there was no reliable record of everyone holding a copy.

That approach made it easy to get work done. It also made it difficult to limit permissions, remove someone’s access or trace a change back to the person who made it.

The same informal approach existed elsewhere. Access to systems and information was granted as people needed it, usually by a manager. Resources had no clearly assigned owners, and there was no consistent process for requesting or approving access.

As the organisation grew, it needed a more deliberate way to manage access. Introducing clear responsibilities and repeatable processes was part of that growth, as well as the work towards ISO 27001.

Understanding what depended on existing access

Removing permissions required care. An account could have more access than its owner needed, while also running a deployment or script that depended on those permissions.

A change that looked straightforward could interrupt someone’s work or break an automation. Before changing access, the system administration team needed to understand how it was being used.

The team had a mandate under the information security management system to implement the changes. Working with team leads and newly assigned system owners meant reviewing existing access, checking why it was needed and considering what would happen when it changed.

This also changed who made the decisions. System owners became responsible for deciding who needed access to their resources. Administrators implemented those decisions in the infrastructure.

Replacing the shared Kubernetes credentials

For Kubernetes, the shared administrator credential was invalidated. Access moved to Rancher, connected to the existing directory through LDAP.

Users signed in with their own accounts and could obtain their own Kubernetes credentials through Rancher. Their team’s directory groups determined which clusters and namespaces they could access, and what they could do there.

This gave the team a way to manage access for each person. Someone who needed access to a project could receive the appropriate permissions without receiving administrator access to the whole cluster.

Actions were also traceable to individual identities in the audit logs. It became possible to see who had made a change, using which account.

Giving access requests an owner

The approval process needed to support the technical changes.

Jira Assets held the records of systems and their owners. Requests and changes were tracked through linked Jira tickets. Someone requesting access had to explain what they needed and why. The resource owner reviewed that need and took responsibility for approving it.

The ticket recorded the request, the approval and the implementation. Directory groups and permissions controlled what the person could actually access.

Together with the activity logs, these records made it possible to follow access from the original request through to its use.

Applying the same approach to data and automation

Database owners reviewed who could access real user data and narrowed those permissions. Test and demo datasets were cleaned of real user information, and people with test or demo access were separated from production user data.

Automations also needed attention. Some relied on personal credentials left behind by former employees. These were replaced with dedicated accounts assigned to teams, with only the permissions needed for the particular task.

Test and production networks were separated, and internet-facing environments were isolated from the internal network. Audit logging and alerts for attempted unauthorised access provided visibility into activity on important systems.

By the end, access had a named owner, a reason and a record. Administrators had a defined process to follow, system owners were responsible for approvals, and users received permissions appropriate to their work.

The hardest part was understanding the dependencies built up around the old access arrangements. Working through those dependencies with the people responsible for the systems was essential to making the changes.