👋 Introduction

Growing pains are hard. Your organization is growing, apps are succeeding, and your engineering team has tripled—now, manage your Amazon Kubernetes Service (EKS) and permissions for an expansive team of non-stop complexity. Growing pains are hard.

At Grip, we encountered similar frustrations with managing our EKS cluster permissions due to our growing number of engineers and the increasing complexity of our systems. The time invested in managing our EKS permissions model grew significantly, encouraging us to seek out a comprehensive and sustainable solution to this problem.

📄 TL;DR

• Kubernetes has 2 main methods for authorization:
  • RBAC - Role-based access control
  • ABAC - Attribute-based access control
• Authentication can be done in multiple methods.
  • EKS is configured to use aws-iam-authenticator which allows Kubernetes to integrate with AWS IAM.
• There are two configurations required in order to define authentication and authorization:
  • Authorization - A RoleBinding or a ClusterRoleBinding
  • Authentication - A ConfigMap named aws-auth which is under the kube-system namespace
• There is no standardized method for providing IAM group access to an EKS cluster or namespace.
• Our solution utilizes an IAM role to authenticate the user group automatically and transparently when kubectl is being used.

🛂 RBAC Overview

If you are familiar with Kubernetes RBAC, you can go ahead and jump to EKS RBAC Integration with IAM

Kubernetes's access control naming convention is similar to AWS IAM, so make sure you don't confuse the two.

• Kubernetes Role - A set of permissions for a namespace

• Kubernetes ClusterRole - A set of permissions for an entire cluster

• Kubernetes RBAC RoleBinding - A binding between one Role and one or more Users or Groups

• Kubernetes RBAC ClusterRoleBinding - A binding between one ClusterRole and one or more Users or Groups

Since this isn’t really a “Getting started with RBAC” kind of post. If you’re still not sure about the terminology, Check out Kubernetes RBAC docs for further reading.

🔑 EKS RBAC Integration with IAM

Now that we’ve finished our overview of Kubernetes RBAC, let’s dive right in and see how it integrates with AWS IAM. The general idea is:

1. aws-iam-authenticator authenticates an IAM identity.

   • This is achieved by token authentication webhook.

2. aws-iam-authenticator translates the authenticated identity to a Kubernetes RBAC identity.

3. Each IAM identity is mapped to a Kubernetes username or added to a group.
4. Kubernetes RBAC uses these translated identities to enforce permissions to different resources.
   • Kubernetes RBAC is not aware of IAM identities.

AWS EKS uses a specific ConfigMap named aws-auth to manage the AWS Roles and AWS Users who are allowed to connect and manage the cluster (or namespace).

This is an example of aws-auth.yaml file:

Authentication and Authorization Flow

👨‍👩‍👦 Adding an IAM group to EKS RBAC

According to AWS official documentation, aws-auth.yaml doesn’t support IAM groups translation to RBAC.

We conducted comprehensive research to verify this fact, and we found that there is no official documentation on the full format specification of aws-auth.yaml. So we read the implementation of aws-iam-authenticator to find alternative solutions.

We have built a full format specification based on the implementation - aws-auth-specification.yaml

As you can see, aws-auth supports three IAM entities mapAccounts, mapUsers and, mapRoles but none of them allow us to map an IAM group.

Therefore, it is not possible to define an IAM group for RBAC using standard methods.

Furthermore, we found multiple open issues on GitHub asking for help with the same issue:

• GitHub[aws-iam-authenticator] - Can I not add an IAM group to my ConfigMap?
• Github[containers-roadmap] - [EKS] [request]: MapGroups in ConfigMap
• GitHub[aws-iam-authenticator] - How to use IAM Groups?

✅ Our Chosen Solution

After comparing the different options, we decided to use the most robust solution we found, which is the “assume role” method.

General Idea

1. Create an IAM role - dev.eks-access.role
2. Create a policy that allows it to manage EKS and attach it to the role - dev.eks-access.policy
3. Add the role ARN under the mapRoles section of aws-auth.yaml.
4. Create a policy that allows assuming the role and attach it to your R&D group - dev.assume-eks-access-role.policy

Solution Downside

The main downside of this solution is that R&D team members must assume the dev.eks-access.role every time they want to interact with the cluster. A common workaround is to use AWS profiles, but this also requires changing the profile before accessing the cluster.

We wanted to get rid of this constraint. After some brainstorming, we assumed that the way aws-iam-authenticator works on our dev machines allows it to assume a role on the fly, and only for Kubernetes-related operations.

Solving the Downside - .kube/config Authentication Configuration

The way that kubectl (and other Kubernetes operations) authenticate to the cluster is using the configuration described in .kube/config.

This way, the kubectl command knows how to authenticate to EKS using the aws-iam-authenticator

users:
- name: arn:aws:eks:us-east-2:123456789123:cluster/dev.my-eks-cluster
  user:
    exec:
      apiVersion: client.authentication.Kubernetes.io/v1alpha1
      args:
      - --region
      - us-east-2
      - eks
      - get-token
      - --cluster-name
      - dev.my-eks-cluster
      command: aws
      env: null

This authentication configuration is equivalent to the following command

aws --region us-east-2 eks get-token --cluster-name dev.my-eks-cluster

So, if we could assume the role only in this context, all the Kubernetes operations will be done with the assumed role, but other AWS operations will be done with the configured R&D user.

And as we expected, both get-token and update-kubeconfig commands support --role-arn flag

OPTIONS
     . . .
    --role-arn (string)  
        Assume this role for credentials when signing the token.
    . . .

Finally! we can just use --role-arn and assume the role on the fly only for Kubernetes operations! This eliminated our previous friction points 🥳

Step-by-Step Guide

This guide gives an example of how this should be done; however, it's suggested that a more strict permission model be implemented for production environments.

Create the Necessary IAM Entities

First, we need to create all the needed IAM entities.

Create dev.eks-access.role Role

ℹ️

We give the root account trust to assume the role. This means that the role will allow everyone in the account to assume the role from the role perspective. This doesn’t mean that everyone in the account can assume the role, because only users with a policy that allows them to assume the role will be able to do so.

Create the role and give it the following trust relationship:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789123:root"
            },
            "Action": "sts:AssumeRole",
            "Condition": {}
        }
    ]
}

Create dev.assume-eks-access-role.policy Policy

This policy allows an IAM entity to assume the dev.eks-access.role.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::123456789123:role/dev.eks-access.role"
        }
    ]
}

Create dev.eks-access.policy Policy

This is an example of a policy that effectively allows full access to access EKS

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "eks:ListClusters",
                "eks:DescribeAddonVersions",
                "eks:CreateCluster"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "eks:*",
            "Resource": "arn:aws:eks:*:123456789123:cluster/*"
        }
    ]
}

Connect the Dots

Inside AWS IAM management console do the following:

• Attach dev.eks-access.policy to dev.eks-access.role
• Attach dev.assume-eks-access-role.policy to your desired group (e.g. dev.rnd.group)

Configure EKS to Work With the Role

Add the role to the cluster aws-auth

eksctl create iamidentitymapping --cluster {CLUSTER_NAME} --arn arn:aws:iam::123456789123:role/dev.eks-access.role --group system:masters --username aws_eks_access_role

⚠️	We give the role system:masters permissions, meaning the role can modify everything in the cluster. Be cautious!

Add the Role to the ClusterRoleBinding

Create a cluster-role-binding.yaml

apiVersion: rbac.authorization.Kubernetes.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: null
  name: entire-cluster-admin-access
roleRef:
  apiGroup: rbac.authorization.Kubernetes.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - apiGroup: rbac.authorization.Kubernetes.io
    kind: User
    name: aws_eks_access_role

⚠️	We give the role cluster-admin permissions, meaning the role can modify everything in the cluster. Be cautious!

Run this command to apply the new ClusterRoleBinding on the Kubernetes cluster:

kubectl apply -f cluster-role-binding.yaml

Update the Kubernetes config file (.kube/config)

After all the entities exist and all configurations are set, all you need to do is to configure .kube/config to be able to access the EKS cluster using this command:

aws eks --region us-east-2 update-kubeconfig --name {CLUSTER_NAME} --role-arn arn:aws:iam::123456789123:role/dev.eks-access.role

🗺️ Other Possible Solutions

These are some other solutions that solve this issue from a different perspective:

• Use Terraform to “manually” give access to all users:
  • This solution isn’t comprehensive, as Terraform isn’t used by all organizations. Source - Can I not add an IAM group to my ConfigMap? · Issue #176 · kubernetes-sigs/aws-iam-authenticator · GitHub
• Deploy iam-eks-user-mapper
  • This project runs as a container in the cluster and modifies the configurations on the fly.
• Use AWS Lambda to periodically update all cluster permissions.

🌯 Wrap Up

This blog post was a comprehensive overview intended to solve a seemingly very simple problem.

By acquiring the relevant knowledge and understanding of some internal implementations, we managed to find a good workaround that provides a great solution to this problem. We hope that other organizations find value in this information, and use it to simplify and streamline their workloads using Kubernetes.

If you are interested in becoming a member of our team, we are hiring!

📖 Related documentation

• Enabling IAM user and role access to your cluster - Amazon EKS
• Using RBAC Authorization | Kubernetes
• Authorization Overview | Kubernetes
• EKS (AWS) AND RBAC, step by step. Introduction | by David De Juan Calvo | Globant | Medium
• GitHub - kubernetes-sigs/aws-iam-authenticator
• Mapping IAM Groups to EKS User Access | by Zach Arnold | Ygrene Tech
• Troubleshooting IAM - Amazon EKS
• Kubernetes RBAC and IAM Integration in Amazon EKS using a Java-based Kubernetes Operator | Containers
• https://aws.amazon.com/premiumsupport/knowledge-center/eks-kubernetes-object-access-error
• Identity and Access Management - EKS Best Practices Guides
• GitHub - keikoproj/aws-auth: Manage the aws-auth config map for EKS Kubernetes clusters