Enabling AWS IAM Group Access to an EKS Cluster Using RBAC
A deep dive into Amazon's Elastic Kubernetes Service (EKS) user authentication and authorization using AWS IAM
đź‘‹ Introduction
Growing pains are hard. Your organization is growing, apps are succeeding, and your engineering team has tripled—now, manage your Amazon Kubernetes Service (EKS) and permissions for an expansive team of non-stop complexity. Growing pains are hard.
At Grip, we encountered similar frustrations with managing our EKS cluster permissions due to our growing number of engineers and the increasing complexity of our systems. The time invested in managing our EKS permissions model grew significantly, encouraging us to seek out a comprehensive and sustainable solution to this problem.
đź“„ TL;DR
- Kubernetes has 2 main methods for authorization:
- Authentication can be done in multiple methods.
- EKS is configured to use aws-iam-authenticator which allows Kubernetes to integrate with AWS IAM.
- There are two configurations required in order to define authentication and authorization:
- Authorization - A
RoleBinding
or aClusterRoleBinding
- Authentication - A
ConfigMap
namedaws-auth
which is under thekube-system
namespace
- Authorization - A
- There is no standardized method for providing IAM group access to an EKS cluster or namespace.
- Our solution utilizes an IAM role to authenticate the user group automatically and transparently when
kubectl
is being used.
🛂 RBAC Overview
If you are familiar with Kubernetes RBAC, you can go ahead and jump to EKS RBAC Integration with IAM
Kubernetes's access control naming convention is similar to AWS IAM, so make sure you don't confuse the two.
- Kubernetes
Role
- A set of permissions for a namespace
- Kubernetes
ClusterRole
- A set of permissions for an entire cluster
- Kubernetes RBAC
RoleBinding
- A binding between oneRole
and one or more Users or Groups
- Kubernetes RBAC
ClusterRoleBinding
- A binding between oneClusterRole
and one or more Users or Groups
Since this isn’t really a “Getting started with RBAC” kind of post. If you’re still not sure about the terminology, Check out Kubernetes RBAC docs for further reading.
🔑 EKS RBAC Integration with IAM
Now that we’ve finished our overview of Kubernetes RBAC, let’s dive right in and see how it integrates with AWS IAM. The general idea is:
aws-iam-authenticator
authenticates an IAM identity.- This is achieved by token authentication webhook.
aws-iam-authenticator
translates the authenticated identity to a Kubernetes RBAC identity.- Each IAM identity is mapped to a Kubernetes username or added to a group.
- Kubernetes RBAC uses these translated identities to enforce permissions to different resources.
- Kubernetes RBAC is not aware of IAM identities.
AWS EKS uses a specific ConfigMap
named aws-auth
to manage the AWS Roles and AWS Users who are allowed to connect and manage the cluster (or namespace).
This is an example of aws-auth.yaml
file:
Authentication and Authorization Flow
👨‍👩‍👦 Adding an IAM group to EKS RBAC
According to AWS official documentation, aws-auth.yaml
doesn’t support IAM groups translation to RBAC.
We conducted comprehensive research to verify this fact, and we found that there is no official documentation on the full format specification of aws-auth.yaml
. So we read the implementation of aws-iam-authenticator
to find alternative solutions.
We have built a full format specification based on the implementation - aws-auth-specification.yaml
As you can see, aws-auth
supports three IAM entities mapAccounts
, mapUsers
and, mapRoles
but none of them allow us to map an IAM group.
Therefore, it is not possible to define an IAM group for RBAC using standard methods.
Furthermore, we found multiple open issues on GitHub asking for help with the same issue:
- GitHub[aws-iam-authenticator] - Can I not add an IAM group to my ConfigMap?
- Github[containers-roadmap] - [EKS] [request]: MapGroups in ConfigMap
- GitHub[aws-iam-authenticator] - How to use IAM Groups?
âś… Our Chosen Solution
After comparing the different options, we decided to use the most robust solution we found, which is the “assume role” method.
General Idea
- Create an IAM role -
dev.eks-access.role
- Create a policy that allows it to manage EKS and attach it to the role -
dev.eks-access.policy
- Add the role ARN under the
mapRoles
section ofaws-auth.yaml
. - Create a policy that allows assuming the role and attach it to your R&D group -
dev.assume-eks-access-role.policy
Solution Downside
The main downside of this solution is that R&D team members must assume the dev.eks-access.role
every time they want to interact with the cluster. A common workaround is to use AWS profiles, but this also requires changing the profile before accessing the cluster.
We wanted to get rid of this constraint. After some brainstorming, we assumed that the way aws-iam-authenticator
works on our dev machines allows it to assume a role on the fly, and only for Kubernetes-related operations.
Solving the Downside - .kube/config
Authentication Configuration
The way that kubectl
(and other Kubernetes operations) authenticate to the cluster is using the configuration described in .kube/config
.
This way, the kubectl
command knows how to authenticate to EKS using the aws-iam-authenticator
users:
- name: arn:aws:eks:us-east-2:123456789123:cluster/dev.my-eks-cluster
user:
exec:
apiVersion: client.authentication.Kubernetes.io/v1alpha1
args:
- --region
- us-east-2
- eks
- get-token
- --cluster-name
- dev.my-eks-cluster
command: aws
env: null
This authentication configuration is equivalent to the following command
aws --region us-east-2 eks get-token --cluster-name dev.my-eks-cluster
So, if we could assume the role only in this context, all the Kubernetes operations will be done with the assumed role, but other AWS operations will be done with the configured R&D user.
And as we expected, both get-token
and update-kubeconfig
commands support --role-arn
flag
OPTIONS
. . .
--role-arn (string)
Assume this role for credentials when signing the token.
. . .
Finally! we can just use --role-arn
and assume the role on the fly only for Kubernetes operations! This eliminated our previous friction points 🥳
Step-by-Step Guide
This guide gives an example of how this should be done; however, it's suggested that a more strict permission model be implemented for production environments.
Create the Necessary IAM Entities
First, we need to create all the needed IAM entities.
Create dev.eks-access.role
Role
ℹ️ |
We give the root account trust to assume the role. This means that the role will allow everyone in the account to assume the role from the role perspective.This doesn’t mean that everyone in the account can assume the role, because only users with a policy that allows them to assume the role will be able to do so. |
Create the role and give it the following trust relationship:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789123:root"
},
"Action": "sts:AssumeRole",
"Condition": {}
}
]
}
Create dev.assume-eks-access-role.policy
Policy
This policy allows an IAM entity to assume the dev.eks-access.role
.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::123456789123:role/dev.eks-access.role"
}
]
}
Create dev.eks-access.policy
Policy
This is an example of a policy that effectively allows full access to access EKS
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"eks:ListClusters",
"eks:DescribeAddonVersions",
"eks:CreateCluster"
],
"Resource": "*"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "eks:*",
"Resource": "arn:aws:eks:*:123456789123:cluster/*"
}
]
}
Connect the Dots
Inside AWS IAM management console do the following:
- Attach
dev.eks-access.policy
todev.eks-access.role
- Attach
dev.assume-eks-access-role.policy
to your desired group (e.g.dev.rnd.group
)
Configure EKS to Work With the Role
Add the role to the cluster aws-auth
eksctl create iamidentitymapping --cluster {CLUSTER_NAME} --arn arn:aws:iam::123456789123:role/dev.eks-access.role --group system:masters --username aws_eks_access_role
⚠️ |
We give the role system:masters permissions, meaning the role can modify everything in the cluster. Be cautious!
|
ClusterRoleBinding
Create a cluster-role-binding.yaml
apiVersion: rbac.authorization.Kubernetes.io/v1
kind: ClusterRoleBinding
metadata:
creationTimestamp: null
name: entire-cluster-admin-access
roleRef:
apiGroup: rbac.authorization.Kubernetes.io
kind: ClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.Kubernetes.io
kind: User
name: aws_eks_access_role
⚠️ |
We give the role cluster-admin permissions, meaning the role can modify everything in the cluster. Be cautious!
|
ClusterRoleBinding
on the Kubernetes cluster:
kubectl apply -f cluster-role-binding.yaml
Update the Kubernetes config file (.kube/config
)
After all the entities exist and all configurations are set, all you need to do is to configure .kube/config
to be able to access the EKS cluster using this command:
aws eks --region us-east-2 update-kubeconfig --name {CLUSTER_NAME} --role-arn arn:aws:iam::123456789123:role/dev.eks-access.role
🗺️ Other Possible Solutions
These are some other solutions that solve this issue from a different perspective:
- Use Terraform to “manually” give access to all users:
- This solution isn’t comprehensive, as Terraform isn’t used by all organizations. Source - Can I not add an IAM group to my ConfigMap? · Issue #176 · kubernetes-sigs/aws-iam-authenticator · GitHub
- Deploy iam-eks-user-mapper
- This project runs as a container in the cluster and modifies the configurations on the fly.
- Use AWS Lambda to periodically update all cluster permissions.
🌯 Wrap Up
This blog post was a comprehensive overview intended to solve a seemingly very simple problem.
By acquiring the relevant knowledge and understanding of some internal implementations, we managed to find a good workaround that provides a great solution to this problem. We hope that other organizations find value in this information, and use it to simplify and streamline their workloads using Kubernetes.
If you are interested in becoming a member of our team, we are hiring!
đź“– Related documentation
- Enabling IAM user and role access to your cluster - Amazon EKS
- Using RBAC Authorization | Kubernetes
- Authorization Overview | Kubernetes
- EKS (AWS) AND RBAC, step by step. Introduction | by David De Juan Calvo | Globant | Medium
- GitHub - kubernetes-sigs/aws-iam-authenticator
- Mapping IAM Groups to EKS User Access | by Zach Arnold | Ygrene Tech
- Troubleshooting IAM - Amazon EKS
- Kubernetes RBAC and IAM Integration in Amazon EKS using a Java-based Kubernetes Operator | Containers
- https://aws.amazon.com/premiumsupport/knowledge-center/eks-kubernetes-object-access-error
- Identity and Access Management - EKS Best Practices Guides
- GitHub - keikoproj/aws-auth: Manage the aws-auth config map for EKS Kubernetes clusters