AWS Cloud Integration using IRSA/EKS Pod Identities
Last updated
Last updated
There are many ways to integrate your AWS Cost and Usage Report (CUR) with Kubecost. This tutorial is intended as the best-practice method for users whose environments meet the following assumptions:
Kubecost will run in a different account than the AWS Payer Account
The IAM permissions will utilize AWS IAM Roles for Service Accounts (IRSA) to avoid shared secrets
The configuration of Kubecost will be done using a cloud-integration.json file, and not via Kubecost UI (following infrastructure as code practices)
If this is not an accurate description of your environment, see our AWS Cloud Integration doc for more options.
Kubecost also supports EKS Pod Identity as an alternative to IRSA. To set up EKS Pod Identities, complete steps 1-4 of the below tutorial fully, then follow Step 5 until you are prompted to move to the optional Step 6 below.
This guide is a one-time setup per AWS payer account and is typically one per organization. It can be automated, but may not be worth the effort given that it will not be needed again.
Kubecost supports multiple AWS payer accounts as well as multiple cloud providers from a single Kubecost primary cluster. For multiple payer accounts, create additional entries inside the array below.
Detail for multiple cloud provider setups is here.
To begin, download the recommended configuration template files from our poc-common-config repo. You will need the following files from this folder:
cloud-integration.json
iam-payer-account-cur-athena-glue-s3-access.json
iam-payer-account-trust-primary-account.json
iam-access-cur-in-payer-account.json
The bottom three files are found in /aws/iam-policies/cur.
Begin by opening cloud_integration.json, which should look like this:
Update athenaWorkgroup to primary, then save the file and close it. The remaining values will be obtained during this tutorial.
Follow the AWS documentation to create a CUR export using the settings below.
For time granularity, select Daily.
Select the checkbox to enable Resource IDs in the report.
Select the checkbox to enable Athena integration with the report.
Select the checkbox to enable the JSON IAM policy to be applied to your bucket.
If this CUR data is only used by Kubecost, it is safe to expire or delete the objects after seven days of retention.
AWS may take up to 24 hours to publish data. Wait until this is complete before continuing to the next step.
While you wait, update the following configuration files:
Update your cloud-integration.json file by providing a projectID value, which will be the AWS payer account number where the CUR is located and where the Kubecost primary cluster is running.
Update your iam-payer-account-cur-athena-glue-s2-access.json file by replacing all instances of CUR_BUCKET_NAME to the name of the bucket you created for CUR data.
As part of the CUR creation process, Amazon creates a CloudFormation template that is used to create the Athena integration. It is created in the CUR S3 bucket under s3-path-prefix/cur-name and typically has the filename crawler-cfn.yml. This .yml is your CloudFormation template. You will need it in order to complete the CUR Athena integration. You can read more about this here.
Your S3 path prefix can be found by going to your AWS Cost and Usage Reports dashboard and selecting your bucket's report. In the Report details tab, you will find the S3 path prefix.
Once Athena is set up with the CUR, you will need to create a new S3 bucket for Athena query results. The bucket used for the CUR cannot be used for the Athena output.
Navigate to the S3 Management Console.
Select Create bucket. The Create Bucket page opens.
Provide a name for your bucket. This is the value for athenaBucketName in your cloud-integration.json file. Use the same region used for the CUR bucket.
Select Create bucket at the bottom of the page.
Navigate to the Amazon Athena dashboard.
Select Settings, then select Manage. The Manage settings window opens.
Set Location of query result to the S3 bucket you just created, then select Save.
Navigate to Athena in the AWS Console. Be sure the region matches the one used in the steps above. Update your cloud-integration.json file with the following values. Use the screenshots below for help.
athenaBucketName: the name of the Athena bucket your created in this step
athenaDatabase: the value in the Database dropdown
athenaRegion: the AWS region value where your Athena query is configured
athenaTable: the partitioned value found in the Table list
For Athena query results written to an S3 bucket only accessed by Kubecost, it is safe to expire or delete the objects after one day of retention.
From the AWS payer account
In iam-payer-account-cur-athena-glue-s3-access.json, replace all ATHENA_RESULTS_BUCKET_NAME instances with your Athena S3 bucket name (the default will look like aws-athena-query-results-xxxx).
In iam-payer-account-trust-primary-account.json, replace SUB_ACCOUNT_222222222 with the account number of the account where the Kubecost primary cluster will run.
In the same location as your downloaded configuration files, run the following command to create the appropriate policy (jq is not required):
Now we can obtain the last value masterPayerARN for cloud-integration.json as the ARN associated with the newly-created IAM role, as seen below in the AWS console:
By arriving at this step, you should have been able to provide all values to your cloud-integration.json file. If any values are missing, reread the tutorial and follow any steps needed to obtain those values.
From the AWS Account where the Kubecost primary cluster will run
In iam-access-cur-in-payer-account.json, update PAYER_ACCOUNT_11111111111 with the AWS account number of the payer account and create a policy allowing Kubecost to assumeRole in the payer account:
Note the output ARN (used in the iamserviceaccount --attach-policy-arn below):
Create a namespace and set environment variables:
If you are using EKS Pod Identity, skip the rest of Step 5 and continue to Step 6.
Enable the OIDC-Provider:
Linking default Kubecost Service Account to an IAM Role
Kubecost's default service account kubecost-cost-analyzer is automatically created in the kubecost namespace upon installation. This service account can be linked to an IAM Role via Annotation + IAM Trust Policy.
In the Helm values for your deployment, add the following section:
Go to the IAM Role and attach the proper IAM trust policy. Use the sample trust policy here. Verify you have replaced the example OIDC URL with your cluster OIDC URL.
Alternative method: Create a new dedicated service account for Kubecost using eksctl
This method creates a new service account via eksctl command line tools, instead of using the default service account. Eksctl automatically creates the trust policy and IAM Role that are linked to the new dedicated Kubernetes service account.
Replace SUB_ACCOUNT_222222222 with the AWS account number where the primary Kubecost cluster will run.
Create the secret (in this setup, there are no actual secrets in this file):
Install Kubecost using the service account and cloud-integration secret:
Add the following section to your Helm values. This will tell Kubecost to use your newly created service account, instead of creating one.
Your cluster must support EKS Pod Identities to use the method below.
Create your pod identity association:
Then update your values.yaml file:
It can take over an hour to process the billing data for large AWS accounts. In the short-term, follow the logs and look for a message similar to (7.7 complete), which should grow gradually to (100.0 complete). Some errors (ERR) are expected, as seen below.
For help with troubleshooting, see the section in our original AWS integration guide.
