AWS Cloud Integration using IAM Roles for Service Accounts (IRSA)
Last updated
Last updated
There are many ways to integrate your AWS Cost and Usage Report (CUR) with Kubecost. This tutorial is intended as the best-practice method for users whose environments meet the following assumptions:
Kubecost will run in a different account than the AWS Payer Account
The IAM permissions will utilize AWS IRSA to avoid shared secrets
The configuration of Kubecost will be done using a cloud-integration.json file, and not via Kubecost UI (following infrastructure as code practices)
If this is not an accurate description of your environment, see our AWS Cloud Integration doc for more options.
This guide is a one-time setup per AWS payer account and is typically one per organization. It can be automated, but may not be worth the effort given that it will not be needed again.
Kubecost supports multiple AWS payer accounts as well as multiple cloud providers from a single Kubecost primary cluster. For multiple payer accounts, create additional entries inside the array below.
Detail for multiple cloud provider setups is here.
To begin, download the recommended configuration template files from our poc-common-config repo. You will need the following files from this folder:
cloud-integration.json
iam-payer-account-cur-athena-glue-s3-access.json
iam-payer-account-trust-primary-account.json
iam-access-cur-in-payer-account.json
Begin by opening cloud_integration.json, which should look like this:
Update athenaWorkgroup to primary, then save the file and close it. The remaining values will be obtained during this tutorial.
Follow the AWS documentation to create a CUR export using the settings below.
For time granularity, select Daily.
Select the checkbox to enable Resource IDs in the report.
Select the checkbox to enable Athena integration with the report.
Select the checkbox to enable the JSON IAM policy to be applied to your bucket.
If this CUR data is only used by Kubecost, it is safe to expire or delete the objects after seven days of retention.
AWS may take up to 24 hours to publish data. Wait until this is complete before continuing to the next step.
While you wait, update the following configuration files:
Update your cloud-integration.json file by providing a projectID value, which will be the AWS payer account number where the CUR is located and where the Kubecost primary cluster is running.
Update your iam-payer-account-cur-athena-glue-s2-access.json file by replacing all instances of CUR_BUCKET_NAME to the name of the bucket you created for CUR data.
As part of the CUR creation process, Amazon creates a CloudFormation template that is used to create the Athena integration. It is created in the CUR S3 bucket under s3-path-prefix/cur-name and typically has the filename crawler-cfn.yml. This .yml is your CloudFormation template. You will need it in order to complete the CUR Athena integration. You can read more about this here.
Your S3 path prefix can be found by going to your AWS Cost and Usage Reports dashboard and selecting your bucket's report. In the Report details tab, you will find the S3 path prefix.
Once Athena is set up with the CUR, you will need to create a new S3 bucket for Athena query results. The bucket used for the CUR cannot be used for the Athena output.
Navigate to the S3 Management Console.
Select Create bucket. The Create Bucket page opens.
Provide a name for your bucket. This is the value for athenaBucketName in your cloud-integration.json file. Use the same region used for the CUR bucket.
Select Create bucket at the bottom of the page.
Navigate to the Amazon Athena dashboard.
Select Settings, then select Manage. The Manage settings window opens.
Set Location of query result to the S3 bucket you just created, then select Save.
Navigate to Athena in the AWS Console. Be sure the region matches the one used in the steps above. Update your cloud-integration.json file with the following values. Use the screenshots below for help.
athenaBucketName: the name of the Athena bucket your created in this step
athenaDatabase: the value in the Database dropdown
athenaRegion: the AWS region value where your Athena query is configured
athenaTable: the partitioned value found in the Table list
For Athena query results written to an S3 bucket only accessed by Kubecost, it is safe to expire or delete the objects after one day of retention.
From the AWS payer account
In iam-payer-account-cur-athena-glue-s3-access.json, replace all ATHENA_RESULTS_BUCKET_NAME instances with your Athena S3 bucket name (the default will look like aws-athena-query-results-xxxx).
In iam-payer-account-trust-primary-account.json, replace SUB_ACCOUNT_222222222 with the account number of the account where the Kubecost primary cluster will run.
In the same location as your downloaded configuration files, run the following command to create the appropriate policy (jq is not required):
Now we can obtain the last value masterPayerARN for cloud-integration.json as the ARN associated with the newly-created IAM role, as seen below in the AWS console:
By arriving at this step, you should have been able to provide all values to your cloud-integration.json file. If any values are missing, reread the tutorial and follow any steps needed to obtain those values.
From the AWS Account where the Kubecost primary cluster will run
In iam-access-cur-in-payer-account.json, update PAYER_ACCOUNT_11111111111 with the AWS account number of the payer account and create a policy allowing Kubecost to assumeRole in the payer account:
Note the output ARN (used in the iamserviceaccount --attach-policy-arn below):
Create a namespace and set environment variables:
Enable the OIDC-Provider:
Create the Kubernetes service account, attaching the assumeRole policy. Replace SUB_ACCOUNT_222222222 with the AWS account number where the primary Kubecost cluster will run.
Create the secret (in this setup, there are no actual secrets in this file):
Install Kubecost using the service account and cloud-integration secret:
It can take over an hour to process the billing data for large AWS accounts. In the short-term, follow the logs and look for a message similar to (7.7 complete), which should grow gradually to (100.0 complete). Some errors (ERR) are expected, as seen below.
For help with troubleshooting, see the section in our original AWS integration guide.
