AWS Cloud Billing Integration
By default, Kubecost pulls on-demand asset prices from the public AWS pricing API. For more accurate pricing, this integration will allow Kubecost to reconcile your current measured Kubernetes spend with your actual AWS bill. This integration also properly accounts for Enterprise Discount Programs, Reserved Instance usage, Savings Plans, Spot usage, and more.
You will need permissions to create the Cost and Usage Report (CUR), and add IAM credentials for Athena and S3. Optional permission is the ability to add and execute CloudFormation templates. Kubecost does not require root access in the AWS account.
Quick Start for IRSA
This guide contains multiple possible methods for connecting Kubecost to AWS billing, based on user environment and preference. Because of this, there may not be a straightforward approach for new users. To address this, a streamlined guide containing best practices can be found here for IRSA environments. This quick start guide has some assumptions to carefully consider, and may not be applicable for all users. See prerequisites in the linked article.
Key AWS terminology
Integrating your AWS account with Kubecost may be a complicated process if you aren’t deeply familiar with the AWS platform and how it interacts with Kubecost. This section provides an overview of some of the key terminology and AWS services that are involved in the process of integration.
Cost and Usage Report: AWS report which tracks cloud spending and writes to an Amazon Simple Storage Service (Amazon S3) bucket for ingestion and long term historical data. The CUR is originally formatted as a CSV, but when integrated with Athena, is converted to Parquet format.
Amazon Athena: Analytics service which queries the CUR S3 bucket for your AWS cloud spending, then outputs data to a separate S3 bucket. Kubecost uses Athena to query for the bill data to perform reconciliation. Athena is technically optional for AWS cloud integration, but as a result, Kubecost will only provide unreconciled costs (on-demand public rates).
S3 bucket: Cloud object storage tool which both CURs and Athena output cost data to. Kubecost needs access to these buckets in order to read that data.
Cost and Usage Report integration
For the below guide, a GitHub repository with sample files can be found here.
Step 1: Setting up a CUR
Follow these steps to set up a Legacy CUR using the settings below.
Select the Legacy CUR export type.
For time granularity, select Daily.
Under 'Additional content', select the Enable resource IDs checkbox.
Under 'Report data integration' select the Amazon Athena checkbox.
For CUR data written to an S3 bucket only accessed by Kubecost, it is safe to expire or delete the objects after seven days of retention.
Remember the name of the bucket you create for CUR data. This will be used in Step 2.
Familiarize yourself with how column name restrictions differ between CURs and Athena tables. AWS may change your CUR name when you upload your CUR to your Athena table in Step 2, documented in AWS' Running Amazon Athena queries. As best practice, use all lowercase letters and only use _
as a special character.
AWS may take up to 24 hours to publish data. Wait until this is complete before continuing to the next step.
If you believe you have the correct permissions, but cannot access the Billing and Cost Management page, have the owner of your organization's root account follow these instructions.
Step 2: Setting up Athena
As part of the CUR creation process, Amazon also creates a CloudFormation template that is used to create the Athena integration. It is created in the CUR S3 bucket, listed in the Objects tab in the path s3-path-prefix/cur-name
and typically has the filename crawler-cfn.yml
. This .yml is your necessary CloudFormation template. You will need it in order to complete the CUR Athena integration. For more information, see the AWS doc Setting up Athena using AWS CloudFormation templates.
Your S3 path prefix can be found by going to your AWS Cost and Usage Reports dashboard and selecting your newly-created CUR. In the 'Report details' tab, you will find the S3 path prefix.
Once Athena is set up with the CUR, you will need to create a new S3 bucket for Athena query results.
Navigate to the S3 Management Console.
Select Create bucket. The Create Bucket page opens.
Use the same region used for the CUR bucket and pick a name that follows the format aws-athena-query-results-.
Select Create bucket at the bottom of the page.
Navigate to the Amazon Athena dashboard.
Select Settings, then select Manage. The Manage settings window opens.
Set Location of query result to the S3 bucket you just created, which will look like s3://aws-athena-query-results..., then select Save.
For Athena query results written to an S3 bucket only accessed by Kubecost, it is safe to expire or delete the objects after 1 day of retention.
Step 3: Setting up IAM permissions
Add via CloudFormation:
Kubecost offers a set of CloudFormation templates to help set your IAM roles up.
If you’re new to provisioning IAM roles, we suggest downloading our templates and using the CloudFormation wizard to set these up. You can learn how to do this in AWS' Creating a stack on the AWS CloudFormation console doc. Open the step below which represents your CUR and management account arrangement, download the .yaml file listed, and upload them as the stack template in the 'Creating a stack' > 'Selecting a stack template' step.
Add manually:
Step 4: Attaching IAM permissions to Kubecost
If you are using the alternative multi-cloud integration method, steps 4 and 5 are not required.
Now that the policies have been created, attach those policies to Kubecost. We support the following methods:
Step 5: Provide CUR config values to Kubecost
These values can either be set from the Kubecost UI or via .Values.kubecostProductConfigs
in the Helm chart. Values for all fields must be provided.
Option 1: Add config values via UI
To add values in the Kubecost UI, select Settings from the left navigation, then scroll to Cloud Cost Settings. Select Update next to External Cloud Cost Configuration (AWS). The Billing Data Export Configuration window opens. Fill in all the below fields:
Athena Region
The AWS region Athena is running in
Athena Database
The name of the database created by the Athena setup
Athena Tablename
The name of the table created by the Athena setup
Athena Result Bucket
An S3 bucket to store Athena query results that you’ve created that Kubecost has permission to access
AWS account ID
The AWS account ID where the Athena CUR is, likely your management account.
When you are done, select Update to confirm.
Option 2: Add config values via Helm
If you set any kubecostProductConfigs
from the Helm chart, all changes via the front end will be overridden on pod restart.
athenaProjectID
: The AWS AccountID where the Athena CUR is, likely your management account.athenaBucketName
: An S3 bucket to store Athena query results that you’ve created that Kubecost has permission to accessThe name of the bucket should match
s3://aws-athena-query-results-*
, so the IAM roles defined above will automatically allow access to itThe bucket can have a Canned ACL of
Private
or other permissions as you see fit.
athenaRegion
: The AWS region Athena is running inathenaDatabase
: The name of the database created by the Athena setupThe athena database name is available as the value (physical id) of
AWSCURDatabase
in the CloudFormation stack created above (in Step 2: Setting up Athena)
athenaTable
: the name of the table created by the Athena setupThe table name is typically the database name with the leading
athenacurcfn_
removed (but is not available as a CloudFormation stack resource). Confirm the table name by visiting the Athena dashboard.
athenaWorkgroup
: The workgroup assigned to be used with Athena. If not specified, defaults toPrimary
Make sure to use only underscore as a delimiter if needed for tables and views. Using a hyphen/dash will not work even though you might be able to create it. See the AWS docs for more info.
If you are using a multi-account setup, you will also need to set
.Values.kubecostProductConfigs.masterPayerARN
to the Amazon Resource Number (ARN) of the role in the management account, e.g.arn:aws:iam::530337586275:role/KubecostRole
.
Troubleshooting
Once you've integrated with the CUR, you can visit Settings > View Full Diagnostics in the UI to determine if Kubecost has been successfully integrated with your CUR. If any problems are detected, you will see a yellow warning sign under the cloud provider permissions status header
You can check pod logs for authentication errors by running: kubectl get pods -n <namespace>
kubectl logs <kubecost-pod-name> -n <namespace> -c cost-model
If you do not see any authentication errors, log in to your AWS console and visit the Athena dashboard. You should be able to find the CUR. Ensure that the database with the CUR matches the athenaTable entered in Step 5. It likely has a prefix with athenacurcfn_
:
You can also check query history to see if any queries are failing:
Common Athena errors
Incorrect bucket in IAM Policy
Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources. You can search in the Athena "Recent queries" dashboard to find additional info about the error.
Resolution: This error is typically caused by the incorrect (Athena results) s3 bucket being specified in the CloudFormation template of Step 3 from above. To resolve the issue, ensure the bucket used for storing the AWS CUR report (Step 1) is specified in the
S3ReadAccessToAwsBillingData
SID of the IAM policy (default: kubecost-athena-access) attached to the user or role used by Kubecost (Default: KubecostUser / KubecostRole). See the following example.
This error can also occur when the management account cross-account permissions are incorrect, however, the solution may differ.
outputLocation is not a valid S3 path
Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources.
Resolution: Please verify that the prefix
s3://
was used when setting theathenaBucketName
Helm value or when configuring the bucket name in the Kubecost UI.
Query not supported
Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources.
Resolution: While rare, this issue was caused by an Athena instance that failed to provision properly on AWS. The solution was to delete the Athena DB and deploy a new one. To verify this is needed, find the failed query ID in the Athena "Recent queries" dashboard and attempt to manually run the query.
HTTPS Response error
Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources.
Resolution: Previously, if you ran a query without specifying a value for query result location, and the query result location setting was not overridden by a workgroup, Athena created a default location for you. Now, before you can run an Athena query in a region in which your account hasn't used Athena previously, you must specify a query result location, or use a workgroup that overrides the query result location setting. While Athena no longer creates a default query results location for you, previously created default
aws-athena-query-results-MyAcctID-MyRegion
locations remain valid and you can continue to use them. The bucket should be in the format of:aws-athena-query-results-MyAcctID-MyRegion
It may also be required to remove and reinstall Kubecost. If doing this please remeber to backup ETL files prior or contact support for additional assistance. See also this AWS doc on specifying a query result location.
Missing Athena column
Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources or in the Kubecost
cost-model
container logs.
Resolution: Verify in AWS' Cost and Usage Reports dashboard that the Resource IDs are enabled as "Report content" for the CUR created in Step 1. If the Resource IDs are not enabled, you will need to re-create the report (this will require redoing Steps 1 and 2 from this doc).
Not a valid S3 path
Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources or in the Kubecost
cost-model
container logs.
Resolution: Verify that
s3://
was included in the bucket name when setting the.Values.kubecostProductConfigs.athenaBucketName
Helm value.
Summary and pricing
AWS services used here are:
Kubecost's cost-model
requires roughly 2 CPU and 10 GB of RAM per 50,000 pods monitored. The backing Prometheus database requires roughly 2 CPU and 25 GB per million metrics ingested per minute. You can pick the EC2 instances necessary to run Kubecost accordingly.
Kubecost can write its cache to disk. Roughly 32 GB per 100,000 pods monitored is sufficient. (Optional: our cache can exist in memory)
Cloudformation (Optional: manual IAM configuration or via Terraform is fine)
EKS (Optional: all K8s flavors are supported)
Last updated