1 of 5

AWS Cloud Billing Integration

By default, Kubecost pulls on-demand asset prices from the public AWS pricing API. For more accurate pricing, this integration will allow Kubecost to reconcile your current measured Kubernetes spend with your actual AWS bill. This integration also properly accounts for Enterprise Discount Programs, Reserved Instance usage, Savings Plans, Spot usage, and more.

You will need permissions to create the Cost and Usage Report (CUR), and add IAM credentials for Athena and S3. Optional permission is the ability to add and execute CloudFormation templates. Kubecost does not require root access in the AWS account.

Quick Start for IRSA

This guide contains multiple possible methods for connecting Kubecost to AWS billing, based on user environment and preference. Because of this, there may not be a straightforward approach for new users. To address this, a streamlined guide containing best practices can be found for IRSA environments. This quick start guide has some assumptions to carefully consider, and may not be applicable for all users. See prerequisites in the linked article.

Key AWS terminology

Integrating your AWS account with Kubecost may be a complicated process if you aren’t deeply familiar with the AWS platform and how it interacts with Kubecost. This section provides an overview of some of the key terminology and AWS services that are involved in the process of integration.

Cost and Usage Report: AWS report which tracks cloud spending and writes to an Amazon Simple Storage Service (Amazon S3) bucket for ingestion and long term historical data. The CUR is originally formatted as a CSV, but when integrated with Athena, is converted to Parquet format.

Amazon Athena: Analytics service which queries the CUR S3 bucket for your AWS cloud spending, then outputs data to a separate S3 bucket. Kubecost uses Athena to query for the bill data to perform . Athena is technically optional for AWS cloud integration, but as a result, Kubecost will only provide unreconciled costs (on-demand public rates).

S3 bucket: Cloud object storage tool which both CURs and Athena output cost data to. Kubecost needs access to these buckets in order to read that data.

Cost and Usage Report integration

For the below guide, a GitHub repository with sample files can be found .

Step 1: Setting up a CUR

Follow to set up a Legacy CUR using the settings below.

Select the Legacy CUR export type.
For time granularity, select Daily.
Under 'Additional content', select the Enable resource IDs checkbox.
Under 'Report data integration' select the Amazon Athena checkbox.

For CUR data written to an S3 bucket only accessed by Kubecost, it is safe to expire or delete the objects after seven days of retention.

Remember the name of the bucket you create for CUR data. This will be used in Step 2.

AWS may take up to 24 hours to publish data. Wait until this is complete before continuing to the next step.

Step 2: Setting up Athena

Your S3 path prefix can be found by going to your AWS Cost and Usage Reports dashboard and selecting your newly-created CUR. In the 'Report details' tab, you will find the S3 path prefix.

Once Athena is set up with the CUR, you will need to create a new S3 bucket for Athena query results.

Select Create bucket. The Create Bucket page opens.
Use the same region used for the CUR bucket and pick a name that follows the format aws-athena-query-results-.
Select Create bucket at the bottom of the page.
Select Settings, then select Manage. The Manage settings window opens.
Set Location of query result to the S3 bucket you just created, which will look like s3://aws-athena-query-results..., then select Save.

For Athena query results written to an S3 bucket only accessed by Kubecost, it is safe to expire or delete the objects after 1 day of retention.

Step 3: Setting up IAM permissions

Add via CloudFormation:

Kubecost offers a set of CloudFormation templates to help set your IAM roles up.

My CUR exists in the same account as Kubecost or the management account

Select Create Stack, then select With new resources (standard) from the dropdown.
On the 'Create stack' page, under 'Prerequisite - Prepare Template', make sure Template is ready has been preselected. Under 'Specify Template', select Upload a template file. Select Choose file, then select your downloaded .yaml file from your file explorer. Select Next.
On the 'Specify stack details' page, enter a name for your stack, then provide the following parameters:
- AthenaCURBucket: The name of the Athena CUR bucket you created in Step 2.
- SpotDataFeedBucketName: (Optional, skip if you have not configured Spot data) The bucket where the Spot data feed is sent
Select Next.
On the 'Configure stack options' page opens, configure any additional options as needed. Select Next.
On the 'Review stack' page, confirm all information, then select I acknowledge that AWS CloudFormation might create IAM resources with custom names.
Select Submit.

My CUR exists in a member account different from Kubecost or the management account

On each sub account running Kubecost:

- Select Create Stack, then select With existing resources (import resources) from the dropdown. On the 'Identify resources' page, select Next.
- Under Template source, choose Upload a template file.
- Select Choose file, which will open your file explorer. Select the .yaml template, and then select Open. Then, select Next.
- On the 'Identify resources' page, provide any additional resources to import. Then, select Next.
- For Stack name, enter a name for your template.
- Set the following parameters:
  - MasterPayerAccountID: The account ID of the management account (formerly called master payer account) where the CUR has been created
  - SpotDataFeedBucketName: The bucket where the Spot data feed is sent
- Select Next.
- Select Next.
- At the bottom of the page, select I acknowledge that AWS CloudFormation might create IAM resources.
- Select Create Stack.

On the management account:

- AthenaCURBucket: The bucket where the CUR is set from Step 1
- KubecostClusterID: An account that Kubecost is running on that requires access to the Athena CUR.

Add manually:

My CUR exists in the same account as Kubecost or the management account

Attach both of the following policies to the same role or user. Use a user if you intend to integrate via ServiceKey, and a role if via IAM annotation (see more below under Via Pod Annotation by EKS). The SpotDataAccess policy statement is optional if the Spot data feed is configured (see “Setting up the Spot Data feed” step below).

My CUR exists in a member account different from Kubecost or the management account

On each sub account running Kubecost, attach both of the following policies to the same role or user. Use a user if you intend to integrate via Service Key, and a role if via IAM annotation (see more below under Via Pod Annotation by EKS). The SpotDataAccess policy statement is optional if the Spot data feed is configured (see “Setting up the Spot Data feed” step below).

On the management account, attach this policy to a role (replace ${AthenaCURBucket} variable):

Then add the following trust statement to the role the policy is attached to on the management account (replace the ${aws-mgmt-account-id} variable with the account you want to assume the role):

Step 4: Attaching IAM permissions to Kubecost

Now that the policies have been created, attach those policies to Kubecost. We support the following methods:

Attach via Service Key and Kubernetes Secret

Option 1: Generate a secret from Helm values:

Note that this will leave your AWS keys unencrypted in your values.yaml. Set the following Helm values:

Option 2: Manually create a secret:

This may be the preferred method if your Helm values are in version control and you want to keep your AWS secrets out of version control.

Create a service-key.json:

Create a Kubernetes secret:

Set the Helm value:

Attach via Service Key in Kubecost UI

To add the Access Key ID and Secret Access Key in the Kubecost UI, select Settings from the left navigation, then scroll to Cloud Cost Settings. Select Update next to External Cloud Cost Configuration (AWS). Fill in the Service key name and Service key secret fields respectively, then select Update.

You may not be allowed to update your Billing Data Export Configuration without filling out all fields first. For explanations of your Athena/AWS-related fields, see Step 5 below.

Attach via pod annotation with eksctl

Prerequisites:

Step 1: Update configuration

Download the following configuration files:

Update the following variables in the files you downloaded:

In cloud-integration.json, update the following values with the information in Step 2: Setting up Athena:

In kubecost-athena-policy.json, replace ${AthenaCURBucket} with your Athena S3 bucket name (configured in Step 2: Setting up Athena).

Step 2: Create policy

In the same location where your downloaded configuration files are, run the following command to create the appropriate policy:

Step 3: Create OIDC provider for your cluster

Step 4: Create required IAM service accounts

Note: Remember to replace 1234567890 with your AWS account ID number.

Step 5: Create required secret to store the configuration

Step 6. Install Kubecost via Helm

Step 5: Provide CUR config values to Kubecost

These values can either be set from the Kubecost UI or via .Values.kubecostProductConfigs in the Helm chart. Values for all fields must be provided.

Option 1: Add config values via UI

To add values in the Kubecost UI, select Settings from the left navigation, then scroll to Cloud Cost Settings. Select Update next to External Cloud Cost Configuration (AWS). The Billing Data Export Configuration window opens. Fill in all the below fields:

When you are done, select Update to confirm.

Option 2: Add config values via Helm

If you set any kubecostProductConfigs from the Helm chart, all changes via the front end will be overridden on pod restart.

athenaProjectID: The AWS AccountID where the Athena CUR is, likely your management account.
athenaBucketName: An S3 bucket to store Athena query results that you’ve created that Kubecost has permission to access
- The name of the bucket should match s3://aws-athena-query-results-*, so the IAM roles defined above will automatically allow access to it
- The bucket can have a Canned ACL of Private or other permissions as you see fit.
athenaRegion: The AWS region Athena is running in
athenaDatabase: The name of the database created by the Athena setup
athenaTable: the name of the table created by the Athena setup
- The table name is typically the database name with the leading athenacurcfn_ removed (but is not available as a CloudFormation stack resource). Confirm the table name by visiting the Athena dashboard.
athenaWorkgroup: The workgroup assigned to be used with Athena. If not specified, defaults to Primary

If you are using a multi-account setup, you will also need to set .Values.kubecostProductConfigs.masterPayerARN to the Amazon Resource Number (ARN) of the role in the management account, e.g. arn:aws:iam::530337586275:role/KubecostRole.

Troubleshooting

Once you've integrated with the CUR, you can visit Settings > View Full Diagnostics in the UI to determine if Kubecost has been successfully integrated with your CUR. If any problems are detected, you will see a yellow warning sign under the cloud provider permissions status header

You can check pod logs for authentication errors by running: kubectl get pods -n <namespace> kubectl logs <kubecost-pod-name> -n <namespace> -c cost-model

If you do not see any authentication errors, log in to your AWS console and visit the Athena dashboard. You should be able to find the CUR. Ensure that the database with the CUR matches the athenaTable entered in Step 5. It likely has a prefix with athenacurcfn_ :

You can also check query history to see if any queries are failing:

Common Athena errors

Incorrect bucket in IAM Policy

Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources. You can search in the Athena "Recent queries" dashboard to find additional info about the error.

Resolution: This error is typically caused by the incorrect (Athena results) s3 bucket being specified in the CloudFormation template of Step 3 from above. To resolve the issue, ensure the bucket used for storing the AWS CUR report (Step 1) is specified in the S3ReadAccessToAwsBillingData SID of the IAM policy (default: kubecost-athena-access) attached to the user or role used by Kubecost (Default: KubecostUser / KubecostRole). See the following example.

This error can also occur when the management account cross-account permissions are incorrect, however, the solution may differ.

outputLocation is not a valid S3 path

Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources.

Resolution: Please verify that the prefix s3:// was used when setting the athenaBucketName Helm value or when configuring the bucket name in the Kubecost UI.

Query not supported

Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources.

Resolution: While rare, this issue was caused by an Athena instance that failed to provision properly on AWS. The solution was to delete the Athena DB and deploy a new one. To verify this is needed, find the failed query ID in the Athena "Recent queries" dashboard and attempt to manually run the query.

HTTPS Response error

Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources.

Missing Athena column

Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources or in the Kubecost cost-model container logs.

Resolution: Verify in AWS' Cost and Usage Reports dashboard that the Resource IDs are enabled as "Report content" for the CUR created in Step 1. If the Resource IDs are not enabled, you will need to re-create the report (this will require redoing Steps 1 and 2 from this doc).

Not a valid S3 path

Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources or in the Kubecost cost-model container logs.

Resolution: Verify that s3:// was included in the bucket name when setting the .Values.kubecostProductConfigs.athenaBucketName Helm value.

Summary and pricing

AWS services used here are:

Kubecost's cost-model requires roughly 2 CPU and 10 GB of RAM per 50,000 pods monitored. The backing Prometheus database requires roughly 2 CPU and 25 GB per million metrics ingested per minute. You can pick the EC2 instances necessary to run Kubecost accordingly.

Kubecost can write its cache to disk. Roughly 32 GB per 100,000 pods monitored is sufficient. (Optional: our cache can exist in memory)

AWS Cloud Integration using IAM Roles for Service Accounts (IRSA)

There are many ways to integrate your AWS Cost and Usage Report (CUR) with Kubecost. This tutorial is intended as the best-practice method for users whose environments meet the following assumptions:

Kubecost will run in a different account than the AWS Payer Account
The IAM permissions will utilize AWS to avoid shared secrets
The configuration of Kubecost will be done using a cloud-integration.json file, and not via Kubecost UI (following infrastructure as code practices)

If this is not an accurate description of your environment, see our doc for more options.

Overview of Kubecost CUR integration

This guide is a one-time setup per AWS payer account and is typically one per organization. It can be automated, but may not be worth the effort given that it will not be needed again.

Basic diagram when the below steps are complete:

Kubecost supports multiple AWS payer accounts as well as multiple cloud providers from a single Kubecost primary cluster. For multiple payer accounts, create additional entries inside the array below.

Detail for multiple cloud provider setups is .

Configuration

Step 1: Download configuration files

cloud-integration.json
iam-payer-account-cur-athena-glue-s3-access.json
iam-payer-account-trust-primary-account.json
iam-access-cur-in-payer-account.json

Begin by opening cloud_integration.json, which should look like this:

Update athenaWorkgroup to primary, then save the file and close it. The remaining values will be obtained during this tutorial.

Step 2: Create a CUR Export (and wait 24 hours)

For time granularity, select Daily.
Select the checkbox to enable Resource IDs in the report.
Select the checkbox to enable Athena integration with the report.
Select the checkbox to enable the JSON IAM policy to be applied to your bucket.

Screenshots from select CUR creation in the above AWS documentation

If this CUR data is only used by Kubecost, it is safe to expire or delete the objects after seven days of retention.

AWS may take up to 24 hours to publish data. Wait until this is complete before continuing to the next step.

While you wait, update the following configuration files:

Update your cloud-integration.json file by providing a projectID value, which will be the AWS payer account number where the CUR is located and where the Kubecost primary cluster is running.
Update your iam-payer-account-cur-athena-glue-s2-access.json file by replacing all instances of CUR_BUCKET_NAME to the name of the bucket you created for CUR data.

Step 3: Setting up Athena

Your S3 path prefix can be found by going to your AWS Cost and Usage Reports dashboard and selecting your bucket's report. In the Report details tab, you will find the S3 path prefix.

Once Athena is set up with the CUR, you will need to create a new S3 bucket for Athena query results. The bucket used for the CUR cannot be used for the Athena output.

Select Create bucket. The Create Bucket page opens.
Provide a name for your bucket. This is the value for athenaBucketName in your cloud-integration.json file. Use the same region used for the CUR bucket.
Select Create bucket at the bottom of the page.
Select Settings, then select Manage. The Manage settings window opens.
Set Location of query result to the S3 bucket you just created, then select Save.

Navigate to Athena in the AWS Console. Be sure the region matches the one used in the steps above. Update your cloud-integration.json file with the following values. Use the screenshots below for help.

athenaBucketName: the name of the Athena bucket your created in this step
athenaDatabase: the value in the Database dropdown
athenaRegion: the AWS region value where your Athena query is configured
athenaTable: the partitioned value found in the Table list

For Athena query results written to an S3 bucket only accessed by Kubecost, it is safe to expire or delete the objects after one day of retention.

Step 4: Setting up payer account IAM permissions

From the AWS payer account

In iam-payer-account-cur-athena-glue-s3-access.json, replace all ATHENA_RESULTS_BUCKET_NAME instances with your Athena S3 bucket name (the default will look like aws-athena-query-results-xxxx).

In iam-payer-account-trust-primary-account.json, replace SUB_ACCOUNT_222222222 with the account number of the account where the Kubecost primary cluster will run.

In the same location as your downloaded configuration files, run the following command to create the appropriate policy (jq is not required):

Now we can obtain the last value masterPayerARN for cloud-integration.json as the ARN associated with the newly-created IAM role, as seen below in the AWS console:

Step 5: Setting up IAM permissions for the primary cluster

By arriving at this step, you should have been able to provide all values to your cloud-integration.json file. If any values are missing, reread the tutorial and follow any steps needed to obtain those values.

From the AWS Account where the Kubecost primary cluster will run

In iam-access-cur-in-payer-account.json, update PAYER_ACCOUNT_11111111111 with the AWS account number of the payer account and create a policy allowing Kubecost to assumeRole in the payer account:

Note the output ARN (used in the iamserviceaccount --attach-policy-arn below):

Create a namespace and set environment variables:

Enable the OIDC-Provider:

Create the Kubernetes service account, attaching the assumeRole policy. Replace SUB_ACCOUNT_222222222 with the AWS account number where the primary Kubecost cluster will run.

Create the secret (in this setup, there are no actual secrets in this file):

Install Kubecost using the service account and cloud-integration secret:

Validation

It can take over an hour to process the billing data for large AWS accounts. In the short-term, follow the logs and look for a message similar to (7.7 complete), which should grow gradually to (100.0 complete). Some errors (ERR) are expected, as seen below.

Troubleshooting

AWS Out of Cluster

Integrating Kubecost with your AWS data provides the ability to allocate out-of-cluster (OOC) costs, e.g. RDS instances and S3 buckets, back to Kubernetes concepts like namespace and deployment as well as reconcile cluster assets back to your billing data. The latter is especially helpful when teams are using Reserved Instances, Savings Plans, or Enterprise Discounts. All billing data remains on your cluster when using this functionality and is not shared externally. Read our Cloud Integrations doc for more information on how Kubecost connects with Cloud Service Providers.

The following guide provides the steps required for enabling OOC costs allocation and accurate pricing, e.g. reserved instance price allocation. In a multi-account organization, all of the following steps will need to be completed in the payer account.

Step 1: Create an AWS Cost and Usage Report (CUR) and integrate it with Kubecost

You can learn how to perform this using our AWS Cloud Integration doc.

Step 2: Tag your resources

Kubecost utilizes AWS tagging to allocate the costs of AWS resources outside of the Kubernetes cluster to specific Kubernetes concepts, such as namespaces, pods, etc. These costs are then shown in a unified dashboard within the Kubecost interface.

To allocate external AWS resources to a Kubernetes concept, use the following tag naming scheme:

Kubernetes Concept

AWS Tag Key

AWS Tag Value

In the kubernetes_label_NAME tag key, the NAME portion should appear exactly as the tag appears inside of Kubernetes. For example, for the tag app.kubernetes.io/name, this tag key would appear as kubernetes_label_app.kubernetes.io/name.

To use an alternative or existing AWS tag schema, you may supply these in your values.yaml under kubecostProductConfigs.labelMappingConfigs.\<aggregation\>\_external_label. Also be sure to set kubecostProductConfigs.labelMappingConfigs.enabled=true.

For more information, consult AWS' Tag your Amazon EC2 resources.

Tags may take several hours to show up in the Cost Allocations Tags section described in the next step.

Custom tags mapping

Tags that contain : in the key may be converted to _ in the Kubecost UI due to Prometheus readability. To use AWS Label Mapping Configs, use this mapping format:

kubecostProductConfigs:
  labelMappingConfigs:
    enabled: true
    namespace_external_label: mycompanybilling_namespace
    cluster_external_label: mycompanybilling_cluster

To view examples of common label mapping configs, see here.

Step 3: Enable user-defined cost allocation tags

In order to make the custom Kubecost AWS tags appear on the CURs, and therefore in Kubecost, individual cost allocation tags must be enabled. Details on which tags to enable can be found in Step 2.

For instructions on enabling user-defined cost allocation tags, consult AWS' Activating user-defined cost allocation tags

Viewing account-level tags

Account-level tags are applied (as labels) to all the Assets built from resources defined under a given AWS account. You can filter AWS resources in the Kubecost Assets View (or API) by account-level tags by adding them ('tag:value') in the Label/Tag filter.

If a resource has a label with the same name as an account-level tag, the resource label value will take precedence.

Modifications incurred on account-level tags may take several hours to update on Kubecost.

Your AWS account will need to support the organizations:ListAccounts and organizations:ListTagsForResource policies to benefit from this feature.

Troubleshooting

In the Kubecost UI, view the Allocations dashboard. If external costs are not shown, open your browser's Developer Tools > Console to see any reported errors.
Query Athena directly to ensure data is available. Note: it can take up to 6 hours for data to be written.
You may need to upgrade your AWS Glue if you are running an old version. See Upgrading to the AWS Glue Data Catalog step-by-step for more info.
Finally, review pod logs from the cost-model container in the cost-analyzer pod and look for auth errors or Athena query results.

AWS Spot Instances

Considerations before configuring Spot pricing

Kubecost uses public pricing from Cloud Service Providers (CSPs) to calculate costs until the actual cloud bill is available, at which point Kubecost will reconcile your Spot prices from your Cost and Usage Report (CUR). This is almost always ready in 48 hours. Most users will likely prefer to configure AWS cloud-integrations instead of configuring the Spot data feed manually as demonstrated in this article.

However, if the majority of costs are due to Spot nodes, it may be useful to configure the Spot pricing data feed as it will increase accuracy for short-term (<48 hour) node costs until the Spot prices from the CUR are available. Note that all other (non-Spot) costs will still be based on public (on-demand) pricing until CUR billing data is reconciled.

Configuring the Spot data feed in Kubecost

With Kubecost, Spot pricing data can be pulled hourly by integrating directly with the AWS Spot feed.

First, to enable the AWS Spot data feed, follow AWS' Spot Instance data feed doc.

While configuring, note the settings used as these values will be needed for the Kubecost configuration.

There are multiple options: this can either be set from the Kubecost UI or via .Values.kubecostProductConfigs in the Helm chart. If you set any kubecostProductConfigs from the Helm chart, all changes via the front end will be deleted on pod restart.

projectID the Account ID of the AWS Account on which the Spot nodes are running.
awsSpotDataRegion region of your Spot data bucket
awsSpotDataBucket the configured bucket for the Spot data feed
awsSpotDataPrefix optional configured prefix for your Spot data feed bucket
spotLabel optional Kubernetes node label name designating whether a node is a Spot node. Used to provide pricing estimates until exact Spot data becomes available from the CUR
spotLabelValue optional Kubernetes node label value designating a Spot node. Used to provide pricing estimates until exact Spot data becomes available from the CUR. For example, if your Spot nodes carry a label lifecycle:spot, then the spotLabel would be lifecycle and the spotLabelValue would be spot

In the UI, you can access these fields via the Settings page, then scroll to Cloud Cost Settings. Next to Spot Instance Configuration, select Update, then fill out all fields.

Spot data feeds are an account level setting, not a payer level. Every AWS Account will have its own Spot data feed. Spot data feed is not currently available in AWS GovCloud.

For Spot data written to an S3 bucket only accessed by Kubecost, it is safe to delete objects after three days of retention.

Configuring IAM

Kubecost requires read access to the Spot data feed bucket. The following IAM policy can be used to grant Kubecost read access to the Spot data feed bucket.

{
  "Version": "2012-10-17",
  "Statement": [
    {
        "Sid": "SpotDataFeed",
        "Effect": "Allow",
        "Action": [
          "s3:ListAllMyBuckets",
          "s3:ListBucket",
          "s3:HeadBucket",
          "s3:HeadObject",
          "s3:List*",
          "s3:Get*"
        ],
        "Resource": "arn:aws:s3:::${SpotDataFeedBucketName}*"
    }
  ]
}

To attach the IAM policy to the Kubecost service account, you can use IRSA or the account's service key.

Option 1: IRSA (IAM Roles for Service Accounts)

If your serviceaccount/kubecost-cost-analyzer already has IRSA annotations attached, be sure to include all policies necessary when running this command.

eksctl create iamserviceaccount \
    --name kubecost-cost-analyzer \
    --namespace kubecost \
    --cluster $CLUSTER_NAME --region $REGION_NAME \
    --attach-policy-arn arn:aws:iam::$ACCOUNT_NUMBER:policy/SpotDataFeed \
    --override-existing-serviceaccounts \
    --approve

Option 2: Service Keys

Create a service-key.json as shown:

{
    "aws_access_key_id": "AWS_service_key_aws_access_key_id",
    "aws_secret_access_key": "AWS_service_key_aws_secret_access_key"
}

Create a K8s secret:

$ kubectl create secret generic cloud-service-key --from-file=service-key.json

Set the following Helm config:

kubecostProductConfigs:
  serviceKeySecretName: "cloud-service-key"

Troubleshooting Spot data feed

No Spot instances detected

Verify the below points:

Make sure data is present in the Spot data feed bucket.
Make sure Project ID is configured correctly. You can cross-verify the values under Helm values in bug report
Check the value of kubecost_node_is_spot in Prometheus:
- "1" means Spot data instance configuration is correct.
- "0" means not configured properly.
Is there a prefix? If so, is it configured in Kubecost?
Make sure the IAM permissions are aligned with https://github.com/kubecost/cloudformation/blob/7feace26637aa2ece1481fda394927ef8e1e3cad/kubecost-single-account-permissions.yaml#L36
Make sure the Spot data feed bucket has all permissions to access by Kubecost
The Spot Instance in the Spot data feed bucket should match the instance in the cluster where the Spot data feed is configured. awsSpotDataBucket has to be present in the right cluster.

AWS Node Price Reconciliation Methodology

Kubecost is capable of aggregating the costs of EC2 compute resources over a given timeframe with a specified duration step size. To achieve this, Kubecost uses Athena queries to gather usage data points with differing price models. The result of this process is a list of resources with their cost by timeframe.

Athena queries

The reconciliation process makes two queries to Athena, one to gather resources that are paid for with either the on-demand model or a savings plan and one query for resources on the reservation price model. The first query includes resources given at a blended rate, which could be on-demand usage or resources that have exceeded the limits of a savings plan. It will also include resources that are part of a savings plan which will have a savings plan effective cost. The second query only includes reserved resources and the cost which reflects the rate they were reserved at.

The queries make use of the following columns from Athena:

line_item_usage_start_date The beginning timestamp of the line item usage. Used to filter resource usage within a date range and to aggregate on usage window.
line_item_usage_end_date The ending timestamp of the line item usage. Used to filter resource usage within a date range and to aggregate on usage window.
line_item_resource_id An ID, also called the provider ID, is given to line items that are instantiated resources.
line_item_line_item_type The type of a line item, used to determine if the resource usage is covered by a savings plan and has a discounted price.
line_item_usage_type What is being used in a line item, for the purposes of a compute resource this, is the type of VM and where it is running
line_item_product_code The service that a line item is from. Used to filter out items that are not from EC2.
reservation_reservation_a_r_n Amazon Resource Name for reservation of line item, the presence of this value is used to identify a resource as being part of a reservation plan.
line_item_unblended_cost The undiscounted cost of a resource.
savings_plan_savings_plan_effective_cost The cost of a resource discounted by a savings plan
reservation_effective_cost The cost of a resource discounted by a reservation

On-Demand/Savings plan query

This query is grouped by six columns:

line_item_usage_start_date
line_item_usage_end_date
line_item_resource_id
line_item_line_item_type
line_item_usage_type
line_item_product_code

The columns line_item_unblended_cost and savings_plan_savings_plan_effective_cost are summed on this grouping. Finally, the query filters out rows that are not within a given date range, have a missing line_item_resource_id, and have a line_item_product_code not equal to "AmazonEC2". The grouping has three important aspects, the timeframe of the line items, the resource as defined by the resource id, and the usage type, which is later used to determine the proper cost of the resources as it was used. This means that line items are grouped according to the resource, the time frame of the usage, and the rate at which the usage was charged.

Reservation query

The reservation query is grouped on five columns:

line_item_usage_start_date
line_item_usage_end_date
reservation_reservation_a_r_n
line_item_resource_id
line_item_product_code

The query is summed on the reservation_effective_cost and filtered by the date window, for missing reservation_reservation_a_r_n values and also removes line items with line_item_product_code not equal to "AmazonEC2". This grouping is on resource id by timeframe removing all non-reservation line items.

Processing query results

The on-demand query is categorized into different resource types: compute, network, storage, and others. The network is identified by the presence of the "byte" in the line_item_usage_type. Compute and storage are identified by the presence of "i-" and "vol-" prefixes in line_item_resource_id respectively. Non compute values are removed from the results. Out of the two costs aggregated by this query the correct one to use is determined by the line_item_line_item_type, if it has a value of "SavingsPlanCoveredUsage", then the savings_plan_savings_plan_effective_cost is used as the cost, and if not then the line_item_unblended_cost is used.

In the reservation query, all of the results are of the compute category and there is only the reservation_effective_cost to use as a cost.

These results are then merged into one set, with the provider id used to associate the cost with other information about the resource.

Why doesn't this match the AWS Cost Explorer?

There are several different ways to look at your node cost data. The default for the cost explorer is Unblended" but it makes the most sense from an allocation perspective to use the amortized rates. Be sure Amortized costs is selected when looking at cost data. Here's an example of how they can vary dramatically on our test cluster.

The t2-mediums here are covered by a savings plan. Unblended, the cost is only $0.06/day for two.

When Amortized costs is selected, the price jumps to $1.50/day

This should closely match our data on the Assets page, for days where we have adjustments come in from the pricing CUR.

AWS Cloud Integration using IAM Roles for Service Accounts (IRSA)

There are many ways to integrate your AWS Cost and Usage Report (CUR) with Kubecost. This tutorial is intended as the best-practice method for users whose environments meet the following assumptions:

Kubecost will run in a different account than the AWS Payer Account
The IAM permissions will utilize AWS to avoid shared secrets
The configuration of Kubecost will be done using a cloud-integration.json file, and not via Kubecost UI (following infrastructure as code practices)

If this is not an accurate description of your environment, see our doc for more options.

Overview of Kubecost CUR integration

This guide is a one-time setup per AWS payer account and is typically one per organization. It can be automated, but may not be worth the effort given that it will not be needed again.

Basic diagram when the below steps are complete:

Detail for multiple cloud provider setups is .

Configuration

Step 1: Download configuration files

To begin, download the recommended configuration template files from our . You will need the following files from this folder:

cloud-integration.json
iam-payer-account-cur-athena-glue-s3-access.json
iam-payer-account-trust-primary-account.json
iam-access-cur-in-payer-account.json

Begin by opening cloud_integration.json, which should look like this:

{
    "aws": [
        {
            "athenaBucketName": "s3://ATHENA_RESULTS_BUCKET_NAME",
            "athenaRegion": "ATHENA_REGION",
            "athenaDatabase": "ATHENA_DATABASE",
            "athenaTable": "ATHENA_TABLE",
            "athenaWorkgroup": "ATHENA_WORKGROUP",
            "projectID": "ATHENA_PROJECT_ID",
            "masterPayerARN": "PAYER_ACCOUNT_ROLE_ARN"
        }
    ]
}

Update athenaWorkgroup to primary, then save the file and close it. The remaining values will be obtained during this tutorial.

Step 2: Create a CUR Export (and wait 24 hours)

Follow the to create a CUR export using the settings below.

For time granularity, select Daily.
Select the checkbox to enable Resource IDs in the report.
Select the checkbox to enable Athena integration with the report.
Select the checkbox to enable the JSON IAM policy to be applied to your bucket.

Screenshots from select CUR creation in the above AWS documentation

If this CUR data is only used by Kubecost, it is safe to expire or delete the objects after seven days of retention.

AWS may take up to 24 hours to publish data. Wait until this is complete before continuing to the next step.

While you wait, update the following configuration files:

Update your cloud-integration.json file by providing a projectID value, which will be the AWS payer account number where the CUR is located and where the Kubecost primary cluster is running.
Update your iam-payer-account-cur-athena-glue-s2-access.json file by replacing all instances of CUR_BUCKET_NAME to the name of the bucket you created for CUR data.

Step 3: Setting up Athena

As part of the CUR creation process, Amazon creates a CloudFormation template that is used to create the Athena integration. It is created in the CUR S3 bucket under s3-path-prefix/cur-name and typically has the filename crawler-cfn.yml. This .yml is your CloudFormation template. You will need it in order to complete the CUR Athena integration. You can read more about this .

Your S3 path prefix can be found by going to your AWS Cost and Usage Reports dashboard and selecting your bucket's report. In the Report details tab, you will find the S3 path prefix.

Once Athena is set up with the CUR, you will need to create a new S3 bucket for Athena query results. The bucket used for the CUR cannot be used for the Athena output.

Navigate to the .
Select Create bucket. The Create Bucket page opens.
Provide a name for your bucket. This is the value for athenaBucketName in your cloud-integration.json file. Use the same region used for the CUR bucket.
Select Create bucket at the bottom of the page.
Navigate to the dashboard.
Select Settings, then select Manage. The Manage settings window opens.
Set Location of query result to the S3 bucket you just created, then select Save.

athenaBucketName: the name of the Athena bucket your created in this step
athenaDatabase: the value in the Database dropdown
athenaRegion: the AWS region value where your Athena query is configured
athenaTable: the partitioned value found in the Table list

For Athena query results written to an S3 bucket only accessed by Kubecost, it is safe to expire or delete the objects after one day of retention.

Step 4: Setting up payer account IAM permissions

From the AWS payer account

In iam-payer-account-cur-athena-glue-s3-access.json, replace all ATHENA_RESULTS_BUCKET_NAME instances with your Athena S3 bucket name (the default will look like aws-athena-query-results-xxxx).

In iam-payer-account-trust-primary-account.json, replace SUB_ACCOUNT_222222222 with the account number of the account where the Kubecost primary cluster will run.

In the same location as your downloaded configuration files, run the following command to create the appropriate policy (jq is not required):

aws iam create-role --role-name kubecost-cur-access \
  --assume-role-policy-document file://iam-payer-account-trust-primary-account.json \
  --output json | jq -r .Role.Arn

Now we can obtain the last value masterPayerARN for cloud-integration.json as the ARN associated with the newly-created IAM role, as seen below in the AWS console:

Step 5: Setting up IAM permissions for the primary cluster

From the AWS Account where the Kubecost primary cluster will run

aws iam create-policy --policy-name kubecost-access-cur-in-payer-account \
  --policy-document file://iam-access-cur-in-payer-account.json \
  --output json |jq -r .Policy.Arn

Note the output ARN (used in the iamserviceaccount --attach-policy-arn below):

arn:aws:iam::SUB_ACCOUNT_222222222:policy/kubecost-access-cur-in-payer-account

Create a namespace and set environment variables:

kubectl create ns kubecost
export CLUSTER_NAME=YOUR_CLUSTER
export AWS_REGION=YOUR_REGION

Enable the OIDC-Provider:

eksctl utils associate-iam-oidc-provider \
    --cluster $CLUSTER_NAME --region $AWS_REGION \
    --approve

Create the Kubernetes service account, attaching the assumeRole policy. Replace SUB_ACCOUNT_222222222 with the AWS account number where the primary Kubecost cluster will run.

eksctl create iamserviceaccount \
    --name kubecost-serviceaccount \
    --namespace kubecost \
    --cluster $CLUSTER_NAME --region $AWS_REGION \
    --attach-policy-arn arn:aws:iam::SUB_ACCOUNT_222222222:policy/kubecost-access-cur-in-payer-account \
    --override-existing-serviceaccounts \
    --approve

Create the secret (in this setup, there are no actual secrets in this file):

kubectl create secret generic cloud-integration -n kubecost --from-file=cloud-integration.json

Install Kubecost using the service account and cloud-integration secret:

helm install kubecost \
  --repo https://kubecost.github.io/cost-analyzer/ cost-analyzer \
  --namespace kubecost \
  --set serviceAccount.name=kubecost-serviceaccount \
  --set serviceAccount.create=false \
  --set kubecostProductConfigs.cloudIntegrationSecret=cloud-integration

Validation

kubectl logs -l app=cost-analyzer --tail -1 --follow |grep -i athena
------------------
Defaulted container "cost-model" out of: cost-model, cost-analyzer-frontend
2023-05-24T19:41:31.63093249Z ERR Failed to lookup reserved instance data: no reservation data available in Athena
2023-05-24T19:41:31.630973097Z ERR Failed to lookup savings plan data: Error fetching Savings Plan Data: QueryAthenaPaginated: athena configuration incomplete
2023-05-24T19:41:34.577437245Z INF Adding AWS Provider to map with key: 440082503234/s3://aws-athena-query-results-
2023-05-24T19:41:34.57901059Z INF ETL: CloudUsage[440082503234/s3://aws-athena-query-results-]: Starting PipelineController
2023-05-24T19:41:34.579037927Z INF CloudCost: IngestionManager: creating integration with key: 440082503234/s3://aws-athena-query-results-
2023-05-24T19:41:34.581131953Z INF RunBuildProcess[CloudCost][440082503234/s3://aws-athena-query-results-]: build[NLzAH]: Starting build back to 2023-02-22 00:00:00 +0000 UTC in blocks of 7d
2023-05-24T19:41:34.581715777Z INF CloudCost[440082503234/s3://aws-athena-query-results-]: ingestor: building window [2023-05-17T00:00:00+0000, 2023-05-24T00:00:00+0000)
2023-05-24T19:41:34.581771159Z INF AthenaIntegration[440082503234/s3://aws-athena-query-results-]: StoreCloudCost: [2023-05-17T00:00:00+0000, 2023-05-24T00:00:00+0000)
2023-05-24T19:41:34.608040758Z ERR ETL:  CloudUsage[440082503234/s3://aws-athena-query-results-]: Build[XPQMa]:  failed to load range from back up 2023-05-18 00:00:00 +0000 UTC - 2023-05-25 00:00:00 +0000 UTC: file does not exist
2023-05-24T19:41:34.60806207Z INF ETL: CloudUsage[440082503234/s3://aws-athena-query-results-]: Build[XPQMa]: QueryCloudUsage [2023-05-18T00:00:00+0000, 2023-05-25T00:00:00+0000)
2023-05-24T19:41:34.588661972Z INF ETL: CloudUsage[440082503234/s3://aws-athena-query-results-]: Run[JkSYH]: QueryCloudUsage [2023-05-21T00:00:00+0000, 2023-05-25T00:00:00+0000)
2023-05-24T19:41:50.528544821Z INF ETL: CloudUsage[440082503234/s3://aws-athena-query-results-]: Build[XPQMa]: coverage [2023-05-18T00:00:00+0000, 2023-05-25T00:00:00+0000) (7.7 complete)

Troubleshooting

For help with troubleshooting, see the section in our original .

AWS Cloud Billing Integration

Quick Start for IRSA

Key AWS terminology

S3 bucket: Cloud object storage tool which both CURs and Athena output cost data to. Kubecost needs access to these buckets in order to read that data.

Cost and Usage Report integration

For the below guide, a GitHub repository with sample files can be found .

Step 1: Setting up a CUR

Follow to set up a Legacy CUR using the settings below.

Select the Legacy CUR export type.
For time granularity, select Daily.
Under 'Additional content', select the Enable resource IDs checkbox.
Under 'Report data integration' select the Amazon Athena checkbox.

For CUR data written to an S3 bucket only accessed by Kubecost, it is safe to expire or delete the objects after seven days of retention.

Remember the name of the bucket you create for CUR data. This will be used in Step 2.

Familiarize yourself with how column name restrictions differ between CURs and Athena tables. AWS may change your CUR name when you upload your CUR to your Athena table in Step 2, documented in AWS' . As best practice, use all lowercase letters and only use _ as a special character.

AWS may take up to 24 hours to publish data. Wait until this is complete before continuing to the next step.

If you believe you have the correct permissions, but cannot access the Billing and Cost Management page, have the owner of your organization's root account follow .

Step 2: Setting up Athena

As part of the CUR creation process, Amazon also creates a CloudFormation template that is used to create the Athena integration. It is created in the CUR S3 bucket, listed in the Objects tab in the path s3-path-prefix/cur-name and typically has the filename crawler-cfn.yml. This .yml is your necessary CloudFormation template. You will need it in order to complete the CUR Athena integration. For more information, see the AWS doc .

Your S3 path prefix can be found by going to your AWS Cost and Usage Reports dashboard and selecting your newly-created CUR. In the 'Report details' tab, you will find the S3 path prefix.

Once Athena is set up with the CUR, you will need to create a new S3 bucket for Athena query results.

Navigate to the .
Select Create bucket. The Create Bucket page opens.
Use the same region used for the CUR bucket and pick a name that follows the format aws-athena-query-results-.
Select Create bucket at the bottom of the page.
Navigate to the dashboard.
Select Settings, then select Manage. The Manage settings window opens.
Set Location of query result to the S3 bucket you just created, which will look like s3://aws-athena-query-results..., then select Save.

For Athena query results written to an S3 bucket only accessed by Kubecost, it is safe to expire or delete the objects after 1 day of retention.

Step 3: Setting up IAM permissions

Add via CloudFormation:

Kubecost offers a set of CloudFormation templates to help set your IAM roles up.

If you’re new to provisioning IAM roles, we suggest downloading our templates and using the CloudFormation wizard to set these up. You can learn how to do this in AWS' doc. Open the step below which represents your CUR and management account arrangement, download the .yaml file listed, and upload them as the stack template in the 'Creating a stack' > 'Selecting a stack template' step.

My CUR exists in the same account as Kubecost or the management account

Download .
Navigate to the .
Select Create Stack, then select With new resources (standard) from the dropdown.
On the 'Create stack' page, under 'Prerequisite - Prepare Template', make sure Template is ready has been preselected. Under 'Specify Template', select Upload a template file. Select Choose file, then select your downloaded .yaml file from your file explorer. Select Next.
On the 'Specify stack details' page, enter a name for your stack, then provide the following parameters:
- AthenaCURBucket: The name of the Athena CUR bucket you created in Step 2.
- SpotDataFeedBucketName: (Optional, skip if you have not configured Spot data) The bucket where the Spot data feed is sent
Select Next.
On the 'Configure stack options' page opens, configure any additional options as needed. Select Next.
On the 'Review stack' page, confirm all information, then select I acknowledge that AWS CloudFormation might create IAM resources with custom names.
Select Submit.

My CUR exists in a member account different from Kubecost or the management account

On each sub account running Kubecost:

Download .
- Navigate to the .
- Select Create Stack, then select With existing resources (import resources) from the dropdown. On the 'Identify resources' page, select Next.
- Under Template source, choose Upload a template file.
- Select Choose file, which will open your file explorer. Select the .yaml template, and then select Open. Then, select Next.
- On the 'Identify resources' page, provide any additional resources to import. Then, select Next.
- For Stack name, enter a name for your template.
- Set the following parameters:
  - MasterPayerAccountID: The account ID of the management account (formerly called master payer account) where the CUR has been created
  - SpotDataFeedBucketName: The bucket where the Spot data feed is sent
- Select Next.
- Select Next.
- At the bottom of the page, select I acknowledge that AWS CloudFormation might create IAM resources.
- Select Create Stack.

On the management account:

Follow the same steps to create a CloudFormation stack as above, but using instead, and with these parameters:
- AthenaCURBucket: The bucket where the CUR is set from Step 1
- KubecostClusterID: An account that Kubecost is running on that requires access to the Athena CUR.

Add manually:

My CUR exists in the same account as Kubecost or the management account

        {
           "Version": "2012-10-17",
           "Statement": [
              {
                 "Sid": "AthenaAccess",
                 "Effect": "Allow",
                 "Action": [
                    "athena:*"
                 ],
                 "Resource": [
                    "*"
                 ]
              },
              {
                 "Sid": "ReadAccessToAthenaCurDataViaGlue",
                 "Effect": "Allow",
                 "Action": [
                    "glue:GetDatabase*",
                    "glue:GetTable*",
                    "glue:GetPartition*",
                    "glue:GetUserDefinedFunction",
                    "glue:BatchGetPartition"
                 ],
                 "Resource": [
                    "arn:aws:glue:*:*:catalog",
                    "arn:aws:glue:*:*:database/athenacurcfn*",
                    "arn:aws:glue:*:*:table/athenacurcfn*/*"
                 ]
              },
              {
                 "Sid": "AthenaQueryResultsOutput",
                 "Effect": "Allow",
                 "Action": [
                    "s3:GetBucketLocation",
                    "s3:GetObject",
                    "s3:ListBucket",
                    "s3:ListBucketMultipartUploads",
                    "s3:ListMultipartUploadParts",
                    "s3:AbortMultipartUpload",
                    "s3:CreateBucket",
                    "s3:PutObject"
                 ],
                 "Resource": [
                    "arn:aws:s3:::aws-athena-query-results-*"
                 ]
              },


              {
                 "Sid": "S3ReadAccessToAwsBillingData",
                 "Effect": "Allow",
                 "Action": [
                    "s3:Get*",
                    "s3:List*"
                 ],
                 "Resource": [
                    "arn:aws:s3:::${AthenaCURBucket}*"
                 ]
              }
           ]
        }
	{
           "Version": "2012-10-17",
           "Statement": [
              {
                 "Sid": "SpotDataAccess",
                 "Effect": "Allow",
                 "Action": [
                    "s3:ListAllMyBuckets",
                    "s3:ListBucket",
                    "s3:HeadBucket",
                    "s3:HeadObject",
                    "s3:List*",
                    "s3:Get*"
                 ],
                 "Resource": "arn:aws:s3:::${SpotDataFeedBucketName}*"
              }
           ]
        }

My CUR exists in a member account different from Kubecost or the management account

	{
               "Version": "2012-10-17",
               "Statement": [
                  {
                     "Sid": "AssumeRoleInMasterPayer",
                     "Effect": "Allow",
                     "Action": "sts:AssumeRole",
                     "Resource": "arn:aws:iam::${MasterPayerAccountID}:role/KubecostRole-${This-account’s-id}"
                  }
               ]
	}

	{
               "Version": "2012-10-17",
               "Statement": [
                  {
                     "Sid": "SpotDataAccess",
                     "Effect": "Allow",
                     "Action": [
                        "s3:ListAllMyBuckets",
                        "s3:ListBucket",
                        "s3:HeadBucket",
                        "s3:HeadObject",
                        "s3:List*",
                        "s3:Get*"
                     ],
                     "Resource": "arn:aws:s3:::${SpotDataFeedBucketName}*"
                  }
               ]
	}

On the management account, attach this policy to a role (replace ${AthenaCURBucket} variable):

	{
               "Version": "2012-10-17",
               "Statement": [
                  {
                     "Sid": "AthenaAccess",
                     "Effect": "Allow",
                     "Action": [
                        "athena:*"
                     ],
                     "Resource": [
                        "*"
                     ]
	},
	{
                     "Sid": "ReadAccessToAthenaCurDataViaGlue",
                     "Effect": "Allow",
                     "Action": [
                        "glue:GetDatabase*",
                        "glue:GetTable*",
                        "glue:GetPartition*",
                        "glue:GetUserDefinedFunction",
                        "glue:BatchGetPartition"
                     ],
                     "Resource": [
                        "arn:aws:glue:*:*:catalog",
                        "arn:aws:glue:*:*:database/athenacurcfn*",
                        "arn:aws:glue:*:*:table/athenacurcfn*/*"
                     ]
                  },
                  {
                     "Sid": "AthenaQueryResultsOutput",
                     "Effect": "Allow",
                     "Action": [
                        "s3:GetBucketLocation",
                        "s3:GetObject",
                        "s3:ListBucket",
                        "s3:ListBucketMultipartUploads",
                        "s3:ListMultipartUploadParts",
                        "s3:AbortMultipartUpload",
                        "s3:CreateBucket",
                        "s3:PutObject"
                     ],
                     "Resource": [
                        "arn:aws:s3:::aws-athena-query-results-*"
                     ]
                  },
                  {
                     "Sid": "S3ReadAccessToAwsBillingData",
                     "Effect": "Allow",
                     "Action": [
                        "s3:Get*",
                        "s3:List*"
                     ],
                     "Resource": [
                        "arn:aws:s3:::${AthenaCURBucket}*"
                     ]
                  }
               ]
	}

Then add the following trust statement to the role the policy is attached to on the management account (replace the ${aws-mgmt-account-id} variable with the account you want to assume the role):

	{
               "Version": "2012-10-17",
               "Statement": [
                  {
                     "Effect": "Allow",
                     "Principal": {
                        "AWS": "arn:aws:iam::${aws-mgmt-account-id}:root"
                     },
                     "Action": [
                        "sts:AssumeRole"
                     ]
                  }
               ]
            }

Step 4: Attaching IAM permissions to Kubecost

If you are using the alternative method, steps 4 and 5 are not required.

Now that the policies have been created, attach those policies to Kubecost. We support the following methods:

Attach via Service Key and Kubernetes Secret

Navigate to the , then select Access Management > Users from the left navigation. Find the Kubecost User and select Security credentials > Create access key. Follow along to receive the Access Key ID and Secret Access Key (AWS will not provide you the Secret Access Key in the future, so make sure you save this value). Then, follow the steps from either Option 1 or Option 2 below, but not both.

Option 1: Generate a secret from Helm values:

Note that this will leave your AWS keys unencrypted in your values.yaml. Set the following Helm values:

kubecostProductConfigs:
  createServiceKeySecret: true
  awsServiceKeyName: <ACCESS_KEY_ID>
  awsServiceKeyPassword: <SECRET_ACCESS_KEY>

Option 2: Manually create a secret:

This may be the preferred method if your Helm values are in version control and you want to keep your AWS secrets out of version control.

Create a service-key.json:

{
    "aws_access_key_id": "<ACCESS_KEY_ID>",
    "aws_secret_access_key": "<SECRET_ACCESS_KEY>"
}

Create a Kubernetes secret:

$ kubectl create secret generic <SECRET_NAME> --from-file=service-key.json --namespace <kubecost>

Set the Helm value:

kubecostProductConfigs:
  serviceKeySecretName: <SECRET_NAME>

Attach via Service Key in Kubecost UI

In the , select Access Management > Users. Find the Kubecost user and select Security credentials > Create access key. Note the Access Key ID and Secret Access Key.
To add the Access Key ID and Secret Access Key in the Kubecost UI, select Settings from the left navigation, then scroll to Cloud Cost Settings. Select Update next to External Cloud Cost Configuration (AWS). Fill in the Service key name and Service key secret fields respectively, then select Update.

You may not be allowed to update your Billing Data Export Configuration without filling out all fields first. For explanations of your Athena/AWS-related fields, see Step 5 below.

Attach via pod annotation with eksctl

Prerequisites:

Amazon EKS cluster set up via
is installed on your device

Step 1: Update configuration

Download the following configuration files:

Update the following variables in the files you downloaded:

In cloud-integration.json, update the following values with the information in Step 2: Setting up Athena:

    "athenaBucketName": "s3://<AWS_cloud_integration_athenaBucketName>",
    "athenaRegion": "<AWS_cloud_integration_athenaRegion>",
    "athenaDatabase": "<AWS_cloud_integration_athenaDatabase>",
    "athenaTable": "<AWS_cloud_integration_athenaTable>",
    "projectID": "<AWS_account_ID>"

In kubecost-athena-policy.json, replace ${AthenaCURBucket} with your Athena S3 bucket name (configured in Step 2: Setting up Athena).

Step 2: Create policy

In the same location where your downloaded configuration files are, run the following command to create the appropriate policy:

aws iam create-policy --policy-name kubecost-athena-policy --policy-document file://kubecost-athena-policy.json

Step 3: Create OIDC provider for your cluster

kubectl create ns kubecost
export YOUR_CLUSTER_NAME=<ENTER_YOUR_ACTUAL_CLUSTER_NAME>
export AWS_REGION=<ENTER_YOUR_AWS_REGION>
eksctl utils associate-iam-oidc-provider \
    --cluster ${YOUR_CLUSTER_NAME} --region ${AWS_REGION} \
    --approve

Step 4: Create required IAM service accounts

Note: Remember to replace 1234567890 with your AWS account ID number.

eksctl create iamserviceaccount \
    --name kubecost-serviceaccount-cur-athena-thanos \
    --namespace kubecost \
    --cluster ${YOUR_CLUSTER_NAME} --region ${AWS_REGION} \
    --attach-policy-arn arn:aws:iam::1234567890:policy/kubecost-athena-policy \
    --override-existing-serviceaccounts \
    --approve

Step 5: Create required secret to store the configuration

kubectl create secret generic cloud-integration -n kubecost --from-file=cloud-integration.json

Step 6. Install Kubecost via Helm

helm upgrade --install kubecost --repo https://kubecost.github.io/cost-analyzer/ cost-analyzer \
--namespace kubecost \
-f https://raw.githubusercontent.com/kubecost/poc-common-configurations/main/aws-attach-roles/values-amazon-primary.yaml

Step 5: Provide CUR config values to Kubecost

These values can either be set from the Kubecost UI or via .Values.kubecostProductConfigs in the Helm chart. Values for all fields must be provided.

Option 1: Add config values via UI

Field

Description

When you are done, select Update to confirm.

Option 2: Add config values via Helm

If you set any kubecostProductConfigs from the Helm chart, all changes via the front end will be overridden on pod restart.

athenaProjectID: The AWS AccountID where the Athena CUR is, likely your management account.
athenaBucketName: An S3 bucket to store Athena query results that you’ve created that Kubecost has permission to access
- The name of the bucket should match s3://aws-athena-query-results-*, so the IAM roles defined above will automatically allow access to it
- The bucket can have a Canned ACL of Private or other permissions as you see fit.
athenaRegion: The AWS region Athena is running in
athenaDatabase: The name of the database created by the Athena setup
- The athena database name is available as the value (physical id) of AWSCURDatabase in the CloudFormation stack created above (in )
athenaTable: the name of the table created by the Athena setup
- The table name is typically the database name with the leading athenacurcfn_ removed (but is not available as a CloudFormation stack resource). Confirm the table name by visiting the Athena dashboard.
athenaWorkgroup: The workgroup assigned to be used with Athena. If not specified, defaults to Primary

Make sure to use only underscore as a delimiter if needed for tables and views. Using a hyphen/dash will not work even though you might be able to create it. See the for more info.

If you are using a multi-account setup, you will also need to set .Values.kubecostProductConfigs.masterPayerARN to the Amazon Resource Number (ARN) of the role in the management account, e.g. arn:aws:iam::530337586275:role/KubecostRole.

Troubleshooting

You can check pod logs for authentication errors by running: kubectl get pods -n <namespace> kubectl logs <kubecost-pod-name> -n <namespace> -c cost-model

You can also check query history to see if any queries are failing:

Common Athena errors

Incorrect bucket in IAM Policy

Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources. You can search in the Athena "Recent queries" dashboard to find additional info about the error.

```
QueryAthenaPaginated: query execution error: no query results available for query <Athena Query ID>
```

And/or the following error will be found in the Kubecost `cost-model` container logs.

```
Permission denied on S3 path: s3://cur-report/cur-report/cur-report/year=2022/month=8

This query ran against the "athenacurcfn_test" database, unless qualified by the query. Please post the error message on our forum  or contact customer support  with Query Id: <Athena Query ID>
```

Resolution: This error is typically caused by the incorrect (Athena results) s3 bucket being specified in the CloudFormation template of Step 3 from above. To resolve the issue, ensure the bucket used for storing the AWS CUR report (Step 1) is specified in the S3ReadAccessToAwsBillingData SID of the IAM policy (default: kubecost-athena-access) attached to the user or role used by Kubecost (Default: KubecostUser / KubecostRole). See the following example.

This error can also occur when the management account cross-account permissions are incorrect, however, the solution may differ.

        {
        "Action": [
            "s3:Get*",
            "s3:List*"
        ],
        "Resource": [
            "arn:aws:s3:::<AWS CUR BUCKET>*"
        ],
        "Effect": "Allow",
        "Sid": "S3ReadAccessToAwsBillingData"
    }

outputLocation is not a valid S3 path

Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources.

Connection test failed for cloud integration config: Fetch error: cloud billing data fetch error: GetCloudCost: error getting Athena columns: QueryAthenaPaginated: start query error: operation error Athena: StartQueryExecution, https response error StatusCode: 400, RequestID: a6059220-5ac8-4c24-97d2-401a2dbfd421, InvalidRequestException: outputLocation is not a valid S3 path.

Resolution: Please verify that the prefix s3:// was used when setting the athenaBucketName Helm value or when configuring the bucket name in the Kubecost UI.

Query not supported

Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources.

QueryAthenaPaginated: start query error: operation error Athena: StartQueryExecution, https response error StatusCode: 400, RequestID: <Athena Query ID>, InvalidRequestException: Queries of this type are not supported

Resolution: While rare, this issue was caused by an Athena instance that failed to provision properly on AWS. The solution was to delete the Athena DB and deploy a new one. To verify this is needed, find the failed query ID in the Athena "Recent queries" dashboard and attempt to manually run the query.

HTTPS Response error

Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources.

QueryAthenaPaginated: start query error: operation error Athena: StartQueryExecution, https response error StatusCode: 400, RequestID: ********************, InvalidRequestException: Unable to verify/create output bucket aws-athena-query-results-test

Resolution: Previously, if you ran a query without specifying a value for query result location, and the query result location setting was not overridden by a workgroup, Athena created a default location for you. Now, before you can run an Athena query in a region in which your account hasn't used Athena previously, you must specify a query result location, or use a workgroup that overrides the query result location setting. While Athena no longer creates a default query results location for you, previously created default aws-athena-query-results-MyAcctID-MyRegion locations remain valid and you can continue to use them. The bucket should be in the format of: aws-athena-query-results-MyAcctID-MyRegion It may also be required to remove and reinstall Kubecost. If doing this please remeber to backup ETL files prior or contact support for additional assistance. See also this AWS doc on .

Missing Athena column

Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources or in the Kubecost cost-model container logs.

QueryAthenaPaginated: query execution error: no query results available for query <Athena Query ID>

Checking the Athena logs we see a syntax error:

SYNTAX_ERROR: line 4:3: Column 'line_item_resource_id' cannot be resolved

This query ran against the "<DB Name>" database, unless qualified by the query

Resolution: Verify in AWS' Cost and Usage Reports dashboard that the Resource IDs are enabled as "Report content" for the CUR created in Step 1. If the Resource IDs are not enabled, you will need to re-create the report (this will require redoing Steps 1 and 2 from this doc).

Not a valid S3 path

Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources or in the Kubecost cost-model container logs.

QueryAthenaPaginated: start query error: operation error Athena: StartQueryExecution, https response error StatusCode: 400, RequestID: <Athena Query ID>, InvalidRequestException: outputLocation is not a valid S3 path.

Resolution: Verify that s3:// was included in the bucket name when setting the .Values.kubecostProductConfigs.athenaBucketName Helm value.

Summary and pricing

AWS services used here are:

Kubecost can write its cache to disk. Roughly 32 GB per 100,000 pods monitored is sufficient. (Optional: our cache can exist in memory)

(Optional: manual IAM configuration or via Terraform is fine)
(Optional: all K8s flavors are supported)