Installing Kubecost

To get started with Kubecost and OpenCost, visit our Installation page which will take you step by step through getting Kubecost set up.

This installation method is available for free and leverages the Kubecost Helm Chart. It provides access to all OpenCost and Kubecost community functionality and can scale to large clusters. This will also provide a token for trialing and retaining data across different Kubecost product tiers.

Alternative installation methods

1. You can also install directly with the Kubecost Helm Chart with Helm v3.1+ using the following commands. This provides the same functionality as the step above but doesn't generate a product token for managing tiers or upgrade trials.

helm upgrade --install kubecost \
  --repo https://kubecost.github.io/cost-analyzer/ cost-analyzer \
  --namespace kubecost --create-namespace

1. You can run Helm Template against the Kubecost Helm Chart to generate local YAML output. This requires extra effort when compared to directly installing the Helm Chart but is more flexible than deploying a flat manifest.

helm template kubecost \
  --repo https://kubecost.github.io/cost-analyzer/ cost-analyzer \
  --namespace kubecost --create-namespace \
  -f your-custom-values.yaml > kubecost.yaml
kubectl apply -f kubecost.yaml

1. You can install via flat manifest. This install path is not recommended because it has limited flexibility for managing your deployment and future upgrades.

kubectl apply -f https://raw.githubusercontent.com/kubecost/cost-analyzer-helm-chart/develop/kubecost.yaml

1. Lastly, you can deploy the open-source OpenCost project directly as a Pod. This install path provides a subset of free functionality and is available here. Specifically, this install path deploys the underlying cost allocation model without the same UI or access to enterprise functionality: cloud provider billing integration, RBAC/SAML support, and scale improvements in Kubecost.

Configuring Kubecost at install

Kubecost has a number of product configuration options that you can specify at install time in order to minimize the number of settings changes required within the product UI. This makes it simple to redeploy Kubecost. These values can be configured under kubecostProductConfigs in our values.yaml. These parameters are passed to a ConfigMap that Kubecost detects and writes to its /var/configs.

Troubleshooting installation

If you encounter any errors while installing Kubecost, first visit our Troubleshoot Install doc. If the error you are experiencing is not already documented here, or a solution is not found, contact our Support team at support@kubecost.com for more help.

Updating Kubecost

Kubecost releases are scheduled on a near-monthly basis. You can keep up to date with new Kubecost updates and patches by following our release notes here.

After installing Kubecost, you will be able to update Kubecost with the following command, which will upgrade you to the most recent version:

helm repo update && helm upgrade kubecost kubecost/cost-analyzer -n kubecost

You can upgrade or downgrade to a specific version of Kubecost with the following command:

helm upgrade kubecost --repo... --version 1.XXX.X

Deleting Kubecost

To uninstall Kubecost and its dependencies, run the following command:

helm uninstall kubecost -n kubecost

Next steps

After successfully installing Kubecost, first time users should review our First Time User Guide to start immediately seeing the benefits of the product while also ensuring their workspace is properly set up.

After successfully installing Kubecost, new users should familiarize themselves with these onboarding steps to begin immediately realizing value. This doc will explain to you the core features and options you will have access to and direct you to other necessary docs groups that will help you get set up.

While certain steps in this article may be optional depending on your setup, these are recommended best practices for seeing the most value out of Kubecost as soon as possible.

Step 1: Integrate with your cloud provider(s)

Many Kubernetes adopters may have billing with cloud service providers (CSPs) that differs from public pricing. By default, Kubecost will detect the CSP of the cluster where it is installed and pull list prices for nodes, storage, and LoadBalancers across all major CSPs: Azure, AWS, and GCP.

However, Kubecost is also able to integrate these CSPs to receive the most accurate billing data. By completing a cloud integration, Kubecost is able to reconcile costs with your actual cloud bill to reflect enterprise discounts, Spot market prices, commitment discounts, and more.

New users should seek to integrate any and all CSPs they use into Kubecost. For an overview of cloud integrations and getting started, see our Cloud Billing Integrations doc. Once you have completed all necessary integrations, return to this article.

Due to the frequency of updates from providers, it can take anywhere from 24 to 48 hours to see adjusted costs.

Step 2: Review your data

Now that your base install and CSP integrations are complete, it's time to determine the accuracy against your cloud bill. Based on different methods of cost aggregation, Kubecost should assess your billing data within a 3-5% margin of error.

Monitoring your cost billing

After enabling port-forwarding, you should have access to the Kubecost UI. Explore the different pages in the left navigation, starting with the Monitor dashboards. These pages, including Allocations, Assets, Clusters, and Cloud Costs, are comprised of different categories of cost spending, and allow you to apply customized queries for specific billing data. These queries can then be saved in the form of reports for future quick access. Each page of the Kubecost UI has more dedicated information in the Navigating the Kubecost UI section.

Step 3: Learn about protecting your data and spend

It's important to take precautions to ensure your billing data is preserved, and you know how to monitor your infrastructure's health.

ETL Backup

Metrics reside in Prometheus, but extracting information for either the UI or through API responses directly from this store is not performant at scale. For this reason, the data is optimized and stored in a structure is called extract, transform, load, or ETL. Kubecost's definition of ETL usually will refer to this ETL process.

Like any other system, backup of critical data is a must, and backing up ETL is no exception. To address this, we offer a number of different options based on your product tier. Descriptions and instructions for our backup functionalities can be found in our ETL Backup doc.

Alerts and Health

Similar to most systems, monitoring health is vital. For this, we offer several means of monitoring the health of both Kubecost and the host cluster.

Alerts

Alerts can be configured to enable a proactive approach to monitoring your spend, and can be distributed across different workplace communication tools including email, Slack, and Microsoft Teams. Alerts can establish budgets for your different types of spend and cost-efficiency, and warn you if those budgets are reached. These Alerts are able to be configured via Helm or directly in your Kubecost UI.

Health

The Health page will display an overall cluster health score which assesses how reliably and efficiently your infrastructure is performing. Scores start at 100 and decrease based on how severe any present errors are.

Step 4: Multi-cluster and Federated setups

Kubecost has multiple ways of supporting multi-cluster environments, which vary based on your Kubecost product tier.

Kubecost Free will only allow you to view a single cluster at a time in the Kubecost UI. However, you can connect multiple different clusters and switch through them using Kubecost's context switcher.

Kubecost Enterprise provides a "single-pane-of-glass" view which combines metrics across all clusters into a shared storage bucket. One cluster is designated as the primary cluster from which you view the UI, with all other clusters considered secondary. Attempting to view the UI through a secondary cluster will not display metrics across your entire environment.

It is recommended to complete the steps above for your primary cluster before adding any secondary clusters. To learn more about advanced multi-cluster/Federated configurations, see our Multi-Cluster doc.

Learning more about Kubecost

After completing these primary steps, you are well on your way to being proficient in Kubecost. However, managing Kubernetes infrastructure can be complicated, and for that we have plenty more documentation to help. For advanced or optional configuration options, see our Next Steps with Kubecost guide which will introduce you to additional concepts.

Kubecost requires a Kubernetes cluster to be deployed.

Supported Kubernetes versions

• Users should be running Kubernetes 1.20+.

• Kubernetes 1.28 is officially supported as of v1.105.

• Versions outside of the stated compatibility range may work, depending on individual configurations, but are untested.

Supported cluster types

• Managed Kubernetes clusters (e.g. EKS, GKE, AKS) most common

• Kubernetes distributions (e.g. OpenShift, DigitalOcean, Rancher, Tanzu)

• Bootstrapped Kubernetes cluster

• On-prem and air-gapped using custom pricing sheets

Supported Cloud Providers

• AWS (Amazon Web Services)

  • All regions supported, as shown in opencost/pkg/cloud/awsprovider.go

  • x86, ARM

• GCP (Google Cloud Platform)

  • All regions supported, as shown in opencost/pkg/cloud/gcpprovider.go

  • x86

• Azure (Microsoft)

  • All regions supported, as shown in opencost/pkg/cloud/azureprovider.go

  • x86

This list is certainly not exhaustive! This is simply a list of observations as to where our users run Kubecost based on their questions and feedback. Please contact us with any questions!

Once you have familiarized yourself with Kubecost and integrated with any cloud providers, it's time to move on to more advanced concepts. This doc provides commonly used product configurations and feature overviews to help get you up and running after the Kubecost product has been installed. You may be redirected to other Kubecost docs to learn more about specific concepts or follow tutorials.

Memory and storage

The default Kubecost installation has a 32Gb persistent volume and a 15-day retention period for Prometheus metrics. This is enough space to retain data for roughly 300 pods, depending on your exact node and container count. See the Kubecost Helm chart configuration options to adjust both the retention period and storage size.

To determine the appropriate disk size, you can use this formula to approximate:

needed_disk_space = retention_time_minutes * ingested_samples_per_minutes * bytes_per_sample

Where ingested samples can be measured as the average over a recent period, e.g. sum(avg_over_time(scrape_samples_post_metric_relabeling[24h])). On average, Prometheus uses around 1.5-2 bytes per sample. So, ingesting 100k samples per minute and retaining them for 15 days would demand around 40 GB. It’s recommended to add another 20-30% capacity for headroom and WAL. More info on disk sizing here.

Setting requests and limits

Users should set and/or update resource requests and limits before taking Kubecost into production at scale. These inputs can be configured in the Kubecost values.yaml for Kubecost modules and subcharts.

The exact recommended values for these parameters depend on the size of your cluster, availability requirements, and usage of the Kubecost product. Suggested values for each container can be found within Kubecost itself on the namespace page. More info on these recommendations is available here.

For best results, run Kubecost for up to seven days on a production cluster, then tune resource requests/limits based on resource consumption.

Configure security of user access

To broaden usage to other teams or departments within your Kubecost environment, basic security measures will usually be required. There are a number of options for protecting your workspace depending on your Kubecost product tier.

Ingress controller

Establishing an ingress controller will allow for control of access for your workspace. Learn more about enabling external access in Kubecost with our Ingress Examples doc.

SSO/SAML/RBAC/OIDC

You can configure SSO and RBAC on a separate baseline deployment, which will not only shorten the deployment time of security features, but it will also avoid unwanted access denial. This is helpful when using only one developer deployment. See our user management guides below:

• SAML

• OIDC

Using an existing node exporter

For teams already running node exporter on the default port, our bundled node exporter may remain in a Pending state. You can optionally use an existing node exporter DaemonSet by setting the prometheus.nodeExporter.enabled and prometheus.serviceAccounts.nodeExporter.create Kubecost Helm chart config options to false. This requires your existing node exporter endpoint to be visible from the namespace where Kubecost is installed. More configs options shown here.

Deploying Kubecost without persistent volumes

You may optionally pass the following Helm flags to install Kubecost and its bundled dependencies without any persistent volumes. However, any time the Prometheus server pod is restarted, all historical billing data will be lost unless Thanos or other long-term storage is enabled in the Kubecost product.

--set prometheus.alertmanager.persistentVolume.enabled=false
--set prometheus.pushgateway.persistentVolume.enabled=false
--set prometheus.server.persistentVolume.enabled=false
--set persistentVolume.enabled=false

Resource efficiency and idle costs

Efficiency and idle costs can teach you more about the cost-value of your Kubernetes spend by showing you how efficiently your resources are used. To learn more about pod resource efficiency and cluster idle costs, see Efficiency and Idle.

See also

• Using an existing Prometheus installation

• Using an existing Grafana installation

• Exposing Kubecost with an Ingress

Often while using and configuring Kubecost, our documentation may ask you to pass certain Helm flag values. There are three different approaches for passing custom Helm values into your Kubecost product, which are explained in this doc. In these examples, we are updating the kubecostProductConfigs.productKey.key Helm value which enables Kubecost Enterprise, however these methods will work for all other Helm flags.

Method 1: Pass exact parameters via --set command-line flags

For example, you can only pass a product key if that is all you need to configure.

Method 2: Pass exact parameters via custom values file

Similar to Method 1, you can create a separate values file that contains only the parameters needed.

Your values.yaml should look like this:

Then run your install command:

Enabling external access to the Kubecost product requires exposing access to port 9090 on the kubecost-cost-analyzer pod. Exposing this endpoint will handle routing to Grafana as well. There are multiple ways to do this, including Ingress or Service definitions.

Common samples below and others can be found on our .

The following example definitions use the NGINX .

Basic auth example

Non-root path example

ALB Example

Once an AWS Load Balancer (ALB) Controller is installed, you can use the following Ingress resource manifest pointed at the Kubecost cost-analyzer service:

Integration with cloud service providers (CSPs) via their respective billing APIs allows Kubecost to display out-of-cluster (OOC) costs (e.g. AWS S3, Google Cloud Storage, Azure Storage Account). Additionally, it allows Kubecost to reconcile Kubecost's in-cluster predictions with actual billing data to improve accuracy.

Kubecost's cloud processes

As indicated above, setting up a cloud integration with your CSP allows Kubecost to pull in additional billing data. The two processes that incorporate this information are reconciliation and CloudCost (formerly known as CloudUsage).

Reconciliation

Reconciliation matches in-cluster assets with items found in the billing data pulled from the CSP. This allows Kubecost to display the most accurate depiction of your in-cluster spending. Additionally, the reconciliation process creates Network assets for in-cluster nodes based on the information in the billing data. The main drawback of this process is that the CSPs have between a 6 to 24-hour delay in releasing billing data, and reconciliation requires a complete day of cost data to reconcile with the in-cluster assets. This requires a 48-hour window between resource usage and reconciliation. If reconciliation is performed within this window, asset cost is deflated to the partially complete cost shown in the billing data.

Cost-based metrics are based on on-demand pricing unless there is definitive data from a CSP that the node is not on-demand. This way estimates are as accurate as possible. If a new reserved instance is provisioned or a node joins a savings plan:

1. Kubecost continues to emit on-demand pricing until the node is added to the cloud bill.

2. Once the node is added to the cloud bill, Kubecost starts emitting something closer to the actual price.

3. For the time period where Kubecost assumed the node was on-demand but it was actually reserved, reconciliation fixes the price in ETL.

Visualize unreconciled costs

Visit Settings, then toggle on Highlight Unreconciled Costs, then select Save at the bottom of the page to apply changes. Now, when you visit your Allocations or Assets dashboards, the most recent 36 hours of data will display hatching to signify unreconciled costs.

CloudCost

CloudCost allows Kubecost to pull in OOC cloud spend from your CSP's billing data, including any services run by the CSP as well as compute resources. By labelling OOC costs, their value can be distributed to your Allocations data as external costs. This allows you to better understand the proportion of OOC cloud spend that your in-cluster usage depends on.

Your cloud billing data is reflected in the aggregate costs of Account, Provider, Invoice Entity, and Service. Aggregating and drilling down into any of these categories will provide a subset of the entire bill, based on the Helm value .values.cloudCost.topNItems, which will log 1,000 values. This subset is each days' top n items by cost. An optional label list can be used to include or exclude items to be pulled from the bill.

CloudCost becomes available as soon as they appear in the billing data, with the 6 to 24-hour delay mentioned above, and are updated as they become more complete.

Managing cloud integrations

You can view your existing cloud integrations and their success status in the Kubecost UI by visiting Settings, then scrolling to Cloud Integrations. To create a new integration or learn more about existing integrations, select View additional details to go to the Cloud Integrations page.

Here, you can view your integrations and filter by successful or failed integrations. For non-successful integrations, Kubecost will display a diagnostic error message in the Status column to contextualize steps toward successful integration.

Select an individual integration to view a side panel that contains the most recent run, next run, refresh rate, and an exportable YAML of Helm configs for its CSP's integration values.

Adding a cloud integration

You can add a new cloud integration by selecting Add Integration. For guides on how to set up an integration for a specific CSP, follow these links to helpful Kubecost documentation:

• Multi-Cloud

• AWS

• GCP

• Azure

Deleting a cloud integration

Select an existing cloud integration, then in the slide panel that appears, select Delete.

Cloud integration configurations

The Kubecost Helm chart provides values that can enable or disable each cloud process on the deployment once a cloud integration has been set up. Turning off either of these processes will disable all the benefits provided by them.

Cloud account name aliasing

Often an integrated cloud account name may be a series of random letter and numbers which do not reflect the account's owner, team, or function. Kubecost allows you to rename cloud accounts to create more readable cloud metrics in your Kubecost UI. After you have successfully integrated your cloud account (see above), you need to manually edit your values.yaml and provide the original account name and your intended rename:

kubecostProductConfigs:
  cloudAccountMapping:
    ACCOUNT_ID: "ACCOUNT_NAME"

You will see these changes reflected in Kubecost's UI on the Overview page under Cloud Costs Breakdown. These example account IDs could benefit from being renamed:

Cloud Stores

The ETL contains a Map of Cloud Stores, each representing an integration with a CSP. Each Cloud Store is responsible for the Cloud Usage and reconciliation pipelines which add OOC costs and adjust Kubecost's estimated cost respectively by cost and usage data pulled from the CSP. Each Cloud Store has a unique identifier called the ProviderKey which varies depending on which CSP is being connected to and ensures that duplicate configurations are not introduced into the ETL. The value of the ProviderKey is the following for each CSP at a scope that the billing data is being for:

• AWS: Account Id

• GCP: Project Id

• Azure: Subscription Id

The ProviderKey can be used as an argument for the endpoints for Cloud Usage and Reconciliation repair APIs, to indicate that the specified operation should only be done on a single Cloud Store rather than all of them, which is the default behavior. Additionally, the Cloud Store keeps track of the Status of the Cloud Connection Diagnostics for each of the Cloud Usage and reconciliation. The Cloud Connection Status is meant to be used as a tool in determining the health of the Cloud Connection that is the basis of each Cloud Store. The Cloud Connection Status has various failure states that are meant to provide actionable information on how to get your Cloud Connection running properly. These are the Cloud Connection Statuses:

• INITIAL_STATUS: The zero value of Cloud Connection Status means that the cloud connection is untested. Once Cloud Connection Status has been changed and it should not return to this value. This status is assigned on creation to the Cloud Store

• MISSING_CONFIGURATION: Kubecost has not detected any method of Cloud Configuration. This value is only possible on the first Cloud Store that is created as a wrapper for the open-source CSP. This status is assigned during failures in Configuration Retrieval.

• INCOMPLETE_CONFIGURATION: Cloud Configuration is missing the required values to connect to the cloud provider. This status is assigned during failures in Configuration Retrieval.

• FAILED_CONNECTION: All required Cloud Configuration values are filled in, but a connection with the CSP cannot be established. This is indicative of a typo in one of the Cloud Configuration values or an issue in how the connection was set up in the CSP's Console. The assignment of this status varies between CSPs but should happen if there if an error is thrown when an interaction with an object from the CSP's SDK occurs.

• MISSING_DATA: The Cloud Integration is properly configured, but the CSP is not returning billing/cost and usage data. This status is indicative of the billing/cost and usage data export of the CSP being incorrectly set up or the export being set up in the last 48 hours and not having started populating data yet. This status is set when a query has been successfully made but the results come back empty. If the CSP already has a SUCCESSFUL_CONNECTION status, then this status should not be set because this indicates that the specific query made may have been empty.

• SUCCESSFUL_CONNECTION: The Cloud Integration is properly configured and returning data. This status is set on any successful query where data is returned

After starting or restarting Cloud Usage or reconciliation, two subprocesses are started: one which fills in historic data over the coverage of the Daily CloudUsage and Asset Store, and one which runs periodically on a predefined interval to collect and process new cost and usage data as it is made available by the CSP. The ETL's status endpoint contains a cloud object that provides information about each Cloud Store including the Cloud Connection Status and diagnostic information about Cloud Usage and Reconciliation. The diagnostic items on the Cloud Usage and Reconciliation are:

• Coverage: The window of time that the historical subprocess has covered

• LastRun: The last time that the process ran, updates each time the periodic subprocess runs

• NextRun: Next scheduled run of the periodic subprocess

• Progress: Ratio of Coverage to Total amount of time to be covered

• RefreshRate: The interval that the periodic subprocess runs

• Resolution: The window size of the process

• StartTime: When the Cloud Process was started

For more information on APIs related to rebuilding and repairing Cloud Usage or reconciliation, see the CloudCost Diagnostic APIs doc.

This document outlines how to set up cloud integration for accounts on multiple cloud service providers (CSPs), or multiple accounts on the same cloud provider. This configuration can be used independently of, or in addition, to other cloud integration configurations provided by Kubecost. Once configured, Kubecost will display cloud assets for all configured accounts and perform reconciliation for all that have their respective accounts configured.

Step 1: Set up cloud cost and usage reporting

For each cloud account that you would like to configure, you will need to make sure that it is exporting cost data to its respective service to allow Kubecost to gain access to it.

• Azure: Set up cost data export following this .

• GCP: Set up BigQuery billing data exports with this .

• AWS: Follow steps 1-3 to set up and configure a Cost and Usage Report (CUR) in our .

• Alibaba: Create a user account with access to the .

Step 2: Create cloud integration secret

The secret should contain a file named cloud-integration.json with the following format (only containing applicable CSPs in your setup):

This method of cloud integration supports multiple configurations per cloud provider simply by adding each cost export to their respective arrays in the .json file. The structure and required values for the configuration objects for each cloud provider are described below. Once you have filled in the configuration object, use the command:

Once the secret is created, set .Values.kubecostProductConfigs.cloudIntegrationSecret to <SECRET_NAME> and upgrade Kubecost via Helm.

Azure

The following values can be located in the Azure Portal under Cost Management > Exports, or Storage accounts:

• azureSubscriptionID is the Subscription ID belonging to the Storage account which stores your exported Azure cost report data.

• azureStorageAccount is the name of the Storage account where the exported Azure cost report data is being stored.

• azureStorageAccessKey can be found by selecting Access Keys from the navigation sidebar then selecting Show keys. Using either of the two keys will work.

• azureStorageContainer is the name that you chose for the exported cost report when you set it up. This is the name of the container where the CSV cost reports are saved in your Storage account.

• azureContainerPath is an optional value which should be used if there is more than one billing report that is exported to the configured container. The path provided should have only one billing export because Kubecost will retrieve the most recent billing report for a given month found within the path.

• azureCloud is an optional value which denotes the cloud where the storage account exists. Possible values are public and gov. The default is public.

Set these values into the following object and add them to the Azure array:

GCP

If you don't already have a GCP service key for any of the projects you would like to configure, you can run the following commands in your command line to generate and export one. Make sure your GCP project is where your external costs are being run.

You can then get your service account key to paste into the UI:

• <KEY_JSON> is the GCP service key created above. This value should be left as a JSON when inserted into the configuration object

• <PROJECT_ID> is the Project ID in the GCP service key.

• <BILLING_DATA_DATASET> requires a BigQuery dataset prefix (e.g. billing_data) in addition to the BigQuery table name. A full example is billing_data.gcp_billing_export_v1_018AIF_74KD1D_534A2.

Set these values into the following object and add it to the GCP array:

Many of these values in this config can be generated using the following command:

AWS

Gather each of these values from the AWS console for each account you would like to configure.

• <ACCESS_KEY_ID> is the ID of the Access Key created in the previous step.

• <ACCESS_KEY_SECRET> is the secret of the Access Key created in the

• <ATHENA_BUCKET_NAME> is the S3 bucket storing Athena query results which Kubecost has permission to access. The name of the bucket should match s3://aws-athena-query-results-*, so the IAM roles defined above will automatically allow access to it. The bucket can have a canned ACL set to Private or other permissions as needed.

• <ATHENA_REGION> is the AWS region Athena is running in

• <ATHENA_DATABASE> is the name of the database created by the Athena setup. The Athena database name is available as the value (physical id) of AWSCURDatabase in the CloudFormation stack created above.

• <ATHENA_TABLE> is the name of the table created by the Athena setup The table name is typically the database name with the leading athenacurcfn_ removed (but is not available as a CloudFormation stack resource).

• <ATHENA_WORKGROUP> is the workgroup assigned to be used with Athena. Default value is Primary.

• <ATHENA_PROJECT_ID>is the AWS AccountID where the Athena CUR is. For example: 530337586277.

• <MASTER_PAYER_ARN> is an optional value which should be set if you are using a multi-account billing set-up and are not accessing Athena through the primary account. It should be set to the ARN of the role in the management (formerly master payer) account, for example: arn:aws:iam::530337586275:role/KubecostRole.

Set these values into the following object and add them to the AWS array in the cloud-integration.json:

Additionally set the kubecostProductConfigs.athenaProjectID Helm value to the AWS account that Kubecost is being installed in.

Alibaba

Kubecost does not support complete integrations with Alibaba, but you will still be able to view accurate list prices for cloud resources. Gather these following values from the Alibaba Cloud Console for your account:

• clusterRegion is the most used region

• accountID is your Alibaba account ID

• serviceKeyName is the RAM user key name

• serviceKeySecret is the RAM user secret

Set these values into the following object and add them to the Alibaba array in your cloud-integration.json:

There are many ways to integrate your AWS Cost and Usage Report (CUR) with Kubecost. This tutorial is intended as the best-practice method for users whose environments meet the following assumptions:

1. Kubecost will run in a different account than the AWS Payer Account

2. The IAM permissions will utilize AWS to avoid shared secrets

3. The configuration of Kubecost will be done using a cloud-integration.json file, and not via Kubecost UI (following infrastructure as code practices)

If this is not an accurate description of your environment, see our doc for more options.

Overview of Kubecost CUR integration

This guide is a one-time setup per AWS payer account and is typically one per organization. It can be automated, but may not be worth the effort given that it will not be needed again.

Kubecost supports multiple AWS payer accounts as well as multiple cloud providers from a single Kubecost primary cluster. For multiple payer accounts, create additional entries inside the array below.

Detail for multiple cloud provider setups is .

Configuration

Step 1: Download configuration files

• cloud-integration.json

• iam-payer-account-cur-athena-glue-s3-access.json

• iam-payer-account-trust-primary-account.json

• iam-access-cur-in-payer-account.json

Begin by opening cloud_integration.json, which should look like this:

Update athenaWorkgroup to primary, then save the file and close it. The remaining values will be obtained during this tutorial.

Step 2: Create a CUR Export (and wait 24 hours)

• For time granularity, select Daily.

• Select the checkbox to enable Resource IDs in the report.

• Select the checkbox to enable Athena integration with the report.

• Select the checkbox to enable the JSON IAM policy to be applied to your bucket.

AWS may take up to 24 hours to publish data. Wait until this is complete before continuing to the next step.

While you wait, update the following configuration files:

• Update your cloud-integration.json file by providing a projectID value, which will be the AWS payer account number where the CUR is located and where the Kubecost primary cluster is running.

• Update your iam-payer-account-cur-athena-glue-s2-access.json file by replacing all instances of CUR_BUCKET_NAME to the name of the bucket you created for CUR data.

Step 3: Setting up Athena

Once Athena is set up with the CUR, you will need to create a new S3 bucket for Athena query results. The bucket used for the CUR cannot be used for the Athena output.

2. Select Create bucket. The Create Bucket page opens.

3. Provide a name for your bucket. This is the value for athenaBucketName in your cloud-integration.json file. Use the same region used for the CUR bucket.

4. Select Create bucket at the bottom of the page.

6. Select Settings, then select Manage. The Manage settings window opens.

7. Set Location of query result to the S3 bucket you just created, then select Save.

Navigate to Athena in the AWS Console. Be sure the region matches the one used in the steps above. Update your cloud-integration.json file with the following values. Use the screenshots below for help.

• athenaBucketName: the name of the Athena bucket your created in this step

• athenaDatabase: the value in the Database dropdown

• athenaRegion: the AWS region value where your Athena query is configured

• athenaTable: the partitioned value found in the Table list

Step 4: Setting up payer account IAM permissions

From the AWS payer account

In iam-payer-account-cur-athena-glue-s3-access.json, replace all ATHENA_RESULTS_BUCKET_NAME instances with your Athena S3 bucket name (the default will look like aws-athena-query-results-xxxx).

In iam-payer-account-trust-primary-account.json, replace SUB_ACCOUNT_222222222 with the account number of the account where the Kubecost primary cluster will run.

In the same location as your downloaded configuration files, run the following command to create the appropriate policy (jq is not required):

Now we can obtain the last value masterPayerARN for cloud-integration.json as the ARN associated with the newly-created IAM role, as seen below in the AWS console:

Step 5: Setting up IAM permissions for the primary cluster

From the AWS Account where the Kubecost primary cluster will run

In iam-access-cur-in-payer-account.json, update PAYER_ACCOUNT_11111111111 with the AWS account number of the payer account and create a policy allowing Kubecost to assumeRole in the payer account:

Note the output ARN (used in the iamserviceaccount --attach-policy-arn below):

Create a namespace and set environment variables:

Enable the OIDC-Provider:

Create the Kubernetes service account, attaching the assumeRole policy. Replace SUB_ACCOUNT_222222222 with the AWS account number where the primary Kubecost cluster will run.

Create the secret (in this setup, there are no actual secrets in this file):

Install Kubecost using the service account and cloud-integration secret:

Validation

It can take over an hour to process the billing data for large AWS accounts. In the short-term, follow the logs and look for a message similar to (7.7 complete), which should grow gradually to (100.0 complete). Some errors (ERR) are expected, as seen below.

Troubleshooting

Integrating Kubecost with your AWS data provides the ability to allocate out-of-cluster (OOC) costs, e.g. RDS instances and S3 buckets, back to Kubernetes concepts like namespace and deployment as well as reconcile cluster assets back to your billing data. The latter is especially helpful when teams are using Reserved Instances, Savings Plans, or Enterprise Discounts. All billing data remains on your cluster when using this functionality and is not shared externally. Read our doc for more information on how Kubecost connects with Cloud Service Providers.

The following guide provides the steps required for enabling OOC costs allocation and accurate pricing, e.g. reserved instance price allocation. In a multi-account organization, all of the following steps will need to be completed in the payer account.

Step 1: Create an AWS Cost and Usage Report (CUR) and integrate it with Kubecost

You can learn how to perform this using our doc.

Step 2: Tag your resources

Kubecost utilizes AWS tagging to allocate the costs of AWS resources outside of the Kubernetes cluster to specific Kubernetes concepts, such as namespaces, pods, etc. These costs are then shown in a unified dashboard within the Kubecost interface.

To allocate external AWS resources to a Kubernetes concept, use the following tag naming scheme:

To use an alternative or existing AWS tag schema, you may supply these in your under kubecostProductConfigs.labelMappingConfigs.\<aggregation\>\_external_label. Also be sure to set kubecostProductConfigs.labelMappingConfigs.enabled=true.

Tags may take several hours to show up in the Cost Allocations Tags section described in the next step.

Custom tags mapping

Tags that contain : in the key may be converted to _ in the Kubecost UI due to Prometheus readability. To use AWS Label Mapping Configs, use this mapping format:

Step 3: Enable user-defined cost allocation tags

In order to make the custom Kubecost AWS tags appear on the CURs, and therefore in Kubecost, individual cost allocation tags must be enabled. Details on which tags to enable can be found in Step 2.

Viewing account-level tags

Account-level tags are applied (as labels) to all the Assets built from resources defined under a given AWS account. You can filter AWS resources in the Kubecost Assets View (or API) by account-level tags by adding them ('tag:value') in the Label/Tag filter.

If a resource has a label with the same name as an account-level tag, the resource label value will take precedence.

Modifications incurred on account-level tags may take several hours to update on Kubecost.

Your AWS account will need to support the organizations:ListAccounts and organizations:ListTagsForResource policies to benefit from this feature.

Troubleshooting

• In the Kubecost UI, view the Allocations dashboard. If external costs are not shown, open your browser's Developer Tools > Console to see any reported errors.

• Query Athena directly to ensure data is available. Note: it can take up to 6 hours for data to be written.

• Finally, review pod logs from the cost-model container in the cost-analyzer pod and look for auth errors or Athena query results.

Considerations before configuring Spot pricing

Kubecost uses public pricing from Cloud Service Providers (CSPs) to calculate costs until the actual cloud bill is available, at which point Kubecost will reconcile your Spot prices from your Cost and Usage Report (CUR). This is almost always ready in 48 hours. Most users will likely prefer to configure instead of configuring the Spot data feed manually as demonstrated in this article.

However, if the majority of costs are due to Spot nodes, it may be useful to configure the Spot pricing data feed as it will increase accuracy for short-term (<48 hour) node costs until the Spot prices from the CUR are available. Note that all other (non-Spot) costs will still be based on public (on-demand) pricing until CUR billing data is reconciled.

Configuring the Spot data feed in Kubecost

With Kubecost, Spot pricing data can be pulled hourly by integrating directly with the AWS Spot feed.

First, to enable the AWS Spot data feed, follow AWS' doc.

While configuring, note the settings used as these values will be needed for the Kubecost configuration.

There are multiple options: this can either be set from the Kubecost UI or via .Values.kubecostProductConfigs in the Helm chart. If you set any kubecostProductConfigs from the Helm chart, all changes via the front end will be deleted on pod restart.

• projectID the Account ID of the AWS Account on which the Spot nodes are running.

• awsSpotDataRegion region of your Spot data bucket

• awsSpotDataBucket the configured bucket for the Spot data feed

• awsSpotDataPrefix optional configured prefix for your Spot data feed bucket

• spotLabel optional Kubernetes node label name designating whether a node is a Spot node. Used to provide pricing estimates until exact Spot data becomes available from the CUR

• spotLabelValue optional Kubernetes node label value designating a Spot node. Used to provide pricing estimates until exact Spot data becomes available from the CUR. For example, if your Spot nodes carry a label lifecycle:spot, then the spotLabel would be lifecycle and the spotLabelValue would be spot

In the UI, you can access these fields via the Settings page, then scroll to Cloud Cost Settings. Next to Spot Instance Configuration, select Update, then fill out all fields.

Spot data feeds are an account level setting, not a payer level. Every AWS Account will have its own Spot data feed. Spot data feed is not currently available in AWS GovCloud.

Configuring IAM

Kubecost requires read access to the Spot data feed bucket. The following IAM policy can be used to grant Kubecost read access to the Spot data feed bucket.

To attach the IAM policy to the Kubecost service account, you can use IRSA or the account's service key.

Option 1: IRSA (IAM Roles for Service Accounts)

Option 2: Service Keys

Create a service-key.json as shown:

Create a K8s secret:

Set the following Helm config:

Troubleshooting Spot data feed

No Spot instances detected

Verify the below points:

• Make sure data is present in the Spot data feed bucket.

• Make sure Project ID is configured correctly. You can cross-verify the values under Helm values in bug report

• Check the value of kubecost_node_is_spot in Prometheus:

  • "1" means Spot data instance configuration is correct.

  • "0" means not configured properly.

• Is there a prefix? If so, is it configured in Kubecost?

• Make sure the IAM permissions are aligned with https://github.com/kubecost/cloudformation/blob/7feace26637aa2ece1481fda394927ef8e1e3cad/kubecost-single-account-permissions.yaml#L36

• Make sure the Spot data feed bucket has all permissions to access by Kubecost

• The Spot Instance in the Spot data feed bucket should match the instance in the cluster where the Spot data feed is configured. awsSpotDataBucket has to be present in the right cluster.

Kubecost needs access to the Microsoft Azure Billing Rate Card API to access accurate pricing data for your Kubernetes resources.

Creating a custom Azure role

Start by creating an Azure role definition. Below is an example definition, replace YOUR_SUBSCRIPTION_ID with the Subscription ID where your Kubernetes cluster lives:

{
    "Name": "KubecostRole",
    "IsCustom": true,
    "Description": "Rate Card query role",
    "Actions": [
        "Microsoft.Compute/virtualMachines/vmSizes/read",
        "Microsoft.Resources/subscriptions/locations/read",
        "Microsoft.Resources/providers/read",
        "Microsoft.ContainerService/containerServices/read",
        "Microsoft.Commerce/RateCard/read"
    ],
    "AssignableScopes": [
        "/subscriptions/YOUR_SUBSCRIPTION_ID"
    ]
}

Save this into a file called myrole.json.

Next, you'll want to register that role with Azure:

az role definition create --verbose --role-definition @myrole.json

Creating an Azure service principal

Next, create an Azure service principal.

az ad sp create-for-rbac --name "KubecostAccess" --role "KubecostRole" --scope "/subscriptions/YOUR_SUBSCRIPTION_ID" --output json

Keep this information which is used in the service-key.json below.

Supplying Azure service principal details to Kubecost

Option 1: Via a Kubernetes Secret (Recommended)

Create a file called service-key.json and update it with the Service Principal details from the above steps:

{
    "subscriptionId": "<Azure Subscription ID>",
    "serviceKey": {
        "appId": "<Entra ID App ID>",
        "displayName": "KubecostAccess",
        "password": "<Entra ID Client Secret>",
        "tenant": "<Entra Tenant ID>"
    }
}

Next, create a Secret for the Azure Service Principal

kubectl create secret generic azure-service-key -n kubecost --from-file=service-key.json

Finally, set the kubecostProductConfigs.serviceKeySecretName Helm value to the name of the Kubernetes secret you created. We use the value azure-service-key in our examples.

Option 2: Via Helm values

In the Helm values file:

kubecostProductConfigs:
  azureSubscriptionID: <Azure Subscription ID>
  azureClientID: <Entra ID App ID>
  azureTenantID: <Entra Tenant ID>
  azureClientPassword: <Entra ID Client Secret>
  azureOfferDurableID: MS-AZR-0003P
  azureBillingRegion: US
  currencyCode: USD
  createServiceKeySecret: true

Or at the command line:

helm upgrade --install kubecost kubecost/cost-analyzer -n kubecost \
  --set kubecostProductConfigs.azureSubscriptionID=<Azure Subscription ID> \
  --set kubecostProductConfigs.azureClientID=<Entra ID App ID> \
  --set kubecostProductConfigs.azureTenantID=<Entra Tenant ID> \
  --set kubecostProductConfigs.azureClientPassword=<Entra ID Client Secret> \
  --set kubecostProductConfigs.azureOfferDurableID=MS-AZR-0003P \
  --set kubecostProductConfigs.azureBillingRegion=US
  --set kubecostProductConfigs.currencyCode=USD
  --set kubecostProductConfigs.createServiceKeySecret=true

Azure billing region, offer durable ID, and currency

Kubecost supports querying the Azure APIs for cost data based on the region, offer durable ID, and currency defined in your Microsoft Azure offer.

Those properties are configured with the following Helm values:

• kubecostProductConfigs.azureBillingRegion

• kubecostProductConfigs.azureOfferDurableID

• kubecostProductConfigs.currencyCode

Be sure to verify your billing information with Microsoft and update the above Helm values to reflect your bill to country, subscription offer durable ID/number, and currency.

See also

The following Microsoft documents are a helpful reference:

• Microsoft Azure Offer Details

• Azure Pricing FAQ

• Geographic availability and currency support for the commercial marketplace

• Azure Portal > Cost Management + Billing > Billing Account Properties

• Understand Cost Management data

Kubecost provides the ability to allocate out-of-cluster (OOC) costs, e.g. Cloud SQL instances and Cloud Storage buckets, back to Kubernetes concepts like namespaces and deployments.

Read the Cloud Billing Integrations doc for more information on how Kubecost connects with cloud service providers.

The following guide provides the steps required for allocating OOC costs in GCP.

Step 1: Enable billing data export

Begin by reviewing Google's documentation on exporting cloud billing data to BigQuery.

GCP users must create a detailed billing export to gain access to all Kubecost CloudCost features including reconciliation. Exports of type "Standard usage cost data" and "Pricing Data" do not have the correct information to support CloudCosts.

Step 2: Create a GCP service account

If your Big Query dataset is in a different project than the one where Kubecost is installed, please see the section on Cross-Project Service Accounts.

Add a service account key to allocate OOC resources (e.g. storage buckets and managed databases) back to their Kubernetes owners. The service account needs the following:

roles/bigquery.user
roles/compute.viewer
roles/bigquery.dataViewer
roles/bigquery.jobUser

If you don't already have a GCP service account with the appropriate rights, you can run the following commands in your command line to generate and export one. Make sure your GCP project is where your external costs are being run.

export PROJECT_ID=$(gcloud config get-value project)
gcloud iam service-accounts create compute-viewer-kubecost --display-name "Compute Read Only Account Created For Kubecost" --format json
gcloud projects add-iam-policy-binding $PROJECT_ID --member serviceAccount:compute-viewer-kubecost@$PROJECT_ID.iam.gserviceaccount.com --role roles/compute.viewer
gcloud projects add-iam-policy-binding $PROJECT_ID --member serviceAccount:compute-viewer-kubecost@$PROJECT_ID.iam.gserviceaccount.com --role roles/bigquery.user
gcloud projects add-iam-policy-binding $PROJECT_ID --member serviceAccount:compute-viewer-kubecost@$PROJECT_ID.iam.gserviceaccount.com --role roles/bigquery.dataViewer
gcloud projects add-iam-policy-binding $PROJECT_ID --member serviceAccount:compute-viewer-kubecost@$PROJECT_ID.iam.gserviceaccount.com --role roles/bigquery.jobUser

Step 3: Connecting GCP service account to Kubecost

After creating the GCP service account, you can connect it to Kubecost in one of two ways before configuring:

Option 3.1: Connect using Workload Identity Federation (recommended)

You can set up an IAM policy binding to bind a Kubernetes service account to your GCP service account as seen below, where:

• NAMESPACE is the namespace Kubecost is installed into

• KSA_NAME is the name of the service account attributed to the Kubecost deployment

gcloud iam service-accounts add-iam-policy-binding compute-viewer-kubecost@$PROJECT_ID.iam.gserviceaccount.com --role roles/iam.workloadIdentityUser --member "serviceAccount:$PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME]"

You will also need to enable the IAM Service Account Credentials API in the GCP project.

Option 3.2: Connect using a service account key

Create a service account key:

gcloud iam service-accounts keys create ./compute-viewer-kubecost-key.json --iam-account compute-viewer-kubecost@$PROJECT_ID.iam.gserviceaccount.com

Once the GCP service account has been connected, set up the remaining configuration parameters.

Step 4. Configuring GCP for Kubecost

You're almost done. Now it's time to configure Kubecost to finalize your connectivity.

Option 4.1: Configuring using values.yaml (recommended)

It is recommended to provide the GCP details in your values.yaml to ensure they are retained during an upgrade or redeploy. First, set the following configs:

kubecostProductConfigs:
  projectID: "$PROJECT_ID"
  bigQueryBillingDataDataset: "YOUR_DATASET.YOUR_TABLE_NAME"

If you've connected using Workload Identity Federation, add these configs:

# Ensure Kubecost deployment runs on nodes that use Workload Identity
nodeSelector:
  iam.gke.io/gke-metadata-server-enabled: "true"
# Add annotations to all kubecost-related serviceaccounts
serviceAccount:
  annotations:
    iam.gke.io/gcp-service-account: "compute-viewer-kubecost@$PROJECT_ID.iam.gserviceaccount.com"

Otherwise, if you've connected using a service account key, create a secret for the GCP service account key you've created and add the following configs:

kubectl create secret generic gcp-secret -n kubecost --from-file=./compute-viewer-kubecost-key.json

kubecostProductConfigs:
  gcpSecretName: "gcp-secret"

Option 4.2: Configuring via the Kubecost UI

In Kubecost, select Settings from the left navigation, and under Cloud Integrations, select Add Cloud Integration > GCP, then provide the relevant information in the GCP Billing Data Export Configuration window:

• GCP Service Key: Optional field. If you've created a service account key, copy the contents of the compute-viewer-kubecost-key.json file and paste them here. If you've connected using Workload Identity federation in Step 3, you should leave this box empty.

• GCP Project Id: The ID of your GCP project.

• GCP Billing Database: Requires a BigQuery dataset prefix (e.g. billing_data) in addition to the BigQuery table name. A full example is billing_data.gcp_billing_export_resource_v1_XXXXXX_XXXXXX_XXXXX

Step 5: Label cloud assets

You can now label assets with the following schema to allocate costs back to their appropriate Kubernetes owner. Learn more here on updating GCP asset labels.

Cluster:    "kubernetes_cluster" :   <clusterID>
Namespace:  "kubernetes_namespace" : <namespace>
Deployment: "kubernetes_deployment": <deployment>
Label:      "kubernetes_label_NAME": <label>
Pod:        "kubernetes_pod":        <pod>
Daemonset:  "kubernetes_daemonset":  <daemonset>
Container:  "kubernetes_container":  <container>

To use an alternative or existing label schema for GCP cloud assets, you may supply these in your values.yaml under the kubecostProductConfigs.labelMappingConfigs.<aggregation>_external_label.

Viewing project-level labels

Project-level labels are applied to all the Assets built from resources defined under a given GCP project. You can filter GCP resources in the Kubecost Cloud Costs Explorer (or API).

If a resource has a label with the same name as a project-level label, the resource label value will take precedence.

Modifications incurred on project-level labels may take several hours to update on Kubecost.

Cross-project service account configuration

Due to organizational constraints, it is common that Kubecost must be run in a separate project from the project containing the billing data Big Query dataset, which is needed for Cloud Integration. Configuring Kubecost in this scenario is still possible, but some of the values in the above script will need to be changed. First, you will need the project id of the projects where Kubecost is installed, and the Big Query dataset is located. Additionally, you will need a GCP user with the permissions iam.serviceAccounts.setIamPolicy for the Kubecost project and the ability to manage the roles listed above for the Big Query Project. With these, fill in the following script to set the relevant variables:

export KUBECOST_PROJECT_ID=<Project ID where kubecost is installed>
export BIG_QUERY_PROJECT_ID=<Project ID where bigquery data is stored>
export SERVICE_ACCOUNT_NAME=<Unique name for your service account>

Once these values have been set, this script can be run and will create the service account needed for this configuration.

gcloud config set project $KUBECOST_PROJECT_ID
gcloud iam service-accounts create $SERVICE_ACCOUNT_NAME --display-name "Cross Project CUR" --format json
gcloud projects add-iam-policy-binding $BIG_QUERY_PROJECT_ID --member serviceAccount:$SERVICE_ACCOUNT_NAME@$KUBECOST_PROJECT_ID.iam.gserviceaccount.com --role roles/compute.viewer
gcloud projects add-iam-policy-binding $BIG_QUERY_PROJECT_ID --member serviceAccount:$SERVICE_ACCOUNT_NAME@$KUBECOST_PROJECT_ID.iam.gserviceaccount.com --role roles/bigquery.user
gcloud projects add-iam-policy-binding $BIG_QUERY_PROJECT_ID --member serviceAccount:$SERVICE_ACCOUNT_NAME@$KUBECOST_PROJECT_ID.iam.gserviceaccount.com --role roles/bigquery.dataViewer
gcloud projects add-iam-policy-binding $BIG_QUERY_PROJECT_ID --member serviceAccount:$SERVICE_ACCOUNT_NAME@$KUBECOST_PROJECT_ID.iam.gserviceaccount.com --role roles/bigquery.jobUser

Now that your service account is created follow the normal configuration instructions.

Troubleshooting

Account labels not showing up in partitions

There are cases where labels applied at the account label do not show up in the date-partitioned data. If account level labels are not showing up, you can switch to querying them unpartitioned by setting an extraEnv in Kubecost: GCP_ACCOUNT_LABELS_NOT_PARTITIONED: true. See here.

InvalidQuery 400 error for GCP integration

In cases where Kubecost does not detect a connection following GCP integration, revisit Step 1 and ensure you have enabled detailed usage cost, not standard usage cost. Kubecost uses detailed billing cost to display your OOC spend, and if it was not configured correctly during installation, you may receive errors about your integration.

Kubecost is capable of aggregating the costs of EC2 compute resources over a given timeframe with a specified duration step size. To achieve this, Kubecost uses Athena queries to gather usage data points with differing price models. The result of this process is a list of resources with their cost by timeframe.

Athena queries

The reconciliation process makes two queries to Athena, one to gather resources that are paid for with either the on-demand model or a savings plan and one query for resources on the reservation price model. The first query includes resources given at a blended rate, which could be on-demand usage or resources that have exceeded the limits of a savings plan. It will also include resources that are part of a savings plan which will have a savings plan effective cost. The second query only includes reserved resources and the cost which reflects the rate they were reserved at.

The queries make use of the following columns from Athena:

• line_item_usage_start_date The beginning timestamp of the line item usage. Used to filter resource usage within a date range and to aggregate on usage window.

• line_item_usage_end_date The ending timestamp of the line item usage. Used to filter resource usage within a date range and to aggregate on usage window.

• line_item_resource_id An ID, also called the provider ID, is given to line items that are instantiated resources.

• line_item_line_item_type The type of a line item, used to determine if the resource usage is covered by a savings plan and has a discounted price.

• line_item_usage_type What is being used in a line item, for the purposes of a compute resource this, is the type of VM and where it is running

• line_item_product_code The service that a line item is from. Used to filter out items that are not from EC2.

• reservation_reservation_a_r_n Amazon Resource Name for reservation of line item, the presence of this value is used to identify a resource as being part of a reservation plan.

• line_item_unblended_cost The undiscounted cost of a resource.

• savings_plan_savings_plan_effective_cost The cost of a resource discounted by a savings plan

• reservation_effective_cost The cost of a resource discounted by a reservation

On-Demand/Savings plan query

This query is grouped by six columns:

1. line_item_usage_start_date

2. line_item_usage_end_date

3. line_item_resource_id

4. line_item_line_item_type

5. line_item_usage_type

6. line_item_product_code

The columns line_item_unblended_cost and savings_plan_savings_plan_effective_cost are summed on this grouping. Finally, the query filters out rows that are not within a given date range, have a missing line_item_resource_id, and have a line_item_product_code not equal to "AmazonEC2". The grouping has three important aspects, the timeframe of the line items, the resource as defined by the resource id, and the usage type, which is later used to determine the proper cost of the resources as it was used. This means that line items are grouped according to the resource, the time frame of the usage, and the rate at which the usage was charged.

Reservation query

The reservation query is grouped on five columns:

1. line_item_usage_start_date

2. line_item_usage_end_date

3. reservation_reservation_a_r_n

4. line_item_resource_id

5. line_item_product_code

The query is summed on the reservation_effective_cost and filtered by the date window, for missing reservation_reservation_a_r_n values and also removes line items with line_item_product_code not equal to "AmazonEC2". This grouping is on resource id by timeframe removing all non-reservation line items.

Processing query results

The on-demand query is categorized into different resource types: compute, network, storage, and others. The network is identified by the presence of the "byte" in the line_item_usage_type. Compute and storage are identified by the presence of "i-" and "vol-" prefixes in line_item_resource_id respectively. Non compute values are removed from the results. Out of the two costs aggregated by this query the correct one to use is determined by the line_item_line_item_type, if it has a value of "SavingsPlanCoveredUsage", then the savings_plan_savings_plan_effective_cost is used as the cost, and if not then the line_item_unblended_cost is used.

In the reservation query, all of the results are of the compute category and there is only the reservation_effective_cost to use as a cost.

These results are then merged into one set, with the provider id used to associate the cost with other information about the resource.

Why doesn't this match the AWS Cost Explorer?

There are several different ways to look at your node cost data. The default for the cost explorer is Unblended" but it makes the most sense from an allocation perspective to use the amortized rates. Be sure Amortized costs is selected when looking at cost data. Here's an example of how they can vary dramatically on our test cluster.

The t2-mediums here are covered by a savings plan. Unblended, the cost is only $0.06/day for two.

When Amortized costs is selected, the price jumps to $1.50/day

This should closely match our data on the Assets page, for days where we have adjustments come in from the pricing CUR.

Connecting your Azure account to Kubecost allows you to view Kubernetes metrics side-by-side with out-of-cluster (OOC) costs (e.g. Azure Database Services). Additionally, it allows Kubecost to reconcile measured Kubernetes spend with your actual Azure bill. This gives teams running Kubernetes a complete and accurate picture of costs. For more information, read Cloud Billing Integrations and this blog post.

To configure Kubecost's Azure Cloud Integration, you will need to set up daily exports of cost reports to Azure storage. Kubecost will then access your cost reports through the Azure Storage API to display your OOC cost data alongside your in-cluster costs.

A GitHub repository with sample files used in below instructions can be found here.

Step 1: Export Azure cost report

Follow Azure's Create and Manage Exported Data tutorial to export cost reports. For Metric, make sure you select Amortized cost (Usage and Purchases). For Export type, make sure you select Daily export of month-to-date costs. Do not select File Partitioning. Also, take note of the Account name and Container specified when choosing where to export the data to. Note that a successful cost export will require Microsoft.CostManagementExports to be registered in your subscription.

Alternatively, you can follow this Kubecost guide.

It will take a few hours to generate the first report, after which Kubecost can use the Azure Storage API to pull that data.

Once the cost export has successfully executed, verify that a non-empty CSV file has been created at this path: <STORAGE_ACCOUNT>/<CONTAINER_NAME>/<OPTIONAL_CONTAINER_PATH>/<COST_EXPORT_NAME>/<DATE_RANGE>/<CSV_FILE>.

Step 2: Provide access to Azure Storage API

Obtain the following values from Azure to provide to Kubecost. These values can be located in the Azure Portal by selecting Storage Accounts, then selecting your specific Storage account for details.

• azureSubscriptionID is the "Subscription ID" belonging to the Storage account which stores your exported Azure cost report data.

• azureStorageAccount is the name of the Storage account where the exported Azure cost report data is being stored.

• azureStorageAccessKey can be found by selecting Access keys in your Storage account left navigation under "Security + networking". Using either of the two keys will work.

• azureStorageContainer is the name that you chose for the exported cost report when you set it up. This is the name of the container where the CSV cost reports are saved in your Storage account.

• azureContainerPath is an optional value which should be used if there is more than one billing report that is exported to the configured container. The path provided should have only one billing export because Kubecost will retrieve the most recent billing report for a given month found within the path.

• azureCloud is an optional value which denotes the cloud where the storage account exist, possible values are public and gov. The default is public.

Next, create a JSON file which must be named cloud-integration.json with the following format:

{
    "azure": [
        {
            "azureSubscriptionID": "AZ_cloud_integration_subscriptionId",
            "azureStorageAccount": "AZ_cloud_integration_azureStorageAccount",
            "azureStorageAccessKey": "AZ_cloud_integration_azureStorageAccessKey",
            "azureStorageContainer": "AZ_cloud_integration_azureStorageContainer",
            "azureContainerPath": "",
            "azureCloud": "public/gov"
        }
    ]
}

Next, create the Secret:

$ kubectl create secret generic <SECRET_NAME> --from-file=cloud-integration.json -n kubecost

Next, ensure the following are set in your Helm values:

kubecostProductConfigs:
  cloudIntegrationSecret: <SECRET_NAME>

Next, upgrade Kubecost via Helm:

$ helm upgrade kubecost kubecost/cost-analyzer -n kubecost -f values.yaml

You can verify a successful configuration by checking the following in the Kubecost UI:

• The Assets dashboard will be broken down by Kubernetes assets.

• The Assets dashboard will no longer show a banner that says "External cloud cost not configured".

• The Diagnostics page (via Settings > View Full Diagnostics) view will show a green checkmark under Cloud Integrations.

Step 3: Tagging Azure resources

Kubecost utilizes Azure tagging to allocate the costs of Azure resources outside of the Kubernetes cluster to specific Kubernetes concepts, such as namespaces, pods, etc. These costs are then shown in a unified dashboard within the Kubecost interface.

To allocate external Azure resources to a Kubernetes concept, use the following tag naming scheme:

In the kubernetes_label_NAME tag key, the NAME portion should appear exactly as the tag appears inside of Kubernetes. For example, for the tag app.kubernetes.io/name, this tag key would appear as kubernetes_label_app.kubernetes.io/name.

To use an alternative or existing Azure tag schema, you may supply these in your values.yaml under the kubecostProductConfigs.labelMappingConfigs.<aggregation>_external_label . Also be sure to set kubecostProductConfigs.labelMappingConfigs.enabled = true

For more details on what Azure resources support tagging, along with what resource type tags are available in cost reports, please review the official Microsoft documentation here.

Troubleshooting and debugging

To troubleshoot a configuration that is not yet working:

• $ kubectl get secrets -n kubecost to verify you've properly configured cloud-integration.json.

• $ helm get values kubecost to verify you've properly set .Values.kubecostProductConfigs.cloudIntegrationSecret

• Verify that a non-empty CSV file has been created at this path in your Azure Portal Storage Account: <STORAGE_ACCOUNT>/<CONTAINER_NAME>/<OPTIONAL_CONTAINER_PATH>/<COST_EXPORT_NAME>/<DATE_RANGE>/<CSV_FILE>. Ensure new CSVs are being generated every day.

• When opening a cost report CSV, ensure that there are rows in the file that do not have a MeterCategory of “Virtual Machines” or “Storage” as these items are ignored because they are in cluster costs. Additionally, make sure that there are items with a UsageDateTime that matches the date you are interested in.

When reviewing logs:

• The following error is reflective of Kubecost's previous Azure Cloud Integration method and can be safely disregarded.

  ERR Error, Failed to locate azure storage config file: /var/azure-storage-config/azure-storage-config.json

By default, Kubecost pulls on-demand asset prices from the public AWS pricing API. For more accurate pricing, this integration will allow Kubecost to reconcile your current measured Kubernetes spend with your actual AWS bill. This integration also properly accounts for Enterprise Discount Programs, Reserved Instance usage, Savings Plans, Spot usage, and more.

You will need permissions to create the Cost and Usage Report (CUR), and add IAM credentials for Athena and S3. Optional permission is the ability to add and execute CloudFormation templates. Kubecost does not require root access in the AWS account.

Quick Start for IRSA

This guide contains multiple possible methods for connecting Kubecost to AWS billing, based on user environment and preference. Because of this, there may not be a straightforward approach for new users. To address this, a streamlined guide containing best practices can be found here for IRSA environments. This quick start guide has some assumptions to carefully consider, and may not be applicable for all users. See prerequisites in the linked article.

Key AWS terminology

Integrating your AWS account with Kubecost may be a complicated process if you aren’t deeply familiar with the AWS platform and how it interacts with Kubecost. This section provides an overview of some of the key terminology and AWS services that are involved in the process of integration.

Cost and Usage Report: AWS report which tracks cloud spending and writes to an Amazon Simple Storage Service (Amazon S3) bucket for ingestion and long term historical data. The CUR is originally formatted as a CSV, but when integrated with Athena, is converted to Parquet format.

Amazon Athena: Analytics service which queries the CUR S3 bucket for your AWS cloud spending, then outputs data to a separate S3 bucket. Kubecost uses Athena to query for the bill data to perform reconciliation. Athena is technically optional for AWS cloud integration, but as a result, Kubecost will only provide unreconciled costs (on-demand public rates).

S3 bucket: Cloud object storage tool which both CURs and Athena output cost data to. Kubecost needs access to these buckets in order to read that data.

Cost and Usage Report integration

For the below guide, a GitHub repository with sample files can be found here.

Step 1: Setting up a CUR

Follow these steps to set up a Legacy CUR using the settings below.

• Select the Legacy CUR export type.

• For time granularity, select Daily.

• Under 'Additional content', select the Enable resource IDs checkbox.

• Under 'Report data integration' select the Amazon Athena checkbox.

Remember the name of the bucket you create for CUR data. This will be used in Step 2.

AWS may take up to 24 hours to publish data. Wait until this is complete before continuing to the next step.

Step 2: Setting up Athena

As part of the CUR creation process, Amazon also creates a CloudFormation template that is used to create the Athena integration. It is created in the CUR S3 bucket, listed in the Objects tab in the path s3-path-prefix/cur-name and typically has the filename crawler-cfn.yml. This .yml is your necessary CloudFormation template. You will need it in order to complete the CUR Athena integration. For more information, see the AWS doc Setting up Athena using AWS CloudFormation templates.

Once Athena is set up with the CUR, you will need to create a new S3 bucket for Athena query results.

1. Navigate to the S3 Management Console.

2. Select Create bucket. The Create Bucket page opens.

3. Use the same region used for the CUR bucket and pick a name that follows the format aws-athena-query-results-.

4. Select Create bucket at the bottom of the page.

5. Navigate to the Amazon Athena dashboard.

6. Select Settings, then select Manage. The Manage settings window opens.

7. Set Location of query result to the S3 bucket you just created, which will look like s3://aws-athena-query-results..., then select Save.

Step 3: Setting up IAM permissions

Add via CloudFormation:

Kubecost offers a set of CloudFormation templates to help set your IAM roles up.

If you’re new to provisioning IAM roles, we suggest downloading our templates and using the CloudFormation wizard to set these up. You can learn how to do this in AWS' Creating a stack on the AWS CloudFormation console doc. Open the step below which represents your CUR and management account arrangement, download the .yaml file listed, and upload them as the stack template in the 'Creating a stack' > 'Selecting a stack template' step.

Add manually:

        {
           "Version": "2012-10-17",
           "Statement": [
              {
                 "Sid": "AthenaAccess",
                 "Effect": "Allow",
                 "Action": [
                    "athena:*"
                 ],
                 "Resource": [
                    "*"
                 ]
              },
              {
                 "Sid": "ReadAccessToAthenaCurDataViaGlue",
                 "Effect": "Allow",
                 "Action": [
                    "glue:GetDatabase*",
                    "glue:GetTable*",
                    "glue:GetPartition*",
                    "glue:GetUserDefinedFunction",
                    "glue:BatchGetPartition"
                 ],
                 "Resource": [
                    "arn:aws:glue:*:*:catalog",
                    "arn:aws:glue:*:*:database/athenacurcfn*",
                    "arn:aws:glue:*:*:table/athenacurcfn*/*"
                 ]
              },
              {
                 "Sid": "AthenaQueryResultsOutput",
                 "Effect": "Allow",
                 "Action": [
                    "s3:GetBucketLocation",
                    "s3:GetObject",
                    "s3:ListBucket",
                    "s3:ListBucketMultipartUploads",
                    "s3:ListMultipartUploadParts",
                    "s3:AbortMultipartUpload",
                    "s3:CreateBucket",
                    "s3:PutObject"
                 ],
                 "Resource": [
                    "arn:aws:s3:::aws-athena-query-results-*"
                 ]
              },


              {
                 "Sid": "S3ReadAccessToAwsBillingData",
                 "Effect": "Allow",
                 "Action": [
                    "s3:Get*",
                    "s3:List*"
                 ],
                 "Resource": [
                    "arn:aws:s3:::${AthenaCURBucket}*"
                 ]
              }
           ]
        }
	{
           "Version": "2012-10-17",
           "Statement": [
              {
                 "Sid": "SpotDataAccess",
                 "Effect": "Allow",
                 "Action": [
                    "s3:ListAllMyBuckets",
                    "s3:ListBucket",
                    "s3:HeadBucket",
                    "s3:HeadObject",
                    "s3:List*",
                    "s3:Get*"
                 ],
                 "Resource": "arn:aws:s3:::${SpotDataFeedBucketName}*"
              }
           ]
        }

	{
               "Version": "2012-10-17",
               "Statement": [
                  {
                     "Sid": "AssumeRoleInMasterPayer",
                     "Effect": "Allow",
                     "Action": "sts:AssumeRole",
                     "Resource": "arn:aws:iam::${MasterPayerAccountID}:role/KubecostRole-${This-account’s-id}"
                  }
               ]
	}

	{
               "Version": "2012-10-17",
               "Statement": [
                  {
                     "Sid": "SpotDataAccess",
                     "Effect": "Allow",
                     "Action": [
                        "s3:ListAllMyBuckets",
                        "s3:ListBucket",
                        "s3:HeadBucket",
                        "s3:HeadObject",
                        "s3:List*",
                        "s3:Get*"
                     ],
                     "Resource": "arn:aws:s3:::${SpotDataFeedBucketName}*"
                  }
               ]
	}

	{
               "Version": "2012-10-17",
               "Statement": [
                  {
                     "Sid": "AthenaAccess",
                     "Effect": "Allow",
                     "Action": [
                        "athena:*"
                     ],
                     "Resource": [
                        "*"
                     ]
	},
	{
                     "Sid": "ReadAccessToAthenaCurDataViaGlue",
                     "Effect": "Allow",
                     "Action": [
                        "glue:GetDatabase*",
                        "glue:GetTable*",
                        "glue:GetPartition*",
                        "glue:GetUserDefinedFunction",
                        "glue:BatchGetPartition"
                     ],
                     "Resource": [
                        "arn:aws:glue:*:*:catalog",
                        "arn:aws:glue:*:*:database/athenacurcfn*",
                        "arn:aws:glue:*:*:table/athenacurcfn*/*"
                     ]
                  },
                  {
                     "Sid": "AthenaQueryResultsOutput",
                     "Effect": "Allow",
                     "Action": [
                        "s3:GetBucketLocation",
                        "s3:GetObject",
                        "s3:ListBucket",
                        "s3:ListBucketMultipartUploads",
                        "s3:ListMultipartUploadParts",
                        "s3:AbortMultipartUpload",
                        "s3:CreateBucket",
                        "s3:PutObject"
                     ],
                     "Resource": [
                        "arn:aws:s3:::aws-athena-query-results-*"
                     ]
                  },
                  {
                     "Sid": "S3ReadAccessToAwsBillingData",
                     "Effect": "Allow",
                     "Action": [
                        "s3:Get*",
                        "s3:List*"
                     ],
                     "Resource": [
                        "arn:aws:s3:::${AthenaCURBucket}*"
                     ]
                  }
               ]
	}

	{
               "Version": "2012-10-17",
               "Statement": [
                  {
                     "Effect": "Allow",
                     "Principal": {
                        "AWS": "arn:aws:iam::${aws-mgmt-account-id}:root"
                     },
                     "Action": [
                        "sts:AssumeRole"
                     ]
                  }
               ]
            }

Step 4: Attaching IAM permissions to Kubecost

Now that the policies have been created, attach those policies to Kubecost. We support the following methods:

kubecostProductConfigs:
  createServiceKeySecret: true
  awsServiceKeyName: <ACCESS_KEY_ID>
  awsServiceKeyPassword: <SECRET_ACCESS_KEY>

{
    "aws_access_key_id": "<ACCESS_KEY_ID>",
    "aws_secret_access_key": "<SECRET_ACCESS_KEY>"
}

$ kubectl create secret generic <SECRET_NAME> --from-file=service-key.json --namespace <kubecost>

kubecostProductConfigs:
  serviceKeySecretName: <SECRET_NAME>

    "athenaBucketName": "s3://<AWS_cloud_integration_athenaBucketName>",
    "athenaRegion": "<AWS_cloud_integration_athenaRegion>",
    "athenaDatabase": "<AWS_cloud_integration_athenaDatabase>",
    "athenaTable": "<AWS_cloud_integration_athenaTable>",
    "projectID": "<AWS_account_ID>"

aws iam create-policy --policy-name kubecost-athena-policy --policy-document file://kubecost-athena-policy.json

kubectl create ns kubecost
export YOUR_CLUSTER_NAME=<ENTER_YOUR_ACTUAL_CLUSTER_NAME>
export AWS_REGION=<ENTER_YOUR_AWS_REGION>
eksctl utils associate-iam-oidc-provider \
    --cluster ${YOUR_CLUSTER_NAME} --region ${AWS_REGION} \
    --approve

eksctl create iamserviceaccount \
    --name kubecost-serviceaccount-cur-athena-thanos \
    --namespace kubecost \
    --cluster ${YOUR_CLUSTER_NAME} --region ${AWS_REGION} \
    --attach-policy-arn arn:aws:iam::1234567890:policy/kubecost-athena-policy \
    --override-existing-serviceaccounts \
    --approve

kubectl create secret generic cloud-integration -n kubecost --from-file=cloud-integration.json

helm upgrade --install kubecost --repo https://kubecost.github.io/cost-analyzer/ cost-analyzer \
--namespace kubecost \
-f https://raw.githubusercontent.com/kubecost/poc-common-configurations/main/aws-attach-roles/values-amazon-primary.yaml

Step 5: Provide CUR config values to Kubecost

These values can either be set from the Kubecost UI or via .Values.kubecostProductConfigs in the Helm chart. Values for all fields must be provided.

Option 1: Add config values via UI

To add values in the Kubecost UI, select Settings from the left navigation, then scroll to Cloud Cost Settings. Select Update next to External Cloud Cost Configuration (AWS). The Billing Data Export Configuration window opens. Fill in all the below fields:

When you are done, select Update to confirm.

Option 2: Add config values via Helm

• athenaProjectID: The AWS AccountID where the Athena CUR is, likely your management account.

• athenaBucketName: An S3 bucket to store Athena query results that you’ve created that Kubecost has permission to access

  • The name of the bucket should match s3://aws-athena-query-results-*, so the IAM roles defined above will automatically allow access to it

  • The bucket can have a Canned ACL of Private or other permissions as you see fit.

• athenaRegion: The AWS region Athena is running in

• athenaDatabase: The name of the database created by the Athena setup

  • The athena database name is available as the value (physical id) of AWSCURDatabase in the CloudFormation stack created above (in Step 2: Setting up Athena)

• athenaTable: the name of the table created by the Athena setup

  • The table name is typically the database name with the leading athenacurcfn_ removed (but is not available as a CloudFormation stack resource). Confirm the table name by visiting the Athena dashboard.

• athenaWorkgroup: The workgroup assigned to be used with Athena. If not specified, defaults to Primary

• If you are using a multi-account setup, you will also need to set .Values.kubecostProductConfigs.masterPayerARN to the Amazon Resource Number (ARN) of the role in the management account, e.g. arn:aws:iam::530337586275:role/KubecostRole.

Troubleshooting

Once you've integrated with the CUR, you can visit Settings > View Full Diagnostics in the UI to determine if Kubecost has been successfully integrated with your CUR. If any problems are detected, you will see a yellow warning sign under the cloud provider permissions status header

You can check pod logs for authentication errors by running: kubectl get pods -n <namespace> kubectl logs <kubecost-pod-name> -n <namespace> -c cost-model

If you do not see any authentication errors, log in to your AWS console and visit the Athena dashboard. You should be able to find the CUR. Ensure that the database with the CUR matches the athenaTable entered in Step 5. It likely has a prefix with athenacurcfn_ :

You can also check query history to see if any queries are failing:

Common Athena errors

Incorrect bucket in IAM Policy

• Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources. You can search in the Athena "Recent queries" dashboard to find additional info about the error.

```
QueryAthenaPaginated: query execution error: no query results available for query <Athena Query ID>
```

And/or the following error will be found in the Kubecost `cost-model` container logs.

```
Permission denied on S3 path: s3://cur-report/cur-report/cur-report/year=2022/month=8

This query ran against the "athenacurcfn_test" database, unless qualified by the query. Please post the error message on our forum  or contact customer support  with Query Id: <Athena Query ID>
```

• Resolution: This error is typically caused by the incorrect (Athena results) s3 bucket being specified in the CloudFormation template of Step 3 from above. To resolve the issue, ensure the bucket used for storing the AWS CUR report (Step 1) is specified in the S3ReadAccessToAwsBillingData SID of the IAM policy (default: kubecost-athena-access) attached to the user or role used by Kubecost (Default: KubecostUser / KubecostRole). See the following example.

        {
        "Action": [
            "s3:Get*",
            "s3:List*"
        ],
        "Resource": [
            "arn:aws:s3:::<AWS CUR BUCKET>*"
        ],
        "Effect": "Allow",
        "Sid": "S3ReadAccessToAwsBillingData"
    }

outputLocation is not a valid S3 path

• Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources.

Connection test failed for cloud integration config: Fetch error: cloud billing data fetch error: GetCloudCost: error getting Athena columns: QueryAthenaPaginated: start query error: operation error Athena: StartQueryExecution, https response error StatusCode: 400, RequestID: a6059220-5ac8-4c24-97d2-401a2dbfd421, InvalidRequestException: outputLocation is not a valid S3 path.

• Resolution: Please verify that the prefix s3:// was used when setting the athenaBucketName Helm value or when configuring the bucket name in the Kubecost UI.

Query not supported

• Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources.

QueryAthenaPaginated: start query error: operation error Athena: StartQueryExecution, https response error StatusCode: 400, RequestID: <Athena Query ID>, InvalidRequestException: Queries of this type are not supported

• Resolution: While rare, this issue was caused by an Athena instance that failed to provision properly on AWS. The solution was to delete the Athena DB and deploy a new one. To verify this is needed, find the failed query ID in the Athena "Recent queries" dashboard and attempt to manually run the query.

HTTPS Response error

• Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources.

QueryAthenaPaginated: start query error: operation error Athena: StartQueryExecution, https response error StatusCode: 400, RequestID: ********************, InvalidRequestException: Unable to verify/create output bucket aws-athena-query-results-test

• Resolution: Previously, if you ran a query without specifying a value for query result location, and the query result location setting was not overridden by a workgroup, Athena created a default location for you. Now, before you can run an Athena query in a region in which your account hasn't used Athena previously, you must specify a query result location, or use a workgroup that overrides the query result location setting. While Athena no longer creates a default query results location for you, previously created default aws-athena-query-results-MyAcctID-MyRegion locations remain valid and you can continue to use them. The bucket should be in the format of: aws-athena-query-results-MyAcctID-MyRegion It may also be required to remove and reinstall Kubecost. If doing this please remeber to backup ETL files prior or contact support for additional assistance. See also this AWS doc on specifying a query result location.

Missing Athena column

• Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources or in the Kubecost cost-model container logs.

QueryAthenaPaginated: query execution error: no query results available for query <Athena Query ID>

Checking the Athena logs we see a syntax error:

SYNTAX_ERROR: line 4:3: Column 'line_item_resource_id' cannot be resolved

This query ran against the "<DB Name>" database, unless qualified by the query

• Resolution: Verify in AWS' Cost and Usage Reports dashboard that the Resource IDs are enabled as "Report content" for the CUR created in Step 1. If the Resource IDs are not enabled, you will need to re-create the report (this will require redoing Steps 1 and 2 from this doc).

Not a valid S3 path

• Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources or in the Kubecost cost-model container logs.

QueryAthenaPaginated: start query error: operation error Athena: StartQueryExecution, https response error StatusCode: 400, RequestID: <Athena Query ID>, InvalidRequestException: outputLocation is not a valid S3 path.

• Resolution: Verify that s3:// was included in the bucket name when setting the .Values.kubecostProductConfigs.athenaBucketName Helm value.

Summary and pricing

AWS services used here are:

• Athena

• S3

• EC2

Kubecost's cost-model requires roughly 2 CPU and 10 GB of RAM per 50,000 pods monitored. The backing Prometheus database requires roughly 2 CPU and 25 GB per million metrics ingested per minute. You can pick the EC2 instances necessary to run Kubecost accordingly.

• EBS

Kubecost can write its cache to disk. Roughly 32 GB per 100,000 pods monitored is sufficient. (Optional: our cache can exist in memory)

• Cloudformation (Optional: manual IAM configuration or via Terraform is fine)

• EKS (Optional: all K8s flavors are supported)

In order to create a Google service account for use with Thanos, navigate to the Google Cloud Platform home page and select IAM & Admin > Service Accounts.

From here, select the option Create Service Account.

Provide a service account name, ID, and description, then select Create and Continue.

You should now be at the Service account permissions (optional) page. Select the first Role dropdown and select Storage Object Creator. Select Add Another Role, then select Storage Object Viewer from the second dropdown. Select Continue.

You should now be prompted to allow specific accounts access to this service account. This should be based on specific internal needs and is not a requirement. You can leave this empty and select Done.

Create a key

Once back to the Service accounts page, select the Actions icon > Manage keys. Then, select the Add Key dropdown and select Create new key. A Create private key window opens.

Select JSON as the Key type and select Create. This will download a JSON service account key entry for use with the Thanos object-store.yaml mentioned in the initial setup step.

Certain features of Kubecost, including Savings Insights like Orphaned Resources and Reserved Instances, require access to the cluster's GCP account. This is usually indicated by a 403 error from Google APIs which is due to 'insufficient authentication scopes'. Viewing this error in the Kubecost UI will display the cause of the error as "ACCESS_TOKEN_SCOPE_INSUFFICIENT".

To obtain access to these features, follow this tutorial which will show you how to configure your Google IAM Service Account and Workload Identity for your application.

Creating a GCP IAM Service Account

1. Creating an API Key

Go to your GCP Console and select APIs & Services > Credentials from the left navigation. Select + Create Credentials > API Key.

On the Credentials page, select the icon in the Actions column for your newly-created API key, then select Edit API key. The Edit API key page opens.

Under ‘API restrictions’, select Restrict key, then from the dropdown, select only Cloud Billing API. Select OK to confirm. Then select Save at the bottom of the page.

2. Configuring Workload Identity

From here, consult Google Cloud's guide to perform the following steps:

• Enable Workload Identity on an existing GCP cluster, or spin up a new cluster which will have Workload Identity enabled by default

• Migrate any existing workloads to Workload Identity

• Configure your applications to use Workload Identity

• Create both a Kubernetes service account (KSA) and an IAM service account (GSA).

• Annotate the KSA with the email of the GSA.

• Update your pod spec to use the annotated KSA, and ensure all nodes on that workload use Workload Identity.

You can stop once you have modified your pod spec (before 'Verify the Workload Identity Setup'). You should now have a GCP cluster with Workload Identity enabled, and both a KSA and a GSA, which are connected via the role roles/iam.workloadIdentityUser.

3. Updating your IAM service account

From here, restart the pod(s) to confirm your changes. You should now have access to all expected Kubecost functionality through your service account with Identity Workload.

Primary and secondary clusters

In an Enterprise multi-cluster setup, the UI is accessed through a designated primary cluster. All other clusters in the environment send metrics to a central object-store with a lightweight agent (aka secondary clusters). The primary cluster is designated by setting the Helm flag .Values.federatedETL.primaryCluster=true, which instructs this cluster to read from the combined folder that was processed by the federator. This cluster will consume additional resources to run the Kubecost UI and backend.

Enterprise Federation

There are two primary methods to aggregate all cluster information back to a single Kubecost UI:

Both methods allow for greater compute efficiency by running the most resource-intensive workloads on a single primary cluster.

For environments that already have a Prometheus instance, ETL Federation may be preferred because only a single Kubecost pod is required.

The below diagrams highlight the two architectures:

Kubecost ETL Federation (Preferred)

Kubecost Thanos Federation

Thanos is a tool to aggregate Prometheus metrics to a central object storage (S3 compatible) bucket. Thanos is implemented as a sidecar on the Prometheus pod on all clusters. Thanos Federation is one of two primary methods to aggregate all cluster information back to a single view as described in our article.

The preferred method for multi-cluster is . The configuration guide below is for Kubecost Thanos Federation, which may not scale as well as ETL Federation in large environments.

This guide will cover how to enable Thanos on your primary cluster, and on any additional secondary clusters.

Configuration

1. Follow steps to enable all required Thanos components on a Kubecost primary cluster, including the Prometheus sidecar.

2. For each additional cluster, only the Thanos sidecar is needed.

Consider the following Thanos recommendations for secondaries:

1. Ensure you provide a unique identifier for prometheus.server.global.external_labels.cluster_id to have additional clusters be visible in the Kubecost product, e.g. cluster-two.

Architecture diagram

This doc provides recommendations to improve the stability and recoverability of your Kubecost data when deploying in a Federated ETL architecture.

Option 1: Increase Prometheus retention

Kubecost can rebuild its extract, transform, load (ETL) data using Prometheus metrics from each cluster. It is strongly recommended to retain local cluster Prometheus metrics that meet an organization's disaster recovery requirements.

Option 2: Metrics backup

For long term storage of Prometheus metrics, we recommend setting up a Thanos sidecar container to push Prometheus metrics to a cloud storage bucket.

Option 3: Bucket versioning

Use your cloud service provider's bucket versioning feature to take frequent snapshots of the bucket holding your Kubecost data (ETL files and Prometheus metrics).

Option 4: Alerting

Aggregator is a new backend for Kubecost. It is used in a configuration without Thanos, replacing the component. Aggregator serves a critical subset of Kubecost APIs, but will eventually be the default model for Kubecost and serve all APIs. Currently, Aggregator supports all major monitoring and savings APIs, and also budgets and reporting.

Aggregator is designed to accommodate queries of large-scale datasets by improving API load times and reducing UI errors. It is not designed to introduce new functionality; it is meant to improve functionality at scale.

Aggregator is currently free for all Enterprise users to configure, and is always able to be rolled back.

Configuring Aggregator

Prerequisites

• Aggregator can only be configured in a Federated ETL environment

• Must be using v1.107.0 of Kubecost or newer

• Your values.yaml file must have set kubecostDeployment.queryServiceReplicas to its default value 0.

• You must have your context set to your primary cluster. Kubecost Aggregator cannot be deployed on secondary clusters.

Tutorial

Select from one of the two templates below and save the content as federated-store.yaml. This will be your configuration template required to set up Aggregator.

Basic configuration:

Advanced configuration (for larger deployments):

There is no baseline for what is considered a larger deployment, which will be dependent on load times in your Kubecost environment.

Once you’ve configured your federated-store.yaml_, create a secret using the following command:

Finally, upgrade your existing Kubecost installation. This command will install Kubecost if it does not already exist:

Validating Aggregator pod is running successfully

When first enabled, the aggregator pod will ingest the last three years (if applicable) of ETL data from the federated-store. This may take several hours. Because the combined folder is ignored, the federator pod is not used here, but can still run if needed. You can run kubectl get pods and ensure the aggregator pod is running, but should still wait for all data to be ingested.

Federated extract, transform, load (ETL) is one of two methods to aggregate all cluster information back to a single display described in our doc. Federated ETL gives teams the benefit of combining multiple Kubecost installations into one view without dependency on Thanos.

There are two primary advantages for using ETL Federation:

1. For environments that already have a Prometheus instance, Kubecost only requires a single pod per monitored cluster

2. Many solutions that aggregate Prometheus metrics (like Thanos), are often expensive to scale in large environments

Kubecost ETL Federation diagram

Sample configurations

This guide has specific detail on how ETL Configuration works and deployment options.

Clusters

The federated ETL is composed of three types of clusters.

• Federated Clusters: The clusters which are being federated (clusters whose data will be combined and viewable at the end of the federated ETL pipeline). These clusters upload their ETL files after they have built them to Federated Storage.

• Federator Clusters: The cluster on which the Federator (see in Other components) is set to run within the core cost-analyzer container. This cluster combines the Federated Cluster data uploaded to federated storage into combined storage.

• Primary Cluster: A cluster where you can see the total Federated data that was combined from your Federated Clusters. These clusters read from combined storage.

These cluster designations can overlap, in that some clusters may be several types at once. A cluster that is a Federated Cluster, Federator Cluster, and Primary Cluster will perform the following functions:

• As a Federated Cluster, push local cluster cost data to be combined from its local ETL build pipeline.

• As a Federator Cluster, run the Federator inside the cost-analyzer, which pulls this local cluster data from S3, combines them, then pushes them back to combined storage.

• As a Primary Cluster, pull back this combined data from combined storage to serve it on Kubecost APIs and/or the Kubecost frontend.

Other components

The Storages referred to here are an S3 (or GCP/Azure equivalent) storage bucket which acts as remote storage for the Federated ETL Pipeline.

• Federated Storage: A set of folders on paths <bucket>/federated/<cluster id> which are essentially ETL backup data, holding a “copy” of Federated Cluster data. Federated Clusters push this data to Federated Storage to be combined by the Federator. Federated Clusters write this data, and the Federator reads this data.

• Combined Storage: A folder on S3 on the path <bucket>/federated/combined which holds one set of ETL data containing all the allocations/assets in all the ETL data from Federated Storage. The Federator takes files from Federated Storage and combines them, adding a single set of combined ETL files to Combined Storage to be read by the Primary Cluster. The Federator writes this data, and the Primary Cluster reads this data.

• The Federator: A component of the cost-model which is run on the Federator Cluster, which can be a Federated Cluster, a Primary Cluster, or neither. The Federator takes the ETL binaries from Federated Storage and merges them, adding them to Combined Storage.

• Federated ETL: The pipeline containing the above components.

Federated ETL architecture

This diagram shows an example setup of the Federated ETL with:

• Three pure Federated Clusters (not classified as any other cluster type): Cluster 1, Cluster 2, and Cluster 3

• One Federator Cluster that is also a Federated Cluster: Cluster 4

• One Primary Cluster that is also a Federated Cluster: Cluster 5

The result is 5 clusters federated together.

Setup

Step 0: Ensure unique cluster IDs

Ensure each federated cluster has a unique clusterName and cluster_id:

Step 1: Storage configuration

1. For any cluster in the pipeline (Federator, Federated, Primary, or any combination of the three), create a file federated-store.yaml with the same format used for Thanos/S3 backup.
2. Add a secret using that file: kubectl create secret generic <secret_name> -n kubecost --from-file=federated-store.yaml. Then set .Values.kubecostModel.federatedStorageConfigSecret to the kubernetes secret name.

Step 2: Cluster configuration (Federated/Federator)

1. For all clusters you want to federate together (i.e. see their data on the Primary Cluster), set .Values.federatedETL.federatedCluster to true. This cluster is now a Federated Cluster, and can also be a Federator or Primary Cluster.

2. For the cluster “hosting” the Federator, set .Values.federatedETL.federator.enabled to true. This cluster is now a Federator Cluster, and can also be a Federated or Primary Cluster.

   • Optional: If you have any Federated Clusters pushing to a store that you do not want a Federator Cluster to federate, add the cluster id under the Federator config section .Values.federatedETL.federator.clusters. If this parameter is empty or not set, the Federator will take all ETL files in the /federated directory and federate them automatically.

   • Multiple Federators federating from the same source will not break, but it’s not recommended.

Step 3: Cluster configuration (Primary)

In Kubecost, the Primary Cluster serves the UI and API endpoints as well as reconciling cloud billing (cloud-integration).

1. For the cluster that will be the Primary Cluster, set .Values.federatedETL.primaryCluster to true. This cluster is now a Primary Cluster, and can also be a Federator or Federated Cluster.

2. Cloud-integration requires .Values.federatedETL.federator.primaryClusterID set to the same value used for .Values.kubecostProductConfigs.clusterName

   • Important: If the Primary Cluster is also to be federated, please wait 2-3 hours for data to populate Federated Storage before setting a Federated Cluster to primary (i.e. set .Values.federatedETL.federatedCluster to true, then wait to set .Values.federatedETL.primaryCluster to true). This allows for maximum certainty of data consistency.

   • If you do not set this cluster to be federated as well as primary, you will not see local data for this cluster.

   • The Primary Cluster’s local ETL will be overwritten with combined federated data.

     • This can be undone by unsetting it as a Primary Cluster and rebuilding ETL.

     • Setting a Primary Cluster may result in a loss of the cluster’s local ETL data, so it is recommended to back up any filestore data that one would want to save to S3 before designating the cluster as primary.

     • Alternatively, a fresh Kubecost install can be used as a consumer of combined federated data by setting it as the Primary but not a Federated Cluster.

Step 4: Verifying successful configuration

1. The Federated ETL should begin functioning. On any ETL action on a Federated Cluster (Load/Put into local ETL store) the Federated Clusters will add data to Federated Storage. The Federator will run 5 minutes after the Federator Cluster startup, and then every 30 minutes after that. The data is merged into the Combined Storage, where it can be read by the Primary.

   • To verify Federated Clusters are uploading their data correctly, check the container logs on a Federated Cluster. It should log federated uploads when ETL build steps run. The S3 bucket can also be checked to see if data is being written to the /federated/<cluster_id> path.

   • To verify the Federator is functioning, check the container logs on the Federator Cluster. The S3 bucket can also be checked to verify that data is being written to /federated/combined.

   • To verify the entire pipeline is working, either query Allocations/Assets or view the respective views on the frontend. Multi-cluster data should appear after:

     • The Federator has run at least once.

     • There was data in the Federated Storage for the Federator to have combined.

Setup with internal certificate authority

If you are using an internal certificate authority (CA), follow this tutorial instead of the above Setup section.

Begin by creating a ConfigMap with the certificate provided by the CA on every agent, including the Federator and any federated clusters, and name the file kubecost-federator-certs.yaml.

Now run the following command, making sure you specify the location for the ConfigMap you created:

kubectl create cm kubecost-federator-certs --from-file=/path/to/kubecost-federator-certs.yaml

Mount the certification on the Federator and any federated clusters by passing these Helm flags to your values.yaml/manifest:

Create a file federated-store.yaml, which will go on all clusters:

Now run the following command (omit kubectl create namespace kubecost if your kubecost namespace already exists, or this command will fail):

See also

Data recovery

Repairing ETL