Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Multi-cloud integrations are only officially supported on Kubecost Enteprise plans.
This document outlines how to set up cloud integration for accounts on multiple cloud service providers (CSPs), or multiple accounts on the same cloud provider. This configuration can be used independently of, or in addition, to other cloud integration configurations provided by Kubecost. Once configured, Kubecost will display cloud assets for all configured accounts and perform reconciliation for all federated clusters that have their respective accounts configured.
For each cloud account that you would like to configure, you will need to make sure that it is exporting cost data to its respective service to allow Kubecost to gain access to it.
Azure: Set up cost data export following this guide.
GCP: Set up BigQuery billing data exports with this guide.
AWS: Follow steps 1-3 to set up and configure a Cost and Usage Report (CUR) in our guide.
Alibaba: Create a user account with access to the QueryInstanceBill API.
The secret should contain a file named cloud-integration.json with the following format (only containing applicable CSPs in your setup):
This method of cloud integration supports multiple configurations per cloud provider simply by adding each cost export to their respective arrays in the .json file. The structure and required values for the configuration objects for each cloud provider are described below. Once you have filled in the configuration object, use the command:
Once the secret is created, set .Values.kubecostProductConfigs.cloudIntegrationSecret to <SECRET_NAME> and upgrade Kubecost via Helm.
A GitHub repository with sample files required can be found here. Select the folder with the name of the cloud service you are configuring.
The following values can be located in the Azure Portal under Cost Management > Exports, or Storage accounts:
azureSubscriptionID is the Subscription ID belonging to the Storage account which stores your exported Azure cost report data.
azureStorageAccount is the name of the Storage account where the exported Azure cost report data is being stored.
azureStorageAccessKey can be found by selecting Access Keys from the navigation sidebar then selecting Show keys. Using either of the two keys will work.
azureStorageContainer is the name that you chose for the exported cost report when you set it up. This is the name of the container where the CSV cost reports are saved in your Storage account.
azureContainerPath is an optional value which should be used if there is more than one billing report that is exported to the configured container. The path provided should have only one billing export because Kubecost will retrieve the most recent billing report for a given month found within the path.
azureCloud is an optional value which denotes the cloud where the storage account exists. Possible values are public and gov. The default is public.
Set these values into the following object and add them to the Azure array:
If you don't already have a GCP service key for any of the projects you would like to configure, you can run the following commands in your command line to generate and export one. Make sure your GCP project is where your external costs are being run.
You can then get your service account key to paste into the UI:
<KEY_JSON> is the GCP service key created above. This value should be left as a JSON when inserted into the configuration object
<PROJECT_ID> is the Project ID in the GCP service key.
<BILLING_DATA_DATASET> requires a BigQuery dataset prefix (e.g. billing_data) in addition to the BigQuery table name. A full example is billing_data.gcp_billing_export_v1_018AIF_74KD1D_534A2.
Set these values into the following object and add it to the GCP array:
Many of these values in this config can be generated using the following command:
For each AWS account that you would like to configure, create an Access Key for the Kubecost user who has access to the CUR. Navigate to IAM Management Console dashboard, and select Access Management > Users. Find the Kubecost user and select Security Credentials > Create Access Key. Note the Access Key ID and Secret access key.
Gather each of these values from the AWS console for each account you would like to configure.
<ACCESS_KEY_ID> is the ID of the Access Key created in the previous step.
<ACCESS_KEY_SECRET> is the secret of the Access Key created in the
<ATHENA_BUCKET_NAME> is the S3 bucket storing Athena query results which Kubecost has permission to access. The name of the bucket should match s3://aws-athena-query-results-*, so the IAM roles defined above will automatically allow access to it. The bucket can have a canned ACL set to Private or other permissions as needed.
<ATHENA_REGION> is the AWS region Athena is running in
<ATHENA_DATABASE> is the name of the database created by the Athena setup. The Athena database name is available as the value (physical id) of AWSCURDatabase in the CloudFormation stack created above.
<ATHENA_TABLE> is the name of the table created by the Athena setup The table name is typically the database name with the leading athenacurcfn_ removed (but is not available as a CloudFormation stack resource).
<ATHENA_WORKGROUP> is the workgroup assigned to be used with Athena. Default value is Primary.
<ATHENA_PROJECT_ID>is the AWS AccountID where the Athena CUR is. For example: 530337586277.
<MASTER_PAYER_ARN> is an optional value which should be set if you are using a multi-account billing set-up and are not accessing Athena through the primary account. It should be set to the ARN of the role in the management (formerly master payer) account, for example: arn:aws:iam::530337586275:role/KubecostRole.
Set these values into the following object and add them to the AWS array in the cloud-integration.json:
Additionally set the kubecostProductConfigs.athenaProjectID Helm value to the AWS account that Kubecost is being installed in.
Kubecost does not support complete integrations with Alibaba, but you will still be able to view accurate list prices for cloud resources. Gather these following values from the Alibaba Cloud Console for your account:
clusterRegion is the most used region
accountID is your Alibaba account ID
serviceKeyName is the RAM user key name
serviceKeySecret is the RAM user secret
Set these values into the following object and add them to the Alibaba array in your cloud-integration.json:
There are many ways to integrate your AWS Cost and Usage Report (CUR) with Kubecost. This tutorial is intended as the best-practice method for users whose environments meet the following assumptions:
Kubecost will run in a different account than the AWS Payer Account
The IAM permissions will utilize AWS to avoid shared secrets
The configuration of Kubecost will be done using a cloud-integration.json file, and not via Kubecost UI (following infrastructure as code practices)
If this is not an accurate description of your environment, see our doc for more options.
This guide is a one-time setup per AWS payer account and is typically one per organization. It can be automated, but may not be worth the effort given that it will not be needed again.
Kubecost supports multiple AWS payer accounts as well as multiple cloud providers from a single Kubecost primary cluster. For multiple payer accounts, create additional entries inside the array below.
Detail for multiple cloud provider setups is .
cloud-integration.json
iam-payer-account-cur-athena-glue-s3-access.json
iam-payer-account-trust-primary-account.json
iam-access-cur-in-payer-account.json
Begin by opening cloud_integration.json, which should look like this:
Update athenaWorkgroup to primary, then save the file and close it. The remaining values will be obtained during this tutorial.
For time granularity, select Daily.
Select the checkbox to enable Resource IDs in the report.
Select the checkbox to enable Athena integration with the report.
Select the checkbox to enable the JSON IAM policy to be applied to your bucket.
If this CUR data is only used by Kubecost, it is safe to expire or delete the objects after seven days of retention.
AWS may take up to 24 hours to publish data. Wait until this is complete before continuing to the next step.
While you wait, update the following configuration files:
Update your cloud-integration.json file by providing a projectID value, which will be the AWS payer account number where the CUR is located and where the Kubecost primary cluster is running.
Update your iam-payer-account-cur-athena-glue-s2-access.json file by replacing all instances of CUR_BUCKET_NAME to the name of the bucket you created for CUR data.
Your S3 path prefix can be found by going to your AWS Cost and Usage Reports dashboard and selecting your bucket's report. In the Report details tab, you will find the S3 path prefix.
Once Athena is set up with the CUR, you will need to create a new S3 bucket for Athena query results. The bucket used for the CUR cannot be used for the Athena output.
Select Create bucket. The Create Bucket page opens.
Provide a name for your bucket. This is the value for athenaBucketName in your cloud-integration.json file. Use the same region used for the CUR bucket.
Select Create bucket at the bottom of the page.
Select Settings, then select Manage. The Manage settings window opens.
Set Location of query result to the S3 bucket you just created, then select Save.
Navigate to Athena in the AWS Console. Be sure the region matches the one used in the steps above. Update your cloud-integration.json file with the following values. Use the screenshots below for help.
athenaBucketName: the name of the Athena bucket your created in this step
athenaDatabase: the value in the Database dropdown
athenaRegion: the AWS region value where your Athena query is configured
athenaTable: the partitioned value found in the Table list
For Athena query results written to an S3 bucket only accessed by Kubecost, it is safe to expire or delete the objects after one day of retention.
From the AWS payer account
In iam-payer-account-cur-athena-glue-s3-access.json, replace all ATHENA_RESULTS_BUCKET_NAME instances with your Athena S3 bucket name (the default will look like aws-athena-query-results-xxxx).
In iam-payer-account-trust-primary-account.json, replace SUB_ACCOUNT_222222222 with the account number of the account where the Kubecost primary cluster will run.
In the same location as your downloaded configuration files, run the following command to create the appropriate policy (jq is not required):
Now we can obtain the last value masterPayerARN for cloud-integration.json as the ARN associated with the newly-created IAM role, as seen below in the AWS console:
By arriving at this step, you should have been able to provide all values to your cloud-integration.json file. If any values are missing, reread the tutorial and follow any steps needed to obtain those values.
From the AWS Account where the Kubecost primary cluster will run
In iam-access-cur-in-payer-account.json, update PAYER_ACCOUNT_11111111111 with the AWS account number of the payer account and create a policy allowing Kubecost to assumeRole in the payer account:
Note the output ARN (used in the iamserviceaccount --attach-policy-arn below):
Create a namespace and set environment variables:
Enable the OIDC-Provider:
Create the Kubernetes service account, attaching the assumeRole policy. Replace SUB_ACCOUNT_222222222 with the AWS account number where the primary Kubecost cluster will run.
Create the secret (in this setup, there are no actual secrets in this file):
Install Kubecost using the service account and cloud-integration secret:
It can take over an hour to process the billing data for large AWS accounts. In the short-term, follow the logs and look for a message similar to (7.7 complete), which should grow gradually to (100.0 complete). Some errors (ERR) are expected, as seen below.
To begin, download the recommended configuration template files from our . You will need the following files from this folder:
Follow the to create a CUR export using the settings below.
As part of the CUR creation process, Amazon creates a CloudFormation template that is used to create the Athena integration. It is created in the CUR S3 bucket under s3-path-prefix/cur-name and typically has the filename crawler-cfn.yml. This .yml is your CloudFormation template. You will need it in order to complete the CUR Athena integration. You can read more about this .
Navigate to the .
Navigate to the dashboard.
For help with troubleshooting, see the section in our original .
Kubecost uses public pricing from Cloud Service Providers (CSPs) to calculate costs until the actual cloud bill is available, at which point Kubecost will reconcile your Spot prices from your Cost and Usage Report (CUR). This is almost always ready in 48 hours. Most users will likely prefer to configure AWS cloud-integrations instead of configuring the Spot data feed manually as demonstrated in this article.
However, if the majority of costs are due to Spot nodes, it may be useful to configure the Spot pricing data feed as it will increase accuracy for short-term (<48 hour) node costs until the Spot prices from the CUR are available. Note that all other (non-Spot) costs will still be based on public (on-demand) pricing until CUR billing data is reconciled.
With Kubecost, Spot pricing data can be pulled hourly by integrating directly with the AWS Spot feed.
First, to enable the AWS Spot data feed, follow AWS' Spot Instance data feed doc.
While configuring, note the settings used as these values will be needed for the Kubecost configuration.
There are multiple options: this can either be set from the Kubecost UI or via .Values.kubecostProductConfigs in the Helm chart. If you set any kubecostProductConfigs from the Helm chart, all changes via the front end will be deleted on pod restart.
projectID the Account ID of the AWS Account on which the Spot nodes are running.
awsSpotDataRegion region of your Spot data bucket
awsSpotDataBucket the configured bucket for the Spot data feed
awsSpotDataPrefix optional configured prefix for your Spot data feed bucket
spotLabel optional Kubernetes node label name designating whether a node is a Spot node. Used to provide pricing estimates until exact Spot data becomes available from the CUR
spotLabelValue optional Kubernetes node label value designating a Spot node. Used to provide pricing estimates until exact Spot data becomes available from the CUR. For example, if your Spot nodes carry a label lifecycle:spot, then the spotLabel would be lifecycle and the spotLabelValue would be spot
In the UI, you can access these fields via the Settings page, then scroll to Cloud Cost Settings. Next to Spot Instance Configuration, select Update, then fill out all fields.
Spot data feeds are an account level setting, not a payer level. Every AWS Account will have its own Spot data feed. Spot data feed is not currently available in AWS GovCloud.
For Spot data written to an S3 bucket only accessed by Kubecost, it is safe to delete objects after three days of retention.
Kubecost requires read access to the Spot data feed bucket. The following IAM policy can be used to grant Kubecost read access to the Spot data feed bucket.
To attach the IAM policy to the Kubecost service account, you can use IRSA or the account's service key.
If your serviceaccount/kubecost-cost-analyzer already has IRSA annotations attached, be sure to include all policies necessary when running this command.
Create a service-key.json as shown:
Create a K8s secret:
Set the following Helm config:
Verify the below points:
Make sure data is present in the Spot data feed bucket.
Make sure Project ID is configured correctly. You can cross-verify the values under Helm values in bug report
Check the value of kubecost_node_is_spot in Prometheus:
"1" means Spot data instance configuration is correct.
"0" means not configured properly.
Is there a prefix? If so, is it configured in Kubecost?
Make sure the IAM permissions are aligned with https://github.com/kubecost/cloudformation/blob/7feace26637aa2ece1481fda394927ef8e1e3cad/kubecost-single-account-permissions.yaml#L36
Make sure the Spot data feed bucket has all permissions to access by Kubecost
The Spot Instance in the Spot data feed bucket should match the instance in the cluster where the Spot data feed is configured. awsSpotDataBucket has to be present in the right cluster.
Kubecost is capable of aggregating the costs of EC2 compute resources over a given timeframe with a specified duration step size. To achieve this, Kubecost uses Athena queries to gather usage data points with differing price models. The result of this process is a list of resources with their cost by timeframe.
The reconciliation process makes two queries to Athena, one to gather resources that are paid for with either the on-demand model or a savings plan and one query for resources on the reservation price model. The first query includes resources given at a blended rate, which could be on-demand usage or resources that have exceeded the limits of a savings plan. It will also include resources that are part of a savings plan which will have a savings plan effective cost. The second query only includes reserved resources and the cost which reflects the rate they were reserved at.
The queries make use of the following columns from Athena:
line_item_usage_start_date The beginning timestamp of the line item usage. Used to filter resource usage within a date range and to aggregate on usage window.
line_item_usage_end_date The ending timestamp of the line item usage. Used to filter resource usage within a date range and to aggregate on usage window.
line_item_resource_id An ID, also called the provider ID, is given to line items that are instantiated resources.
line_item_line_item_type The type of a line item, used to determine if the resource usage is covered by a savings plan and has a discounted price.
line_item_usage_type What is being used in a line item, for the purposes of a compute resource this, is the type of VM and where it is running
line_item_product_code The service that a line item is from. Used to filter out items that are not from EC2.
reservation_reservation_a_r_n Amazon Resource Name for reservation of line item, the presence of this value is used to identify a resource as being part of a reservation plan.
line_item_unblended_cost The undiscounted cost of a resource.
savings_plan_savings_plan_effective_cost The cost of a resource discounted by a savings plan
reservation_effective_cost The cost of a resource discounted by a reservation
This query is grouped by six columns:
line_item_usage_start_date
line_item_usage_end_date
line_item_resource_id
line_item_line_item_type
line_item_usage_type
line_item_product_code
The columns line_item_unblended_cost and savings_plan_savings_plan_effective_cost are summed on this grouping. Finally, the query filters out rows that are not within a given date range, have a missing line_item_resource_id, and have a line_item_product_code not equal to "AmazonEC2". The grouping has three important aspects, the timeframe of the line items, the resource as defined by the resource id, and the usage type, which is later used to determine the proper cost of the resources as it was used. This means that line items are grouped according to the resource, the time frame of the usage, and the rate at which the usage was charged.
The reservation query is grouped on five columns:
line_item_usage_start_date
line_item_usage_end_date
reservation_reservation_a_r_n
line_item_resource_id
line_item_product_code
The query is summed on the reservation_effective_cost and filtered by the date window, for missing reservation_reservation_a_r_n values and also removes line items with line_item_product_code not equal to "AmazonEC2". This grouping is on resource id by timeframe removing all non-reservation line items.
The on-demand query is categorized into different resource types: compute, network, storage, and others. The network is identified by the presence of the "byte" in the line_item_usage_type. Compute and storage are identified by the presence of "i-" and "vol-" prefixes in line_item_resource_id respectively. Non compute values are removed from the results. Out of the two costs aggregated by this query the correct one to use is determined by the line_item_line_item_type, if it has a value of "SavingsPlanCoveredUsage", then the savings_plan_savings_plan_effective_cost is used as the cost, and if not then the line_item_unblended_cost is used.
In the reservation query, all of the results are of the compute category and there is only the reservation_effective_cost to use as a cost.
These results are then merged into one set, with the provider id used to associate the cost with other information about the resource.
There are several different ways to look at your node cost data. The default for the cost explorer is Unblended" but it makes the most sense from an allocation perspective to use the amortized rates. Be sure Amortized costs is selected when looking at cost data. Here's an example of how they can vary dramatically on our test cluster.
The t2-mediums here are covered by a savings plan. Unblended, the cost is only $0.06/day for two.
When Amortized costs is selected, the price jumps to $1.50/day
This should closely match our data on the Assets page, for days where we have adjustments come in from the pricing CUR.
By default, Kubecost pulls on-demand asset prices from the public AWS pricing API. For more accurate pricing, this integration will allow Kubecost to reconcile your current measured Kubernetes spend with your actual AWS bill. This integration also properly accounts for Enterprise Discount Programs, Reserved Instance usage, Savings Plans, Spot usage, and more.
You will need permissions to create the Cost and Usage Report (CUR), and add IAM credentials for Athena and S3. Optional permission is the ability to add and execute CloudFormation templates. Kubecost does not require root access in the AWS account.
This guide contains multiple possible methods for connecting Kubecost to AWS billing, based on user environment and preference. Because of this, there may not be a straightforward approach for new users. To address this, a streamlined guide containing best practices can be found here for IRSA environments. This quick start guide has some assumptions to carefully consider, and may not be applicable for all users. See prerequisites in the linked article.
Integrating your AWS account with Kubecost may be a complicated process if you aren’t deeply familiar with the AWS platform and how it interacts with Kubecost. This section provides an overview of some of the key terminology and AWS services that are involved in the process of integration.
Cost and Usage Report: AWS report which tracks cloud spending and writes to an Amazon Simple Storage Service (Amazon S3) bucket for ingestion and long term historical data. The CUR is originally formatted as a CSV, but when integrated with Athena, is converted to Parquet format.
Amazon Athena: Analytics service which queries the CUR S3 bucket for your AWS cloud spending, then outputs data to a separate S3 bucket. Kubecost uses Athena to query for the bill data to perform reconciliation. Athena is technically optional for AWS cloud integration, but as a result, Kubecost will only provide unreconciled costs (on-demand public rates).
S3 bucket: Cloud object storage tool which both CURs and Athena output cost data to. Kubecost needs access to these buckets in order to read that data.
For the below guide, a GitHub repository with sample files can be found here.
Follow these steps to set up a Legacy CUR using the settings below.
Select the Legacy CUR export type.
For time granularity, select Daily.
Under 'Additional content', select the Enable resource IDs checkbox.
Under 'Report data integration' select the Amazon Athena checkbox.
For CUR data written to an S3 bucket only accessed by Kubecost, it is safe to expire or delete the objects after seven days of retention.
Remember the name of the bucket you create for CUR data. This will be used in Step 2.
Familiarize yourself with how column name restrictions differ between CURs and Athena tables. AWS may change your CUR name when you upload your CUR to your Athena table in Step 2, documented in AWS' Running Amazon Athena queries. As best practice, use all lowercase letters and only use _ as a special character.
AWS may take up to 24 hours to publish data. Wait until this is complete before continuing to the next step.
If you believe you have the correct permissions, but cannot access the Billing and Cost Management page, have the owner of your organization's root account follow these instructions.
As part of the CUR creation process, Amazon also creates a CloudFormation template that is used to create the Athena integration. It is created in the CUR S3 bucket, listed in the Objects tab in the path s3-path-prefix/cur-name and typically has the filename crawler-cfn.yml. This .yml is your necessary CloudFormation template. You will need it in order to complete the CUR Athena integration. For more information, see the AWS doc Setting up Athena using AWS CloudFormation templates.
Your S3 path prefix can be found by going to your AWS Cost and Usage Reports dashboard and selecting your newly-created CUR. In the 'Report details' tab, you will find the S3 path prefix.
Once Athena is set up with the CUR, you will need to create a new S3 bucket for Athena query results.
Navigate to the S3 Management Console.
Select Create bucket. The Create Bucket page opens.
Use the same region used for the CUR bucket and pick a name that follows the format aws-athena-query-results-.
Select Create bucket at the bottom of the page.
Navigate to the Amazon Athena dashboard.
Select Settings, then select Manage. The Manage settings window opens.
Set Location of query result to the S3 bucket you just created, which will look like s3://aws-athena-query-results..., then select Save.
For Athena query results written to an S3 bucket only accessed by Kubecost, it is safe to expire or delete the objects after 1 day of retention.
Kubecost offers a set of CloudFormation templates to help set your IAM roles up.
If you’re new to provisioning IAM roles, we suggest downloading our templates and using the CloudFormation wizard to set these up. You can learn how to do this in AWS' Creating a stack on the AWS CloudFormation console doc. Open the step below which represents your CUR and management account arrangement, download the .yaml file listed, and upload them as the stack template in the 'Creating a stack' > 'Selecting a stack template' step.
If you are using the alternative multi-cloud integration method, steps 4 and 5 are not required.
Now that the policies have been created, attach those policies to Kubecost. We support the following methods:
These values can either be set from the Kubecost UI or via .Values.kubecostProductConfigs in the Helm chart. Values for all fields must be provided.
To add values in the Kubecost UI, select Settings from the left navigation, then scroll to Cloud Cost Settings. Select Update next to External Cloud Cost Configuration (AWS). The Billing Data Export Configuration window opens. Fill in all the below fields:
When you are done, select Update to confirm.
If you set any kubecostProductConfigs from the Helm chart, all changes via the front end will be overridden on pod restart.
athenaProjectID: The AWS AccountID where the Athena CUR is, likely your management account.
athenaBucketName: An S3 bucket to store Athena query results that you’ve created that Kubecost has permission to access
The name of the bucket should match s3://aws-athena-query-results-*, so the IAM roles defined above will automatically allow access to it
The bucket can have a Canned ACL of Private or other permissions as you see fit.
athenaRegion: The AWS region Athena is running in
athenaDatabase: The name of the database created by the Athena setup
The athena database name is available as the value (physical id) of AWSCURDatabase in the CloudFormation stack created above (in Step 2: Setting up Athena)
athenaTable: the name of the table created by the Athena setup
The table name is typically the database name with the leading athenacurcfn_ removed (but is not available as a CloudFormation stack resource). Confirm the table name by visiting the Athena dashboard.
athenaWorkgroup: The workgroup assigned to be used with Athena. If not specified, defaults to Primary
Make sure to use only underscore as a delimiter if needed for tables and views. Using a hyphen/dash will not work even though you might be able to create it. See the AWS docs for more info.
If you are using a multi-account setup, you will also need to set .Values.kubecostProductConfigs.masterPayerARN to the Amazon Resource Number (ARN) of the role in the management account, e.g. arn:aws:iam::530337586275:role/KubecostRole.
Once you've integrated with the CUR, you can visit Settings > View Full Diagnostics in the UI to determine if Kubecost has been successfully integrated with your CUR. If any problems are detected, you will see a yellow warning sign under the cloud provider permissions status header
You can check pod logs for authentication errors by running: kubectl get pods -n <namespace> kubectl logs <kubecost-pod-name> -n <namespace> -c cost-model
If you do not see any authentication errors, log in to your AWS console and visit the Athena dashboard. You should be able to find the CUR. Ensure that the database with the CUR matches the athenaTable entered in Step 5. It likely has a prefix with athenacurcfn_ :
You can also check query history to see if any queries are failing:
Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources. You can search in the Athena "Recent queries" dashboard to find additional info about the error.
Resolution: This error is typically caused by the incorrect (Athena results) s3 bucket being specified in the CloudFormation template of Step 3 from above. To resolve the issue, ensure the bucket used for storing the AWS CUR report (Step 1) is specified in the S3ReadAccessToAwsBillingData SID of the IAM policy (default: kubecost-athena-access) attached to the user or role used by Kubecost (Default: KubecostUser / KubecostRole). See the following example.
This error can also occur when the management account cross-account permissions are incorrect, however, the solution may differ.
Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources.
Resolution: Please verify that the prefix s3:// was used when setting the athenaBucketName Helm value or when configuring the bucket name in the Kubecost UI.
Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources.
Resolution: While rare, this issue was caused by an Athena instance that failed to provision properly on AWS. The solution was to delete the Athena DB and deploy a new one. To verify this is needed, find the failed query ID in the Athena "Recent queries" dashboard and attempt to manually run the query.
Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources.
Resolution: Previously, if you ran a query without specifying a value for query result location, and the query result location setting was not overridden by a workgroup, Athena created a default location for you. Now, before you can run an Athena query in a region in which your account hasn't used Athena previously, you must specify a query result location, or use a workgroup that overrides the query result location setting. While Athena no longer creates a default query results location for you, previously created default aws-athena-query-results-MyAcctID-MyRegion locations remain valid and you can continue to use them. The bucket should be in the format of: aws-athena-query-results-MyAcctID-MyRegion It may also be required to remove and reinstall Kubecost. If doing this please remeber to backup ETL files prior or contact support for additional assistance. See also this AWS doc on specifying a query result location.
Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources or in the Kubecost cost-model container logs.
Resolution: Verify in AWS' Cost and Usage Reports dashboard that the Resource IDs are enabled as "Report content" for the CUR created in Step 1. If the Resource IDs are not enabled, you will need to re-create the report (this will require redoing Steps 1 and 2 from this doc).
Symptom: A similar error to this will be shown on the Diagnostics page under Pricing Sources or in the Kubecost cost-model container logs.
Resolution: Verify that s3:// was included in the bucket name when setting the .Values.kubecostProductConfigs.athenaBucketName Helm value.
AWS services used here are:
Kubecost's cost-model requires roughly 2 CPU and 10 GB of RAM per 50,000 pods monitored. The backing Prometheus database requires roughly 2 CPU and 25 GB per million metrics ingested per minute. You can pick the EC2 instances necessary to run Kubecost accordingly.
Kubecost can write its cache to disk. Roughly 32 GB per 100,000 pods monitored is sufficient. (Optional: our cache can exist in memory)
Cloudformation (Optional: manual IAM configuration or via Terraform is fine)
EKS (Optional: all K8s flavors are supported)
Connecting your Azure account to Kubecost allows you to view Kubernetes metrics side-by-side with out-of-cluster (OOC) costs (e.g. Azure Database Services). Additionally, it allows Kubecost to reconcile measured Kubernetes spend with your actual Azure bill. This gives teams running Kubernetes a complete and accurate picture of costs. For more information, read Cloud Billing Integrations and this blog post.
To configure Kubecost's Azure Cloud Integration, you will need to set up daily exports of cost reports to Azure storage. Kubecost will then access your cost reports through the Azure Storage API to display your OOC cost data alongside your in-cluster costs.
A GitHub repository with sample files used in below instructions can be found here.
Follow Azure's Create and Manage Exported Data tutorial to export cost reports. For Metric, make sure you select Amortized cost (Usage and Purchases). For Export type, make sure you select Daily export of month-to-date costs. Do not select File Partitioning. Also, take note of the Account name and Container specified when choosing where to export the data to. Note that a successful cost export will require Microsoft.CostManagementExports to be registered in your subscription.
Alternatively, you can follow this Kubecost guide.
It will take a few hours to generate the first report, after which Kubecost can use the Azure Storage API to pull that data.
Once the cost export has successfully executed, verify that a non-empty CSV file has been created at this path: <STORAGE_ACCOUNT>/<CONTAINER_NAME>/<OPTIONAL_CONTAINER_PATH>/<COST_EXPORT_NAME>/<DATE_RANGE>/<CSV_FILE>.
If you have sensitive data in an existing Azure Storage account, it is recommended to create a separate Azure Storage account to store your cost data export.
For more granular billing data it is possible to scope Azure cost exports to resource groups, management groups, departments, or enrollments. AKS clusters will create their own resource groups which can be used. This functionality can then be combined with Kubecost multi-cloud to ingest multiple scoped billing exports.
Obtain the following values from Azure to provide to Kubecost. These values can be located in the Azure Portal by selecting Storage Accounts, then selecting your specific Storage account for details.
azureSubscriptionID is the "Subscription ID" belonging to the Storage account which stores your exported Azure cost report data.
azureStorageAccount is the name of the Storage account where the exported Azure cost report data is being stored.
azureStorageAccessKey can be found by selecting Access keys in your Storage account left navigation under "Security + networking". Using either of the two keys will work.
azureStorageContainer is the name that you chose for the exported cost report when you set it up. This is the name of the container where the CSV cost reports are saved in your Storage account.
azureContainerPath is an optional value which should be used if there is more than one billing report that is exported to the configured container. The path provided should have only one billing export because Kubecost will retrieve the most recent billing report for a given month found within the path.
azureCloud is an optional value which denotes the cloud where the storage account exist, possible values are public and gov. The default is public.
Next, create a JSON file which must be named cloud-integration.json with the following format:
Additional details about the cloud-integration.json file can be found in our multi-cloud integration doc.
Next, create the Secret:
Next, ensure the following are set in your Helm values:
Next, upgrade Kubecost via Helm:
You can verify a successful configuration by checking the following in the Kubecost UI:
The Assets dashboard will be broken down by Kubernetes assets.
The Assets dashboard will no longer show a banner that says "External cloud cost not configured".
The Diagnostics page (via Settings > View Full Diagnostics) view will show a green checkmark under Cloud Integrations.
If there are no in-cluster costs for a particular day, then there will not be out-of-cluster costs either
Kubecost utilizes Azure tagging to allocate the costs of Azure resources outside of the Kubernetes cluster to specific Kubernetes concepts, such as namespaces, pods, etc. These costs are then shown in a unified dashboard within the Kubecost interface.
To allocate external Azure resources to a Kubernetes concept, use the following tag naming scheme:
In the kubernetes_label_NAME tag key, the NAME portion should appear exactly as the tag appears inside of Kubernetes. For example, for the tag app.kubernetes.io/name, this tag key would appear as kubernetes_label_app.kubernetes.io/name.
To use an alternative or existing Azure tag schema, you may supply these in your values.yaml under the kubecostProductConfigs.labelMappingConfigs.<aggregation>_external_label . Also be sure to set kubecostProductConfigs.labelMappingConfigs.enabled = true
For more details on what Azure resources support tagging, along with what resource type tags are available in cost reports, please review the official Microsoft documentation here.
To troubleshoot a configuration that is not yet working:
$ kubectl get secrets -n kubecost to verify you've properly configured cloud-integration.json.
$ helm get values kubecost to verify you've properly set .Values.kubecostProductConfigs.cloudIntegrationSecret
Verify that a non-empty CSV file has been created at this path in your Azure Portal Storage Account: <STORAGE_ACCOUNT>/<CONTAINER_NAME>/<OPTIONAL_CONTAINER_PATH>/<COST_EXPORT_NAME>/<DATE_RANGE>/<CSV_FILE>. Ensure new CSVs are being generated every day.
When opening a cost report CSV, ensure that there are rows in the file that do not have a MeterCategory of “Virtual Machines” or “Storage” as these items are ignored because they are in cluster costs. Additionally, make sure that there are items with a UsageDateTime that matches the date you are interested in.
When reviewing logs:
The following error is reflective of Kubecost's previous Azure Cloud Integration method and can be safely disregarded.
ERR Error, Failed to locate azure storage config file: /var/azure-storage-config/azure-storage-config.json
Integrating Kubecost with your AWS data provides the ability to allocate out-of-cluster (OOC) costs, e.g. RDS instances and S3 buckets, back to Kubernetes concepts like namespace and deployment as well as reconcile cluster assets back to your billing data. The latter is especially helpful when teams are using Reserved Instances, Savings Plans, or Enterprise Discounts. All billing data remains on your cluster when using this functionality and is not shared externally. Read our Cloud Integrations doc for more information on how Kubecost connects with Cloud Service Providers.
The following guide provides the steps required for enabling OOC costs allocation and accurate pricing, e.g. reserved instance price allocation. In a multi-account organization, all of the following steps will need to be completed in the payer account.
You can learn how to perform this using our AWS Cloud Integration doc.
Kubecost utilizes AWS tagging to allocate the costs of AWS resources outside of the Kubernetes cluster to specific Kubernetes concepts, such as namespaces, pods, etc. These costs are then shown in a unified dashboard within the Kubecost interface.
To allocate external AWS resources to a Kubernetes concept, use the following tag naming scheme:
| Kubernetes Concept | AWS Tag Key | AWS Tag Value |
|---|---|---|
In the kubernetes_label_NAME tag key, the NAME portion should appear exactly as the tag appears inside of Kubernetes. For example, for the tag app.kubernetes.io/name, this tag key would appear as kubernetes_label_app.kubernetes.io/name.
To use an alternative or existing AWS tag schema, you may supply these in your values.yaml under kubecostProductConfigs.labelMappingConfigs.\<aggregation\>\_external_label. Also be sure to set kubecostProductConfigs.labelMappingConfigs.enabled=true.
For more information, consult AWS' Tag your Amazon EC2 resources.
Tags may take several hours to show up in the Cost Allocations Tags section described in the next step.
Tags that contain : in the key may be converted to _ in the Kubecost UI due to Prometheus readability. To use AWS Label Mapping Configs, use this mapping format:
To view examples of common label mapping configs, see here.
In order to make the custom Kubecost AWS tags appear on the CURs, and therefore in Kubecost, individual cost allocation tags must be enabled. Details on which tags to enable can be found in Step 2.
For instructions on enabling user-defined cost allocation tags, consult AWS' Activating user-defined cost allocation tags
Account-level tags are applied (as labels) to all the Assets built from resources defined under a given AWS account. You can filter AWS resources in the Kubecost Assets View (or API) by account-level tags by adding them ('tag:value') in the Label/Tag filter.
If a resource has a label with the same name as an account-level tag, the resource label value will take precedence.
Modifications incurred on account-level tags may take several hours to update on Kubecost.
Your AWS account will need to support the organizations:ListAccounts and organizations:ListTagsForResource policies to benefit from this feature.
In the Kubecost UI, view the Allocations dashboard. If external costs are not shown, open your browser's Developer Tools > Console to see any reported errors.
Query Athena directly to ensure data is available. Note: it can take up to 6 hours for data to be written.
You may need to upgrade your AWS Glue if you are running an old version. See Upgrading to the AWS Glue Data Catalog step-by-step for more info.
Finally, review pod logs from the cost-model container in the cost-analyzer pod and look for auth errors or Athena query results.
Integration with cloud service providers (CSPs) via their respective billing APIs allows Kubecost to display out-of-cluster (OOC) costs (e.g. AWS S3, Google Cloud Storage, Azure Storage Account). Additionally, it allows Kubecost to reconcile Kubecost's in-cluster predictions with actual billing data to improve accuracy.
If you are using Kubecost Cloud, do not attempt to modify your install using information from this article. You need to consult Kubecost Cloud's specific cloud integration procedures which can be found here.
As indicated above, setting up a cloud integration with your CSP allows Kubecost to pull in additional billing data. The two processes that incorporate this information are reconciliation and CloudCost (formerly known as CloudUsage).
Reconciliation matches in-cluster assets with items found in the billing data pulled from the CSP. This allows Kubecost to display the most accurate depiction of your in-cluster spending. Additionally, the reconciliation process creates Network assets for in-cluster nodes based on the information in the billing data. The main drawback of this process is that the CSPs have between a 6 to 24-hour delay in releasing billing data, and reconciliation requires a complete day of cost data to reconcile with the in-cluster assets. This requires a 48-hour window between resource usage and reconciliation. If reconciliation is performed within this window, asset cost is deflated to the partially complete cost shown in the billing data.
Cost-based metrics are based on on-demand pricing unless there is definitive data from a CSP that the node is not on-demand. This way estimates are as accurate as possible. If a new reserved instance is provisioned or a node joins a savings plan:
Kubecost continues to emit on-demand pricing until the node is added to the cloud bill.
Once the node is added to the cloud bill, Kubecost starts emitting something closer to the actual price.
For the time period where Kubecost assumed the node was on-demand but it was actually reserved, reconciliation fixes the price in ETL.
The reconciled assets will inherit the labels from the corresponding items in the billing data. If there exist identical label keys between the original assets and those of the billing data items, the label value of the original asset will take precedence.
Visit Settings, then toggle on Highlight Unreconciled Costs, then select Save at the bottom of the page to apply changes. Now, when you visit your Allocations or Assets dashboards, the most recent 36 hours of data will display hatching to signify unreconciled costs.
As of v1.106 of Kubecost, CloudCost is enabled by default, and Cloud Usage is disabled. Upgrading Kubecost will not affect the UI or hinder performance relating to this.
CloudCost allows Kubecost to pull in OOC cloud spend from your CSP's billing data, including any services run by the CSP as well as compute resources. By labelling OOC costs, their value can be distributed to your Allocations data as external costs. This allows you to better understand the proportion of OOC cloud spend that your in-cluster usage depends on.
Your cloud billing data is reflected in the aggregate costs of Account, Provider, Invoice Entity, and Service. Aggregating and drilling down into any of these categories will provide a subset of the entire bill, based on the Helm value .values.cloudCost.topNItems, which will log 1,000 values. This subset is each days' top n items by cost. An optional label list can be used to include or exclude items to be pulled from the bill.
CloudCost becomes available as soon as they appear in the billing data, with the 6 to 24-hour delay mentioned above, and are updated as they become more complete.
You can view your existing cloud integrations and their success status in the Kubecost UI by visiting Settings, then scrolling to Cloud Integrations. To create a new integration or learn more about existing integrations, select View additional details to go to the Cloud Integrations page.
Here, you can view your integrations and filter by successful or failed integrations. For non-successful integrations, Kubecost will display a diagnostic error message in the Status column to contextualize steps toward successful integration.
Select an individual integration to view a side panel that contains the most recent run, next run, refresh rate, and an exportable YAML of Helm configs for its CSP's integration values.
You can add a new cloud integration by selecting Add Integration. For guides on how to set up an integration for a specific CSP, follow these links to helpful Kubecost documentation:
Select an existing cloud integration, then in the slide panel that appears, select Delete.
The Kubecost Helm chart provides values that can enable or disable each cloud process on the deployment once a cloud integration has been set up. Turning off either of these processes will disable all the benefits provided by them.
Often an integrated cloud account name may be a series of random letter and numbers which do not reflect the account's owner, team, or function. Kubecost allows you to rename cloud accounts to create more readable cloud metrics in your Kubecost UI. After you have successfully integrated your cloud account (see above), you need to manually edit your values.yaml and provide the original account name and your intended rename:
You will see these changes reflected in Kubecost's UI on the Overview page under Cloud Costs Breakdown. These example account IDs could benefit from being renamed:
The ETL contains a Map of Cloud Stores, each representing an integration with a CSP. Each Cloud Store is responsible for the Cloud Usage and reconciliation pipelines which add OOC costs and adjust Kubecost's estimated cost respectively by cost and usage data pulled from the CSP. Each Cloud Store has a unique identifier called the ProviderKey which varies depending on which CSP is being connected to and ensures that duplicate configurations are not introduced into the ETL. The value of the ProviderKey is the following for each CSP at a scope that the billing data is being for:
AWS: Account Id
GCP: Project Id
Azure: Subscription Id
The ProviderKey can be used as an argument for the endpoints for Cloud Usage and Reconciliation repair APIs, to indicate that the specified operation should only be done on a single Cloud Store rather than all of them, which is the default behavior. Additionally, the Cloud Store keeps track of the Status of the Cloud Connection Diagnostics for each of the Cloud Usage and reconciliation. The Cloud Connection Status is meant to be used as a tool in determining the health of the Cloud Connection that is the basis of each Cloud Store. The Cloud Connection Status has various failure states that are meant to provide actionable information on how to get your Cloud Connection running properly. These are the Cloud Connection Statuses:
INITIAL_STATUS: The zero value of Cloud Connection Status means that the cloud connection is untested. Once Cloud Connection Status has been changed and it should not return to this value. This status is assigned on creation to the Cloud Store
MISSING_CONFIGURATION: Kubecost has not detected any method of Cloud Configuration. This value is only possible on the first Cloud Store that is created as a wrapper for the open-source CSP. This status is assigned during failures in Configuration Retrieval.
INCOMPLETE_CONFIGURATION: Cloud Configuration is missing the required values to connect to the cloud provider. This status is assigned during failures in Configuration Retrieval.
FAILED_CONNECTION: All required Cloud Configuration values are filled in, but a connection with the CSP cannot be established. This is indicative of a typo in one of the Cloud Configuration values or an issue in how the connection was set up in the CSP's Console. The assignment of this status varies between CSPs but should happen if there if an error is thrown when an interaction with an object from the CSP's SDK occurs.
MISSING_DATA: The Cloud Integration is properly configured, but the CSP is not returning billing/cost and usage data. This status is indicative of the billing/cost and usage data export of the CSP being incorrectly set up or the export being set up in the last 48 hours and not having started populating data yet. This status is set when a query has been successfully made but the results come back empty. If the CSP already has a SUCCESSFUL_CONNECTION status, then this status should not be set because this indicates that the specific query made may have been empty.
SUCCESSFUL_CONNECTION: The Cloud Integration is properly configured and returning data. This status is set on any successful query where data is returned
After starting or restarting Cloud Usage or reconciliation, two subprocesses are started: one which fills in historic data over the coverage of the Daily CloudUsage and Asset Store, and one which runs periodically on a predefined interval to collect and process new cost and usage data as it is made available by the CSP. The ETL's status endpoint contains a cloud object that provides information about each Cloud Store including the Cloud Connection Status and diagnostic information about Cloud Usage and Reconciliation. The diagnostic items on the Cloud Usage and Reconciliation are:
Coverage: The window of time that the historical subprocess has covered
LastRun: The last time that the process ran, updates each time the periodic subprocess runs
NextRun: Next scheduled run of the periodic subprocess
Progress: Ratio of Coverage to Total amount of time to be covered
RefreshRate: The interval that the periodic subprocess runs
Resolution: The window size of the process
StartTime: When the Cloud Process was started
For more information on APIs related to rebuilding and repairing Cloud Usage or reconciliation, see the CloudCost Diagnostic APIs doc.
Certain features of Kubecost, including Savings Insights like Orphaned Resources and Reserved Instances, require access to the cluster's GCP account. This is usually indicated by a 403 error from Google APIs which is due to 'insufficient authentication scopes'. Viewing this error in the Kubecost UI will display the cause of the error as "ACCESS_TOKEN_SCOPE_INSUFFICIENT".
To obtain access to these features, follow this tutorial which will show you how to configure your Google IAM Service Account and Workload Identity for your application.
Go to your GCP Console and select APIs & Services > Credentials from the left navigation. Select + Create Credentials > API Key.
On the Credentials page, select the icon in the Actions column for your newly-created API key, then select Edit API key. The Edit API key page opens.
Under ‘API restrictions’, select Restrict key, then from the dropdown, select only Cloud Billing API. Select OK to confirm. Then select Save at the bottom of the page.
From here, consult Google Cloud's guide to perform the following steps:
Enable Workload Identity on an existing GCP cluster, or spin up a new cluster which will have Workload Identity enabled by default
Migrate any existing workloads to Workload Identity
Configure your applications to use Workload Identity
Create both a Kubernetes service account (KSA) and an IAM service account (GSA).
Annotate the KSA with the email of the GSA.
Update your pod spec to use the annotated KSA, and ensure all nodes on that workload use Workload Identity.
You can stop once you have modified your pod spec (before 'Verify the Workload Identity Setup'). You should now have a GCP cluster with Workload Identity enabled, and both a KSA and a GSA, which are connected via the role roles/iam.workloadIdentityUser.
In the GCP Console, select IAM & Admin > IAM. Find your newly-created GSA and select the Edit Principal pencil icon. You will need to provide the following roles to this service account:
BigQuery Data Viewer
BigQuery Job User
BigQuery User
Compute Viewer
Service Account Token Creator
Select Save.
The following roles need to be added to your IAM service account:
roles/bigquery.user
roles/compute.viewer
roles/bigquery.dataViewer
roles/bigquery.jobUser
roles/iam.serviceAccountTokenCreator
Use this command to add each role individually to the GSA:
From here, restart the pod(s) to confirm your changes. You should now have access to all expected Kubecost functionality through your service account with Identity Workload.
In order to create a Google service account for use with Thanos, navigate to the and select IAM & Admin > Service Accounts.
From here, select the option Create Service Account.
Provide a service account name, ID, and description, then select Create and Continue.
You should now be at the Service account permissions (optional) page. Select the first Role dropdown and select Storage Object Creator. Select Add Another Role, then select Storage Object Viewer from the second dropdown. Select Continue.
You should now be prompted to allow specific accounts access to this service account. This should be based on specific internal needs and is not a requirement. You can leave this empty and select Done.
Once back to the Service accounts page, select the Actions icon > Manage keys. Then, select the Add Key dropdown and select Create new key. A Create private key window opens.
Select JSON as the Key type and select Create. This will download a JSON service account key entry for use with the Thanos object-store.yaml mentioned in the initial setup step.
Kubecost needs access to the Microsoft Azure Billing Rate Card API to access accurate pricing data for your Kubernetes resources.
You can also get this functionality plus external costs by completing the full .
Start by creating an Azure role definition. Below is an example definition, replace YOUR_SUBSCRIPTION_ID with the Subscription ID where your Kubernetes cluster lives:
Save this into a file called myrole.json.
Next, you'll want to register that role with Azure:
Next, create an Azure service principal.
Keep this information which is used in the service-key.json below.
Next, create a Secret for the Azure Service Principal
When managing the service account key as a Kubernetes Secret, the secret must reference the service account key JSON file, and that file must be named service-key.json.
Finally, set the kubecostProductConfigs.serviceKeySecretName Helm value to the name of the Kubernetes secret you created. We use the value azure-service-key in our examples.
Or at the command line:
Kubecost supports querying the Azure APIs for cost data based on the region, offer durable ID, and currency defined in your Microsoft Azure offer.
Those properties are configured with the following Helm values:
kubecostProductConfigs.azureBillingRegion
kubecostProductConfigs.azureOfferDurableID
kubecostProductConfigs.currencyCode
Be sure to verify your billing information with Microsoft and update the above Helm values to reflect your bill to country, subscription offer durable ID/number, and currency.
The following Microsoft documents are a helpful reference:
Kubecost provides the ability to allocate out-of-cluster (OOC) costs, e.g. Cloud SQL instances and Cloud Storage buckets, back to Kubernetes concepts like namespaces and deployments.
Read the doc for more information on how Kubecost connects with cloud service providers.
The following guide provides the steps required for allocating OOC costs in GCP.
A GitHub repository with sample files used in the below instructions can be found .
Begin by reviewing on exporting cloud billing data to BigQuery.
GCP users must create a to gain access to all Kubecost CloudCost features including . Exports of type "Standard usage cost data" and "Pricing Data" do not have the correct information to support CloudCosts.
If you are using the alternative method, Step 2 is not required.
If your Big Query dataset is in a different project than the one where Kubecost is installed, please see the section on .
Add a service account key to allocate OOC resources (e.g. storage buckets and managed databases) back to their Kubernetes owners. The service account needs the following:
If you don't already have a GCP service account with the appropriate rights, you can run the following commands in your command line to generate and export one. Make sure your GCP project is where your external costs are being run.
After creating the GCP service account, you can connect it to Kubecost in one of two ways before configuring:
NAMESPACE is the namespace Kubecost is installed into
KSA_NAME is the name of the service account attributed to the Kubecost deployment
Create a service account key:
Once the GCP service account has been connected, set up the remaining configuration parameters.
You're almost done. Now it's time to configure Kubecost to finalize your connectivity.
If you've connected using Workload Identity Federation, add these configs:
Otherwise, if you've connected using a service account key, create a secret for the GCP service account key you've created and add the following configs:
When managing the service account key as a Kubernetes secret, the secret must reference the service account key JSON file, and that file must be named compute-viewer-kubecost-key.json.
In Kubecost, select Settings from the left navigation, and under Cloud Integrations, select Add Cloud Integration > GCP, then provide the relevant information in the GCP Billing Data Export Configuration window:
GCP Service Key: Optional field. If you've created a service account key, copy the contents of the compute-viewer-kubecost-key.json file and paste them here. If you've connected using Workload Identity federation in Step 3, you should leave this box empty.
GCP Project Id: The ID of your GCP project.
GCP Billing Database: Requires a BigQuery dataset prefix (e.g. billing_data) in addition to the BigQuery table name. A full example is billing_data.gcp_billing_export_resource_v1_XXXXXX_XXXXXX_XXXXX
Be careful when handling your service key! Ensure you have entered it correctly into Kubecost. Don't lose it or let it become publicly available.
Google generates special labels for GKE resources (e.g. "goog-gke-node", "goog-gke-volume"). Values with these labels are excluded from OOC costs because Kubecost already includes them as in-cluster assets. Thus, to make sure all cloud assets are included, we recommend installing Kubecost on each cluster where insights into costs are required.
If a resource has a label with the same name as a project-level label, the resource label value will take precedence.
Modifications incurred on project-level labels may take several hours to update on Kubecost.
Due to organizational constraints, it is common that Kubecost must be run in a separate project from the project containing the billing data Big Query dataset, which is needed for Cloud Integration. Configuring Kubecost in this scenario is still possible, but some of the values in the above script will need to be changed. First, you will need the project id of the projects where Kubecost is installed, and the Big Query dataset is located. Additionally, you will need a GCP user with the permissions iam.serviceAccounts.setIamPolicy for the Kubecost project and the ability to manage the roles listed above for the Big Query Project. With these, fill in the following script to set the relevant variables:
Once these values have been set, this script can be run and will create the service account needed for this configuration.
Now that your service account is created follow the normal configuration instructions.
InvalidQuery 400 error for GCP integrationIn cases where Kubecost does not detect a connection following GCP integration, revisit Step 1 and ensure you have enabled detailed usage cost, not standard usage cost. Kubecost uses detailed billing cost to display your OOC spend, and if it was not configured correctly during installation, you may receive errors about your integration.
| Field | Description |
|---|---|
| Kubernetes Concept | Azure Tag Key | Azure Tag Value |
|---|---|---|
| Value | Default | Description |
|---|---|---|
Create a file called and update it with the Service Principal details from the above steps:
In the :
You can set up an to bind a Kubernetes service account to your GCP service account as seen below, where:
You will also need to enable the in the GCP project.
It is recommended to provide the GCP details in your to ensure they are retained during an upgrade or redeploy. First, set the following configs:
You can now label assets with the following schema to allocate costs back to their appropriate Kubernetes owner. Learn more on updating GCP asset labels.
To use an alternative or existing label schema for GCP cloud assets, you may supply these in your under the kubecostProductConfigs.labelMappingConfigs.<aggregation>_external_label.
Project-level labels are applied to all the Assets built from resources defined under a given GCP project. You can filter GCP resources in the Kubecost (or ).
There are cases where labels applied at the account label do not show up in the date-partitioned data. If account level labels are not showing up, you can switch to querying them unpartitioned by setting an extraEnv in Kubecost: GCP_ACCOUNT_LABELS_NOT_PARTITIONED: true. See .
Cluster
kubernetes_cluster
cluster-name
Namespace
kubernetes_namespace
namespace-name
Deployment
kubernetes_deployment
deployment-name
Label
kubernetes_label_NAME*
label-value
DaemonSet
kubernetes_daemonset
daemonset-name
Pod
kubernetes_pod
pod-name
Container
kubernetes_container
container-name
.Values.kubecostModel.etlAssetReconciliationEnabled
true
Enables reconciliation processes and endpoints. This Helm value corresponds to the ETL_ASSET_RECONCILIATION_ENABLED environment variable.
.Values.kubecostModel.etlCloudUsage
true
Enables Cloud Usage processes and endpoints. This Helm value corresponds to the ETL_CLOUD_USAGE_ENABLED environment variable.
.Values.kubecostModel.etlCloudRefreshRateHours
6
The interval at which the run loop executes for both reconciliation and Cloud Usage. Reducing this value will decrease resource usage and billing data access costs, but will result in a larger delay in the most current data being displayed. This Helm value corresponds to the ETL_CLOUD_REFRESH_RATE_HOURS environment variable.
.Values.kubecostModel.etlCloudQueryWindowDays
7
The maximum number of days that will be queried from a cloud integration in a single query. Reducing this value can help to reduce memory usage during the build process, but will also result in more queries which can drive up billing data access costs. This Helm value corresponds to the ETL_CLOUD_QUERY_WINDOW_DAYS environment variable.
.Values.kubecostModel.etlCloudRunWindowDays
3
The number of days into the past each run loop will query. Reducing this value will reduce memory load, however, it can cause Kubecost to miss updates to the CUR, if this has happened the day will need to be manually repaired. This Helm value corresponds to the ETL_CLOUD_RUN_WINDOW_DAYS environment variable.
Athena Region
The AWS region Athena is running in
Athena Database
The name of the database created by the Athena setup
Athena Tablename
The name of the table created by the Athena setup
Athena Result Bucket
An S3 bucket to store Athena query results that you’ve created that Kubecost has permission to access
AWS account ID
The AWS account ID where the Athena CUR is, likely your management account.
Cluster
kubernetes_cluster
cluster-name
Namespace
kubernetes_namespace
namespace-name
Deployment
kubernetes_deployment
deployment-name
Label
kubernetes_label_NAME*
label-value
DaemonSet
kubernetes_daemonset
daemonset-name
Pod
kubernetes_pod
pod-name
Container
kubernetes_container
container-name
