Search…
⌃K
Links

AWS Cloud Integration

By default, Kubecost pulls On-Demand asset prices from the public AWS pricing API. For more accurate pricing, this integration will allow Kubecost to reconcile your current measured Kubernetes spend with your actual AWS bill. This integration also properly accounts for Enterprise Discount Programs, Reserved Instance usage, Savings Plans, spot usage, and more.
You will need permissions to create the Cost and Usage Report (CUR), and add IAM credentials for Athena and S3. Optional permission is the ability to add and execute CloudFormation templates. Kubecost does not require root access in the AWS account.
A Github repository with sample files which follow the below instructions can be found here.

Cost and Usage Report integration

Step 1: Setting up the CUR

Follow these steps to set up a CUR. For time granularity, select Daily. Be sure to enable Resource IDs and Athena integration when creating the CUR.
Remember the name of the bucket you create for CUR data. This will be used in Step 2.
Note: If you believe you have the correct permissions, but cannot access the Billing and Cost Management page, have the owner of your organization's root account follow these instructions
AWS may take up to 24 hours to publish data. Wait until this is complete before continuing to the next step.

Step 2: Setting up Athena

As part of the CUR creation process, Amazon also creates a CloudFormation template that is used to create the Athena integration. It is created in the CUR S3 bucket under your-billing-prefix/cur-name and typically has the filename crawler-cfn.yml. You will need to deploy this CloudFormation template in order to complete the CUR Athena integration. You can read more about this here.
Once Athena is set up with the CUR, you will need to create a new S3 bucket for Athena query results.
  1. 1.
    Navigate to https://console.aws.amazon.com/s3
  2. 2.
    Select Create Bucket
  3. 3.
    Be sure to use the same region as was used for the CUR bucket and pick a name that follows the format aws-athena-query-results-*
  4. 4.
    Select Create Bucket
  5. 5.
    Navigate to https://console.aws.amazon.com/athena
  6. 6.
    Select Settings
  7. 7.
    Set Query result location to the S3 bucket you just created

Step 3: Setting up IAM permissions

Add via CloudFormation:

Kubecost offers a set of CloudFormation templates to help set your IAM roles up. If you’re new to provisioning IAM roles, we suggest downloading our templates and using the CloudFormation wizard to set these up, as explained here. Download template files from the URLs provided below and upload them as the stack template in the Creating a stack > Selecting a stack template step.
My Kubernetes clusters all run in the same account as the master payer account.
My Kubernetes clusters run in different accounts from the master payer account

Add manually:

My Kubernetes clusters run in the same account as the master payer account
My Kubernetes clusters run in different accounts

Step 4: Attaching IAM permissions to Kubecost

Note: If you are using the alternative multi-cloud integration method, steps 4 and 5 are not required. The use of "Attach via Pod Annotation on EKS" authentication is optional.
Now that the policies have been created, attach those policies to Kubecost. We support the following methods:
Attach via ServiceKey And Kubernetes Secret
Attach via Service Key on Kubecost frontend
Attach via Pod Annotation on EKS

Step 5: Provide CUR config values to Kubecost

These values can either be set from the kubecost frontend or via .Values.kubecostProductConfigs in the helm chart. Note that if you set any kubecostProductConfigs from the Helm chart, all changes via the frontend will be overridden on pod restart.
  • athenaProjectID e.g. "530337586277" # The AWS AccountID where the Athena CUR is. Generally your master payer account.
  • athenaBucketName An S3 bucket to store Athena query results that you’ve created that Kubecost has permission to access
    • The name of the bucket should match s3://aws-athena-query-results-*, so the IAM roles defined above will automatically allow access to it
    • The bucket can have a Canned ACL of Private or other permissions as you see fit.
  • athenaRegion The aws region athena is running in
  • athenaDatabase the name of the database created by the Athena setup
    • The athena database name is available as the value (physical id) of AWSCURDatabase in the CloudFormation stack created above (in Step 2: Setting up Athena)
  • athenaTable the name of the table created by the Athena setup
    • The table name is typically the database name with the leading athenacurcfn_ removed (but is not available as a CloudFormation stack resource)
  • athenaWorkgroup The workgroup assigned to be used with Athena. If not specified, defaults to Primary
Note: Make sure use only underscore as a delimiter if needed for tables and views, using dash will not work even though you might be able to create it see docs.
  • If you are using a multi-account setup, you will also need to set .Values.kubecostProductConfigs.masterPayerARN to the Amazon Resource Number (ARN) of the role in the masterpayer account, e.g. arn:aws:iam::530337586275:role/KubecostRole.

Troubleshooting

Once you've integrated with the CUR, you can visit /diagnostics.html in Kubecost to determine if Kubecost has been successfully integrated with your CUR. If any problems are detected, you will see a yellow warning sign under the cloud provider permissions status header:
Screen Shot 2020-12-06 at 9 37 40 PM
You can check pod logs for authentication errors by running: kubectl get pods -n <namespace> kubectl logs <kubecost-pod-name> -n <namespace> -c cost-model
If you do not see any authentication errors, log in to your AWS console and visit the Athena dashboard. You should be able to find the CUR. Ensure that the database with the CUR matches the athenaTable entered in step 4-- it likely has a prefix with athenacurfn_ :
Screen Shot 2020-12-06 at 9 43 31 PM
You can also check query history to see if any queries are failing:
Screen Shot 2020-12-06 at 9 43 50 PM

Common Athena errors.

Incorrect bucket in IAM Policy.

  • Symptom: A similar error to this will be shown on the diagnostics page under "Pricing Sources" on the "Diagnostics" page. You can search for the in the Athena "Recent queries" dashboard to find additional info about the error.
    QueryAthenaPaginated: query execution error: no query results available for query <Athena Query ID>
    And / or the following error will be found in the Kubecost cost-model container logs.
    Permission denied on S3 path: s3://cur-report/cur-report/cur-report/year=2022/month=8
    This query ran against the "athenacurcfn_test" database, unless qualified by the query. Please post the error message on our forum or contact customer support with Query Id: <Athena Query ID>
  • Resolution: This error is typically caused by the incorrect (Athena results) s3 bucket being specified in the cloudformation template of step 3 from above. To resolve the issue ensure the bucket used for storing the AWS CUR report (step 1) is specified in the S3ReadAccessToAwsBillingData SID of the IAM policy (default: kubecost-athena-access) attached to the user or role used by Kubecost (Default: KubecostUser / KubecostRole). See the following example. NOTE: This error can also occur when master payer cross account permissions are incorrect, that solution may differ.
    {
    "Action": [
    "s3:Get*",
    "s3:List*"
    ],
    "Resource": [
    "arn:aws:s3:::<AWS CUR BUCKET>*"
    ],
    "Effect": "Allow",
    "Sid": "S3ReadAccessToAwsBillingData"
    }

Query not supported

  • Symptom: A similar error to this will be shown on the diagnostics page under "Pricing Sources" on the "Diagnostics" page.
QueryAthenaPaginated: start query error: operation error Athena: StartQueryExecution, https response error StatusCode: 400, RequestID: <Athena Query ID>, InvalidRequestException: Queries of this type are not supported
  • Resolution: While rare, this issue was caused by and Athena instance which failed to provision properly on AWS. The solution was to delete the Athena DB and deploy a new one. To verify this is needed, find the failed query ID in the Athena "Recent queries" dashboard and attempt to manually run the query.

HTTPS Response error

  • Symptom: A similar error to this will be shown on the diagnostics page under "Pricing Sources" on the "Diagnostics" page.
QueryAthenaPaginated: start query error: operation error Athena: StartQueryExecution, https response error StatusCode: 400, RequestID: ********************, InvalidRequestException: Unable to verify/create output bucket aws-athena-query-results-test
  • Resolution: Previously, if you ran a query without specifying a value for Query result location, and the query result location setting was not overridden by a workgroup, Athena created a default location for you. Now, before you can run an Athena query in a region in which your account hasn't used Athena previously, you must specify a query result location, or use a workgroup that overrides the query result location setting. While Athena no longer creates a default query results location for you, previously created default aws-athena-query-results-MyAcctID-MyRegion locations remain valid and you can continue to use them. https://docs.aws.amazon.com/athena/latest/ug/querying.html#query-results-specify-location The bucket should be in the format of: aws-athena-query-results-MyAcctID-MyRegion It may also be required to remove and reinstall Kubecost. If doing this please remeber to backup ETL files prior or contact support for additional assistance.

Summary and pricing

AWS services used here are:
Kubecost's cost-model requires roughly 2 CPU and 10 GB of RAM per 50,000 pods monitored. The backing Prometheus database requires roughly 2 CPU and 25 GB per million metrics ingested per minute. You can pick the EC2 instances necessary to run Kubecost accordingly.
Kubecost can write its cache to disk. Roughly 32 GB per 100,000 pods monitored is sufficient. (Optional: our cache can exist in memory)
  • Cloudformation (Optional: manual IAM configuration or via Terraform is fine)
  • EKS (Optional: all K8s flavors are supported)