kubecost

Product and technical docs

View project on GitHub

Kubecost provides the ability to allocate out of clusters costs, e.g. Cloud SQL instances and Cloud Storage buckets, back to Kubernetes concepts like namespace and deployment. All data remains on your cluster when using this functionality and is not shared externally.

The following guide provides the steps required for allocating out of cluster costs.

Step 1: Enable billing data export

https://cloud.google.com/billing/docs/how-to/export-data-bigquery

Step 2: Visit Kubecost setup page and provide configuration info

Add a service key to allocate out of cluster resources (e.g. storage buckets and managed databases) back to their Kubernetes owners. The service account needs the following:

roles/bigquery.user
roles/compute.viewer
roles/bigquery.dataViewer
roles/bigquery.jobUser

If you don’t already have a GCP service key, you can run the following commands in your command line to generate and export one. Make sure your gcloud project is where your external costs are being run.

export PROJECT_ID=$(gcloud config get-value project)
gcloud iam service-accounts create compute-viewer-kubecost --display-name "Compute Read Only Account Created For Kubecost" --format json
gcloud projects add-iam-policy-binding $PROJECT_ID --member serviceAccount:compute-viewer-kubecost@$PROJECT_ID.iam.gserviceaccount.com --role roles/compute.viewer
gcloud projects add-iam-policy-binding $PROJECT_ID --member serviceAccount:compute-viewer-kubecost@$PROJECT_ID.iam.gserviceaccount.com --role roles/bigquery.user
gcloud projects add-iam-policy-binding $PROJECT_ID --member serviceAccount:compute-viewer-kubecost@$PROJECT_ID.iam.gserviceaccount.com --role roles/bigquery.dataViewer
gcloud projects add-iam-policy-binding $PROJECT_ID --member serviceAccount:compute-viewer-kubecost@$PROJECT_ID.iam.gserviceaccount.com --role roles/bigquery.jobUser
gcloud iam service-accounts keys create ./compute-viewer-kubecost-key.json --iam-account compute-viewer-kubecost@$PROJECT_ID.iam.gserviceaccount.com 

You can then get your service account key to paste into the UI (be careful with this!):

 cat compute-viewer-kubecost-key.json 

In Kubecost, navigate to the settings page and click “update” for the “External Cloud Cost Configuration (GCP)” setting, then follow the remaining instructions found at the “Add Key” link:

GCP out of cluster key entry

BigQuery dataset requires a BigQuery dataset prefix (e.g. billing_data) in addition to the BigQuery table name. A full example is billing_data.gcp_billing_export_v1_018AIF_74KD1D_534A2.

These config values can alternatively be provided via a values file.

Step 3: Label cloud assets

You can now label assets with the following schema to allocate costs back to their appropriate Kubernetes owner. Learn more here on updating GCP asset labels.

Cluster:    "kubernetes_cluster" :   <clusterID>
Namespace:  "kubernetes_namespace" : <namespace>
Deployment: "kubernetes_deployment": <deployment>
Label:      "kubernetes_label_NAME": <label>
Pod:        "kubernetes_pod":        <pod>
Daemonset:  "kubernetes_daemonset":  <daemonset>
Container:  "kubernetes_container":  <container>

To use an alternative or existing label schema for GCP cloud assets, you may supply these in your values.yaml under the “kubecostProductConfigs.labelMappingConfigs.<aggregation>_external_label”

Note: Google generates special labels for GKE resources (e.g. “goog-gke-node”, “goog-gke-volume”). Values with these labels are excluded from out-of-cluster costs because Kubecost already includes them as in-cluster assets. Thus, to make sure all cloud assets are included, we recommend installing Kubecost on each cluster where insights into costs are required.