Comment on page

Secondary Clusters Guide

Secondary clusters use a minimal Kubecost deployment to send their metrics to a central storage-bucket (aka durable storage) that is accessed by the primary cluster to provide a single-pane-of-glass view into all aggregated cluster costs globally. This aggregated cluster view is exclusive to Kubecost Enterprise.
Kubecost's UI will appear broken when set to a secondary cluster. It should only be used for troubleshooting.
This guide explains settings that can be tuned in order to run the minimum Kubecost components to run Kubecost more efficiently.
See the Additional resources section below for complete examples in our GitHub repo.

Kubecost Global

Disable product caching and reduce query concurrency with the following parameters:
--set kubecostModel.warmCache=false
--set kubecostModel.warmSavingsCache=false
--set kubecostModel.etl=false
--set kubecostModel.etlCloudAsset=false
--set kubecostModel.maxQueryConcurrency=1


Grafana is not needed on secondary clusters.
--set global.grafana.enabled=false
--set global.grafana.proxy=false


Kubecost and its accompanying Prometheus collect a reduced set of metrics that allow for lower resource/storage usage than a standard Prometheus deployment.
The following configuration options further reduce resource consumption when not using the Kubecost frontend:
--set prometheus.server.retention=2d
Potentially reducing retention even further, metrics are sent to the storage-bucket every 2 hours.
You can tune prometheus.server.persistentVolume.size depending on scale, or outright disable persistent storage.


Disable Thanos components. These are only used for troubleshooting on secondary clusters. See this guide for troubleshooting via kubectl logs.
Secondary clusters write to the global storage-bucket via the thanos-sidecar on the prometheus-server pod.
--set thanos.compact.enabled=false
--set thanos.bucket.enabled=false
--set thanos.query.enabled=false
--set thanos.queryFrontend.enabled=false


You can disable node-exporter and the service account if cluster/node rightsizing recommendations are not required.
node-export must be disabled if there is an existing DaemonSet. More info here.

Helm values

For reference, this secondary-clusters.yaml snippet is a list of the most common settings for efficient secondary clusters:
clusterName: kubecostProductConfigs_clusterName
# productKey not needed on secondary clusters
warmCache: false
warmSavingsCache: false
etl: false
etlCloudAsset: false
maxQueryConcurrency: 1
enabled: false
proxy: false
# cluster_id should be unique for all clusters and the same value as .kubecostProductConfigs.clusterName
cluster_id: kubecostProductConfigs_clusterName
retention: 2d
# nodeExporter:
# enabled: false
# serviceAccounts:
# nodeExporter:
# create: false
enabled: false
enabled: false
enabled: false
enabled: false
enabled: false

Additional resources

You can find complete installation guides and sample files on our repo.
Additional considerations for properly tuning resource consumption is here.
©2023 Stackwatch, Inc. All rights reserved. Designed & built in California.