This feature is only officially supported on Kubecost Enterprise plans.
Thanos is a tool to aggregate Prometheus metrics to a central object storage (S3 compatible) bucket. Thanos is implemented as a sidecar on the Prometheus pod on all clusters. Thanos Federation is one of two primary methods to aggregate all cluster information back to a single view as described in our Multi-Cluster article.
The preferred method for multi-cluster is ETL Federation. The configuration guide below is for Kubecost Thanos Federation, which may not scale as well as ETL Federation in large environments.
This guide will cover how to enable Thanos on your primary cluster, and on any additional secondary clusters.
Follow steps here to enable all required Thanos components on a Kubecost primary cluster, including the Prometheus sidecar.
For each additional cluster, only the Thanos sidecar is needed.
Consider the following Thanos recommendations for secondaries:
Ensure you provide a unique identifier for prometheus.server.global.external_labels.cluster_id
to have additional clusters be visible in the Kubecost product, e.g. cluster-two
.
cluster_id
can be replaced with another label (e.g. cluster
) by modifying .Values.kubecostModel.promClusterIDLabel.
Follow the same verification steps available here.
Sample configurations for each cloud provider can be found here.
This feature is only officially supported on Kubecost Enterprise plans.
Kubecost leverages Thanos and durable storage for three different purposes:
Centralize metric data for a global multi-cluster view into Kubernetes costs via a Prometheus sidecar
Allow for unlimited data retention
Backup Kubecost ETL data
To enable Thanos, follow these steps:
This step creates the object-store.yaml file that contains your durable storage target (e.g. GCS, S3, etc.) configuration and access credentials. The details of this file are documented thoroughly in Thanos documentation.
We have guides for using cloud-native storage for the largest cloud providers. Other providers can be similarly configured.
Use the appropriate guide for your cloud provider:
Create a secret with the .yaml file generated in the previous step:
Each cluster needs to be labelled with a unique Cluster ID, which is done in two places.
values-clusterName.yaml
The Thanos subchart includes thanos-bucket
, thanos-query
, thanos-store
, thanos-compact
, and service discovery for thanos-sidecar
. These components are recommended when deploying Thanos on the primary cluster.
These values can be adjusted under the thanos
block in values-thanos.yaml. Available options are here: thanos/values.yaml
The thanos-store
container is configured to request 2.5GB memory, this may be reduced for smaller deployments. thanos-store
is only used on the primary Kubecost cluster.
To verify installation, check to see all Pods are in a READY state. View Pod logs for more detail and see common troubleshooting steps below.
Thanos sends data to the bucket every 2 hours. Once 2 hours have passed, logs should indicate if data has been sent successfully or not.
You can monitor the logs with:
Monitoring logs this way should return results like this:
As an aside, you can validate the Prometheus metrics are all configured with correct cluster names with:
To troubleshoot the IAM Role Attached to the serviceaccount, you can create a Pod using the same service account used by the thanos-sidecar (default is kubecost-prometheus-server
):
s3-pod.yaml
This should return a list of objects (or at least not give a permission error).
If a cluster is not successfully writing data to the bucket, review thanos-sidecar
logs with the following command:
Logs in the following format are evidence of a successful bucket write:
/stores
endpointIf thanos-query can't connect to both the sidecar and the store, you may want to directly specify the store gRPC service address instead of using DNS discovery (the default). You can quickly test if this is the issue by running:
kubectl edit deployment kubecost-thanos-query -n kubecost
and adding
--store=kubecost-thanos-store-grpc.kubecost:10901
to the container args. This will cause a query restart and you can visit /stores
again to see if the store has been added.
If it has, you'll want to use these addresses instead of DNS more permanently by setting .Values.thanos.query.stores in values-thanos.yaml.
A common error is as follows, which means you do not have the correct access to the supplied bucket:
Assuming pods are running, use port forwarding to connect to the thanos-query-http
endpoint:
Then navigate to http://localhost:8080 in your browser. This page should look very similar to the Prometheus console.
If you navigate to Stores using the top navigation bar, you should be able to see the status of both the thanos-store
and thanos-sidecar
which accompanied the Prometheus server:
Also note that the sidecar should identify with the unique cluster_id
provided in your values.yaml in the previous step. Default value is cluster-one
.
The default retention period for when data is moved into the object storage is currently 2h. This configuration is based on Thanos suggested values. By default, it will be 2 hours before data is written to the provided bucket.
Instead of waiting 2h to ensure that Thanos was configured correctly, the default log level for the Thanos workloads is debug
(it's very light logging even on debug). You can get logs for the thanos-sidecar
, which is part of the prometheus-server
Pod, and thanos-store
. The logs should give you a clear indication of whether or not there was a problem consuming the secret and what the issue is. For more on Thanos architecture, view this resource.
Kubecost v1.67.0+ uses Thanos 0.15.0. If you're upgrading to Kubecost v1.67.0+ from an older version and using Thanos, with AWS S3 as your backing storage for Thanos, you'll need to make a small change to your Thanos Secret in order to bump the Thanos version to 0.15.0 before you upgrade Kubecost.
Thanos 0.15.0 has over 10x performance improvements, so this is recommended.
This is simplified if you're using our default values-thanos.yaml, which has the new configs already.
For the Thanos Secret you're using, the encrypt-sse
line needs to be removed. Everything else should stay the same.
For example, view this sample config:
The easiest way to do this is to delete the existing secret and upload a new one:
kubectl delete secret -n kubecost kubecost-thanos
Update your secret .YAML file as above, and save it as object-store.yaml.
kubectl create secret generic kubecost-thanos -n kubecost --from-file=./object-store.yaml
Once this is done, you're ready to upgrade!
Your values-thanos.yaml needs to be updated to the new defaults . The PR bumps the image version, adds the component, and increases concurrency.