This document will describe why your Kubecost instance’s data can be useful to share with us, what content is in the data, and how to share it.
Kubecost product releases are tested and verified against a combination of generated/synthetic Kubernetes cluster data and examples of customer data that have been shared with us. Customers who share snapshots of their data with us help to ensure that product changes handle their specific use cases and scales. Because the Kubecost product for many customers is run as an on-prem service, with no data sharing back to us, we do not inherently have this data for many of our customers.
Sharing data with us requires an ETL backup executed by the customer in their own environment before the resulting data can be sent out. Kubecost's ETL is a computed cache built upon Prometheus metrics and cloud billing data, on which nearly all API requests made by the user and the Kubecost frontend currently rely. Therefore, the ETL data will contain metric data and identifying information for that metric (e.g. a container name, pod name, namespace, and cluster name) during a time window, but will not contain other information about containers, pods, clusters, cloud resources, etc. You can read more about these metric details in our Kubecost Metrics doc.
The full methodology for creating the ETL backup can be found in our ETL Backup doc. Once these files have been backed up, the content will look as follows before compressing the data:
Once the data is downloaded to the local disk from either the automated or manual ETL backup methods, the data must be converted to a gzip file. A suggested method for downloading the ETL backup and compressing it quickly is to use this script. Check out the tar syntax in that script if doing this manually without the script. When the compressed ETL backup is ready to share, please work with a Kubecost support engineer on sharing the file with us. Our most common approach is to use a Google Drive folder with access limited to you and the support engineer, but we recognize not all companies are open to this and will work with you to determine the most business-appropriate method.
If you are interested in reviewing the contents of the data, either before or after sending the ETL backup to us, you can find an example Golang implementation on how to read the raw ETL data.
We do not recommend enabling ETL Backup in conjunction with Federated ETL.
Kubecost's extract, transform, load (ETL) data is a computed cache based on Prometheus's metrics, from which the user can perform all possible Kubecost queries. The ETL data is stored in a persistent volume mounted to the kubecost-cost-analyzer pod.
There are a number of reasons why you may want to backup this ETL data:
To ensure a copy of your Kubecost data exists, so you can restore the data if needed
To reduce the amount of historical data stored in Prometheus/Thanos, and instead retain historical ETL data
Beginning in v1.100, this feature is enabled by default if you have Thanos enabled. To opt out, set .Values.kubecostModel.etlBucketConfigSecret="".
Kubecost provides cloud storage backups for ETL backing storage. Backups are not the typical approach of "halt all reads/writes and dump the database." Instead, the backup system is a transparent feature that will always ensure that local ETL data is backed up, and if local data is missing, it can be retrieved from backup storage. This feature protects users from accidental data loss by ensuring that previously backed-up data can be restored at runtime.
Durable backup storage functionality is supported with a Kubecost Enterprise plan.
When the ETL pipeline collects data, it stores daily and hourly (if configured) cost metrics on a configured storage. This defaults to a PV-based disk storage, but can be configured to use external durable storage on the following providers:
AWS S3
Azure Blob Storage
Google Cloud Storage
This configuration secret follows the same layout documented for Thanos here.
You will need to create a file named object-store.yaml using the chosen storage provider configuration (documented below), and run the following command to create the secret from this file:
The file must be named object-store.yaml.
If Kubecost was installed via Helm, ensure the following value is set.
If you are using an existing disk storage option for your ETL data, enabling the durable backup feature will retroactively back up all previously stored data*. This feature is also fully compatible with the existing S3 backup feature.
If you are using a memory store for your ETL data with a local disk backup (kubecostModel.etlFileStoreEnabled: false), the backup feature will simply replace the local backup. In order to take advantage of the retroactive backup feature, you will need to update to file store (kubecostModel.etlFileStoreEnabled: true). This option is now enabled by default in the Helm chart.
The simplest way to backup Kubecost's ETL is to copy the pod's ETL store to your local disk. You can then send that file to any other storage system of your choice. We provide a script to do that.
To restore the backup, untar the results of the ETL backup script into the ETL directory pod.
There is also a Bash script available to restore the backup in Kubecost's etl-backup repo.
Currently, this feature is still in development, but there is currently a status card available on the Diagnostics page that will eventually show the status of the backup system:
In some scenarios like when using Memory store, setting kubecostModel.etlHourlyStoreDurationHours to a value of 48 hours or less will cause ETL backup files to become truncated. The current recommendation is to keep etlHourlyStoreDurationHours at its default of 49 hours.
This feature is only supported on Kubecost Enterprise plans.
The query service replica (QSR) is a scale-out query service that reduces load on the cost-model pod. It allows for improved horizontal scaling by being able to handle queries for larger intervals, and multiple simultaneous queries.
The query service will forward /model/allocation and /model/assets requests to the Query Services StatefulSet.
The diagram below demonstrates the backing architecture of this query service and its functionality.
There are three options that can be used for the source ETL Files:
For environments that have Kubecost Federated ETL enabled, this store will be used, no additional configuration is required.
For single cluster environments, QSR can target the ETL backup store. To learn more about ETL backups, see the ETL Backup doc.
Alternatively, an object-store containing the ETL dataset to be queried can be configured using a secret kubecostDeployment.queryServiceConfigSecret. The file name of the secret must be object-store.yaml. Examples can be found in our Configuring Thanos doc.
QSR uses persistent volume storage to avoid excessive S3 transfers. Data is retrieved from S3 hourly as new ETL files are created and stored in these PVs. The databaseVolumeSize should be larger than the size of the data in the S3 bucket.
When the pods start, data from the object-store is synced and this can take a significant time in large environments. During the sync, parts of the Kubecost UI will appear broken or have missing data. You can follow the pod logs to see when the sync is complete.
The default of 100Gi is enough storage for 1M pods and 90 days of retention. This can be adjusted:
Once the data store is configured, set kubecostDeployment.queryServiceReplicas to a non-zero value and perform a Helm upgrade.
Once QSR has been enabled, the new pods will automatically handle all API requests to /model/allocation and /model/assets.
