Kubecost's ETL is a computed cache based on Prometheus's metrics, from which the user can perform all possible Kubecost queries. The ETL data is stored in a
PersistentVolumemounted to the
There are a number of reasons why you may want to backup this ETL data:
- To ensure a copy of your Kubecost data exists, so that you can restore the data if needed
- If you would like to reduce the amount of data stored in Prometheus (15 day retention window by default) or Thanos
We provide cloud storage backups for ETL backing storage. Backups are not the typical approach of "halt all reads/writes and dump the database." Instead, the backup system is a transparent feature that will always ensure that local ETL data is backed up, and if local data is missing, it can be retrieved from backup storage. This feature protects users from accidental data loss by ensuring that previously backed up data can be restored at runtime.
Note: Durable backup storage functionality is only part of Kubecost Enterprise.
When the ETL pipeline collects data, it stores both daily and hourly (if configured) cost metrics on a configured storage. This defaults to a persistent volume based disk storage, but can be configured to use external durable storage on the following providers:
- AWS S3
- Azure Blob Storage
- Google Cloud Storage
You will need to create a file named
object-store.yamlusing the chosen storage provider configuration (documented below), and run the following command to create the secret from this file:
kubectl create secret generic <YOUR_SECRET_NAME> -n kubecost --from-file=object-store.yaml
Note: The file must be named
.Values.kubecostModel.etlBucketConfigSecret=kubecost-thanoswill enable the backup feature. This will backup all ETL data to the same bucket being used by Thanos.
"private_key": "-----BEGIN PRIVATE KEY-----\...\n-----END PRIVATE KEY-----\n",
"client_email": "[email protected]",
If Kubecost was installed via Helm, ensure the following value is set.
If you are using an existing disk storage option for your ETL data, enabling the durable backup feature will retroactively back up all previously stored data*. This feature is also fully compatible with the existing S3 backup feature.
* If you are using a memory store for your ETL data with a local disk backup (
kubecostModel.etlFileStoreEnabled: false), the backup feature will simply replace the local backup. In order to take advantage of the retroactive backup feature, you will need to update to file store (
kubecostModel.etlFileStoreEnabled: true). This option is now enabled by default in the Helm chart.
To restore the backup, untar the results of the etl-backup script into the ETL directory pod.
kubectl cp -c cost-model <untarred-results-of-script> <kubecost-namespace>/<kubecost-pod-name>/var/configs/db/etl
Currently, this feature is still in development, but there is currently a status card available on the diagnostics page that will eventually show the status of the backup system:
Diagnostic ETL Backup Status
In some scenarios like when using Memory store, setting
kubecostModel.etlHourlyStoreDurationHoursto a value of
48hours or less will cause ETL backup files to become truncated. The current recomendation is to keep etlHourlyStoreDurationHours at its default of