Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Federated ETL Architecture is only officially supported on Kubecost Enterprise plans.
This doc provides recommendations to improve the stability and recoverability of your Kubecost data when deploying in a Federated ETL architecture.
Kubecost can rebuild its extract, transform, load (ETL) data using Prometheus metrics from each cluster. It is strongly recommended to retain local cluster Prometheus metrics that meet an organization's disaster recovery requirements.
For long term storage of Prometheus metrics, we recommend setting up a Thanos sidecar container to push Prometheus metrics to a cloud storage bucket.
You can configure the Thanos sidecar following this example or this example. Additionally, ensure you configure the following:
object-store.yaml so the Thanos sidecar has permissions to read/write to the cloud storage bucket
.Values.prometheus.server.global.external_labels.cluster_id so Kubecost is able to distinguish which metric belongs to which cluster in the Thanos bucket.
Use your cloud service provider's bucket versioning feature to take frequent snapshots of the bucket holding your Kubecost data (ETL files and Prometheus metrics).
Configure Prometheus Alerting rules or Alertmanager to get notified when you are losing metrics or when metrics deviate beyond a known standard.
Kubecost v1.67.0+ uses Thanos 0.15.0. If you're upgrading to Kubecost v1.67.0+ from an older version and using Thanos, with AWS S3 as your backing storage for Thanos, you'll need to make a small change to your Thanos Secret in order to bump the Thanos version to 0.15.0 before you upgrade Kubecost.
Thanos 0.15.0 has over 10x performance improvements, so this is recommended.
Your values-thanos.yaml needs to be updated to the new defaults here. The PR bumps the image version, adds the query-frontend component, and increases concurrency.
This is simplified if you're using our default values-thanos.yaml, which has the new configs already.
For the Thanos Secret you're using, the encrypt-sse line needs to be removed. Everything else should stay the same.
For example, view this sample config:
The easiest way to do this is to delete the existing secret and upload a new one:
kubectl delete secret -n kubecost kubecost-thanos
Update your secret .YAML file as above, and save it as object-store.yaml.
kubectl create secret generic kubecost-thanos -n kubecost --from-file=./object-store.yaml
Once this is done, you're ready to upgrade!
Aggregator is a new backend for Kubecost. It is used in a Federated ETL configuration without Thanos, replacing the Federator component. Aggregator serves a critical subset of Kubecost APIs, but will eventually be the default model for Kubecost and serve all APIs. Currently, Aggregator supports all major monitoring and savings APIs, and also budgets and reporting.
Existing documentation for Kubecost APIs will use endpoints for non-Aggregator environments unless otherwise specified, but will still be compatible after configuring Aggregator.
Aggregator is designed to accommodate queries of large-scale datasets by improving API load times and reducing UI errors. It is not designed to introduce new functionality; it is meant to improve functionality at scale.
Aggregator is currently free for all Enterprise users to configure, and is always able to be rolled back.
Aggregator can only be configured in a Federated ETL environment
Must be using v1.107.0 of Kubecost or newer
Your values.yaml file must have set kubecostDeployment.queryServiceReplicas to its default value 0.
You must have your context set to your primary cluster. Kubecost Aggregator cannot be deployed on secondary clusters.
Select from one of the two templates below and save the content as federated-store.yaml. This will be your configuration template required to set up Aggregator.
The name of the .yaml file used to create the secret must be named federated-store.yaml or Aggregator will not start.
Basic configuration:
Advanced configuration (for larger deployments):
There is no baseline for what is considered a larger deployment, which will be dependent on load times in your Kubecost environment.
Once you’ve configured your federated-store.yaml_, create a secret using the following command:
Next, you will need to create an additional cloud-integration secret. Follow this tutorial on creating cloud integration secrets to generate your cloud-integration.json file, then run the following command:
Finally, upgrade your existing Kubecost installation. This command will install Kubecost if it does not already exist:
Upgrading your existing Kubecost using your configured federated-store.yaml_ file above will reset all existing Helm values configured in your values.yaml. If you wish to preserve any of those changes, append your values.yaml by adding the contents of your federated-store.yaml file into it, then replacing federated-store.yaml with values.yaml in the upgrade command below:
When first enabled, the aggregator pod will ingest the last three years (if applicable) of ETL data from the federated-store. This may take several hours. Because the combined folder is ignored, the federator pod is not used here, but can still run if needed. You can run kubectl get pods and ensure the aggregator pod is running, but should still wait for all data to be ingested.
Kubecost uses a shared storage bucket to store metrics from clusters, known as durable storage, in order to provide a single-pane-of-glass for viewing cost across many clusters. Multi-cluster is an enterprise feature of Kubecost.
There are multiple methods to provide Kubecost access to an S3 bucket. This guide has two examples:
Using a Kubernetes secret
Attaching an AWS Identity and Access Management (IAM) role to the service account used by Prometheus
Both methods require an S3 bucket. Our example bucket is named kc-thanos-store.
This is a simple S3 bucket with all public access blocked. No other bucket configuration changes should be required.
Once created, add an IAM policy to access this bucket. See our AWS Thanos IAM Policy doc for instructions.
To use the Kubernetes secret method for allowing access, create a YAML file named object-store.yaml with contents similar to the following example. See region to endpoint mappings here.
Instead of using a secret key in a file, many will want to use this method.
Attach the policy to the Thanos pods service accounts. Your object-store.yaml should follow the format below when using this option, which does not contain the secret_key and access_key fields.
Then, follow this AWS guide to enable attaching IAM roles to pods.
You can define the IAM role to associate with a service account in your cluster by creating a service account in the same namespace as Kubecost and adding an annotation to it of the form eks.amazonaws.com/role-arn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/<IAM_ROLE_NAME> as described here.
Once that annotation has been created, configure the following:
You can encrypt the S3 bucket where Kubecost data is stored in AWS via S3 and KMS. However, because Thanos can store potentially millions of objects, it is suggested that you use bucket-level encryption instead of object-level encryption. More details available in these external docs:
Visit the Configuring Thanos doc for troubleshooting help.
This feature is only officially supported on Kubecost Enterprise plans.
Kubecost leverages Thanos and durable storage for three different purposes:
Centralize metric data for a global multi-cluster view into Kubernetes costs via a Prometheus sidecar
Allow for unlimited data retention
Backup Kubecost ETL data
To enable Thanos, follow these steps:
This step creates the object-store.yaml file that contains your durable storage target (e.g. GCS, S3, etc.) configuration and access credentials. The details of this file are documented thoroughly in Thanos documentation.
We have guides for using cloud-native storage for the largest cloud providers. Other providers can be similarly configured.
Use the appropriate guide for your cloud provider:
Create a secret with the .yaml file generated in the previous step:
Each cluster needs to be labelled with a unique Cluster ID, which is done in two places.
values-clusterName.yaml
The Thanos subchart includes thanos-bucket, thanos-query, thanos-store, thanos-compact, and service discovery for thanos-sidecar. These components are recommended when deploying Thanos on the primary cluster.
These values can be adjusted under the thanos block in values-thanos.yaml. Available options are here: thanos/values.yaml
The thanos-store container is configured to request 2.5GB memory, this may be reduced for smaller deployments. thanos-store is only used on the primary Kubecost cluster.
To verify installation, check to see all Pods are in a READY state. View Pod logs for more detail and see common troubleshooting steps below.
Thanos sends data to the bucket every 2 hours. Once 2 hours have passed, logs should indicate if data has been sent successfully or not.
You can monitor the logs with:
Monitoring logs this way should return results like this:
As an aside, you can validate the Prometheus metrics are all configured with correct cluster names with:
To troubleshoot the IAM Role Attached to the serviceaccount, you can create a Pod using the same service account used by the thanos-sidecar (default is kubecost-prometheus-server):
s3-pod.yaml
This should return a list of objects (or at least not give a permission error).
If a cluster is not successfully writing data to the bucket, review thanos-sidecar logs with the following command:
Logs in the following format are evidence of a successful bucket write:
/stores endpointIf thanos-query can't connect to both the sidecar and the store, you may want to directly specify the store gRPC service address instead of using DNS discovery (the default). You can quickly test if this is the issue by running:
kubectl edit deployment kubecost-thanos-query -n kubecost
and adding
--store=kubecost-thanos-store-grpc.kubecost:10901
to the container args. This will cause a query restart and you can visit /stores again to see if the store has been added.
If it has, you'll want to use these addresses instead of DNS more permanently by setting .Values.thanos.query.stores in values-thanos.yaml.
A common error is as follows, which means you do not have the correct access to the supplied bucket:
Assuming pods are running, use port forwarding to connect to the thanos-query-http endpoint:
Then navigate to http://localhost:8080 in your browser. This page should look very similar to the Prometheus console.
If you navigate to Stores using the top navigation bar, you should be able to see the status of both the thanos-store and thanos-sidecar which accompanied the Prometheus server:
Also note that the sidecar should identify with the unique cluster_id provided in your values.yaml in the previous step. Default value is cluster-one.
The default retention period for when data is moved into the object storage is currently 2h. This configuration is based on Thanos suggested values. By default, it will be 2 hours before data is written to the provided bucket.
Instead of waiting 2h to ensure that Thanos was configured correctly, the default log level for the Thanos workloads is debug (it's very light logging even on debug). You can get logs for the thanos-sidecar, which is part of the prometheus-server Pod, and thanos-store. The logs should give you a clear indication of whether or not there was a problem consuming the secret and what the issue is. For more on Thanos architecture, view this resource.
This feature is only officially supported on Kubecost Enterprise plans.
Thanos is a tool to aggregate Prometheus metrics to a central object storage (S3 compatible) bucket. Thanos is implemented as a sidecar on the Prometheus pod on all clusters. Thanos Federation is one of two primary methods to aggregate all cluster information back to a single view as described in our Multi-Cluster article.
The preferred method for multi-cluster is ETL Federation. The configuration guide below is for Kubecost Thanos Federation, which may not scale as well as ETL Federation in large environments.
This guide will cover how to enable Thanos on your primary cluster, and on any additional secondary clusters.
Follow steps here to enable all required Thanos components on a Kubecost primary cluster, including the Prometheus sidecar.
For each additional cluster, only the Thanos sidecar is needed.
Consider the following Thanos recommendations for secondaries:
Ensure you provide a unique identifier for prometheus.server.global.external_labels.cluster_id to have additional clusters be visible in the Kubecost product, e.g. cluster-two.
cluster_id can be replaced with another label (e.g. cluster) by modifying .Values.kubecostModel.promClusterIDLabel.
Follow the same verification steps available here.
Sample configurations for each cloud provider can be found here.
In order to create an AWS IAM policy for use with Thanos:
Navigate to the AWS console and select IAM.
Select Policies in the Navigation menu, then select Create Policy.
Add the following JSON in the policy editor:
Make sure to replace <your-bucket-name> with the name of your newly-created S3 bucket.
4. Select Review policy and name this policy, e.g. kc-thanos-store-policy.
Navigate to Users in IAM control panel, then select Add user.
Provide a username (e.g. kubecost-thanos-service-account) and select Programmatic access.
Select Attach existing policies directly, search for the policy name provided in Step 4, then create the user.
Capture your Access Key ID and secret in the view below:
If you don’t want to use a service account, IAM credentials retrieved from an instance profile are also supported. You must get both access key and secret key from the same method (i.e. both from service or instance profile). More info on retrieving credentials here.
Kubecost Free can now be installed on an unlimited number of individual clusters. Larger teams will benefit from using Kubecost Enterprise to better manage many clusters. See pricing for more details.
In an Enterprise multi-cluster setup, the UI is accessed through a designated primary cluster. All other clusters in the environment send metrics to a central object-store with a lightweight agent (aka secondary clusters). The primary cluster is designated by setting the Helm flag .Values.federatedETL.primaryCluster=true, which instructs this cluster to read from the combined folder that was processed by the federator. This cluster will consume additional resources to run the Kubecost UI and backend.
As of Kubecost 1.108, agent health is monitored by a diagnostic pod that collects information from the local cluster and sends it to an object-store. This data is then processed by the Primary cluster and accessed via the UI and API.
Because the UI is only accessible through the primary cluster, Helm flags related to UI display are not applied to secondary clusters.
This feature is only supported for Kubecost Enterprise.
There are two primary methods to aggregate all cluster information back to a single Kubecost UI:
Both methods allow for greater compute efficiency by running the most resource-intensive workloads on a single primary cluster.
For environments that already have a Prometheus instance, ETL Federation may be preferred because only a single Kubecost pod is required.
The below diagrams highlight the two architectures:
Kubecost ETL Federation (Preferred)
Kubecost Thanos Federation
Federated ETL is only officially supported for Kubecost Enterprise plans.
Federated extract, transform, load (ETL) is one of two methods to aggregate all cluster information back to a single display described in our Multi-Cluster doc. Federated ETL gives teams the benefit of combining multiple Kubecost installations into one view without dependency on Thanos.
There are two primary advantages for using ETL Federation:
For environments that already have a Prometheus instance, Kubecost only requires a single pod per monitored cluster
Many solutions that aggregate Prometheus metrics (like Thanos), are often expensive to scale in large environments
This guide has specific detail on how ETL Configuration works and deployment options.
Alternatively, the most common configurations can be found in our poc-common-configurations repo.
The federated ETL is composed of three types of clusters.
Federated Clusters: The clusters which are being federated (clusters whose data will be combined and viewable at the end of the federated ETL pipeline). These clusters upload their ETL files after they have built them to Federated Storage.
Federator Clusters: The cluster on which the Federator (see in Other components) is set to run within the core cost-analyzer container. This cluster combines the Federated Cluster data uploaded to federated storage into combined storage.
Primary Cluster: A cluster where you can see the total Federated data that was combined from your Federated Clusters. These clusters read from combined storage.
These cluster designations can overlap, in that some clusters may be several types at once. A cluster that is a Federated Cluster, Federator Cluster, and Primary Cluster will perform the following functions:
As a Federated Cluster, push local cluster cost data to be combined from its local ETL build pipeline.
As a Federator Cluster, run the Federator inside the cost-analyzer, which pulls this local cluster data from S3, combines them, then pushes them back to combined storage.
As a Primary Cluster, pull back this combined data from combined storage to serve it on Kubecost APIs and/or the Kubecost frontend.
The Storages referred to here are an S3 (or GCP/Azure equivalent) storage bucket which acts as remote storage for the Federated ETL Pipeline.
Federated Storage: A set of folders on paths <bucket>/federated/<cluster id> which are essentially ETL backup data, holding a “copy” of Federated Cluster data. Federated Clusters push this data to Federated Storage to be combined by the Federator. Federated Clusters write this data, and the Federator reads this data.
Combined Storage: A folder on S3 on the path <bucket>/federated/combined which holds one set of ETL data containing all the allocations/assets in all the ETL data from Federated Storage. The Federator takes files from Federated Storage and combines them, adding a single set of combined ETL files to Combined Storage to be read by the Primary Cluster. The Federator writes this data, and the Primary Cluster reads this data.
The Federator: A component of the cost-model which is run on the Federator Cluster, which can be a Federated Cluster, a Primary Cluster, or neither. The Federator takes the ETL binaries from Federated Storage and merges them, adding them to Combined Storage.
Federated ETL: The pipeline containing the above components.
This diagram shows an example setup of the Federated ETL with:
Three pure Federated Clusters (not classified as any other cluster type): Cluster 1, Cluster 2, and Cluster 3
One Federator Cluster that is also a Federated Cluster: Cluster 4
One Primary Cluster that is also a Federated Cluster: Cluster 5
The result is 5 clusters federated together.
Ensure each federated cluster has a unique clusterName and cluster_id:
Add a secret using that file: kubectl create secret generic <secret_name> -n kubecost --from-file=federated-store.yaml. Then set .Values.kubecostModel.federatedStorageConfigSecret to the kubernetes secret name.
For all clusters you want to federate together (i.e. see their data on the Primary Cluster), set .Values.federatedETL.federatedCluster to true. This cluster is now a Federated Cluster, and can also be a Federator or Primary Cluster.
For the cluster “hosting” the Federator, set .Values.federatedETL.federator.enabled to true. This cluster is now a Federator Cluster, and can also be a Federated or Primary Cluster.
Optional: If you have any Federated Clusters pushing to a store that you do not want a Federator Cluster to federate, add the cluster id under the Federator config section .Values.federatedETL.federator.clusters. If this parameter is empty or not set, the Federator will take all ETL files in the /federated directory and federate them automatically.
Multiple Federators federating from the same source will not break, but it’s not recommended.
In Kubecost, the Primary Cluster serves the UI and API endpoints as well as reconciling cloud billing (cloud-integration).
For the cluster that will be the Primary Cluster, set .Values.federatedETL.primaryCluster to true. This cluster is now a Primary Cluster, and can also be a Federator or Federated Cluster.
Cloud-integration requires .Values.federatedETL.federator.primaryClusterID set to the same value used for .Values.kubecostProductConfigs.clusterName
Important: If the Primary Cluster is also to be federated, please wait 2-3 hours for data to populate Federated Storage before setting a Federated Cluster to primary (i.e. set .Values.federatedETL.federatedCluster to true, then wait to set .Values.federatedETL.primaryCluster to true). This allows for maximum certainty of data consistency.
If you do not set this cluster to be federated as well as primary, you will not see local data for this cluster.
The Primary Cluster’s local ETL will be overwritten with combined federated data.
This can be undone by unsetting it as a Primary Cluster and rebuilding ETL.
Setting a Primary Cluster may result in a loss of the cluster’s local ETL data, so it is recommended to back up any filestore data that one would want to save to S3 before designating the cluster as primary.
Alternatively, a fresh Kubecost install can be used as a consumer of combined federated data by setting it as the Primary but not a Federated Cluster.
The Federated ETL should begin functioning. On any ETL action on a Federated Cluster (Load/Put into local ETL store) the Federated Clusters will add data to Federated Storage. The Federator will run 5 minutes after the Federator Cluster startup, and then every 30 minutes after that. The data is merged into the Combined Storage, where it can be read by the Primary.
To verify Federated Clusters are uploading their data correctly, check the container logs on a Federated Cluster. It should log federated uploads when ETL build steps run. The S3 bucket can also be checked to see if data is being written to the /federated/<cluster_id> path.
To verify the Federator is functioning, check the container logs on the Federator Cluster. The S3 bucket can also be checked to verify that data is being written to /federated/combined.
To verify the entire pipeline is working, either query Allocations/Assets or view the respective views on the frontend. Multi-cluster data should appear after:
The Federator has run at least once.
There was data in the Federated Storage for the Federator to have combined.
If you are using an internal certificate authority (CA), follow this tutorial instead of the above Setup section.
Begin by creating a ConfigMap with the certificate provided by the CA on every agent, including the Federator and any federated clusters, and name the file kubecost-federator-certs.yaml.
Now run the following command, making sure you specify the location for the ConfigMap you created:
kubectl create cm kubecost-federator-certs --from-file=/path/to/kubecost-federator-certs.yaml
Mount the certification on the Federator and any federated clusters by passing these Helm flags to your values.yaml/manifest:
Create a file federated-store.yaml, which will go on all clusters:
Now run the following command (omit kubectl create namespace kubecost if your kubecost namespace already exists, or this command will fail):
When using ETL Federation, there are several methods to recover Kubecost data in the event of data loss. See our Backups and Alerting doc for more details regarding these methods.
In the event of missing or inaccurate data, you may need to rebuild your ETL pipelines. This is a documented procedure. See the Repair Kubecost ETLs doc for information and troubleshooting steps.
Start by . The following example uses a bucket named thanos-bucket. Next, download a service account JSON file from Google's service account manager ().
Now create a YAML file named object-store.yaml in the following format, using your bucket name and service account details:
Note: Because this is a YAML file, it requires this specific indention.
Warning: Do not apply a retention policy to your Thanos bucket, as it will prevent Thanos compaction from completing.
To use Azure Storage as Thanos object store, you need to precreate a storage account from Azure portal or using Azure CLI. Follow the instructions from the .
Now create a .YAML file named object-store.yaml with the following format:
