Kubecost Cloud is a SaaS solution that offers easier deployment and more convenient maintenance as an alternative to our on-prem product. You can get started with a free trial here.

Functionality

What features are currently available?

Kubecost Cloud will provide you with everything you need in order to visualize your cost spend data and begin saving, with many more features to come soon. Kubecost Cloud currently supports:

• Allocations, Assets, and Cloud Cost dashboards for multiple Kubernetes clusters

• Cloud integration for AWS, GCP, and Azure billing

• Savings page with container right-sizing, cluster right-sizing, and abandoned workload detection

• SSO via Google, Microsoft, Okta, and GitHub

• Saved reports of your monitoring dashboard data

• Team and user management

• Overview homepage

What features are coming next?

Kubecost Cloud's functionality will be quickly expanded. Here's a sneak peek at which features we plan to implement next:

• Clusters dashboard

• Shared and idle costs

• RBAC

• Expanded Savings page options

Is Kubecost Cloud right for me?

Kubecost Cloud can help teams oversee cloud spend like our existing product, but there are more reasons this hosted version can be advantageous:

• Simplicity: Kubecost will handle your upgrades, scaling, and maintenance.

• Cost effective: Kubecost Cloud will empower users to think big, start small, and scale quickly. Save on team time when you don’t host on-prem.

• Security: We take responsibility for ensuring information on our servers is secure and private.

Requirements

Kubecost Cloud has been tested with deployments of up to 1,000 nodes.

Environments supported

• Kubernetes 1.8+

• Helm 3.1+

• Does not support air-gapped environments

Resource requirements

• In a small Kube cluster (less than 20 nodes), the Kubecost Cloud Agent total resource usage is approximately:

  • 2 GiB RAM

  • .5 CPUs

• The network costs DaemonSet will add a per node resource usage at:

  • 20 MiB RAM

  • .05 CPU

Cluster requirements:

• Supports all major cloud Kubernetes services (EKS, AKS, GKE)

• Supports on-prem Kubernetes clusters

• Supports self-managed clusters on AWS, GCP, and Azure.

Using existing Kubecost documentation

Kubecost Cloud brings lots of the same functionality from our existing on-prem product, but there are several distinct differences in their composition that may prevent you from consulting standard Kubecost documentation for assistance. We do not recommend you consult our other live documentation resources for help, and instead contact us directly at cloud-support@kubecost.com with questions. Hyperlinks embedded in this group of docs may take you to live Kubecost web properties for additional information where relevant.

All information found in this group of docs will be applicable for your installation and experience. Over time, we hope to expand this group to assist with troubleshooting and support.

Integration with cloud service providers (CSPs) via their respective billing APIs allows Kubecost to display out-of-cluster (OOC) costs (e.g. AWS S3, Google Cloud Storage, Azure Storage Account). Additionally, it allows Kubecost to reconcile Kubecost's in-cluster estimates with actual billing data to improve accuracy.

To learn more about how Kubecost provides accurate cost data or how to manage existing cloud integrations, read below. Otherwise, see any of the following articles to get started on integrating a specific CSP:

• Kubecost Cloud AWS Integration

• Kubecost Cloud GCP Integration

• Kubecost Cloud Azure Integration

Reconciliation

Reconciliation matches in-cluster assets with items found in the billing data pulled from the CSP. This allows Kubecost to display the most accurate depiction of your in-cluster spending. Additionally, the reconciliation process creates Network assets for in-cluster nodes based on the information in the billing data. The main drawback of this process is that the CSPs have between a 6 to 24-hour delay in releasing billing data, and reconciliation requires a complete day of cost data to reconcile with the in-cluster assets. This requires a 48-hour window between resource usage and reconciliation. If reconciliation is performed within this window, asset cost is deflated to the partially complete cost shown in the billing data.

Cost-based metrics are based on on-demand pricing unless there is definitive data from a CSP that the node is not on-demand. This way estimates are as accurate as possible. If a new reserved instance is provisioned or a node joins a savings plan:

1. Kubecost continues to emit on-demand pricing until the node is added to the cloud bill.

2. Once the node is added to the cloud bill, Kubecost starts emitting something closer to the actual price.

3. For the time period where Kubecost assumed the node was on-demand but it was actually reserved, reconciliation adjusts the price to reflect the actual cost.

Managing cloud integrations

You can view your existing cloud integrations and their success status in the Kubecost UI by visiting Settings, then scrolling to Cloud Integrations. To create a new integration or learn more about existing integrations, select View additional details to go to the Cloud Integrations page.

Here, you can view your existing integrations. For non-successful integrations, Kubecost will display a diagnostic error message in the Status column to contextualize steps toward successful integration.

Select an individual integration to view a side panel that contains details about that integration. You will be able to select Edit to adjust any details about your integration that were configured during the original setup. You can also delete an integration by selecting the red trash can icon next to the name of the integration.

Kubecost Cloud provides the ability to allocate out of cluster (OOC) costs back to Kubernetes concepts like namespaces and deployments. The following guide provides the steps required for allocating OOC costs in AWS.

Prerequisites

Before beginning your integration, you need to create a Cost and Usage Report (CUR) through AWS. Consult AWS' documentation for step-by-step instructions if needed during this process. When creating your CUR, make sure to configure these settings:

• Time granularity is set to Daily

• Resource IDs and Athena are enabled

Remember the name of the S3 bucket that is created for this CUR. AWS may require up to 24 hours to publish data. Wait until you have received data before proceeding with this integration.

Adding an integration

In the Kubecost Cloud UI, begin by selecting Settings in the left navigation. Scroll down to Cloud Integrations, then select View Additional Details. The Cloud Integrations dashboard opens. Select + Add Integration. Then, select AWS Integration from the slide panel.

Step 1: Setting up a CUR

If your CUR has been properly set up and is now providing data after following the on-screen instructions in the Kubecost UI, select Continue.

Step 2: Setting up Athena

It's important to set up an Athena integration so Kubecost can perform reconciliation for providing accurate billing data. The on-screen instructions of the Kubecost Cloud UI are repeated here:

As part of the CUR creation process, Amazon also creates a CloudFormation template that is used to create the Athena integration. It is created in the CUR S3 bucket under s3-path-prefix/cur-name and typically has the filename crawler-cfn.yml. This .yml is your necessary CloudFormation template. You will need it in order to complete the CUR Athena integration. You can read more about this

Once Athena is set up with the CUR, you will need to create a new S3 bucket for Athena query results:

• Select Create bucket. The Create Bucket page opens.

• Use the same region used for the CUR bucket and pick a name that follows the format aws-athena-query-results-*.

• Select Create bucket at the bottom of the page.

• Select Settings, then select Manage. The 'Manage settings' window opens.

• Set Location of query result to the S3 bucket you just created, then select Save.

When you have completed all the above steps, select Continue in the Kubecost Cloud UI.

Step 3: Setting up IAM permissions

Before continuing with the integration in the Kubecost Cloud UI, you need to set up IAM permissions in AWS.

• Select Create Stack, then select With existing resources (import resources) from the dropdown. On the 'Identify resources' page, select Next.

• Under Template source, choose Upload a template file.

• Select Choose file, which will open your file explorer. Select the .yaml template, and then select Open. Then, select Next.

• On the 'Identify resources' page, provide any additional resources to import. Then, select Next.

• For Stack name, enter a name for your template.

• Set the following parameters:

  • MasterPayerAccountID: The account ID of the management account (formerly called master payer account) where the CUR has been created

  • SpotDataFeedBucketName: Optional. The bucket where the Spot data feed is sent to

• Review all provided information, then select Import Resources.

• At the bottom of the page, select I acknowledge that AWS CloudFormation might create IAM resources.

• Select Create Stack.

Step 4: Provide CUR config values to Kubecost

You will be prompted to provide values for several different fields to finalize your integration. See this table for working definitions of each field:

When you have provided all mandatory fields, select Create Integration to finalize. Be patient while your integration is set up. The Status should initially display as Unknown. This is normal. You should eventually see the integration's Status change from Pending to Successful.

Kubecost Cloud provides the ability to allocate out of cluster (OOC) costs back to Kubernetes concepts like namespaces and deployments. The following guide provides the steps required for allocating OOC costs in GCP.

Prerequisites

Before you interact with Kubecost Cloud, you will need to export your cloud billing data in GCP to BigQuery. For help, consult Google's documentation on the subject.

After this, it is also recommend to use a detailed billing export in order to gain access to Kubecost's cloud integration functionality such as reconciliation for the most accurate spend data.

You will need to prepare the following fields:

• GCP Project Id: The ID of your GCP project.

• GCP Dataset: BigQuery dataset prefix

• GCP Table: BigQuery table name.

If you are having trouble determining these values, consider this example. A dataset billing_data.gcp_billing_export_v1_018AIF_74KD1D_534A2 under a project project-1 will have the following values:

• Project ID: project-1

• Dataset Name: billing_data

• Table: gcp_billing_export_v1_018AIF_74KD1D_534A2

Adding an integration

In the Kubecost Cloud UI, begin by selecting Settings in the left navigation. Scroll down to Cloud Integrations, then select View Additional Details. The Cloud Integrations dashboard opens. Select + Add Integration. Then, select GCP Integration from the slide panel.

Step 1: Enable billing data export

After completing the Prerequisites section, you will fill out the three fields in this step with the respective values (Project Id, Dataset, and Table). All fields are mandatory. Then, select Continue.

Step 2: Create a GCP service account

You will need to give your Kubecost GCP service account certain permissions in your project roles/bigquery.jobUser. This can be performed either through the gcloud CLI or the GCP Console. Follow the instructions as they appear directly in the UI. Then, select Continue.

Step 3: Assign Kubecost service account access to BigQuery dataset

Finally, you must give the Kubecost service account direct access to your BigQuery dataset via the BigQuery Data Viewer role. Like Step 2, this can be performed either through the gcloud CLI or the GCP Console. Follow the instructions as they appear directly in the UI. Then, select Continue.

Finalizing your integration

After completing Step 3, you will see an overview of details for your integration. You can correct any details by selecting Edit. The Status should initially display as Unknown. This is normal. If everything looks correct, select Close to return to the Cloud Integrations dashboard. Your integration will now appear as a line item.

This doc will show you how to register for Kubecost Cloud, invite members to and manage your team(s), and create and remove clusters.

Accessing Kubecost Cloud

Creating a user account

You can create a new user account in moments. On the login page, provide a Name, Email, and Password to register. You can also register using an active Google, Microsoft, Okta, or GitHub account via SSO.

Managing teams

You can access information about your team by selecting Settings from the left navigation. You should see all teams you either own or are a member of. Here you will be able to create a team, manage existing members, and invite new members.

Begin by selecting Create Team. You will be prompted to choose a name for your team, then you can immediately begin inviting others to join.

Creating an invitation

In the row of the team you’d like to create an invite for, select Invite Members. Then, add your team member’s email to the Invite Member box, then select Invite.

There is currently no limit to the number of members that can be added to a team.

Accepting an invitation

Invitations to join are sent out via email. To join a team, you must follow the invitation link and register for Kubecost Cloud (see above). Once logged in, you will see a banner at the top of your page which will allow you to officially join the team. You can also accept an invite on the Settings page under Manage Teams.

Editing your team

When you select Invite Member, the Edit Team window appears. Here, you can see a list of all active members, as well as emails with pending invitations. You can remove members and cancel outgoing invitations. Members will not be alerted when they are removed from a team.

If you are a member of multiple teams, you will see a green checkmark icon next to the team you are currently viewing cost data for. You can switch teams by selecting Switch next to the team name, or by selecting Switch Team in the lower left navigation.

Managing clusters

Adding a cluster

Install prerequisites

• kubectl

• Helm v3.1+

Agent install

If no clusters are currently under management, you will find instructions on the Allocations page for installing the Kubecost Agent on your cluster. You can also find these instructions in Settings > Add Cluster.

Choose a unique ID for your cluster. This does not need to be the same name as your cluster, but it does need to be unique within your team.

Execute the following command to install the Kubecost Cloud agent to your cluster. The agent key will be pre-populated in the install command in the Kubecost Cloud UI.

helm upgrade --install kubecost-cloud \
--repo https://kubecost.github.io/kubecost-cloud-agent/ kubecost-cloud-agent \
--namespace kubecost-cloud --create-namespace \
-f https://raw.githubusercontent.com/kubecost/kubecost-cloud-agent/main/values-cloud-agent.yaml \
--set imageVersion="lunar-sandwich.v0.1.2" \
--set cloudAgentKey="AGENTKEY" \
--set cloudAgentClusterId="cluster-1" \
--set cloudReportingServer="collector.app.kubecost.com:31357" \
--set networkCosts.enabled=true

After 5-10 minutes, you should see your cluster connected. Data should automatically begin appearing in your Allocations and Assets dashboards.

You can view your connected clusters in the Settings page under Manage Clusters, which will display the unique ID, Provider, and Agent version.

Removing a cluster

Remove the agent from the cluster to stop reporting new metrics to Kubecost Cloud.

Example based on default Helm install command:

export release=kubecost-cloud
export namespace=kubecost-cloud
helm uninstall ${release} --namespace ${namespace}

If you modified the Helm release name or namespace, you will need to update the command accordingly.

After five minutes of no longer receiving data, the cluster will disappear from Manage Clusters. Any data previously received will be available for the remainder of the retention period.

Troubleshooting

GKE Autopilot rejects Kubecost Cloud agent

When attempting to install the Kubecost Cloud agent on a GKE Autopilot cluster, you may receive an error related to the network costs daemonSet:

Error: admission webhook "
gkepolicy.common-webhooks.networking.gke.io
" denied the request: GKE Warden rejected the request because it violates one or more constraints.

To work around this problem, modify your install command to disable the network costs daemonSet. That setting change will look like this:

--set networkCosts.enabled=false

Without network costs installed, you will be missing visibility into the networking layer in your environment. Kubecost is actively working with GCP to get our agent added to this list of autopilot partner workloads.

Kubecost Cloud's UI shares many similarities in feature functionality with self-hosted Kubecost. However, while Kubecost Cloud is expanding in utility, there may be distinct differences that require users to consult specific product documentation depending on which version of Kubecost they are using. This docs group contains information on the multiple Kubecost Cloud pages and dashboards.

The Assets page shows Kubernetes cluster costs broken down by the individual backing assets in your cluster (e.g. cost by node, disk, and other assets). It’s used to identify spend drivers over time and to audit Allocation data. This view can also optionally show out-of-cluster assets by service, tag/label, etc.

UI overview

Date range

You can control the window of allocation spend by selecting Last 7 days (the default option), and choosing the time window you want to view spend for. When using custom dates instead of a preset, select Apply to make changes.

Aggregate By

Here you can aggregate cost by namespace, deployment, service, and other native Kubernetes concepts. While selecting Single Aggregation, you will only be able to categorize by one concept at a time. While selecting Multi Aggregation, you will be able to filter for multiple concepts at the same time.

Edit Report

Selecting Edit Report will provide filtering and visualization options.

Resolution

Choose one of the following ways to display your query data on the dashboard:

• Daily: Default, displays data as a bar graph with daily steps

• Entire window: Displays all cost data as a semicircle graph with proportionate costs of your aggregate

Cost metric

View either cumulative or run rate costs measured over the selected time window based on the resources allocated.

• Cumulative Cost: represents the actual/historical spend captured by the Kubecost agent over the selected time window

• Rate metrics: Monthly, daily, or hourly "run rate" cost, also used for projected cost figures, based on samples in the selected time window

Filters

Filter resources by namespace, cluster, and/or Kubernetes label to more closely investigate a rise in spend or key cost drivers at different aggregations such as deployments or pods. When a filter is applied, only resources with this matching value will be shown.Comma-separated lists are supported to filter by multiple values, e.g. label filter equals kube-system,kubecost. Wild card filters are also supported, indicated by a * following the filter, e.g. label=kube* to return any namespace beginning with kube.

Additional options

The three horizontal dots icon provides additional means of handling your query data. You can open a saved report or download your query data as a CSV file.

The Collections page allows users to create groups of external cloud costs and Kubernetes objects in order to receive a unified view of all cloud spending. This is to leverage existing tags applied to cloud resources, without needing to rename them to match Kubernetes names. Collections will also avoid duplicate costs.

Kubernetes costs, also referred to as in-cluster costs, refer to spending on resources including nodes, disks, and network costs. They are monitored via the Assets dashboard. Cloud costs, also referred to as out-of-cluster or external costs, refer to spending on third party services such as services offered by cloud service providers. They are monitored via the Cloud Cost Explorer.

Creating a new collection

To begin, select Add a new Collection in the top right. If you have not created any collections yet, you should see a Add a Collection icon in the middle of your screen. You can select this instead. The ‘New Collection’ page opens. The headline for your page should be an auto-generated title. This will be the name of your collection, which you can change by selecting the pencil icon. You can provide a name and a category for your collection. Categories are custom labels you can add to multiple collections to allow for filtering of that category, if multiple collections are intended for the same team or function. Name and category can always be edited later.

Once you are satisfied with the name of your collection, you can add Kubernetes and cloud costs.

All Kubernetes and cloud spend sources will appear on this page under one of two columns: Costs in Collection, or Costs Not in Collection. When first creating your collection, there should be no costs already added. Therefore, you should select Costs Not in Collection to view a list of all available cost sources, or select Explore Costs.

Costs are then divided into two categories: Kubernetes and Cloud (see above for an explanation of these costs). You can toggle between the two groups using the Domain dropdown on the left of the page. For each group, you will see a total cost value displayed on the right of the page, above your cost table.

Costs are organized in a table, listed in descending order starting with the highest values. To add an individual item, hover your cursor over the item until you see a green Add button visible on the right of the page. Select Add to include that cost into your collection. You can add all your listed Kubernetes or cloud costs into a collection by selecting Add All, displayed next to your total cost. This is only available once you've added at least one filter. Once an item has been added to the collection, changes will be automatically saved.

You can filter and recategorize your cost table using Aggregate By and Add Filters. Aggregate By allows you to organize your costs by a listed category, and supports single and multi-aggregation. Add Filters allows for flexible filtering of your table items, including filtering by custom labels.

After having added items to your collection, selecting Costs in Collection will provide a complete list of all items, as well as key cost metrics including total and percentage costs of both Kubernetes and cloud items.

In the event there is cost overlap from conflicting Kubernetes and cloud costs, they will be reflected in a special Overlap category. Kubecost automatically subtracts the overlapping costs so that the totals seen for the collection are accurate and do not contain duplicate costs. Currently, overlap is not considered for Network costs and Load Balancer costs coming from workloads running on Azure and GCP clusters.

Managing collections

The Collections page will list all existing collections with a chart displaying cost over time and total cost. You can begin editing a collection by selecting it. Selecting the three vertical dots in the top right of the collection tile will allow you to delete it. You can also adjust your Collections display by adjusting the window of time, aggregating by item or kind, or filtering.

The Reports dashboard allows you to save defined queries across your different monitoring dashboards, including Allocations, Assets, and Cloud Costs, for quick access. All saved reports will display their type, window duration, and what they aggregate by.

Creating a report

Start by selecting Create a report, then select which dashboard you want to query for. You can also go directly to any dashboard and set up your query first. On either dashboard, once you have the parameters you want to save, select Save Report. You can choose a custom name for your report, but one will auto-generate based on your parameters, then select Save. The query will now show on the Reports dashboard.

Deleting a report

Reports can be deleted directly from the Reports dashboard by selecting the three dots icon under Actions, then selecting Delete. You can also delete a report by selecting it, then selecting Unsave Report on the corresponding dashboard. Confirm with Unsave.

The Cloud Cost Explorer is a dashboard which provides visualization and filtering of your cloud spending. This dashboard includes the costs for all assets in your connected cloud accounts by pulling from those providers' Cost and Usage Reports (CURs) or other cloud billing reports.

For help on integrating one or several cloud service providers (CSPs), see the corresponding documentation:

UI Overview

Date range

You can adjust your displayed metrics using the date range feature, represented by Last 7 days, the default range. This will control the time range of metrics that appear. Select the date range of the report by setting specific start and end dates, or by using one of the preset options.

Aggregate filters

You can adjust your displayed metrics by aggregating your cost by category. Supported fields are Account, Provider, Invoice Entity, Service, Item, as well as custom labels. The Cloud Cost Explorer dashboard supports single and multi-aggregation. See the table below for descriptions of each field.

Edit

Selecting the Edit button will allow for additional filtering and pricing display options for your cloud data.

Add filters

You can filter displayed dashboard metrics by selecting Edit, then adding a filter. Filters can be created for the following categories (see descriptions of each category in the Aggregate filters table above):

• Service

• Account

• Invoice Entity

• Provider

• Labels

Cost Metric

Additional options

Selecting Save will save your current Cloud Costs query for access on the Reports page.

Select the three horizontal dots icon to view additional options. Select Open Report to open any existing Cloud Costs reports. Selecting Download CSV will download the current query as a CSV file.

Cost metrics by CSP

The current Cloud Cost schema is optimistic in that it provides space for cost metrics that may not yet be available from some providers. As the FOCUS Spec gains more adoption among CSPs, all fields will be populated with values that match the definitions. For now, some values on certain providers are being populated with their nearest approximate. This section outlines how each value is populated on each CSP.

Kubecost Cloud provides the ability to allocate out of cluster (OOC) costs back to Kubernetes concepts like namespaces and deployments. The following guide provides the steps required for allocating OOC costs in Azure.

Adding an integration

In the Kubecost Cloud UI, begin by selecting Settings in the left navigation. Scroll down to Cloud Integrations, then select View Additional Details. The Cloud Integrations dashboard opens. Select + Add Integration. Then, select Azure Integration from the slide panel.

Step 1: Export Azure Cost Report

Read Microsoft's tutorial for creating and managing exported data. Follow the on-screen instructions in the Kubecost Cloud UI when exporting your cost report to ensure it's properly configured. Then, select Continue.

Step 2: Provide access to Azure Storage API

In the UI, provide Kubecost with the following values which can be located in the Azure Portal by selecting Storage Accounts:

• Azure Subscription ID: Subscription ID belonging to the Storage Account which stores your exported Azure cost report data.

• Azure Storage Account: Name of the Storage account where the exported Azure cost report data is being stored.

• Azure Access Key: Found by selecting Access keys in the Storage account left navigation under "Security + networking".

• Azure Storage Container: The name that you chose for the exported cost report when you set it up. This is the name of the container where the CSV cost reports are saved in your Storage account.

• Azure Container Path (optional): An optional value which should be used if there is more than one billing report that is exported to the configured container. The path provided should have only one billing export because Kubecost will retrieve the most recent billing report for a given month found within the path.

• Azure Cloud (optional): An optional value which denotes the cloud where the storage account exist, possible values are public and gov. The default is public.

When all required values have been provided, select Create Integration. Be patient while your integration is set up. The Status should initially display as Unknown. This is normal. You should eventually see the integration's Status change from Pending to Successful.

The following APIs are available in Kubecost Cloud:

Monitoring APIs

• Allocation API

• Assets API

• Cloud Cost API

Savings APIs

• Cluster Right-Sizing Recommendations API

• Request Right-Sizing Recommendations API

• Abandoned Workloads API

Using Kubecost APIs

Using the window parameter

The window parameter exists for all monitoring APIs as well as select savings APIs which refers to the date range of data for Kubecost to sample. Acceptable formats for using window parameter include:

• Units of time (ex:"15m", "24h", "7d", "48h", etc.)

• Relative time references (ex: "today", "yesterday", "week", "month", "lastweek", "lastmonth")

• Start and end unix timestamps (ex: "1586822400,1586908800")

• Start and end UTC RFC3339 pairs (ex: "2020-04-01T00:00:00Z,2020-04-03T00:00:00Z")

Kubecost Cloud does not release updates on a scheduled timeline like our self-hosted product. Instead, Kubecost Cloud product updates will be documented here periodically.

8/6/24

• Fix agent issue with cloud-agent:0.1.4 to properly respect prometheus resolution on metrics queries.

4/30/24

New Features:

• View spending trends on the Allocations and Cloud Costs Explorer pages

Bug Fixes:

• Fixed issue where new Cloud Cost reports could not be saved

• Fixed issue where Collections exports could not be downloaded

4/26/24

New Features:

• Cloud Costs page: performance improvements through pagination and streamlined API calls

• Cloud Costs page: autocomplete now available for filters and aggregations

• Clusters page: split Cluster into Cluster Name and Agent ID

• Clusters page: enhance "Add Cluster" and "Update Cluster" modals

• Improve Azure cost export downloads to be faster and more resilient to failures

Bug Fixes:

• Fixed issue where costs from the Kubernetes domain could not be added to Collections

• Fixed issue where the agent version was incorrectly formatted on the Clusters page

4/16/24

New Features:

• Allocations page: new window and filter selectors, autocomplete now available for filters and aggregations

• Improved performance for complex queries

Bug Fixes:

• Fixed issue where Cluster Status Notifications could not be configured

• Fixed issue where Contains(Prefix/Suffix) filters for labels would not return the expected results

3/6/24

Bug Fixes:

• Fixed issue where Cloud Costs without resource IDs would be under-counted within a Collection

2/26/24

New Features:

• Collections page allows users to create a single report containing both Kubernetes and cloud Costs

• Updated left navigation includes color palette and moves dashboards under 'Monitor'

• Kubecost Cloud agent now leverages 3rd party verified TLS certificates instead of self-signed certificates

• Improved accuracy of Load Balancer and Persistent Volume costs by reconciling each individual entity against cloud costs.

12/21/23

New Features:

• Cluster Status Alerts: get notified when a Kubecost Cloud agent stops reporting

• New filtering options added on the Allocations page: Department/Environment/Owner/Product/Team. Filtering by any of the properties listed above is equivalent to filtering by the following allocation label names: “department”, “env”, “owner”, “app”, “team”. The label names are not customizable at the moment.

Bug Fixes:

• Fixed issue where Cloud Cost reports could not be created directly from the Reports page

• Fixed issue where Allocation aggregation by Service would not return the expected results

• Fixed issue where Allocation aggregation by Department/Environment/Owner/Product/Team would not return the expected results. The aggregation options listed above will now aggregate by the following allocation label names: “department”, “env”, “owner”, “app”, “team”. The label names are not customizable at the moment.

12/8/23

New features:

• Alerts page

  • Get alerted when there is a significant change in your spending

• Auto-send reports

  • Schedule .csv or .pdf reports directly to your email

• Support for OpenShift Clusters

Bug fixes:

• Fixed issue where applying multiple filters in the Cloud Cost Explorer Page resulted in only the last filter being applied

• Fixed issue where an error with one of the savings insights would block access to the entire Savings page

• Fixed issue where Cloud Cost Reports with certain properties could not be saved correctly

• Fixed issue where Azure non-compute cloud items were not returned on the Cloud Cost Explorer page

• Fixed issue where the Assets page would fail to load due to NaN costs

Assets API (Querying)

GET http://app.kubecost.com/query/assets/query

The Assets API retrieves backing cost data broken down by individual assets in your cluster but also provides various aggregations of this data.

Path Parameters

Assets API (Total)

GET http://app.kubecost.com/query/assets/totals

The /totals endpoint does not aggregate, and accepts only a window and an optional filter, returning a single total cost.

Path Parameters

Kubecost values your feedback when it comes to ensuring Kubecost Cloud meets your expectations and performs seamlessly. Below is our approach to support in order to ensure customer feedback is acknowledged and assisted with.

Categorization of support requests

Issues, bugs, and other customer requests are classified into one of three categories based on level of severity:

Tier 1

These are business critical issues, involving total failure or severe degradation of services.

Tier 2

These are degraded service incidents, involving partial failure of mild degradation of services.

Tier 3

These are general issues, not considered business critical or partially degraded. This includes assistance with the product, implementation questions, and feature requests.

Service level agreement model

See below how response time and additional support actions are handled based on each tier:

Shared responsibility model

Kubecost responsibility

Kubecost manages the Kubecost Cloud environment with standby support at all times. Kubecost will assist you with deploying the Cloud Agent, and will provide support for other customer issues. We take responsibility for ensuring information on our servers follows best practices for security and privacy.

User responsibility

Users will be responsible for managing their cloud or on-prem environment, installing the Kubecost Cloud Agent in their clusters, and creating and granting permissions to billing exports. The Kubecost support team is available at cloud-support@kubecost.com to assist with any Agent or billing export issues that occur.

Submitting bug reports

If you encounter a bug or performance issue with Kubecost Cloud, reach out to cloud-support@kubecost.com. You can alternatively describe the problem on our Slack in #support.

Checking Kubecost status

You can view the operational status of Kubecost Cloud at status.kubecost.com. Outages are monitored if they occur for swift resolution. On this page, you can sign up for updates and view past outages.

Kubecost Cloud supports an advanced filtering language for all APIs.

Filter operators

The supported filter operators are:

• : Equality

  • For string fields (e.g. namespace): equality

  • For slice/array fields (e.g. services): slice contains at least one value equal (equivalent to ~:)

  • For map fields (e.g. labels): map contains key (equivalent to ~:)

• !: Inequality, or "not contains" if an array type

• ~: Contains

  • For string fields: contains

  • For slice fields: slice contains at least one value equal (equivalent to :)

  • For map fields: map contains key (equivalent to :)

• !~: NotContains, inverse of ~:

• <~: ContainsPrefix

  • For string fields: string starts with

  • For slice fields: slice contains at least one value that starts with

  • For map fields: map contains at least one key that starts with

• !<~: NotContainsPrefix, inverse of <~:

• ~>: ContainsSuffix

  • For string fields: strings ends with

  • For slice fields: slice contains at least one value that ends with

  • For map fields: map contains at least one key that ends with

• !~>: NotContainsSuffix, inverse of ~>:

Filter values are strings surrounded by ". Multiple values can be separated by commas ,.

Individual filters can be joined by + (representing logical AND) or | (representing logical OR). To use + and | in the same filter expression, scope must be denoted via ( and ). See examples.

Examples

Here are some example filters to see how the filtering language works:

• namespace:"kubecost"+container:"cost-model" Return only results that are in the kubecost namespace and are for the cost-model container.

• cluster:"cluster-one"+label[app]:"cost-analyzer" Return only results in cluster cluster-one that are labeled with app=cost-analyzer.

• cluster!:"cluster-one" Ignore results from cluster cluster-one

• namespace:"kubecost","kube-system" Return only results from namespaces kubecost and kube-system.

• namespace!:"kubecost","kube-system" Return results for all namespaces except kubecost and kube-system.

For example, in an Allocation query:

The format is essentially: <filter field> <filter op> <filter value>

Formal grammar and implementation

The Abandoned Workloads page can detect workloads which have not sent or received a meaningful rate of traffic over a configurable duration.

You can access the Abandoned Workloads page by selecting Savings in the left navigation, then selecting Remedy abandoned workloads.

The Abandoned Workloads page will display front and center an estimated savings amount per month based on a number of detected workloads considered abandoned, defined by two values:

• Traffic threshold (bytes/sec): This slider will determine a meaningful rate of traffic (bytes in and out per second) to detect activity of workloads. Only workloads below the threshold will be taken into account, therefore, as you increase the threshold, you should observe the total detected workloads increase.

• Window (days): From the main dropdown, you will be able to select the duration of time to check for activity. Presets include 2 days, 7 days, and 30 days. As you increase the duration, you should observe the total detected workloads increase.

Filtering your abandoned workloads

Beneath your total savings value and slider scale, you will see a dashboard containing all abandoned workloads. The total number of workloads detected should appear in the top right of the table.

You can filter your workloads through four dropdowns; across clusters, namespaces, owners, and owner kinds.

Selecting an individual line item will expand the item, providing you with additional traffic data for that item.

Kubecost Cloud can provide recommendations for right-sizing your container requests to ensure they are as cost-effective as possible. Recommendations are provided for all namespaces within your cluster.

Viewing request recommendations

You can access request right-sizing by selecting Savings in the left navigation, then select the Right-size your container requests panel.

On the Container Request Right-sizing Recommendations page, you will see a table containing all namespaces/controller pairs and the cluster and container associated with each. You will also see the requested and recommended RAM/CPU, the current efficiency, and finally estimated monthly savings by adopting recommendations.

Configuring your recommendations

The displayed right-sizing recommendations are calculated by taking into account your environment profile. You can optionally configure this for more optimal results by selecting Customize above the table.

• Window: The range of time Kubecost will read for resource activity to determine its recommendations.

• Profile: Refers to the type of environment. The selected value for Profile may restrict you from customizing certain other values.

  • Production: Stable container activity, will provide some extra space for potential spikes in activity.

  • Development: Container can tolerate small amount of instability, will run somewhat close to capacity.

  • High availability: Container should avoid instability at all costs, will size container with lots of extra space to account for unexpected spikes in activity.

• CPU/RAM recommendation algorithm: The algorithm used to compute the recommendations. Currently, the only supported option is Percentile.

• CPU/RAM target utilization: These can be set to limit recommended utilization of resources below a percentage threshold.

• CPU/RAM percentile: Percentage of data points that will be sampled within your window range. Outlier data will be filtered out when determining recommendations.

• Add Filters: Filter the table of namespaces/controllers to be equal or not equal to values of one or several different categories such as cluster, label, or pod. For example, if you want to only see namespace/cluster pairs within the namespace kube-system, select Namespace from the first dropdown, select Equals from the second dropdown, then provide the value "kube-system" in the text box. Select the plus icon to confirm your filter. Multiple filters can be applied.

Select Save to confirm your customization. The table should update accordingly.

Alerts allow teams to receive updates on real-time Kubernetes spend. This doc gives an overview of how to configure alerts sent through email, Slack, and Microsoft Teams. Alerts are either created to monitor specific data sets and trends, or they must be toggled on or off.

Currently, only spend changes in allocation costs are supported in Kubecost Cloud. This alert type detects sudden changes in allocation spending (spending relating to any Kubernetes objects) when compared to a previous interval.

Creating an alert

To begin, select Create Alert. The 'Create New Alert' slide panel opens. Provide the following fields:

• Alert Type: Currently only can be set to Allocation Spend Change.

• Window: Spend over selected interval will be compared against the previous interval. Supports Daily, Weekly, and Monthly.

• Cost Threshold (%): Percent change to trigger alert. Must be configured as a negative value to detect sudden decreases in spend.

• Aggregation: The Kubernetes object type to monitor spend for.

• Filter: Optional, filter to a specific selected Kubernetes object.

• Recipients: Choose which platform(s) you want your alert sent through. Kubecost Alerts can be sent via Slack/Microsoft Teams webhook URLs, or by email. All three platforms do not need values provided for them, but may trigger global recipients if left blank (see below).

Before you finalize your alert, you can select Test Alert, which will send a test alert across the provided webhooks/emails. This is useful for ensuring your alert has been configured correctly. If the alert was sent successfully, you can finalize your changes by selecting Submit. Your alert will now appear on the Alerts page, and can be tested, edited, or deleted at any time by selecting the corresponding icons in your alert's line.

Global recipients

Global recipients specify a default fallback recipient for each type of message. If an alert does not define any email recipients, its messages will be sent to any emails specified in the Global Recipients email list. Likewise, if an alert does not define a webhook, its messages will be sent to the webhook, if one is present. Alerts that do define recipients will ignore the global setting for recipients of that type.

You can define global recipients by selecting Edit in the Global Recipients box, then selecting the desired platform type and providing a corresponding value. Confirm by selecting Save. You can only provide one webhook per platform as a global recipient.

Cluster Status Alerts

Cluster Status notifications, if enabled, provide notice if and when the cluster agent(s) installed have stopped reporting data to Kubecost Cloud.

To configure, select Add in the Cluster Status Alerts box. The 'Cluster Status' slide panel opens. Provide any Slack/Microsoft Teams webhooks or email addresses to which you want the alert sent. Confirm by selecting Enable.

Kubecost Cloud can provide recommendations for right-sizing your clusters to ensure they are configured in the most cost-effective way. Recommendations are available for any and all clusters.

Viewing cluster recommendations

You can access cluster right-sizing by selecting Savings in the left navigation, then select the Right-size your cluster nodes panel.

Kubecost will offer two recommendations: simple (uses one node type) and complex (uses two or more node types). Kubecost may hide the complex recommendation when it is more expensive than the simple recommendation, and present a single recommendation instead. These recommendations and their metrics will be displayed in a chart next to your existing configuration in order to compare values like total cost, node count, and usage.

You can toggle on Show advanced metrics to view more details about your cluster resource consumption.

Configuring your recommendations

Kubecost Cloud provides its right-sizing recommendations based on the characteristics of your cluster. You have the option to edit certain properties to generate relevant recommendations. Select Filter to configure your settings in the Cluster Sizing Settings window.

There are multiple dropdown menus to consider:

• In the Cluster dropdown, you can select the individual cluster you wish to apply right-sizing recommendations to.

• In the Profile dropdown, select the most relevant category of your cluster. You can select Production, Development, or High Availability.

  • Production: Stable cluster activity, will provide some extra space for potential spikes in activity.

  • Development: Cluster can tolerate small amount of instability, will run cluster somewhat close to capacity.

  • High availability: Cluster should avoid instability at all costs, will size cluster with lots of extra space to account for unexpected spikes in activity.

• In the Architecture dropdown, select either x86 or ARM. You may only see x86 as an option. This is normal. At the moment, ARM architecture recommendations are only supported on AWS clusters.

With this information provided, Kubecost can provide the most accurate recommendations for running your clusters efficiently.

The Savings page provides miscellaneous functionality to help you use resources more effectively and assess wasteful spending. In the center of the page, you will see your estimated monthly savings available.

Savings insights

The Savings page provides an array of panels containing different insights capable of lowering your Kubernetes and cloud spend. The current list of Savings features includes:

• Right-size your cluster nodes

• Right-size your container requests

• Remedy abandoned workloads

Kubecost Cloud is available for licensing on GCP Marketplace and can be installed in minutes. This guide will take you through licensing through GCP Marketplace, and next steps for setting up Kubecost Cloud. Kubecost currently offers 30 days of Kubecost Cloud free without licensing fees.

Prerequisites

• Set up a Google Cloud account with an attached billing account.

• Set up a Kubecost Cloud account

GCP Marketplace licensing guide

On the Product details page for Kubecost, select Subscribe. You will be taken to an Order Summary page.

Under "1. Select Plan", the default plan should be Cloud Pro, and the default usage fee should be USD 0.167 per node per day. You can use the Pricing Calculator in the right sidebar to determine estimated costs by providing estimated timeframe of usage from 1 day to 1 year, and total node count.

Under "2. Purchase Details", select the billing account you wish to associate Kubecost Cloud with from the dropdown.

Under "3. Terms", read and agree to the terms and conditions, which include Google Cloud Marketplace Terms of Service as well as the Kubecost Terms of Service. Then, select Subscribe. Wait a moment while your order request is processed. Select Go to Product Page in the pop-up which should appear once the order has been sent to Kubecost. If you have not already, select Sign up with provider on the product page and provide all necessary user info to get your Kubecost Cloud account set up. Purchase orders should be automatically processed. Refresh the product page until you see Manage on Provider. Selecting this will take you from GCP Marketplace to the Kubecost Cloud login page.

You should now have access to the Kubecost Cloud dashboard.

Next steps

After having licensed Kubecost Cloud, you are able to install the Kubecost Cloud Agent onto all clusters you want to receive cost metrics for. See our existing Installation and Onboarding guide for help getting started.

The Allocations page allows you to aggregate cost by namespace, deployment, service, and other native Kubernetes concepts. While selecting Single Aggregation, you will only be able to categorize by one concept at a time. While selecting Multi Aggregation, you will be able to filter for multiple concepts at the same time.

Costs aggregations are also visible by other meaningful organizational concepts, e.g. Team, Department, and Product. These aggregations are based on Kubernetes labels, referenced at both the pod and namespace-level, with labels at the pod-level being favored over the namespace label when both are present. Workloads without the relevant label will be shown as __unallocated__.

UI Overview

Date range

You can control the window of allocation spend by selecting Last 7 days (the default option), and choosing the time window you want to view spend for. When using custom dates instead of a preset, select Apply to make changes.

Aggregate By

Here you can aggregate cost by namespace, deployment, service, and other native Kubernetes concepts. While selecting Single Aggregation, you will only be able to categorize by one concept at a time. While selecting Multi Aggregation, you will be able to filter for multiple concepts at the same time.

Costs aggregations are also visible by other meaningful organizational concepts, e.g. Team, Department, and Product. These aggregations are based on Kubernetes labels, referenced at both the pod and namespace-level, with labels at the pod-level being favored over the namespace label when both are present. Workloads without the relevant label will be shown as __unallocated__.

Edit Report

Selecting Edit Report will provide more options of filtering and visualizing your window query.

Idle costs

For an overview of what idle costs are and how they are calculated, see the bottom of the page.

Customize how you wish idle costs to be displayed in your report chart:

• Hide: Hides idle costs.

• Separate By Cluster: Associates idle costs with the clusters they are a part of.

• Separate By Node: Associates idle costs with the nodes they are scheduled to.

Chart

View Allocations data in the following formats:

1. Cost over time: Cost per aggregation broken down over days or hours depending on date range

2. Cost: Total cost per aggregation over date range

3. Proportional cost: Cost per aggregate displayed as a percentage of total cost over date range

4. Cost Treemap: Hierarchically structured view of costs in current aggregation

While Cost over time is selected, hover over any interval to receive a breakdown of your spending.

Cost metric

View either cumulative or run rate costs measured over the selected time window based on the resources allocated.

• Cumulative Cost: represents the actual/historical spend captured by the Kubecost agent over the selected time window

• Rate metrics: Monthly, daily, or hourly "run rate" cost, also used for projected cost figures, based on samples in the selected time window

Costs allocations are based on the following:

1. Resources allocated, i.e. max of resource requests and usage

2. The cost of each resource

3. The amount of time resources were provisioned

For more information, refer to the OpenCost spec.

Filters

Filter resources by namespace, cluster, and/or Kubernetes label to more closely investigate a rise in spend or key cost drivers at different aggregations such as deployments or pods. When a filter is applied, only resources with this matching value will be shown. These filters are also applied to external out-of-cluster asset tags:

Comma-separated lists are supported to filter by multiple categories, e.g. namespace filter equals kube-system,kubecost. Wild card filters are also supported, indicated by a * following the filter, e.g. namespace=kube* to return any namespace beginning with kube.

Additional options

The three horizontal dots icon provides additional means of handling your query data. You can open a saved report or download your query data as a CSV file.

Idle

Cluster idle cost is defined as the difference between the cost of allocated resources and the cost of the hardware they run on. Allocation is defined as the max of usage and requests. It can also be expressed as follows:

idle_cost = sum(cluster_cost) - (cpu_allocation_cost + ram_allocation_cost + gpu_allocation_cost) where allocation = max(request, usage)

Idle costs can also be thought of as the cost of the space that the Kubernetes scheduler could schedule pods, without disrupting any existing workloads, but it is not currently.

Kubecost Cloud uses an agent to gather metrics and send them to our SaaS platform. The agent requires an Agent package and a daemonSet:

Kubecost Agent package

• Cost-model: Provides cost allocation calculations and metrics, reads from and scraped by Prometheus server.

• Prometheus server: Short-term time-series data store (14 days or less)

• ConfigMap-Reload: Updates Prometheus when changes are made.

Network costs DaemonSet

• Used to allocate costs to the workload responsible for egress costs

• Enabled on install (to learn how to uninstall the DaemonSet, see below)

Architecture overview

Disabling the network costs DaemonSet

The network costs DaemonSet will be installed to your Kubecost Cloud by default, however you can manually disable it by running this Helm upgrade command:

helm upgrade --install kubecost-cloud \
--repo https://kubecost.github.io/kubecost-cloud-agent/ kubecost-cloud-agent \
--namespace kubecost-cloud --create-namespace \
-f https://raw.githubusercontent.com/kubecost/kubecost-cloud-agent/main/values-cloud-agent.yaml \
--set imageVersion="lunar-sandwich.v0.1.2" \
--set cloudAgentKey="AGENTKEY" \
--set cloudAgentClusterId="cluster-1" \
--set cloudReportingServer="collector.app.kubecost.com:31357" \
--set networkCosts.enabled=false

Container Request Right Sizing Recommendation API

GET app.kubecost.com/query/savings/requestSizingV2

The container request right sizing recommendation API provides recommendations for based on configurable parameters and estimates the savings from implementing those recommendations on a per-container, per-controller level. If the cluster-level resources stay static, then there may not be significant savings from applying Kubecost's recommendations until you reduce your cluster resources. Instead, your idle allocation will increase.

Query Parameters

Kubecost Cloud has select endpoints available for API access. The default API access endpoint will be api.app.kubecost.com. Obtaining access involves generating an API key value, which is then used to generate a personal access token. The token, together with your team ID, will then allow you to make queries.

Access tutorial

Step 1: Generating an API key value

To get started, visit Settings in the Kubecost Cloud UI, then under 'My API Key', select Generate Key. A key value will generate in a text box with a lifespan of one month. Save this value, as it will not be retrievable later. Only one key can exist at any given time for a given user. Remove the existing key to generate a new one.

Step 2: Generating a personal access token

With your generated API key value, run the following command to generate a personal access token:

The generated token is short-lived and expires after ten minutes. This token can be used in token authentication for API requests to all available Kubecost endpoints.

Step 3: Generating your team ID value

You will also need to retrieve your team ID value. This is a value associated with your . You can find this ID at the bottom of Settings, in the 'Team Info' section.

Step 4: Using the API endpoints

To verify your token is working, authenticate to the Kubecost Cloud API access route using the personal access token generated in step 2 and the team ID retrieved in Step 3. For example:

Keep in mind that the TOKEN_VALUE must be generated every ten minutes while using the API.

Abandoned Workloads API

GET http://app.kubecost.com/query/savings/abandonedWorkloads

The Abandoned Workloads API suggests cluster workloads that have been abandoned based on network traffic levels.

Path Parameters

Allocation API (Querying)

GET http://app.kubecost.com/query/allocation/query

The Allocation API is the preferred way to query for costs and resources allocated to Kubernetes workloads and optionally aggregated by Kubernetes concepts like namespace, controller, and label.

Path Parameters

Allocation API (Total)

GET http://app.kubecost.com/query/allocation/totals

The Allocation API is the preferred way to query for costs and resources allocated to Kubernetes workloads. The /totals endpoint does not aggregate, and accepts only a window and an optional filter, returning a single total cost.

Path Parameters

Cloud Cost API (Querying)

GET http://app.kubecost.com/query/cloudcost/query

The Cloud Cost API is the preferred way to query for costs and resources related to cloud provider services.

Path Parameters

Cloud Costs API (Total)

GET http://app.kubecost.com/query/cloudcost/totals

The /totals endpoint does not aggregate, and accepts only a window and an optional filter, returning a single total cost.

Path Parameters

Kubecost's Cluster Right-Sizing Recommendation API can monitor the resource utilization of your clusters and offer cost-effective right-sizing solutions.

Cluster Right-Sizing Recommendation API

GET http://app.kubecost.com/query/savings/clusterSizingETL

Path Parameters