The Kubecost Allocations dashboard allows you to quickly see allocated spend across all native Kubernetes concepts, e.g. namespace, k8s label, and service. It also allows for allocating cost to organizational concepts like team, product/project, department, or environment. This document explains the metrics presented and describes how you can control the data displayed in this view.

Configuring your query

Kubecost provides a variety of options for configuring your allocations queries to view the information you need. Below is a table of the major configuration options, with in-depth explanations in this article for how they work.

Date Range

Select the date range of the report, called the window, by setting specific start and end dates, or by using one of the preset options. You can use Select Start and Select End to establish custom date ranges as well.

Step size

Step size refers to the length of time of each group of data displayed on your dashboard across the window. Options are Default, Daily, Weekly, Monthly, and Quarterly. When retaining long periods of data through custom configurations (such as Prometheus), consider using larger step sizes to avoid potential display errors. The step size when selecting Default is dependent on the size of your window.

Aggregate By filters

Here you can aggregate cost by namespace, deployment, service, and other native Kubernetes concepts. While selecting Single Aggregation, you will only be able to select one concept at a time. While selecting Multi Aggregation, you will be able to filter for multiple concepts at the same time.

Costs aggregations are also visible by other meaningful organizational concepts, e.g. Team, Department, and Product. These aggregations are based on Kubernetes labels, referenced at both the pod and namespace-level, with labels at the pod-level being favored over the namespace label when both are present. The Kubernetes label name used for these concepts can be configured in Settings or in values.yaml after setting kubecostProductConfigs.labelMappingConfigs.enabled to true. Workloads without the relevant label will be shown as __unallocated__.

To find what pods are not part of the relevant label set, you can either apply an __unallocated__ label filter in this allocation view or explore variations of the following kubectl commands:

kubectl get pods -l 'app notin (prometheus, cost-analyzer, ...)' --all-namespaces
kubectl get pods --show-labels -n <TARGET_NAMESPACE>

Edit report

The Edit icon has additional options for configuring your query such as how to display your data, adding filters, and configuring shared resources.

Idle Costs

Allocating idle costs proportionately distributes slack or idle cluster costs to tenants. Idle refers to resources that are provisioned but not being fully used or requested by a tenant.

As an example, if your cluster is only 25% utilized, as measured by the max of resource usage and requests, applying idle costs would proportionately increase the cost of each pod/namespace/deployment by 4x. This feature can be enabled by default in Settings.

The idle costs dropdown allows you to choose how you wish your idle costs to be displayed:

• Hide: Hide idle costs completely.

• Separate: Idle costs appear as their own cost, visualized as a gray-colored bar in your display table.

• Share By Cluster: Idle costs are grouped by the cluster they belong to.

• Share By Node: Idle costs are grouped by the node they belong to.

To learn more about sharing idle costs, see here.

Chart

View Allocation data in the following formats:

1. Cost: Total cost per aggregation over date range

2. Cost over time: Cost per aggregation broken down over days or hours depending on date range

3. Efficiency over time: Shows resource efficiency over given date range

4. Proportional cost: Cost per aggregate displayed as a percentage of total cost over date range

5. Cost Treemap: Hierarchically structured view of costs in current aggregation

You can select Edit > Chart > Cost over time from the dropdown to have your data displayed on a per-day basis. Hovering over any day's data will provide a breakdown of your spending.

Cost metric

View either cumulative or run rate costs measured over the selected time window based on the resources allocated.

• Cumulative Cost: represents the actual/historical spend captured by the Kubecost agent over the selected time window

• Rate metrics: Monthly, daily, or hourly "run rate" cost, also used for projected cost figures, based on samples in the selected time window

Costs allocations are based on the following:

1. Resources allocated, i.e. max of resource requests and usage

2. The cost of each resource

3. The amount of time resources were provisioned

For more information, refer to the OpenCost spec.

Filters

Filter resources by namespace, clusterID, and/or Kubernetes label to more closely investigate a rise in spend or key cost drivers at different aggregations such as deployments or pods. When a filter is applied, only resources with this matching value will be shown. These filters are also applied to external out-of-cluster (OOC) asset tags. Supported filters are as follows:

Comma-separated lists are supported to filter by multiple categories, e.g. namespace filter equals kube-system,kubecost. Wild card filters are also supported, indicated by a * following the filter, e.g. namespace=kube* to return any namespace beginning with kube.

Advanced filtering

You can also implement more advanced forms of filtering to include or exclude values including prefixes or suffixes for any of the above categories in the table. Selecting the filtering dropdown (default Equals) will show you all available filtering options. These are reflective of Kubecost's v2 filtering language.

Shared resources

Select how shared costs set on the settings page will be shared among allocations. Pick from default shared resources, or select a custom shared resource. A custom shared resource can be selected in the Configure custom shared resources feature at the bottom of the Edit window.

Additional options

The three horizontal dots icon (directly next to Save) will provide additional options for handling your report:

• Open Report: Allows you to open one of your saved reports without first navigating to the Reports page

• Alerts: Send one of four reports routinely: recurring, efficiency, budget, and spend change

• Download CSV: Download your current report as a CSV file

• Download PDF: Download your current report as a PDF file

Cost metrics table

Cost allocation metrics are available for both in-cluster and OOC resources:

Cost efficiency table example

Additional options column

The rightmost column in the Allocations metrics table allows you to perform additional actions on individual line items (functionality will vary based on how you aggregate):

• Inspect: Opens an advanced cost overview of the namespace in a new tab.

• Inspect Shared Costs: Opens an advanced cost overview of your shared costs in a new tab.

• View Right-Sizing: Opens the Container Request Right-Sizing Recommendations page in a new tab.

This document describes how Kubecost calculates network costs.

Network cost calculation methodology

Kubecost uses best-effort to allocate network transfer costs to the workloads generating those costs. The level of accuracy has several factors described below.

There are two primary factors when determining how network costs are calculated:

Base functionality

A default installation of Kubecost will use the onDemand rates for internet egress and proportionally assign those costs by pod using the metric container_network_transmit_bytes_total. This is not exactly the same as costs obtained via the network costs DaemonSet, but will be approximately similar.

Network costs DaemonSet

When you enable the network costs DaemonSet, Kubecost has the ability to attribute the network-byte traffic to specific pods. This will allow the most accurate cost distribution, as Kubecost has per-pod metrics for source and destination traffic.

Cloud integration

Kubecost uses cloud integration to pull actual cloud provider billing information. Without enabling cloud integration, these prices will be based on public onDemand pricing.

Cloud providers allocate data transfers as line-items on a per-node basis. Kubecost will allocate network transfer costs based on each pod's share of container_network_transmit_bytes_total of its node.

This will result in a accurate node-based costs. However, it is only estimating the actual pod/application responsible for the network-transfer costs.

Both cloud integration and network cost DaemonSet

Enabling both cloud-integration and the networkCosts DaemonSet allows Kubecost to give the most accurate data transfer costs to each pod.

Limitations

At this time, there is a minor limitation where Kubecost cannot determine accurate costs for pods that use hostNetwork. These pods, today, will share all costs with the costs with the node.

For teams interested in reducing their Kubernetes costs, it's beneficial to first understand how provisioned resources have been used. There are two major concepts to start with: pod resource efficiency and cluster idle costs.

Efficiency

Pod resource efficiency is defined as the resource utilization versus the resource request over a given time window. It is cost-weighted and can be expressed as follows:

(((CPU Usage / CPU Requested) * CPU Cost) + ((RAM Usage / RAM Requested) * RAM Cost)) / (RAM Cost + CPU Cost)

where CPU Usage = rate(container_cpu_usage_seconds_total) over the time window RAM Usage = avg(container_memory_working_set_bytes) over the time window

For example, if a pod is requesting 2CPU and 1GB, using 500mCPU and 500MB, CPU on the node costs $10/CPU, and RAM on the node costs $1/GB, we have ((0.5/2) * 20 + (0.5/1) * 1) / (20 + 1) = 5.5 / 21 = 26%

Idle

Cluster idle cost is defined as the difference between the cost of allocated resources and the cost of the hardware they run on. Allocation is defined as the max of usage and requests. It can also be expressed as follows:

idle_cost = sum(cluster_cost) - (cpu_allocation_cost + ram_allocation_cost + gpu_allocation_cost)

where allocation = max(request, usage)

Node idle cost can be expressed as:

idle_cost = sum(node_cost) - (cpu_allocation_cost + ram_allocation_cost + gpu_allocation_cost)

where allocation = max(request, usage)

So, idle costs can also be thought of as the cost of the space that the Kubernetes scheduler could schedule pods, without disrupting any existing workloads, but it is not currently.

Sharing idle

Idle can be charged back to pods on a cost-weighted basis or viewed as a separate line item. As an example, consider the following representations:

• [ ... ] = cluster

• ( ... ) = node

• wN = workload

• -- = idle capacity

Then, a cluster might look like:

[ ( w1, w2, w3, w4, --, --), (w5, --, --, --, --, --) ]

In total, there are 12 units of resources, and idle can be shared as follows:

• Separate: In this single cluster across two nodes, there are 7 total idles.

• Share By Node: The first node has 4 resources used and 2 idle. The second node has 1 resource used and 5 idle. If you share idle by node, then w1-4 will share 2 idles, and w5 will get 5 idles.

• Share By Cluster: The single cluster has 5 resources used and 7 idle. If you share idle by cluster, then w1-5 will share the 7 idles.

Distributing idle when aggregating

If for example you are aggregating by namespace, idle costs will be distributed to each namespace proportional to how much that namespace costs. Specifically:

namespace_cpu_idle_cost = (namespace_cpu_cost / (total_cpu_cost - idle_cpu_cost)) * idle_cpu_cost

This same principle applies for ram, and also applies to any aggregation that is used (e.g. Deployment, Label, Service, Team).

Target values for efficiency and idle

The most common pattern for cost reduction is to ensure service owners tune the efficiency of their pods, and ensure cluster owners scale resources to appropriately minimize idle.

It's recommended to target idle in the following ranges:

• CPU: 50%-65%

• Memory: 45%-60%

• Storage: 65%-80%

Target figures are highly dependent on the predictability and distribution of your resource usage (e.g. P99 vs median), the impact of high utilization on your core product/business metrics, and more. While too low resource utilization is wasteful, too high utilization can lead to latency increases, reliability issues, and other negative behavior.