AWS Distro for Open Telemetry

See also

• Amazon Managed Service for Prometheus (AMP) Overview

• AWS Agentless AMP

• AMP with Kubecost Prometheus (remote_write)

Overview

This guide will walk you through the steps to deploy Kubecost with AWS Distro for Open Telemetry (ADOT) to collect metrics from your Kubernetes clusters utilizing the EKS-Optimized license.

Kubecost EKS-Optimized allows for 15 days of query history. Unlock unlimited history with Kubecost Enterprise.

Architecture Diagram

Prerequisites

1. An AWS Managed Prometheus Workspace is required to use ADOT

2. AWS IAM permissions to add permissions for Kubecost to read from the workspace

Before following this guide, make sure you've reviewed AWS' Set up metrics ingestion using AWS Distro for Open Telemetry on an Amazon Elastic Kubernetes Service cluster to enable the ADOT collector daemonSet.

This guide assumes that the Kubecost Helm release name and the Kubecost namespace have the same value (usually this will be kubecost), which allows a global find and replace on YOUR_NAMESPACE.

Configuration

Clone this repository that contains all of the configuration files you will need to deploy Kubecost with ADOT.

git clone https://github.com/kubecost/poc-common-configurations.git
cd poc-common-configurations/aws-amp/adot

Update all configuration files with your cluster name (replace all instances of YOUR_CLUSTER_NAME_HERE). The examples use cluster_id for the key of the key:value pair for the cluster name. You can use any key you want, including what is likely already being used.

ADOT configuration

There are many options for deploying the ADOT daemonSet. At a minimum, Kubecost needs the provided scrape config to be added to the ADOT Prometheus ConfigMap. This sample ConfigMap also contains cAdvisor metrics, which is required by Kubecost.

kubectl apply -f example-configs/prometheus-daemonset.yaml -n adot-col

Alternatively, you can add these items to your existing ConfigMap.

For the Kubecost scrape_configs job, honor_labels: true must be set. Without this, you will likely only see the kube-system or kubecost namespace in the UI.

Kubecost AWS IAM setup

1. Create the Kubecost namespace:

   kubectl create ns YOUR_NAMESPACE

2. Create the AWS IAM policy to allow Kubecost to query metrics from AMP:

   aws iam create-policy --policy-name kubecost-read-amp-metrics --policy-document file://iam-read-amp-metrics.json

3. (Optional) Create the AWS IAM policy to allow Kubecost to find savings in the AWS Account:

   aws iam create-policy --policy-name DescribeResources --policy-document file://iam-describeCloudResources.json

4. (Optional) Create the AWS IAM policy to allow Kubecost to write to find account-level tags:

   aws iam create-policy --policy-name OrganizationListAccountTags --policy-document file://iam-listAccounts-tags.json

Kubecost primary installation

1. Configure the Kubecost Service Account:

   eksctl create iamserviceaccount \
       --name kubecost-sa \
       --namespace YOUR_NAMESPACE \
       --cluster YOUR_CLUSTER_NAME_HERE --region YOUR_REGION \
       --attach-policy-arn arn:aws:iam::AWS_ACCOUNT_NUMBER:policy/kubecost-read-amp-metrics \
       --attach-policy-arn arn:aws:iam::AWS_ACCOUNT_NUMBER:policy/OrganizationListAccountTags \
       --attach-policy-arn arn:aws:iam::AWS_ACCOUNT_NUMBER:policy/DescribeResources \
       --override-existing-serviceaccounts --approve

2. Install Kubecost:

   aws ecr-public get-login-password --region us-east-1 | helm registry login --username AWS --password-stdin public.ecr.aws
   helm install YOUR_NAMESPACE \
       oci://public.ecr.aws/kubecost/cost-analyzer \
       -f values-kubecost-primary.yaml
       -f https://raw.githubusercontent.com/kubecost/cost-analyzer-helm-chart/develop/cost-analyzer/values-eks-cost-monitoring.yaml

Kubecost agent installation

This assumes you have created the IAM policies above. If using multiple AWS accounts, you will need to create the policies in each account (section titled Kubecost AWS IAM setup).

1. Create the Kubecost namespace:

   kubectl create ns YOUR_NAMESPACE

2. Configure the Kubecost Service Account:

   eksctl create iamserviceaccount \
       --name kubecost-sa \
       --namespace YOUR_NAMESPACE \
       --cluster YOUR_CLUSTER_NAME_HERE --region YOUR_REGION \
       --attach-policy-arn arn:aws:iam::AWS_ACCOUNT_NUMBER:policy/kubecost-read-amp-metrics \
       --attach-policy-arn arn:aws:iam::AWS_ACCOUNT_NUMBER:policy/OrganizationListAccountTags \
       --attach-policy-arn arn:aws:iam::AWS_ACCOUNT_NUMBER:policy/DescribeResources \
       --override-existing-serviceaccounts --approve

3. Deploy the Kubecost agent:

   aws ecr-public get-login-password --region us-east-1 | helm registry login --username AWS --password-stdin public.ecr.aws
   helm install YOUR_NAMESPACE \
       oci://public.ecr.aws/kubecost/cost-analyzer \
       -f values-kubecost-agent.yaml
       -f https://raw.githubusercontent.com/kubecost/cost-analyzer-helm-chart/develop/cost-analyzer/values-eks-cost-monitoring.yaml

ADOT daemonSet quick install

See this example .yaml file for an all-in-one ADOT DS config.

Troubleshooting

For more help troubleshooting, see our Amazon Managed Service for Prometheus (AMP) Overview doc.