AWS Agentless AMP

Overview

Using AMP allows multi-cluster Kubecost with EKS-Optimized licenses.

This guide will walk you through the steps to deploy Kubecost with AWS Agentless AMP to collect metrics from your Kubernetes cluster.

Keep in mind that "agentless" refers to the Prometheus scraper, not the Kubecost agent. The Kubecost agent is still required to collect metrics from the cluster.

The guide below assumes a multi-cluster setup will be used, which is supported with the EKS-Optimized license that is enabled by following the below guide.

Prerequisites

Follow this Using an AWS managed collector guide to enable the managed collector.

This guide assumes that the Kubecost Helm release name and the Kubecost namespace are equal, which allows a global find-and-replace on $KUBECOST_NAMESPACE.

Architecture diagram

Agentless AMP Configuration

AMP setup

Clone this poc-common-configurations repository that contains all of the configuration files you will need to deploy Kubecost with AWS Agentless AMP.
```
git clone https://github.com/kubecost/poc-common-configurations.git
cd poc-common-configurations/aws/amp-agentless
```
Update all configuration files with your cluster name (replace all YOUR_CLUSTER_NAME_HERE).

Build the configuration variables:

CLUSTER_NAME=YOUR_CLUSTER_NAME_HERE
CLUSTER_REGION=us-east-2
KUBECOST_NAMESPACE=kubecost
WORKSPACE_ID=ws-YOUR_WORKSPACE_ID
AWS_ACCOUNT_ID=11111111111
WORKSPACE_ARN=$(aws amp describe-workspace --workspace-id $WORKSPACE_ID --output json | jq -r .workspace.arn)
CLUSTER_JSON=$(aws eks describe-cluster --name $CLUSTER_NAME --region $CLUSTER_REGION --output json)
CLUSTER_ARN=$(echo $CLUSTER_JSON | jq -r .cluster.arn)
SECURITY_GROUP_IDS=$(echo $CLUSTER_JSON | jq -r .cluster.resourcesVpcConfig.clusterSecurityGroupId)
SUBNET_IDS=$(echo $CLUSTER_JSON | jq -r '.cluster.resourcesVpcConfig.subnetIds | @csv')

Create the Kubecost scraper:

KUBECOST_SCRAPER_OUTPUT=$(aws amp create-scraper --output json \
    --alias kubecost-scraper \
    --source eksConfiguration="{clusterArn=$CLUSTER_ARN, securityGroupIds=[$SECURITY_GROUP_IDS],subnetIds=[$SUBNET_IDS]}" \
    --scrape-configuration configurationBlob="$(base64 scraper-kubecost-with-networking.yaml|tr -d '\n')" \
    --destination ampConfiguration="{workspaceArn=$WORKSPACE_ARN}")
echo $KUBECOST_SCRAPER_OUTPUT
KUBECOST_SCRAPER_ID=$(echo $KUBECOST_SCRAPER_OUTPUT|jq -r .scraperId)
echo $KUBECOST_SCRAPER_ID

Get the ARN of the scraper:

ARN_PART=$(aws amp describe-scraper --output json --region $CLUSTER_REGION --scraper-id $KUBECOST_SCRAPER_ID | jq -r .scraper.roleArn | cut -d'_' -f2)
ROLE_ARN_KUBECOST_SCRAPER="arn:aws:iam::$AWS_ACCOUNT_ID:role/AWSServiceRoleForAmazonPrometheusScraper_$ARN_PART"
echo $ROLE_ARN_KUBECOST_SCRAPER

Add the ARN of the scraper to the kube-system/aws-auth configMap:

eksctl create iamidentitymapping \
    --cluster $CLUSTER_NAME --region $CLUSTER_REGION \
    --arn $ROLE_ARN_KUBECOST_SCRAPER \
    --username aps-collector-user

Create a scraper for cAdvisor and node exporter. Node exporter is optional. cAdvisor is required, but may already be available.

CADVSIOR_SCRAPER_OUTPUT=$(aws amp create-scraper --output json \
    --alias cadvisor-scraper \
    --source eksConfiguration="{clusterArn=$CLUSTER_ARN, securityGroupIds=[$SECURITY_GROUP_IDS],subnetIds=[$SUBNET_IDS]}" \
    --scrape-configuration configurationBlob="$(base64 scraper-cadvisor-node-exporter.yaml|tr -d '\n')" \
    --destination ampConfiguration="{workspaceArn=$WORKSPACE_ARN}")
echo $CADVSIOR_SCRAPER_OUTPUT
CADVSIOR_SCRAPER_ID=$(echo $CADVSIOR_SCRAPER_OUTPUT|jq -r .scraperId)
echo $CADVSIOR_SCRAPER_ID

Get the ARN of the scraper:

ARN_PART=$(aws amp describe-scraper --output json --region $CLUSTER_REGION --scraper-id $CADVSIOR_SCRAPER_ID | jq -r .scraper.roleArn | cut -d'_' -f2)
ROLE_ARN_CADVSIOR_SCRAPER="arn:aws:iam::$AWS_ACCOUNT_ID:role/AWSServiceRoleForAmazonPrometheusScraper_$ARN_PART"
echo $ROLE_ARN_CADVSIOR_SCRAPER

Add the ARN of the scraper to the kube-system/aws-auth configmap:

eksctl create iamidentitymapping \
    --cluster $CLUSTER_NAME --region $CLUSTER_REGION \
    --arn $ROLE_ARN_CADVSIOR_SCRAPER \
    --username aps-collector-user

Apply the agentless RBAC permissions:
```
kubectl apply -f rbac.yaml
```

Kubecost primary cluster installation

Create the Kubecost namespace:
```
kubectl create ns $KUBECOST_NAMESPACE
```

Create the AWS IAM policy to allow Kubecost to query metrics from AMP:

aws iam create-policy --policy-name kubecost-read-amp-metrics --policy-document file://iam-read-amp-metrics.json

(Optional) Create the AWS IAM policy to allow Kubecost to find savings in the AWS Account:

aws iam create-policy --policy-name DescribeResources --policy-document file://iam-describeCloudResources.json

(Optional) Create the AWS IAM policy to allow Kubecost to write to find account-level tags:

aws iam create-policy --policy-name OrganizationListAccountTags --policy-document file://iam-listAccounts-tags.json

Configure the Kubecost Service Account:

If the following fails, be sure that IRSA is enabled on your EKS cluster. https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

eksctl create iamserviceaccount \
--name kubecost-sa \
--namespace $KUBECOST_NAMESPACE \
--cluster $CLUSTER_NAME --region $CLUSTER_REGION \
--attach-policy-arn arn:aws:iam::$AWS_ACCOUNT_ID:policy/kubecost-read-amp-metrics \
--attach-policy-arn arn:aws:iam::$AWS_ACCOUNT_ID:policy/OrganizationListAccountTags \
--attach-policy-arn arn:aws:iam::$AWS_ACCOUNT_ID:policy/DescribeResources \
--override-existing-serviceaccounts --approve

Update the placeholder values such as YOUR_CLUSTER_NAME_HERE in values-kubecost-primary.yaml.

Install Kubecost on your primary:

aws ecr-public get-login-password --region us-east-1 | helm registry login --username AWS --password-stdin public.ecr.aws
helm install $KUBECOST_NAMESPACE -n $KUBECOST_NAMESPACE \
    oci://public.ecr.aws/kubecost/cost-analyzer \
    -f https://raw.githubusercontent.com/kubecost/cost-analyzer-helm-chart/develop/cost-analyzer/values-eks-cost-monitoring.yaml \
    -f values-kubecost-primary.yaml

Kubecost agents installation

Follow the above AMP setup section to configure the scraper(s) on each cluster.

This assumes you have created the AWS IAM policies above. If using multiple AWS accounts, you will need to create the policies in each account.

Update the placeholder values such as YOUR_CLUSTER_NAME_HERE in values-kubecost-agent.yaml.
Create the Kubecost namespace:
```
kubectl create ns $KUBECOST_NAMESPACE
```

Configure the Kubecost Service Account:

eksctl create iamserviceaccount \
    --name kubecost-sa \
    --namespace $KUBECOST_NAMESPACE \
    --cluster $CLUSTER_NAME --region $CLUSTER_REGION \
    --attach-policy-arn arn:aws:iam::$AWS_ACCOUNT_ID:policy/kubecost-read-amp-metrics \
    --attach-policy-arn arn:aws:iam::$AWS_ACCOUNT_ID:policy/OrganizationListAccountTags \
    --attach-policy-arn arn:aws:iam::$AWS_ACCOUNT_ID:policy/DescribeResources \
    --override-existing-serviceaccounts --approve

Deploy the Kubecost agent:

aws ecr-public get-login-password --region us-east-1 | helm registry login --username AWS --password-stdin public.ecr.aws
helm install $KUBECOST_NAMESPACE -n $KUBECOST_NAMESPACE \
    oci://public.ecr.aws/kubecost/cost-analyzer \
    -f https://raw.githubusercontent.com/kubecost/cost-analyzer-helm-chart/develop/cost-analyzer/values-eks-cost-monitoring.yaml \
    -f values-kubecost-agent.yaml

Troubleshooting

It will take a few minutes for the scrapers start.

For more help troubleshooting, see our Amazon Managed Service for Prometheus (AMP) Overview doc.

PreviousAmazon Managed Service for Prometheus NextAWS Distro for Open Telemetry

Last updated 8 months ago

See also

Overview

Prerequisites

Architecture diagram

Agentless AMP Configuration

AMP setup

Kubecost primary cluster installation

Kubecost agents installation

Troubleshooting