Kubecost's Kubescaler implements continuous request right-sizing: the automatic application of Kubecost's high-fidelity recommendations to your containers' resource requests. This provides an easy way to automatically improve your allocation of cluster resources by improving efficiency.

Kubescaler can be enabled and configured on a per-workload basis so that only the workloads you want edited will be edited.

Setup

Kubescaler is part of Cluster Controller, and should be configured after the Cluster Controller is enabled.

Usage

Kubescaler is configured on a workload-by-workload basis via annotations. Currently, only deployment workloads are supported.

Notable Helm values:

Supported workload types

Kubescaler supports:

• apps/v1 Deployments

• apps/v1 DaemonSets

• batch/v1 CronJobs (K8s v1.21+). No attempt will be made to autoscale a CronJob until it has run at least once.

Kubescaler cannot support:

• "Uncontrolled" Pods. Learn more here.

Example

export NS="kubecost"
export DEP="kubecost-cost-analyzer"
export AN_ENABLE="request.autoscaling.kubecost.com/enabled=true"
export AN_FREQ="request.autoscaling.kubecost.com/frequencyMinutes=660"
export AN_TCPU="cpu.request.autoscaling.kubecost.com/targetUtilization=0.9"
export AN_TMEM="memory.request.autoscaling.kubecost.com/targetUtilization=0.9"
export AN_WINDOW="request.autoscaling.kubecost.com/recommendationQueryWindow=3d"

kubectl annotate -n "${NS}" deployment "${DEP}" "${AN_ENABLE}"
kubectl annotate -n "${NS}" deployment "${DEP}" "${AN_FREQ}"
kubectl annotate -n "${NS}" deployment "${DEP}" "${AN_TCPU}"
kubectl annotate -n "${NS}" deployment "${DEP}" "${AN_TMEM}"
kubectl annotate -n "${NS}" deployment "${DEP}" "${AN_WINDOW}"

Kubescaler will take care of the rest. It will apply the best-available recommended requests to the annotated controller every 11 hours. If the recommended requests exceed the current limits, the update is currently configured to set the request to the current limit.

To check current requests for your Deployments, use the following command:

kubectl get deployment -n "kubecost" -o=jsonpath="{range .items[*]}"deployment/"{.metadata.name}{'\n'}{range .spec.template.spec.containers[*]}{.name}{'\t'}{.resources.requests}{'\n'}{end}{'\n'}{end}"

Cluster turndown is an automated scale down and scaleup of a Kubernetes cluster's backing nodes based on a custom schedule and turndown criteria. This feature can be used to reduce spend during down hours and/or reduce surface area for security reasons. The most common use case is to scale non-production (prod) environments (e.g. development (dev) clusters) to zero during off hours.

How it works

Setup

Prerequisites

• kubectl

• Enable the Cluster Controller

You will receive full turndown functionality once the Cluster Controller is enabled via a provider service key setup and Helm upgrade. Review the Cluster Controller doc linked above under Prerequisites for more information, then return here when you've confirmed the Cluster Controller is running.

Verify the pod is running

You can verify that the cluster-turndown pod is running with the following command:

$ kubectl get pods -l app=cluster-turndown -n turndown

Setting a turndown schedule

Turndown uses a Kubernetes Custom Resource Definition to create schedules. Here is an example resource located at artifacts/example-schedule.yaml:

apiVersion: kubecost.com/v1alpha1
kind: TurndownSchedule
metadata:
  name: example-schedule
  finalizers:
  - "finalizer.kubecost.com"
spec:
  start: 2020-03-12T00:00:00Z
  end: 2020-03-12T12:00:00Z
  repeat: daily

This definition will create a schedule that starts by turning down at the designated start date-time and turning back up at the designated end date-time. Both the start and end times should be in RFC3339 format, i.e. times based on offsets to UTC. There are three possible values for repeat:

• none: Single schedule turndown and turnup.

• daily: Start and end times will reschedule every 24 hours.

• weekly: Start and end times will reschedule every 7 days.

To create this schedule, you may modify example-schedule.yaml to your desired schedule and run:

$ kubectl apply -f artifacts/example-schedule.yaml

Currently, updating a resource is not supported, so if the scheduling of the example-schedule.yaml fails, you will need to delete the resource via:

$ kubectl delete tds example-schedule

Then make the modifications to the schedule and re-apply.

Viewing a turndown schedule

The turndownschedule resource can be listed via kubectl as well:

$ kubectl get turndownschedules

or using the shorthand:

$ kubectl get tds

Details regarding the status of the turndown schedule can be found by outputting as a JSON or YAML:

$ kubectl get tds example-schedule -o yaml

apiVersion: kubecost.com/v1alpha1
kind: TurndownSchedule
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"kubecost.com/v1alpha1","kind":"TurndownSchedule","metadata":{"annotations":{},"finalizers":["finalizer.kubecost.com"],"name":"example-schedule"},"spec":{"end":"2020-03-17T00:35:00Z","repeat":"daily","start":"2020-03-17T00:20:00Z"}}
  creationTimestamp: "2020-03-17T00:18:39Z"
  finalizers:
  - finalizer.kubecost.com
  generation: 1
  name: example-schedule
  resourceVersion: "33573"
  selfLink: /apis/kubecost.com/v1alpha1/turndownschedules/example-schedule
  uid: d9b16aed-67e4-11ea-b591-42010a8e0075
spec:
  end: "2020-03-17T00:35:00Z"
  repeat: daily
  start: "2020-03-17T00:20:00Z"
status:
  current: scaledown
  lastUpdated: "2020-03-17T00:36:39Z"
  nextScaleDownTime: "2020-03-18T00:21:38Z"
  nextScaleUpTime: "2020-03-18T00:36:38Z"
  scaleDownId: 38ebf595-4e2b-46e9-951a-1e3ceff30536
  scaleDownMetadata:
    repeat: daily
    type: scaledown
  scaleUpID: 869ec89f-a8d8-450b-9ebb-71cd4d7fbaf8
  scaleUpMetadata:
    repeat: daily
    type: scaleup
  state: ScheduleSuccess

The status field displays the current status of the schedule including next schedule times, specific schedule identifiers, and the overall state of schedule.

• state: The state of the turndown schedule. This can be:

  • ScheduleSuccess: The schedule has been set and is waiting to run.

  • ScheduleFailed: The scheduling failed due to a schedule already existing, scheduling for a date-time in the past.

  • ScheduleCompleted: For schedules with repeat: none, the schedule will move to a completed state after turn up.

• current: The next action to run.

• lastUpdated: The last time the status was updated on the schedule.

• nextScaleDownTime: The next time a turndown will be executed.

• nextScaleUpTime: The next time at turn up will be executed.

• scaleDownId: Specific identifier assigned by the internal scheduler for turndown.

• scaleUpId: Specific identifier assigned by the internal scheduler for turn up.

• scaleDownMetadata: Metadata attached to the scaledown job, assigned by the turndown scheduler.

• scaleUpMetadata: Metadata attached to the scale up job, assigned by the turndown scheduler.

Canceling a turndown schedule

A turndown can be canceled before turndown actually happens or after. This is performed by deleting the resource:

$ kubectl delete tds example-schedule

Canceling while turndown is currently scaling down or scaling up will result in a delayed cancellation, as the schedule must complete its operation before processing the deletion/cancellation.

If the turndown schedule is canceled between a turndown and turn up, the turn up will occur automatically upon cancellation.

Using cluster turndown via UI

Cluster turndown has limited functionality via the Kubecost UI. To access cluster turndown in the UI, you must first enable Kubecost Actions. Once this is completed, you will be able to create and delete turndown schedules instantaneously for your supported clusters. Read more about turndown's UI functionality in this section of the above Kubecost Actions doc. Review the entire doc for more information on Kubecost Actions functionality and limitations.

Limitations

• The internal scheduler only allows one schedule at a time to be used. Any additional schedule resources created will fail (kubectl get tds -o yaml will display the status).

• Do not attempt to kubectl edit a turndown schedule. This is currently not supported. Recommended approach for modifying is to delete and then create a new schedule.

• There is a 20-minute minimum time window between start and end of turndown schedule.

In v1.94 of Kubecost, the turndownschedules.kubecost.k8s.io/v1alpha1 Custom Resource Definition (CRD) was moved to turndownschedules.kubecost.com/v1alpha1 to adhere to Kubernetes policy for CRD domain namespacing. This is a breaking change for users of Cluster Controller's turndown functionality. Please follow this guide for a successful migration of your turndown schedule resources.

Note: As part of this change, the CRD was updated to use apiextensions.k8s.io/v1 because v1beta1 was removed in K8s v1.22. If using Kubecost v1.94+, Cluster Controller's turndown functionality will not work on K8s versions before the introduction of apiextensions.k8s.io/v1.

Scenario 1: You have deployed Cluster Controller but don't use turndown

In this situation, you've deployed Kubecost's Cluster Controller at some point using --set clusterController.enabled=true, but you don't use the turndown functionality.

That means that this command should return one line:

kubectl get crd turndownschedules.kubecost.k8s.io

And this command should return no resources:

kubectl get turndownschedules.kubecost.k8s.io

This situation is easy! You can do nothing, and turndown should continue to behave correctly because kubectl get turndownschedule and related commands will correctly default to the new turndownschedules.kubecost.com/v1alpha1 CRD after you upgrade to Kubecost v1.94 or higher.

If you would like to be fastidious and clean up the old CRD, simply run kubectl delete crd turndownschedules.kubecost.k8s.io after upgrading Kubecost to v1.94 or higher.

Scenario 2: You currently use turndown

In this situation, you've deployed Kubecost's Cluster Controller at some point using --set clusterController.enabled=true and you have at least one turndownschedule.kubecost.k8s.io resource currently present in your cluster.

That means that this command should return one line:

kubectl get crd turndownschedules.kubecost.k8s.io

And this command should return at least one resource:

kubectl get turndownschedules.kubecost.k8s.io

We have a few steps to perform if you want Cluster Controller's turndown functionality to continue to behave according to your already-defined turndown schedules.

1. Upgrade Kubecost to v1.94 or higher with --set clusterController.enabled=true

2. Make sure the new CRD has been defined after your Kubecost upgrade

   This command should return a line:

   Copy

   kubectl get crd turndownschedules.kubecost.com

3. Copy your existing turndownschedules.kubecost.k8s.io resources into the new CRD

   Copy

   kubectl get turndownschedules.kubecost.k8s.io -o yaml \
       | sed 's|kubecost.k8s.io|kubecost.com|' \
       | kubectl apply -f -

4. (optional) Delete the old turndownschedules.kubecost.k8s.io CRD

   Because the CRDs have a finalizer on them, we have to follow this workaround to remove the finalizer from our old resources. This lets us clean up without locking up.

   Copy

   kubectl patch \
       crd/turndownschedules.kubecost.k8s.io \
       -p '{"metadata":{"finalizers":[]}}' \
       --type=merge

   Note: The following command may be unnecessary because Helm should automatically remove the turndownschedules.kubecost.k8s.io resource during the upgrade. The removal will remain in a pending state until the finalizer patch above is implemented.

   Copy

   kubectl delete crd turndownschedules.kubecost.k8s.io

Kubecost's Cluster Controller allows you to access additional Savings features through automated processes. To function, the Cluster Controller requires write permission to certain resources on your cluster, and for this reason, the Cluster Controller is disabled by default.

The Cluster Controller enables features like:

Feature functionality

The Cluster Controller can be enabled on any cluster type, but certain functionality will only be enabled based on the cloud service provider (CSP) of the cluster and its type:

• The Cluster Controller can only be enabled on your primary cluster.

• The Controller itself and container RRS are available for all cluster types and configurations.

• Cluster turndown, cluster right-sizing, and Kubecost Actions are only available for GKE, EKS, and Kops-on-AWS clusters, after setting up a provider service key.

Therefore, the 'Provider service key setup' section below is optional depending on your cluster environment, but will limit functionality if you choose to skip it. Read the caution banner in the below section for more details.

Provider service key setup

Deploying

You can now enable the Cluster Controller in the Helm chart by finding the clusterController Helm flag and setting enabled: true

You may also enable via --set when running Helm install:

Verify the Cluster Controller is running

You can verify that the Cluster Controller is running by issuing the following:

Once the Cluster Controller has been enabled successfully, you should automatically have access to the listed Savings features.