This feature is in currently in alpha. Please read the documentation carefully.
Kubecost's Kubescaler implements continuous request right-sizing: the automatic application of Kubecost's high-fidelity recommendations to your containers' resource requests. This provides an easy way to automatically improve your allocation of cluster resources by improving efficiency.
Kubescaler can be enabled and configured on a per-workload basis so that only the workloads you want edited will be edited.
Kubescaler is part of Cluster Controller, and should be configured after the Cluster Controller is enabled.
Kubescaler is configured on a workload-by-workload basis via annotations. Currently, only deployment workloads are supported.
Annotation | Description | Example(s) |
---|---|---|
Notable Helm values:
Helm value | Description | Example(s) |
---|---|---|
Kubescaler supports:
apps/v1 Deployments
apps/v1 DaemonSets
batch/v1 CronJobs (K8s v1.21+). No attempt will be made to autoscale a CronJob until it has run at least once.
Kubescaler cannot support:
"Uncontrolled" Pods. Learn more here.
Kubescaler will take care of the rest. It will apply the best-available recommended requests to the annotated controller every 11 hours. If the recommended requests exceed the current limits, the update is currently configured to set the request to the current limit.
To check current requests for your Deployments, use the following command:
Cluster turndown is currently in beta. Please read the documentation carefully.
Cluster turndown is an automated scale down and scaleup of a Kubernetes cluster's backing nodes based on a custom schedule and turndown criteria. This feature can be used to reduce spend during down hours and/or reduce surface area for security reasons. The most common use case is to scale non-production (prod) environments (e.g. development (dev) clusters) to zero during off hours.
If you are upgrading from a pre-1.94 version of the Kubecost Helm chart, you will have to migrate your custom resources. turndownschedules.kubecost.k8s.io
has been changed to turndownschedules.kubecost.com
and finalizers.kubecost.k8s.io
has been changed to finalizers.kubecost.com
. See the for an explanation.
Cluster turndown is only available for clusters on GKE, EKS, or Kops-on-AWS.
You can verify that the cluster-turndown
pod is running with the following command:
Turndown uses a Kubernetes Custom Resource Definition to create schedules. Here is an example resource located at artifacts/example-schedule.yaml:
none
: Single schedule turndown and turnup.
daily
: Start and end times will reschedule every 24 hours.
weekly
: Start and end times will reschedule every 7 days.
To create this schedule, you may modify example-schedule.yaml to your desired schedule and run:
Currently, updating a resource is not supported, so if the scheduling of the example-schedule.yaml fails, you will need to delete the resource via:
Then make the modifications to the schedule and re-apply.
The turndownschedule
resource can be listed via kubectl
as well:
or using the shorthand:
Details regarding the status of the turndown schedule can be found by outputting as a JSON or YAML:
The status
field displays the current status of the schedule including next schedule times, specific schedule identifiers, and the overall state of schedule.
state
: The state of the turndown schedule. This can be:
ScheduleSuccess
: The schedule has been set and is waiting to run.
ScheduleFailed
: The scheduling failed due to a schedule already existing, scheduling for a date-time in the past.
ScheduleCompleted
: For schedules with repeat: none, the schedule will move to a completed state after turn up.
current
: The next action to run.
lastUpdated
: The last time the status was updated on the schedule.
nextScaleDownTime
: The next time a turndown will be executed.
nextScaleUpTime: The next time at turn up will be executed.
scaleDownId
: Specific identifier assigned by the internal scheduler for turndown.
scaleUpId
: Specific identifier assigned by the internal scheduler for turn up.
scaleDownMetadata
: Metadata attached to the scaledown job, assigned by the turndown scheduler.
scaleUpMetadata
: Metadata attached to the scale up job, assigned by the turndown scheduler.
A turndown can be canceled before turndown actually happens or after. This is performed by deleting the resource:
Canceling while turndown is currently scaling down or scaling up will result in a delayed cancellation, as the schedule must complete its operation before processing the deletion/cancellation.
If the turndown schedule is canceled between a turndown and turn up, the turn up will occur automatically upon cancellation.
The internal scheduler only allows one schedule at a time to be used. Any additional schedule resources created will fail (kubectl get tds -o yaml
will display the status).
Do not attempt to kubectl edit
a turndown schedule. This is currently not supported. Recommended approach for modifying is to delete and then create a new schedule.
There is a 20-minute minimum time window between start and end of turndown schedule.
In v1.94 of Kubecost, the turndownschedules.kubecost.k8s.io/v1alpha1
Custom Resource Definition (CRD) was to turndownschedules.kubecost.com/v1alpha1
to adhere to . This is a breaking change for users of Cluster Controller's turndown functionality. Please follow this guide for a successful migration of your turndown schedule resources.
Note: As part of this change, the CRD was updated to use
apiextensions.k8s.io/v1
becausev1beta1
was removed in K8s v1.22. If using Kubecost v1.94+, Cluster Controller's turndown functionality will not work on K8s versions before the introduction ofapiextensions.k8s.io/v1
.
In this situation, you've deployed Kubecost's Cluster Controller at some point using --set clusterController.enabled=true
, but you don't use the turndown functionality.
That means that this command should return one line:
And this command should return no resources:
This situation is easy! You can do nothing, and turndown should continue to behave correctly because kubectl get turndownschedule
and related commands will correctly default to the new turndownschedules.kubecost.com/v1alpha1
CRD after you upgrade to Kubecost v1.94 or higher.
If you would like to be fastidious and clean up the old CRD, simply run kubectl delete crd turndownschedules.kubecost.k8s.io
after upgrading Kubecost to v1.94 or higher.
In this situation, you've deployed Kubecost's Cluster Controller at some point using --set clusterController.enabled=true
and you have at least one turndownschedule.kubecost.k8s.io
resource currently present in your cluster.
That means that this command should return one line:
And this command should return at least one resource:
We have a few steps to perform if you want Cluster Controller's turndown functionality to continue to behave according to your already-defined turndown schedules.
Upgrade Kubecost to v1.94 or higher with --set clusterController.enabled=true
Make sure the new CRD has been defined after your Kubecost upgrade
This command should return a line:
Copy your existing turndownschedules.kubecost.k8s.io
resources into the new CRD
(optional) Delete the old turndownschedules.kubecost.k8s.io
CRD
Note: The following command may be unnecessary because Helm should automatically remove the
turndownschedules.kubecost.k8s.io
resource during the upgrade. The removal will remain in a pending state until the finalizer patch above is implemented.
The Cluster Controller is currently in beta. Please read the documentation carefully.
Kubecost's Cluster Controller allows you to access additional Savings features through automated processes. To function, the Cluster Controller requires write permission to certain resources on your cluster, and for this reason, the Cluster Controller is disabled by default.
The Cluster Controller enables features like:
The Cluster Controller can be enabled on any cluster type, but certain functionality will only be enabled based on the cloud service provider (CSP) of the cluster and its type:
The Cluster Controller can only be enabled on your primary cluster.
The Controller itself and container RRS are available for all cluster types and configurations.
Cluster turndown, cluster right-sizing, and Kubecost Actions are only available for GKE, EKS, and Kops-on-AWS clusters, after setting up a provider service key.
Therefore, the 'Provider service key setup' section below is optional depending on your cluster environment, but will limit functionality if you choose to skip it. Read the caution banner in the below section for more details.
If you are enabling the Cluster Controller for a GKE/EKS/Kops AWS cluster, follow the specialized instructions for your CSP(s) below. If you aren't using a GKE/EKS Kops AWS cluster, skip ahead to the section below.
You can now enable the Cluster Controller in the Helm chart by finding the clusterController
Helm flag and setting enabled: true
You may also enable via --set
when running Helm install:
You can verify that the Cluster Controller is running by issuing the following:
Once the Cluster Controller has been enabled successfully, you should automatically have access to the listed Savings features.
Enable the
You will receive full turndown functionality once the Cluster Controller is enabled via a provider service key setup and Helm upgrade. Review the Cluster Controller doc linked above under Prerequisites for more information, then return here when you've .
This definition will create a schedule that starts by turning down at the designated start
date-time and turning back up at the designated end
date-time. Both the start
and end
times should be in format, i.e. times based on offsets to UTC. There are three possible values for repeat
:
Cluster turndown has limited functionality via the Kubecost UI. To access cluster turndown in the UI, you must first enable . Once this is completed, you will be able to create and delete turndown schedules instantaneously for your supported clusters. Read more about turndown's UI functionality in of the above Kubecost Actions doc. Review the entire doc for more information on Kubecost Actions functionality and limitations.
Because the CRDs have a finalizer on them, we have to follow to remove the finalizer from our old resources. This lets us clean up without locking up.
The following command performs the steps required to set up a service account. .
To use , provide the following required parameters:
For EKS cluster provisioning, if using eksctl
, make sure that you use the --managed
option when creating the cluster. Unmanaged node groups should be upgraded to managed. .
request.autoscaling.kubecost.com/enabled
Whether to autoscale the workload. See note on KUBESCALER_RESIZE_ALL_DEFAULT
.
true
, false
request.autoscaling.kubecost.com/frequencyMinutes
How often to autoscale the workload, in minutes. If unset, a conservative default is used.
73
request.autoscaling.kubecost.com/scheduleStart
Optional augmentation to the frequency parameter. If both are set, the workload will be resized on the scheduled frequency, aligned to the start. If frequency is 24h and the start is midnight, the workload will be rescheduled at (about) midnight every day. Formatted as RFC3339.
2022-11-28T00:00:00Z
cpu.request.autoscaling.kubecost.com/targetUtilization
Target utilization (CPU) for the recommendation algorithm. If unset, the backing recommendation service's default is used.
0.8
memory.request.autoscaling.kubecost.com/targetUtilization
Target utilization (Memory/RAM) for the recommendation algorithm. If unset, the backing recommendation service's default is used.
0.8
request.autoscaling.kubecost.com/recommendationQueryWindow
Value of the window
parameter to be used when acquiring recommendations. See Request sizing API for explanation of window parameter. If setting up autoscaling for a CronJob, it is strongly recommended to set this to a value greater than the duration between Job runs. For example, if you have a weekly CronJob, this parameter should be set to a value greater than 7d
to ensure a recommendation is available.
2d
clusterController.kubescaler.resizeAllDefault
If true, Kubescaler will switch to default-enabled for all workloads unless they are annotated with request.autoscaling.kubecost.com/enabled=false
. This is recommended for low-stakes clusters where you want to prioritize workload efficiency without reworking deployment specs for all workloads.
true