Managing Cluster Autoscaling Configurations

You can create an autoscale configuration for a cluster so that the compute shapes and numbers of the worker nodes are automatically increased or decreased, based on the CPU utilization thresholds.

Autoscaling allows you to maintain optimum performance of your cluster, while keeping costs as low as possible. Autoscaling monitors your CPU utilization and automatically adjusts the CPU capacity, based on the configuration parameters you set.

Note

When a cluster is autoscaled, the new details are reflected in Apache Ambari or Cloudera Manager. To register that change with Apache Ambari or Cloudera Manager, a new cluster admin password is created when you create an autoscale configuration. The password is deleted when the autoscale configuration is deleted.

For more information, see the following:

How Autoscaling Works

The autoscale feature collects data about the CPU utilization of worker or compute only worker nodes in a cluster. Two autoscale trigger types are available:

  • Metrics: This configuration includes the parameters for scaling up (switching to the next larger compute shape) and scaling down (switching to the next smaller compute shape) or scaling out (adding more nodes to the cluster) and scaling in (removing nodes from the cluster). A scale-up or scale-out configuration specifies a duration and a percentage, so that when the average CPU usage exceeds the specified percentage for the specified duration, the node is scaled up or out. A scale-down or scale-in configuration specifies a duration and a percentage, so that when the average CPU usage falls below the specified percentage for the specified duration, the node is scaled down or in.

    The average usage is based on the entire duration specified in the configuration. The autoscale action is triggered at the end of the specified duration. For example, if the scale-up configuration is set to 60% for 6 hours, then the average CPU usage for the entire six hours must exceed 60% for the six-hour duration. The usage might fall below or rise above 60% for brief periods in that six-hour window, but the scale up action is triggered only after the data for the entire six hours is evaluated and averaged, and that average exceeds the percentage specified in the configuration.

    If you want the cluster to autoscale more frequently, based on more frequent fluctuations in CPU activity, use shorter duration values. Legal values for scaling durations are 5 to 60 minutes or 1 to 24 hours. Enter hours as units of 60 minutes. For example, 60, 120, 180, 240, and so on, to 1440 minutes.

    Autoscale durations are mapped to Oracle Cloud Infrastructure Monitoring Query Language (MQL) interval values, where the ranges of values allowed for interval are 1m-60m, 1h-24h, and 1d. (Notice that while the minimum MQL interval is one minute, the minimum Big Data Service interval is five minutes.) See the "Interval Query Component" section in Monitoring Query Language (MQL) Reference.

    Autoscale takes advantage of Oracle Cloud Infrastructure alarms, and the autoscale duration value is also used as the notification interval for the autoscale alarm. (See Managing Alarms.) If the conditions for an autoscale action are still in effect after another interval, then the alarm triggers another autoscale.

  • Schedule: Oracle supports two types of schedule-based policies, schedule-based vertical scaling and schedule-based horizontal scaling.

    As part of schedule-based vertical scaling, you specify the target shape and the shape configuration (OCPU count and memory size).

    As part of schedule-based horizontal scaling, you specify the target node count.

    Note

    All scheduled-based triggers/conditions associated with a cluster must be more than four hours apart.

    Note

    Scheduled-based conditions are snoozed for 15 minutes (up to a maximum of two hours), if the cluster's life cycle state isn't active when the trigger fires.

If you use a flex shape with either type of autoscale configuration, you can add or remove the exact number of OCPUs and control the memory usage during each autoscale operation. With a flex shape, you can also set minimum and maximum boundaries. See Planning the Cluster Layout, Shape, and Storage in the Oracle Cloud Infrastructure documentation for the available shapes.

We recommend that you constantly tune the autoscale values to meet your needs. See the recommendations for tuning alarms in the "Routinely Tune Your Alarms" section in Best Practices for Your Alarms.

Prerequisites

Review the Big Data Service cluster autoscaling prerequisites.

Quota

The tenancy must have a quota that you can scale up or scale out worker or compute only worker nodes. If not, the autoscaling operation fails. See Viewing Your Service Limits, Quotas, and Usage.

Network

When the cluster was created, one of the following options was selected:

  • Deploy Oracle-managed Service gateway and NAT gateway (Quick Start)

    If the cluster was created with this option selected, you can configure and use autoscale.

  • Use the gateways in your selected Customer VCN (Customizable)

    If the cluster was created with this option selected:

Autoscaling Types

You can autoscale a cluster horizontally or vertically when specified metric thresholds are exceeded.

Horizontal scaling adds or removes nodes to the cluster. Vertical scaling changes the shape of a node in the cluster.

For example in case of vertical scaling, when thresholds are met the shapes of all the worker nodes in the cluster are automatically scaled up to the next higher VM.Standard shape, scaled down to the next lower VM.Standard shape, or for Flex shapes to the configured OCPU and memory values.

In case of horizontal scaling, when thresholds are met the number worker nodes in the cluster are automatically scaled out or scaled in depending on the configured rules.

ODH clusters support both vertical and horizontal scaling. However, horizontal scaling applies only to compute only worker nodes and Kafka broker nodes.

CDH clusters support only vertical scaling.