Introducing Cluster Add-ons

Essential Add-ons

Essential cluster add-ons are core components of a Kubernetes cluster, and are required for a cluster to operate correctly. Essential cluster add-ons include:

• CoreDNS: The CoreDNS add-on is a general-purpose authoritative DNS server that is modular and pluggable. Kubernetes Engine creates clusters with CoreDNS as the DNS server. The kubelet process on each worker node directs individual containers to the DNS server to translate DNS names to IP addresses.
• kube-proxy: The kube-proxy add-on is the Kubernetes network proxy that maintains network rules and routes requests.
• a CNI plugin for pod networking: One of the following CNI plugin add-ons to implement network connectivity for pods running on worker nodes:
  • OCI VCN-Native Pod Networking CNI plugin
  • flannel CNI plugin

  CNI plugins configure network interfaces, provision IP addresses, and maintain connectivity.

Essential cluster add-ons are deployed by default in new Kubernetes clusters. When using enhanced clusters, you can configure essential cluster add-ons using Kubernetes Engine.

Optional Add-ons

Optional cluster add-ons are components that you can choose to deploy on a Kubernetes cluster. Optional add-ons extend core Kubernetes functionality to improve cluster manageability and performance. Examples of optional cluster add-ons include:

• Kubernetes Dashboard: The optional Kubernetes Dashboard add-on is a web-based management interface that enables you to deploy, edit, observe, and troubleshoot containerized applications. We do not recommend deploying the Kubernetes Dashboard on production clusters due to the lack of extensible authentication support. For more information, see Accessing a Cluster Using the Kubernetes Dashboard.
• Tiller (not recommended): The optional Tiller add-on is the server portion of Helm. With Tiller running in the cluster, you can use Helm to manage Kubernetes resources. Tiller was removed from Helm in version 3 (and later versions) due to known security risks. Because of those security risks, we strongly recommend that you do not deploy Tiller on production clusters. For the same reason, the Tiller add-on is not shown in the Console. If you decide that you do want to deploy the Tiller add-on despite the security risks, use the OCI CLI or API.
• Database Operator: The optional Oracle Database Operator for Kubernetes add-on extends the Kubernetes API with custom resources and controllers for automating Oracle Database lifecycle management. The Oracle Database Operator supports a variety of database configurations and lifecycle operations. For more information, see the Oracle Database Operator for Kubernetes documentation on GitHub.
• Weblogic Operator: The optional WebLogic Kubernetes Operator add-on supports the running of WebLogic Server and Fusion Middleware Infrastructure domains on Kubernetes. For more information, see the WebLogic Kubernetes Operator documentation on GitHub.
• Certificate Manager: The optional Certificate Manager add-on, also known as cert-manager, adds certificates and certificate issuers to Kubernetes clusters as resource types. The Certificate Manager also simplifies the process of obtaining, using, and renewing those certificates. For more information, see the cert-manager documentation on GitHub.
• Cluster Autoscaler: The optional Cluster Autoscaler add-on automatically resizes a cluster's managed node pools based on application workload demands using the Kubernetes Cluster Autoscaler. Deploying the Kubernetes Cluster Autoscaler as a cluster add-on rather than as a standalone program simplifies configuration and ongoing maintenance, and enables you to specify that you want Oracle to update the Kubernetes Cluster Autoscaler automatically. For more information, see Working with the Cluster Autoscaler as a Cluster Add-on.
• Istio: The optional Istio add-on provides automated baseline traffic resilience, service metrics collection, distributed tracing, traffic encryption, protocol upgrades, and advanced routing functionality for all service-to-service communication. For more information, see Working with Istio as a Cluster Add-on.
• Native Ingress Controller: The optional OCI native ingress controller add-on load balances and routes incoming traffic to service pods running on worker nodes in a Kubernetes cluster. For more information, see Working with the OCI Native Ingress Controller as a Cluster Add-on.
• Kubernetes Metrics Server: The optional Kubernetes Metrics Server is a cluster-wide aggregator of resource usage data. The Kubernetes Metrics Server collects resource metrics from the kubelet running on each worker node and exposes them in the Kubernetes API server through the Kubernetes Metrics API. For more information, see Working with the Kubernetes Metrics Server as a Cluster Add-on.
• NVIDIA GPU Plugin: The optional NVIDIA GPU Plugin add-on is a convenient way to manage the NVIDIA Device Plugin for Kubernetes. The NVIDIA Device Plugin for Kubernetes is the NVIDIA implementation of the Kubernetes device plug-in framework for exposing the number of NVIDIA GPUs on each worker node, and tracking the health of those GPUs. For more information about the NVIDIA Device Plugin for Kubernetes, see the NVIDIA/k8s-device-plugin documentation on github.

Optional cluster add-ons are not deployed by default. When using enhanced clusters, you can choose whether to deploy and configure an increasing number of optional cluster add-ons using Kubernetes Engine.

Updating Add-on Versions

When you enable a cluster add-on, you can specify either that you want Oracle to update the add-on automatically when new versions become available, or that you want to choose a particular version of the add-on to be deployed.

• If you specify that you want the add-on to be automatically updated (default behavior), Oracle deploys the most recent version of the add-on that supports the Kubernetes version specified for the cluster. If a new version of the add-on is subsequently released, Oracle automatically updates the add-on, provided the updated add-on is compatible with the versions of Kubernetes currently supported by Kubernetes Engine.

  If you configure an add-on using one or more of the approved key/value pair configuration arguments (see Cluster Add-on Configuration Arguments), the configuration is retained when the add-on is updated automatically. However, note that any other customizations you might have made to the add-on are discarded when the add-on is updated automatically.

  Bear in mind that it is your responsibility to upgrade clusters to run versions of Kubernetes supported by Kubernetes Engine. To take advantage of automatic add-on updates, we recommend you upgrade clusters in a timely manner to ensure that clusters are always running supported versions of Kubernetes. If a cluster is running an older version of Kubernetes that Kubernetes Engine no longer supports, cluster add-ons might not be updated automatically.

• If you specify that you want to choose the version of the add-on to deploy, Oracle deploys the add-on version that you choose. It is your responsibility to verify that the add-on version is compatible with the Kubernetes version running on the cluster.

  Bear in mind that it is your responsibility to upgrade clusters to run versions of Kubernetes supported by Kubernetes Engine. When Oracle announces that Kubernetes Engine is to cease support for an older version of Kubernetes, it is your responsibility to both upgrade any cluster running that older Kubernetes version, and also (if necessary) to update the add-on to a version that is compatible with the newer Kubernetes version.

Having enabled a cluster add-on and specified that you want Oracle to update the add-on automatically, you can subsequently specify instead that you want to deploy a particular version of the add-on. Similarly, if you have specified that you want to choose the version of the add-on to deploy, you can subsequently specify instead that you want Oracle to update the add-on automatically.

Note that in the case of the OCI VCN-Native Pod Networking CNI plugin add-on, regardless of whether you choose to have Oracle update the add-on automatically or you choose to update the add-on yourself, the updates are only applied when worker nodes are next rebooted. See Updating the OCI VCN-Native Pod Networking CNI plugin.

Notes about Cluster Add-ons

Note the following when configuring cluster add-ons:

• When creating a new cluster, you cannot disable essential cluster add-ons. However, when editing an existing cluster, you can choose to disable an essential add-on. If you do disable an essential cluster add-on, you are taking responsibility for deploying and configuring an alternative add-on to provide equivalent functionality.
• When creating a new cluster, essential cluster add-ons are set to automatically update. However, you can choose to not have an essential add-on automatically updated. If you do so, you are taking responsibility for keeping the add-on up-to-date.
• When you enable a cluster add-on, you can configure the add-on by specifying one or more key/value pairs to pass as arguments to the cluster add-on. For example, for the Kubernetes Dashboard, you specify a value of 3 for the numOfReplicas key.
• When you disable a cluster add-on using the Console, the add-on is not removed from the cluster but is simply not used. To completely remove the add-on, use the CLI or the API.
• Kubernetes Engine regularly reconciles the configuration of essential cluster add-ons deployed on basic and enhanced clusters with the corresponding default cluster add-on configurations. If you configure an essential cluster add-on deployed on an enhanced cluster using one or more of the approved key/value pair configuration arguments (see Cluster Add-on Configuration Arguments), Kubernetes Engine merges your customizations with the default configuration. However, Kubernetes Engine discards all other customizations made to essential cluster add-on configurations, and restores the default configurations.

  In the case of basic clusters, note that customizations to essential cluster add-ons are not guaranteed to persist. If you make a customization to an essential cluster add-on that causes a conflict during the reconciliation process, the customization is reverted. To retain essential cluster add-on customizations, upgrade basic clusters to enhanced clusters.