Oracle Cloud Infrastructure Documentation

Working with Self-Managed Nodes

Find out how to set up and use self-managed nodes with Kubernetes Engine.

A self-managed node is a worker node hosted on a compute instance (or instance pool) that you've created yourself in Compute service, rather than on a compute instance that Kubernetes Engine has created for you. Self-managed nodes are often referred to as Bring Your Own Nodes (BYON). Unlike managed nodes and virtual nodes (which are grouped into managed node pools and virtual node pools respectively), self-managed nodes are not grouped into node pools.

You use the Compute service to create the compute instances on which to host self-managed nodes. Using the Compute service enables you to configure compute instances for specialized workloads, including compute shape and image combinations that are not available for managed nodes and virtual nodes. For example, you might want instances with shapes designed for hardware-accelerated workloads (such as GPU shapes), or shapes designed for high-performance computing (HPC) workloads that require high frequency processor cores (such as HPC and Optimized shapes). You might want to connect many such instances with a high-bandwidth, ultra low-latency network to form an Oracle Cloud Infrastructure cluster network (see Using RDMA Cluster Networks).

If you want to simplify administration and manage multiple self-managed nodes as a group, use the Compute service to create a compute instance pool to host one or more self-managed nodes.

When creating a compute instance (or instance pool) to host a self-managed node, you specify the Kubernetes cluster to which to add the instance. You can only add self-managed nodes to enhanced clusters.

Both the cluster to which you add a self-managed node, and the image you use for the compute instance hosting the self-managed node, must meet certain requirements. See Cluster Requirements and Image Requirements respectively.

At a high level, these are the steps to follow to create a compute instance to host a self-managed node and add it to an existing cluster:

When the compute instance is created, it is added to the cluster as a self-managed node.

Note the following:

  • If you delete the cluster to which you have added self-managed nodes, the compute instances hosting the self-managed nodes are not terminated. Containers currently running on the nodes might continue to run, provided they are not dependent on the Kubernetes control plane. If you delete a cluster to which you have added self-managed nodes, it is your responsibility to terminate the compute instances hosting those self-managed nodes.
  • As well as using the Compute service to create individual compute instances to host individual self-managed nodes, you can also use the Compute service to create a compute instance pool to host one or more self-managed nodes. First, you define an instance configuration that includes the cluster's Kubernetes API private endpoint and base64-encoded CA certificate in a cloud-init script (just as if you were creating an individual compute instance). You then use the instance configuration to create one or more instances in an instance pool, each of which can host a self-managed node. You can also use the instance configuration as a template for launching individual instances that are not part of an instance pool. For more information, see Creating an Instance Configuration and Creating Instance Pools.
  • To upgrade the version of Kubernetes running on a self-managed node, you have to replace the existing self-managed node with a new self-managed node. See Upgrading Self-Managed Nodes to a Newer Kubernetes Version by Replacing an Existing Self-Managed Node.
  • By default, self-managed nodes use the flannel CNI plugin for pod networking. If you want to use the OCI VCN-Native Pod Networking CNI plugin for pod networking instead, use the CLI or API to specify the necessary parameters (see Creating Self-Managed Nodes). To use the OCI VCN-Native Pod Networking CNI plugin for pod networking (rather than the flannel CNI plugin), the cluster's control plane nodes must be running Kubernetes version 1.27.10 (or later). For more information about the OCI VCN-Native Pod Networking CNI plugin and the flannel CNI plugin, see Pod Networking.

Using self-managed nodes with cluster networks

When you use the Compute service to create a compute instance pool to host one or more self-managed nodes, you can manage the instance pool as an Oracle Cloud Infrastructure cluster network (see Cluster Networks with Instance Pools). Compute instances within the cluster network are connected by a high-bandwidth, ultra low-latency, remote direct memory access (RDMA) network. For more information about using an RDMA network with Kubernetes Engine, see Running RDMA (remote direct memory access) GPU workloads on OKE on github.

Notable features and capabilities not supported when using self-managed nodes

Some features and capabilities are not available, or not yet available, when using self-managed nodes.

Note that because self-managed nodes are not grouped into node pools, any functionality related to node pools does not apply.

Feature not supported Additional information
Self-managed node metrics on the Kubernetes Engine Metrics page in the Console Kubernetes metrics for self-managed nodes are not shown on the Kubernetes Engine service's Metrics page in the Console.

Use kubectl to view Kubernetes metrics (such as Kubernetes node state) for self-managed nodes, and the Compute service's Console Metrics page to view compute metrics (such as memory utilization) for compute instances hosting self-managed nodes.
Using images other than the OKE OL7 and OL8 images Not yet available.
Kubernetes skew policy enforcement The Kubernetes skew policy that a cluster's control plane nodes must be no more than two minor versions (or three minor versions, starting from Kubernetes version 1.28) ahead of worker nodes is not enforced.
Kubernetes Cluster Autoscaler Not yet available.
Kubernetes Engine cordon and drain options You cannot specify Kubernetes Engine cordon and drain options for self-managed nodes. Use native Kubernetes cordon and drain commands instead.
Node cycling when upgrading or updating worker nodes Not yet available.