#Node Management

This document explains how to manage worker nodes using Cluster API Machine resources.

#Prerequisites

#Overview

Worker nodes are managed through Cluster API Machine resources, providing declarative and automated node lifecycle management. The deployment process involves:

1. IP-Hostname Pool Configuration - Network settings for worker nodes
2. Machine Template Setup - VM specifications
3. Bootstrap Configuration - Node initialization and join settings
4. Machine Deployment - Orchestration of node creation and management

#Worker Node Deployment

#Step 1: Configure IP-Hostname Pool

The IP-Hostname Pool defines the network configuration for worker node virtual machines. You must plan and configure the IP addresses, hostnames, DNS servers, and other network parameters before deployment.

Example:

Create a DCSIpHostnamePool named <worker-iphostname-pool-name>:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DCSIpHostnamePool
metadata:
  name: <worker-iphostname-pool-name>
  namespace: cpaas-system
spec:
  pool:
  - ip: "<worker-ip-1>"
    mask: "<worker-mask>"
    gateway: "<worker-gateway>"
    dns: "<worker-dns>"
    hostname: "<worker-hostname-1>"
    machineName: "<worker-machine-name-1>"
  - ip: "<worker-ip-2>"
    mask: "<worker-mask>"
    gateway: "<worker-gateway>"
    dns: "<worker-dns>"
    hostname: "<worker-hostname-2>"
    machineName: "<worker-machine-name-2>"
  - ip: "<worker-ip-3>"
    mask: "<worker-mask>"
    gateway: "<worker-gateway>"
    dns: "<worker-dns>"
    hostname: "<worker-hostname-3>"
    machineName: "<worker-machine-name-3>"

Key parameters:

#Step 2: Configure Machine Template

The DCSMachineTemplate defines the specifications for worker node virtual machines, including VM templates, compute resources, storage configuration, and network settings.

Example:

Create a DCSMachineTemplate named <worker-dcs-machine-template-name>:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: DCSMachineTemplate
metadata:
  name: <worker-dcs-machine-template-name>
  namespace: cpaas-system
spec:
  template:
    spec:
      vmTemplateName: <vm-template-name>
      location:
        type: folder
        name: <folder-name>
      resource: # Optional, if not specified, uses template defaults
        type: cluster # cluster | host. Optional
        name: <cluster-name> # Optional
      vmConfig:
        dvSwitchName: <dv-switch-name> # Optional
        portGroupName: <port-group-name> # Optional
        dcsMachineCpuSpec:
          quantity: <worker-cpu>
        dcsMachineMemorySpec: # MB
          quantity: <worker-memory>
        dcsMachineDiskSpec: # GB
        - quantity: 0
          datastoreClusterName: <datastore-cluster-name>
          systemVolume: true
        - quantity: 100
          datastoreClusterName: <datastore-cluster-name>
          path: /var/lib/kubelet
          format: xfs
        - quantity: 100
          datastoreClusterName: <datastore-cluster-name>
          path: /var/lib/containerd
          format: xfs
        - quantity: 100
          datastoreClusterName: <datastore-cluster-name>
          path: /var/cpaas
          format: xfs
      ipHostPoolRef:
        name: <worker-iphostname-pool-name>

Key parameters:

*Required when parent object is specified

#Step 3: Configure Bootstrap Template

The KubeadmConfigTemplate defines the bootstrap configuration for worker nodes, including user accounts, SSH keys, system files, and kubeadm join settings.

Example:

apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
  name: <worker-kubeadm-config-template>
  namespace: cpaas-system
spec:
  template:
    spec:
      format: ignition
      users:
      - name: boot
        sshAuthorizedKeys:
        - "<ssh-authorized-keys>"
      files:
      - path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json
        owner: "root:root"
        permissions: "0644"
        content: |
          {
            "apiVersion": "kubelet.config.k8s.io/v1beta1",
            "kind": "KubeletConfiguration",
            "protectKernelDefaults": true,
            "staticPodPath": null,
            "tlsCertFile": "/etc/kubernetes/pki/kubelet.crt",
            "tlsPrivateKeyFile": "/etc/kubernetes/pki/kubelet.key",
            "streamingConnectionIdleTimeout": "5m",
            "clientCAFile": "/etc/kubernetes/pki/ca.crt"
          }
      preKubeadmCommands:
      - while ! ip route | grep -q "default via"; do sleep 1; done; echo "NetworkManager started"
      - mkdir -p /run/cluster-api && restorecon -Rv /run/cluster-api
      - if [ -f /etc/disk-setup.sh ]; then bash /etc/disk-setup.sh; fi
      postKubeadmCommands:
      - chmod 600 /var/lib/kubelet/config.yaml
      joinConfiguration:
        patches:
          directory: /etc/kubernetes/patches
        nodeRegistration:
          kubeletExtraArgs:
            provider-id: PROVIDER_ID
            volume-plugin-dir: "/opt/libexec/kubernetes/kubelet-plugins/volume/exec/"

#Step 4: Configure Machine Deployment

The MachineDeployment orchestrates the creation and management of worker nodes by referencing the previously configured DCSMachineTemplate and KubeadmConfigTemplate resources. It manages the desired number of nodes and handles rolling updates.

Example:

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: <worker-machine-deployment-name>
  namespace: cpaas-system
spec:
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
  clusterName: <cluster-name>
  replicas: 3
  selector:
    matchLabels: null
  template:
    spec:
      nodeDrainTimeout: 1m
      nodeDeletionTimeout: 5m
      bootstrap:
        configRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
          kind: KubeadmConfigTemplate
          name: <worker-kubeadm-config-template-name>
          namespace: cpaas-system
      clusterName: <cluster-name>
      infrastructureRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
        kind: DCSMachineTemplate
        name: <worker-dcs-machine-template-name>
        namespace: cpaas-system
      version: <worker-kubernetes-version>

Key parameters:

#Node Management Operations

This section covers common operational tasks for managing worker nodes, including scaling, updates, upgrades, and template modifications.

#Scaling Worker Nodes

Worker node scaling allows you to adjust cluster capacity based on workload demands. The Cluster API manages the node lifecycle automatically through the MachineDeployment resource.

#Adding Worker Nodes

Increase the number of worker nodes to handle increased workload or add new capacity.

Use Case: Scale up cluster to add more compute resources

Prerequisites:

• Verify the IP pool has sufficient available IP addresses for new nodes
• Ensure the DCS platform has adequate resources to provision new VMs

Procedure:

1. Check Current Node Status

   View the current machines in the cluster:

   # List all machines in the cluster
   kubectl get machines -n cpaas-system
   
   # List machines for a specific MachineDeployment
   kubectl get machines -n cpaas-system -l cluster.x-k8s.io/deployment-name=<worker-machine-deployment-name>

2. Verify IP Pool Capacity

   Before scaling, ensure the IP pool has enough available entries:

   kubectl get dcsiphostnamepool -n cpaas-system <worker-iphostname-pool-name> -o yaml

   Check that the pool contains at least as many entries as the desired replica count.

   WARNING

   IP Pool Requirement If the IP pool has insufficient entries, add more IP entries to the pool before scaling. Refer to the IP Pool Configuration section for guidance on adding entries.

3. Scale Up the MachineDeployment

   Update the replicas field to the desired number of nodes:

   kubectl patch machinedeployment <worker-machine-deployment-name> -n cpaas-system \
     --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value": <new-replica-count>}]'

   Example: Scale from 3 to 5 nodes

   kubectl patch machinedeployment worker-pool-1 -n cpaas-system \
     --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value": 5}]'

4. Monitor the Scaling Progress

   Watch the machine creation process:

   # Watch machines being created
   kubectl get machines -n cpaas-system -w
   
   # Check MachineDeployment status
   kubectl get machinedeployment <worker-machine-deployment-name> -n cpaas-system

   The Cluster API controller will automatically create new machines based on the MachineDeployment template.

5. Verify Nodes Joined the Cluster

   Switch to the target cluster context and verify the new nodes:

   # Switch to target cluster context
   kubectl config use-context <target-cluster-context>
   
   # Check all nodes are Ready
   kubectl get nodes

   The new nodes should appear in the list and transition to Ready status.

#Removing Worker Nodes

Decrease the number of worker nodes to reduce cluster capacity or remove underutilized resources.

Use Case: Scale down cluster to reduce costs or adjust to reduced workload

Procedure:

1. Identify Nodes to Remove

   View the current machines in the MachineDeployment:

   kubectl get machines -n cpaas-system -l cluster.x-k8s.io/deployment-name=<worker-machine-deployment-name>

2. Scale Down the MachineDeployment

   Update the replicas field to reduce the node count:

   kubectl patch machinedeployment <worker-machine-deployment-name> -n cpaas-system \
     --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value": <new-replica-count>}]'

   Example: Scale from 5 to 3 nodes

   kubectl patch machinedeployment worker-pool-1 -n cpaas-system \
     --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value": 3}]'

3. Monitor the Removal Progress

   Watch the machine deletion process:

   kubectl get machines -n cpaas-system -w

   The Cluster API controller will:

   • Drain the selected nodes (evict pods if possible)
   • Delete the underlying VMs from the DCS platform
   • Remove the machine resources

4. Verify Nodes Removed

   Switch to the target cluster context:

   kubectl config use-context <target-cluster-context>
   kubectl get nodes

   The removed nodes should no longer appear in the list.

#Deleting Specific Nodes

Remove a specific unhealthy or problematic node while maintaining the replica count.

Use Case: Replace a single unhealthy node without scaling the entire deployment

Procedure:

1. Identify the Unhealthy Machine

   Find the machine corresponding to the unhealthy node:

   # List all machines
   kubectl get machines -n cpaas-system
   
   # Check machine status
   kubectl get machine <machine-name> -n cpaas-system -o yaml

2. Annotate the Machine for Deletion

   Mark the machine for deletion:

   kubectl patch machine <machine-name> -n cpaas-system \
     --type='merge' -p='{"metadata": {"annotations": {"cluster.x-k8s.io/delete-machine": "true"}}}'

3. Wait for Machine Replacement

   The Cluster API controller will:

   • Delete the annotated machine
   • Create a new machine to maintain the desired replica count
   • The new machine will automatically join the cluster

4. Monitor the Replacement

   # Watch machine status
   kubectl get machines -n cpaas-system -w

5. Verify Node Replacement

   kubectl config use-context <target-cluster-context>
   kubectl get nodes

   The new node should appear and transition to Ready status.

#Upgrading Machine Infrastructure

To upgrade worker machine specifications (CPU, memory, disk, VM template), follow these steps:

1. Create New Machine Template

   • Copy the existing DCSMachineTemplate referenced by your MachineDeployment
   • Modify the required values (CPU, memory, disk, VM template, etc.)
   • Give the new template a unique name
   • Apply the new DCSMachineTemplate to the cluster

2. Update Machine Deployment

   • Modify the MachineDeployment resource
   • Update the spec.template.spec.infrastructureRef.name field to reference the new template
   • Apply the changes

3. Rolling Update

   • The system will automatically trigger a rolling update
   • Worker nodes will be replaced with the new specifications
   • Monitor the update progress through the MachineDeployment status

#Updating Bootstrap Templates

Bootstrap templates (KubeadmConfigTemplate) are used by MachineDeployment and MachineSet resources. Changes to existing templates do not automatically trigger rollouts of existing machines; only new machines use the updated template.

Update Process:

1. Export Existing Template

   kubectl get KubeadmConfigTemplate <template-name> -o yaml > new-template.yaml

2. Modify Configuration

   • Update the desired fields in the exported YAML
   • Change the metadata.name to a new unique name
   • Remove extraneous metadata fields (resourceVersion, uid, creationTimestamp, etc.)

3. Create New Template

   kubectl apply -f new-template.yaml

4. Update MachineDeployment

   • Modify the MachineDeployment resource
   • Update spec.template.spec.bootstrap.configRef.name to reference the new template
   • Apply the changes to trigger a rolling update

#Upgrading Kubernetes Version

Kubernetes version upgrades require coordinated updates to both the MachineDeployment and the underlying VM template to ensure compatibility.

Upgrade Process:

1. Update Machine Template

   • Create a new DCSMachineTemplate with an updated vmTemplateName that supports the target Kubernetes version
   • Ensure the VM template includes the correct Kubernetes binaries and dependencies

2. Update MachineDeployment

   • Modify the MachineDeployment resource with the following changes:
     • Update spec.template.spec.version to the target Kubernetes version
     • Update spec.template.spec.infrastructureRef.name to reference the new machine template
     • Optionally update spec.template.spec.bootstrap.configRef.name if bootstrap configuration changes are needed

3. Monitor Upgrade

   • The system will perform a rolling upgrade of worker nodes
   • Verify that new nodes join the cluster with the correct Kubernetes version
   • Monitor cluster health throughout the upgrade process