CDF is flooding /var/log/messages with Setting volume ownership.

  • KM03765307
  • 30-Nov-2020
  • 30-Mar-2021

Summary

This KB article describes how to fix the issue listed in CPE case OCTCR19S1153357. CDF is flooding /var/log/messages with Setting volume ownership.

Reference

Root Cause:

  The kubelet service prints lots of warning messages in /var/log/messages, making /var/log/messages unreadable.

Impact:

  This patch applies to the kubelet service only. When the kubelet service restarts, the node will change to NotReady status for a few seconds.

Impact Versions: 

  CDF2020.08, CDF2020.11, CDF2021.02

Installation Packages:

  BYOK

 

Solution: 

1. Log in to one of master nodes.

2. Check the kubelet version. 

    ${K8S_HOME}/bin/kubelet --version

3. Download the new kubelet server binary.   

    if your kubelet version is "Kubernetes v1.17.9", download the "kubernetes-server-linux-amd64.tar.gz" file 

        from: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.17.md#client-binaries

    if your kubelet version is "Kubernetes v1.19.0", download the "kubernetes-server-linux-amd64.tar.gz" file 

        from: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.19.md#client-binaries

4. Upload the downloaded "kubernetes-server-linux-amd64.tar.gz" file to all of the master nodes and worker nodes.

5. Unzip file with command "tar -xvzf kubernetes-server-linux-amd64.tar.gz" on each node. 

6. Run following commands on all of the master nodes and worker nodes one after another: 

    1)  Run command "systemctl stop kubelet" to stop kubelet.

    2)  Run command "mv ${K8S_HOME}/bin/kubelet ${K8S_HOME}/bin/kubelet.bak" to back up the old binary.

    3)  Copy file "kubernetes/server/bin/kubelet" to path: ${K8S_HOME}/bin/kubelet

        cp {Path of tar}/kubernetes/server/bin/kubelet ${K8S_HOME}/bin/

    4)  Run command "systemctl daemon-reload".

    5)  Run command "systemctl restart kubelet".

7. After you have performed the above steps on all of the nodes, run kube-status.sh to check your cluster status.

 

Rollback                                                                                                        

1. Run the following commands on all of the master and worker nodes one by one to rollback the change.

    1)  Run command "systemctl stop kubelet" to stop kubelet.

    2)  Run command "mv ${K8S_HOME}/bin/kubelet.bak ${K8S_HOME}/bin/kubelet"

    3)  Run command "systemctl daemon-reload"

    4)  Run command "systemctl restart kubelet"

 

Notes

1. If you extend the cluster by adding new nodes, you must perform the steps on each new node also.

2. After upgrading from CDF 2020.08 to 2020.11, you must perform these steps again.