Search Engine: Elastic

Article ID: 132920, created on Jul 17, 2018, last review on Dec 7, 2018

  • Applies to:
  • Operations Automation 7.3
  • Operations Automation 7.4

Symptoms

  1. kubelet service does not start on GDPR host after updating kube* packages to version 1.11:

    [root@k8s ~]# systemctl status kubelet
    ● kubelet.service - kubelet: The Kubernetes Node Agent
       Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
      Drop-In: /etc/systemd/system/kubelet.service.d
               └─10-kubeadm.conf
       Active: activating (auto-restart) (Result: exit-code) since Tue 2018-07-17 11:39:26 +07; 1s ago
         Docs: http://kubernetes.io/docs/
      Process: 1869 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255)
     Main PID: 1869 (code=exited, status=255)
    
  2. The following error appears in /var/log/messages:

    Jul 17 11:39:46.699 kubelet: F0717 11:39:46.698922    2022 server.go:190] failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file "/var/lib/kubelet/config.yaml", error: open /var/lib/kube
    let/config.yaml: no such file or directory
    Jul 17 11:39:46.703 systemd: kubelet.service: main process exited, code=exited, status=255/n/a
    Jul 17 11:39:46.705 systemd: Unit kubelet.service entered failed state.
    Jul 17 11:39:46.705 systemd: kubelet.service failed.
    
  3. /var/log/yum.log contains information about recent update of kubelet, kubectl and kubeadm packages:

    Jul 17 11:38:30 Updated: kubelet-1.11.0-0.x86_64
    Jul 17 11:38:32 Updated: kubectl-1.11.0-0.x86_64
    Jul 17 11:38:34 Updated: kubeadm-1.11.0-0.x86_64
    

Cause

Kubernetes 1.11 (or higher) packages are not supported by the moment. The support for newer Kubernetes packages is planned in the future.

Resolution

  1. Exclude kubeadm* kubectl* and kubelet* packages from list of available to update packages in /etc/yum.conf:

    [root@k8s ~]# echo "exclude=kubeadm-1.1[1-9]* kubectl-1.1[1-9]* kubelet-1.1[1-9]*" >> /etc/yum.conf
    

    This way the version will be fixed to 1.10 and all minor updates for Kubernetes packages will be fetched from the repository.

  2. Downgrade kube* packages:

    [root@k8s ~]# yum downgrade kubelet kubectl kubeadm
    Loaded plugins: fastestmirror, langpacks
    Loading mirror speeds from cached hostfile
     * base: ftp.nsc.ru
     * extras: ftp.nsc.ru
     * updates: ftp.nsc.ru
    Resolving Dependencies
    --> Running transaction check
    ---> Package kubeadm.x86_64 0:1.10.5-0 will be a downgrade
    ---> Package kubeadm.x86_64 0:1.11.0-0 will be erased
    ---> Package kubectl.x86_64 0:1.10.5-0 will be a downgrade
    ---> Package kubectl.x86_64 0:1.11.0-0 will be erased
    ---> Package kubelet.x86_64 0:1.10.5-0 will be a downgrade
    ---> Package kubelet.x86_64 0:1.11.0-0 will be erased
    --> Finished Dependency Resolution
    
    Dependencies Resolved
    
  3. Reload systemd manager configuration:

    [root@k8s ~]# systemctl daemon-reload
    
  4. Restart the services:

    [root@k8s ~]# systemctl stop kubelet
    [root@k8s ~]# systemctl stop docker
    [root@k8s ~]# systemctl stop iptables
    [root@k8s ~]# systemctl start iptables
    [root@k8s ~]# systemctl start docker
    [root@k8s ~]# systemctl start kubelet
    
  5. Make sure that kubelet is running:

    [root@k8s ~]# systemctl status kubelet
    ● kubelet.service - kubelet: The Kubernetes Node Agent
       Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
      Drop-In: /etc/systemd/system/kubelet.service.d
               └─10-kubeadm.conf
       Active: active (running) since Tue 2018-07-17 11:41:08 +07; 19min ago
         Docs: http://kubernetes.io/docs/
     Main PID: 2833 (kubelet)
       CGroup: /system.slice/kubelet.service
               └─2833 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cn...
    
  6. Check status of pods:

    [root@k8s ~]# kubectl get pods
    NAME                            READY     STATUS    RESTARTS   AGE
    gdpr-backend-646bf9c4fb-qfhng   1/1       Running   19         49d
    
  7. Check that GDPR application is working from panel.

5356b422f65bdad1c3e9edca5d74a1ae caea8340e2d186a540518d08602aa065 e12cea1d47a3125d335d68e6d4e15e07 dd79f16c76b9dca75205ab5e9f5465ea 1941880841f714e458ae4dc3d9f3062d 55fe109b4b4fe3fbb893f22dbb85a41a

Email subscription for changes to this article
Save as PDF