Search Engine: Elastic

Article ID: 133488, created on Dec 7, 2018, last review on Dec 7, 2018

  • Applies to:
  • Operations Automation 7.4

Symptoms

The task Upgrading resources of application 'UX1 Marketplace' fails with the error:

Chart with name 'ux1-marketplace' is in 'FAILED' state and cannot be updated automatically

The UX1 pod stuck in the Pending state:

[root@linmn01 ~]# kubectl get pod
NAME                                         READY     STATUS    RESTARTS   AGE
gdpr-backend-749f99895-fzqth                 1/1       Running   1          176d
gdpr-backend-859d498b5b-wr9fl                0/1       Pending   0          18d
ux1-marketplace-connector-58964fdb66-nvwlf   0/1       Pending   0          1h
ux1-marketplace-connector-7c847644b4-7t4kh   1/1       Running   0          18d
ux1-marketplace-elastic-97cb69689-r72gh      1/1       Running   0          18d

Cause

The upgrade failed due to the lack of resources for the pod:

[root@linmn01 ~]# kubectl describe pod ux1-marketplace-connector-58964fdb66-nvwlf
...
Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  2m (x398 over 1h)  default-scheduler  0/1 nodes are available: 1 Insufficient cpu.

Resolution

In order to fix the issue and complete the task the below steps have to be applied:

  1. Add the required system resources to the Kubernetes node. Note that, during helm chart upgrade, additional resources are required, since Helm upgrades the chart by creating new pod (which requires resources), and only when a new pod is created, the old one is removed.

  2. Get a list of the UX1 revisions:

    [root@linmn01 ~]# helm history ux1-marketplace
    REVISION        UPDATED                         STATUS          CHART                           DESCRIPTION
    1               Mon Nov 12 04:57:30 2018        SUPERSEDED      ux1-marketplace-1.2.0-502       Install complete
    2               Mon Nov 26 08:58:15 2018        SUPERSEDED      ux1-marketplace-1.4.1-662       Upgrade complete
    3               Fri Nov 30 01:04:32 2018        FAILED          ux1-marketplace-1.4.1-662       Upgrade "ux1-marketplace" failed: timed out waiting for t...
    
  3. Rollback the UX1 chart to previous revision as follows:

    [root@linmn01 ~]# helm rollback ux1-marketplace 2
    
  4. Restart the kubelet service on the kubernetes node:

    [root@k8s ~]# systemctl restart kubelet
    
  5. Resubmit the task.

adc6deaa66054d8a194d131ba07f2785 caea8340e2d186a540518d08602aa065 e12cea1d47a3125d335d68e6d4e15e07 5356b422f65bdad1c3e9edca5d74a1ae aab95f5cf9bcfa920cc1dda8487f084a 55fe109b4b4fe3fbb893f22dbb85a41a 1941880841f714e458ae4dc3d9f3062d

Email subscription for changes to this article
Save as PDF