Search Engine: Elastic

Article ID: 132730, created on Jun 4, 2018, last review on Jun 4, 2018

  • Applies to:
  • Operations Automation 7.2
  • Operations Automation 7.3
  • Operations Automation 7.4

Symptoms

Odin Automation CP becomes unavailable in 5 minutes after pau service startup.

The service does not report anything on the start attempt:

[root@osscore ~]# service pau start
[root@osscore ~]#

while in a good case it should be:

[root@osscore ~]# service pau start
Starting pau (via systemctl):                              [  OK  ]
[root@osscore ~]#

Checking /var/log/pa/console.log for "Controller Boot Thread" gives:

180604 18:51:35 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)
180604 18:56:37 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[

Cause

One of the Wildfly deployments has troubles getting launched, which puts down the whole application server.

Resolution

  1. Make sure that it is not hardware resources shortage that prevents the successful startup. For this purpose, check the CPU/RAM amount available on the node, and if it is rather low, try to increase the JBoss blocking timeout. Edit /usr/local/pem/wildfly-10.1.0.Final/standalone/configuration/standalone-full-ha.xml file, find the system-properties section and add the line:

    <property name="jboss.as.management.blocking.timeout" value="300"/>
    

    300 (5 minutes) is the default timeout, try to increase it twice.

  2. Check standalone-full-ha.xml for the list of deployments and try to disable them one by one to see if helps. To disable, either remove the corresponding section in the file or comment it:

    • originally:

          <deployments>
          <deployment name="whl-endpoint-war.war" runtime-name="whl-endpoint-war.war">
              <content sha1="cfffa73de5dd20a8346885bbe8fb4b78b3700b77"/>
          </deployment>
          <deployment name="core-ear.ear" runtime-name="core-ear.ear">
              <content sha1="94fa7e6901c123da5b268b71abff0e23e93b31e6"/>
          </deployment>
          <deployment name="pui-war.war" runtime-name="pui-war.war">
              <content sha1="41706b47047eda73b95af09c60f6d95324982a14"/>
          </deployment>
      
    • commented pui-war.war section:

          <deployments>
          <deployment name="whl-endpoint-war.war" runtime-name="whl-endpoint-war.war">
              <content sha1="cfffa73de5dd20a8346885bbe8fb4b78b3700b77"/>
          </deployment>
          <deployment name="core-ear.ear" runtime-name="core-ear.ear">
              <content sha1="94fa7e6901c123da5b268b71abff0e23e93b31e6"/>
          </deployment>
          <!-- <deployment name="pui-war.war" runtime-name="pui-war.war">
              <content sha1="41706b47047eda73b95af09c60f6d95324982a14"/>
          </deployment> -->
      

    Do the same for each application one by one until the service starts correctly.

  3. Investigate why the application could not be deployed. After the service is up and running it is possible to run the deployment manually :

    [root@osscore ~]# bash /usr/local/pem/wildfly-10.1.0.Final/bin/jboss-cli.sh
    You are disconnected at the moment. Type 'connect' to connect to the server or 'upported commands.
    [disconnected /] connect
    [standalone@localhost:9990 /] deploy --force /usr/local/pem/u/bss-war.war
    WFLYCTL0344: Operation timed out awaiting service container stability
    

    The likely reason is a problem with remote services . For example, for Cloud Infrastructure endpoint it could be OACI Instance Manager API outage, for BSS application - Billing database unavailability, etc.

  4. After the issues are resolved, it is necessary to deploy the removed applications again and restart the service.

31987597efff5a3a9ce779cc203bbe5e 8fc71f07abe5b233fea1ae0377cd5e3d 5356b422f65bdad1c3e9edca5d74a1ae 1941880841f714e458ae4dc3d9f3062d dd79f16c76b9dca75205ab5e9f5465ea 55fe109b4b4fe3fbb893f22dbb85a41a

Email subscription for changes to this article
Save as PDF