Symptoms
After Odin Automation updates or hotfixes installation, a task to deploy pui-war package on UI node(s) fails or gets rescheduled. The examples of possible task outputs could be:
Failed to process executeUpdateSequence at xpath tail HOOKS/HOOK[@when='finally' and @upgrade='yes']/child::* for pkgId: 5217, ex message is: Failed to process EXEC command for package pui-war of type other having pkgId: 5217, exception message: Invalid exitCode, HCL out: PUI rootpath is /usr/local/pem/u/pui
, HCL err: Traceback (most recent call last):
File "<string>", line 1, in <module>
File "deployPUI.py", line 166, in deployPUI
...
"response-headers" => {
"operation-requires-reload" => true,
"process-state" => "reload-required"
}
}
{
"outcome" => "failed",
"failure-description" => "WFLYCTL0212: Duplicate resource [
(\"socket-binding-group\" => \"standard-sockets\"),
(\"remote-destination-outbound-socket-binding\" => \"pau-remote-ejb0\")
]",
"rolled-back" => true,
"response-headers" => {"process-state" => "reload-required"}
}
or:
Failed to process executeUpdateSequence at xpath tail HOOKS/HOOK[@when='finally' and @upgrade='yes']/child::* for pkgId: 5217, ex message is: Failed to process EXEC command for package pui-war of type other having pkgId: 5217, exception message: Invalid exitCode, HCL out: Starting pau: /
[ OK ]
PUI rootpath is /usr/local/pem/u/pui
, HCL err: Traceback (most recent call last):
File "<string>", line 1, in <module>
File "deployPUI.py", line 174, in deployPUI
phases.get('deploy_war')()
File "deployPUI.py", line 155, in deploy_war
bootstrap.execCLI(jbossdir, '-c', 'deploy pui-war.war --force')
File "/usr/local/pem/u/pui/../bootstrap.py", line 185, in execCLI
"CLI return code is: " + str(cmd.returncode) + ", stderr: " + str(err) + ", stdout: " + str(out))
Exception: CLI return code is: 1, stderr: , stdout: {"WFLYCTL0062: Composite operation failed and was rolled back. Steps that failed:" => {"Operation step-2" => {"WFLYCTL0412: Required services that are not installed:" => ["jboss.deployment.unit.\"pui-war.war\".beanmanager","org.wildfly.remoting.endpoint.outbound-connection.pau-remote-ejb-connection0","org.wildfly.remoting.endpoint.outbound-connection.pau-remote-ejb-connection1"],"WFLYCTL0180: Services with missing/unavailable dependencies" => ["jboss.deployment.unit.\"pui-war.war\".weld.weldClassIntrospector is missing [jboss.deployment.unit.\"pui-war.war\".beanmanager]","jboss.deployment.unit.\"pui-war.war\".batch.environment is missing [jboss.deployment.unit.\"pui-war.war\".beanmanager]","jboss.ejb3.dd-based-ejb-client-context.\"pui-war.war\".internal-remoting-profile is missing [org.wildfly.remoting.endpoint.outbound-connection.pau-remote-ejb-connection1, org.wildfly.remoting.endpoint.outbound-connection.pau-remote-ejb-connection0]"]}}}
How to investigate the reasons of the failures?
Resolution
pui-war is an OA package carrying the UI service capabilities. It's a packed Java web application that gets deployed into the Wildfly application server on each UI node. Every upgrade/update or UI-related hotfix provides a new version of the package, so the old version gets undeployed, while the new one is deployed.
All deployment files are located inside /usr/local/pem/u/pui directory on the UI node.
deployPUI.py script includes the invoked stages.
tune-pui.cli is an important script that contains the applied changes to the main Wildfly configuration that is available at /usr/local/pem/wildfly-10.1.0.Final/standalone/configuration/standalone-ha.xml (the Wildfly version in the path may differ, depending on the OA version).
Note: each PUI redeployment run leaves a backup of the configuration file in the /usr/local/pem/wildfly-10.1.0.Final/standalone/configuration/standalone_xml_history/ directory.
The main log to study redeployment attempts history is /var/log/pa/console.log on the UI node. The log gets saved on each service restart attempt (the last 5 attempts are kept). The following lines indicate the start and end of the redeployment:
180203 02:10:04 INFO [org.jboss.as.server] (management-handler-thread - 7) WFLYSRV0009: Undeployed "pui-war.war" (runtime-name: "pui-war.war")
...
180203 03:10:15 INFO [org.jboss.as.server] (management-handler-thread - 4) WFLYSRV0010: Deployed "pui-war.war" (runtime-name : "pui-war.war")