Search Engine: Elastic

Article ID: 133473, created on Dec 3, 2018, last review on Dec 3, 2018

  • Applies to:
  • Operations Automation

Symptoms

Task Collect disk usage statistics on host fails with the following output:

Request has been timed out, details:
system exception, ID 'IDL:omg.org/CORBA/TIMEOUT:1.0'
TAO exception, minor code = 3e (timeout during recv; low 7 bits of errno: 62 Timer expired), completed = MAYBE

The corresponding HCL request was successfully generated on the Management node. It can be seen in /var/log/pa/core.log by the task ID:

Jun  6 01:02:30.528 : DBG [task:3990046:1224 1:12460:7f58b4ff9700 lib ]: [ InteractiveHostOps::perform] ===> ENTRY
Jun  6 01:02:30.677 : DBG [task:3990046:1224 p:-default-threadpool;-w:-Idle:973 pau]: c.p.p.tracer entry: com.parallels.pa.service.host.ejb.HostMngmtBean.processHostOps(56, [[B@47e1ef77], 25200)
Jun  6 01:02:30.677 : DBG [task:3990046:1224 p:-default-threadpool;-w:-Idle:973 pau]: c.p.p.s.h.e.HostMngmtBean Entering processHostOps, hostId: 56, timeout: 25200
Jun  6 01:02:30.677 : DBG [task:3990046:1224 p:-default-threadpool;-w:-Idle:973 pau]: c.p.p.tracer entry: com.parallels.pa.service.host.ejb.HostMngmtBean.getHostById(56)
Jun  6 01:02:30.677 : DBG [task:3990046:1224 p:-default-threadpool;-w:-Idle:973 pau]: c.p.p.s.h.e.HostMngmtBean Entering getHostById, hostId: 56
Jun  6 01:02:30.678 : DBG [task:3990046:1224 p:-default-threadpool;-w:-Idle:973 pau]: c.p.p.tracer exit: com.parallels.pa.service.host.ejb.HostMngmtBean.getHostById returning com.parallels.pa.service.host.domain.Host@6454e5a9
Jun  6 01:02:30.895 : DBG [task:3990046:1224 p:-default-threadpool;-w:-Idle:986 pau]: c.p.p.tracer entry: com.parallels.pa.service.host.ejb.HCLSenderBean.sendHCL(56, [[B@5b6b5384], 25200)
Jun  6 01:02:30.895 : DBG [task:3990046:1224 p:-default-threadpool;-w:-Idle:986 pau]: c.p.p.s.h.e.HCLSenderBean Entering sendHCL, hostId: 56, timeout: 25200

And has been received by the target node:

Jun  6 01:02:31 HEWSL01 : DBG [task:3990046:1224 56:190477:b2ad5b90 pleskd ]: [ createRequestHandler] Begin request: POST /process Accept-Encoding:gzip, deflate Authorization:eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.ewogICJleHAiIDogMTUyODI2MTM1MCwKICAibmJmIiA6IDE1MjgyMzYxNTAsCiAgImlhdCIgOiAxNTI4MjM2MTUwLAogICJpc3MiIDogIjFlZTU0Y2Y5NTg2
OTAwNGNkMzkwZjJlZGI4NDYzYWNjM2Y1YzZkNmEzZjI1Y2Y4ODI2OTQyNWFiZGViMDRhMDQiLAogICJhdWQiIDogIjczOGE4MmZmMmJjMjgxMjhhNTUzZjg2
MDQ1ODg1MTlhYjQzZDZjYWFiNDE0YjU0YjllYTk3ZGNiYWE3ZjAzMDciLAogICJoY2xfaG9zdF9pcCIgOiAiMTAuMjM5LjUuNjUiCn0 Content-Type:application/xml X-Auth-Host56:eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.ewogICJleHAiIDogMTUyODI2MTM1MCwKICAibmJmIiA6IDE1MjgyMzYxNTAsCiAgImlhdCIgOiAxNTI4MjM2MTUwLAogICJpc3MiIDogIjFlZTU0Y2Y5NTg2
OTAwNGNkMzkwZjJlZGI4NDYzYWNjM2Y1YzZkNmEzZjI1Y2Y4ODI2OTQyNWFiZGViMDRhMDQiLAogICJhdWQiIDogIjczOGE4MmZmMmJjMjgxMjhhNTUzZjg2
MDQ1ODg1MTlhYjQzZDZjYWFiNDE0YjU0YjllYTk3ZGNiYWE3ZjAzMDciLAogICJoY2xfaG9zdF9pcCIgOiAiMTAuMjM5LjUuNjUiCn0 X-Logging-Context:dGFzazozOTkwMDQ2OjEyMjQ= Co
Jun  6 01:02:32 HEWSL01 : DBG [task:3990046:1224 56:190477:b2ad5b90 lib ]: [ HCL::HCLProcessor::process] ===> ENTRY

The default timeout for "Collect disk usage statistics" tasks in OA is set as 25200 (7 hours). Task was running more than 7 hours on the target node, therefore, it failed on OA side:

Jun  6 08:02:30.992 : ERR [task:3990046:1224 1:12460:7f58b4ff9700 lib ]: [ InteractiveHostOps::perform] ExSystem: module_id:'Common', ex_type_id:'1',Message:'Internal error: HTTP POST to 'https://10.239.5.65:8352/process' failed: timeout expired..', deprecated_codes = (0, 0), properties = { reason: 'HTTP POST to 'https://10.239.5.65:8352/process' failed: timeout expired.', }

Cause

The situation is caused by a high load on the target node or connectivity issues.

Resolution

Fix the issue and resubmit the task.

5356b422f65bdad1c3e9edca5d74a1ae caea8340e2d186a540518d08602aa065 e12cea1d47a3125d335d68e6d4e15e07

Email subscription for changes to this article
Save as PDF