Symptoms

Control Panel became unavailable at some time, login was not possible even via local UI on the management node. The issue was mitigated by restart of pa-agent and pau services.

In core.log the following line could be found at the time near the outage:

DBG [bss-rest-endpoints:5701 pa-rest-task-117:1506 pau]: c.p.p.tracer entry: com.parallels.pa.service.bss.pc.ejb.ProductCatalogProxyBean.getMarketplaceSalesSubcategories(com.parallels.pa.service.bss.util.BSSAuthInvocationContext@39d2cc3a, sort(+sortOrder,+name), null)

This method never exited, which could be tracked by finding (or not) of the following entry:

DBG [bss-rest-endpoints:5095 pa-rest-task-117:1506 pau]: c.p.p.tracer exit: com.parallels.pa.service.bss.pc.ejb.ProductCatalogProxyBean.getMarketplaceSalesSubcategories returning [com.parallels.pa.service.bss.pc.rest.BssSalesCategory@1c8be8df]

Some time after that OOM occurred as per console.log with either of two signatures:

java.lang.OutOfMemoryError: Java heap space
Dumping heap to /var/log/pa/java_pid16578.hprof ...
*** JBossAS process (16578) received KILL signal ***

or:

java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to /var/log/pa/java_pid10737.hprof ...
ERROR [org.xnio.listener] (default I/O-1) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: GC overhead limit exceeded

Cause

The cause of the issue is inconsistency in BA database. It is recognized as software-related issue #PBA-80301: CP was not available.

Resolution

The issue could be workarounded by resolving inconsistency in BA database directly.

Please contact your Technical Account Manager or a member of Pooled Technica Associates in order to clarify status of the issue and apply the workaround.

Internal content

Link on internal Article