Symptoms

Provider's Control Panel is unavailable. It can be fixed by the restart of the pau service on the UI node.

In console.log on the UI server, multiple messages with OutOfMemoryError could be observed:

Caused by: java.lang.OutOfMemoryError: Java heap space

In pui.log, the information of huge amount of alive sessions could be found:

10:20:30.001 [Abandoned sessions cleanup timer] DEBUG c.p.p.c.u.c.Context$SessionCleanupTask - Session cleanup: result of static static storages cleaning task:  alive=4473 hibernated=6 killed=3 zombie=1
10:20:30.010 [Abandoned sessions cleanup timer] DEBUG c.p.p.c.u.c.Context$SessionCleanupTask - Session cleanup: result of static static storages cleaning task:  alive=4476 hibernated=6 killed=3 zombie=0
10:20:30.100 [Abandoned sessions cleanup timer] DEBUG c.p.p.c.u.c.Context$SessionCleanupTask - Session cleanup: result of static static storages cleaning task:  alive=4477 hibernated=6 killed=6 zombie=0

Cause

The Out Of Memory condition that provoked the Control Panel outage was caused by the incredibly high value of User session timeout (minutes) in PCP > System > Settings > System Properties > General. In this case, the value was set to 1440, which is 24 hours.

As a result, abandoned sessions (i.e. a user worked in panel and then closed the tab) still are considered as active until the timeout of one day is reached.

Information regarding each active session is stored in the cleanup task's queue and consumes about 400KB memory per session. So in order to keep alive around 4000 sessions, it is required to reserve about 1.5 GB of heap:

400000*4000 = 1.6*10^9 ~ 1.5GB

Resolution

In order to workaround this misconfiguration, restart the pau service on the UI node.

This issue can be fixed by decreasing the PCP > System > Settings > System Properties > General > User session timeout (minutes) parameter. The default value is 15 minutes. One hour timeout is also widely used.

Moreover, if there is an intention to use CCPv2, the value of 30 minutes or less is recommended.

Another possible solution is to increase the heap size (Xmx) in /usr/local/pem/wildfly-8.2.0.Final/bin/standalone.conf in the following line:

export JAVA_OPTS="-server -Xms1024m -Xmx1024m -XX:MaxPermSize=128m -XX:+UseParallelOldGC -Dplesk.root=/usr/local/pem -Dfile.encoding=UTF-8 -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true "

Internal content