Symptoms
Hosting CP does not load and gives Service Unavailable message.
Checking /var/log/poa-ui.log
on the UI server for the corresponding session, the action flow is stuck on the following line:
2016-08-31 22:25:46,235 ec-70 DEBUG CORBA #10006: Plesk.APS._APSManagerStub.getApplicationXMLMetadata(user_id=1 (su_user_id=1)) >>> entry
There is no corresponding getApplicationXMLMetadata
call appearing in /var/log/poa.log
on the OA Management Node.
SaaS service controller is alive, but has too many connections in CLOSE_WAIT status as seen in netstat
output:
# pgrep -f SaaS
19415
# netstat -antp | grep 19415
...
tcp 1 0 10.124.45.159:42005 10.124.45.159:5432 CLOSE_WAIT 19415/SoLoader
tcp 1 0 10.124.45.159:45992 10.124.45.159:5432 CLOSE_WAIT 19415/SoLoader
tcp 1 0 10.124.45.159:45942 10.124.45.159:5432 CLOSE_WAIT 19415/SoLoader
tcp 1 0 10.124.45.159:42473 10.124.45.159:5432 CLOSE_WAIT 19415/SoLoader
tcp 1 0 10.124.45.159:33984 10.124.45.159:5432 CLOSE_WAIT 19415/SoLoader
tcp 1 0 10.124.45.159:33966 10.124.45.159:5432 CLOSE_WAIT 19415/SoLoader
tcp 1 0 10.124.45.159:45974 10.124.45.159:5432 CLOSE_WAIT 19415/SoLoader
...
OA Management Node runs in a Virtuozzo container, and there are the following errors on the hardware node:
[root@vz ~]# dmesg |tail
[29780191.405447] Fatal resource shortage: kmemsize, UB 101.
[29780191.405828] Fatal resource shortage: kmemsize, UB 101.
[29780191.406227] Fatal resource shortage: kmemsize, UB 101.
where 101 is the container ID of MN.
Cause
Insufficient UBC resources result in SaaS service controller outage.
Resolution
For Virtuozzo version 4.7 and 6.0, remove all UBC parameters from the container's configuration file and set only RAM and SWAP limits:
# vzctl set 101 --ram 8G --swap 16G --vm_overcommit 0 --save
vm_overcommit
setting will remove all limits for memory allocation, while memory usage will be limited to 8GB.
For older Virtuozzo versions, SLM limit can be set for the container:
# vzctl set 101 --slmmode slm --slmmemorylimit 8G --save