Symptoms
It is impossible to login to CCP for legacy VPS hosting subscriptions, the screen loads forever or shows white screen.
/var/log/poa-ui.log
on UI server shows exceptions for getVEBySubscription methods:
2016-05-13 08:24:32,277 6d8a3b00eb -5158 DEBUG CORBA #1202626: Plesk._VEManagerStub.getVEBySubscription(user_id=65411 (su_user_id=65411)) >>> entry
2016-05-13 08:28:10,187 6d8a3b00eb 8407) DEBUG CORBA #1203180: Plesk._VEManagerStub.getVEBySubscription(user_id=65411 (su_user_id=65411)) <<< exit [138169] by exception: IDL:omg.org/CORBA/COMM_FAILURE:1.0
There are a lot of tasks to create/remove backups getting rescheduled or failed with message:
Request has been timed out, details:
system exception, ID 'IDL:omg.org/CORBA/TIMEOUT:1.0'
VPS backup nodes or hardware nodes do not respond by API, the following messages appear in /var/log/poa.log
in big amount:
May 13 16:26:17 mn: DBG [task:40934208:20697 1:22568:eebb3b90 VPSManager ]: [ Network::NonBlockingTcpConnector::poll_read] Timeout during VZAgent communication: poll_rea
The amount of connections of VPSManager service controller on OA Management Node to Virtuozzo nodes is close to 128 (the hardcoded limit of threads):
[root@mn ~]# netstat -antp | grep 22568 | egrep -c ':4433|:4434'
126
Cause
VPSManager service controller threads get exhausted in case of PVA Agent unavailability on VPS nodes, especially backup nodes. When daily backup tasks run, there is a big amount of connections getting established, each occupying an SC thread. This leads to a complete unavailability of the service controller, causing VPS subscription CCP load eternally.
Resolution
Fix the availability of PVA Agent on the nodes. The following command restarts the service:
# pvaagent restart