Symptoms

It is impossible to login to CCP for legacy VPS hosting subscriptions, the screen loads forever or shows white screen.

/var/log/poa-ui.log on UI server shows exceptions for getVEBySubscription methods:

2016-05-13 08:24:32,277 6d8a3b00eb -5158 DEBUG  CORBA                #1202626: Plesk._VEManagerStub.getVEBySubscription(user_id=65411 (su_user_id=65411)) >>> entry

2016-05-13 08:28:10,187 6d8a3b00eb 8407) DEBUG  CORBA                #1203180: Plesk._VEManagerStub.getVEBySubscription(user_id=65411 (su_user_id=65411)) <<< exit [138169] by exception: IDL:omg.org/CORBA/COMM_FAILURE:1.0

There are a lot of tasks to create/remove backups getting rescheduled or failed with message:

Request has been timed out, details:
system exception, ID 'IDL:omg.org/CORBA/TIMEOUT:1.0'

VPS backup nodes or hardware nodes do not respond by API, the following messages appear in /var/log/poa.log in big amount:

May 13 16:26:17 mn: DBG [task:40934208:20697 1:22568:eebb3b90 VPSManager ]: [ Network::NonBlockingTcpConnector::poll_read] Timeout during VZAgent communication: poll_rea

The amount of connections of VPSManager service controller on OA Management Node to Virtuozzo nodes is close to 128 (the hardcoded limit of threads):

[root@mn ~]# netstat -antp | grep 22568 | egrep -c ':4433|:4434'
126

Cause

VPSManager service controller threads get exhausted in case of PVA Agent unavailability on VPS nodes, especially backup nodes. When daily backup tasks run, there is a big amount of connections getting established, each occupying an SC thread. This leads to a complete unavailability of the service controller, causing VPS subscription CCP load eternally.

Resolution

Fix the availability of PVA Agent on the nodes. The following command restarts the service:

# pvaagent restart

Internal content

Link on internal Article