Symptoms

Backups are not getting removed from the backup node, so the disk space consumption increases significantly every day.

The error reported in IM log (/var/log/pa/vps.log or /var/log/IM/PACI-im.log) for a backup removal operation is:

2017-04-24 04:50:00,972 (51c57357-447c-4077-9e28-16081f728d24) INFO  GenericVm2VfTask [Shared executor thread #7 @2 @BACKGROUND] - done_with_message(-2147258367, Please try again. If the problem persists, con
tact your system administrator for assistance.)
2017-04-24 04:50:00,973 (51c57357-447c-4077-9e28-16081f728d24) WARN  GenericVm2VfTask [Shared executor thread #7 @2 @BACKGROUND] - VM2VF operation [Delete VE Backup] (reqId=20807) finished with rc=-2147258367 (0x80037001)

The following error is faced in /var/log/parallels.log on the backup node:

04-24 04:30:00.522 F /disp:897009:623706/ Task '25Task_RemoveVmBackupSource' with uuid = {c94ca7dc-35ca-4d38-8219-ef1ab0ccc567} was started. Flags = 0 {6cd6bd4f-5f86-4812-833e-570f82dc081a} {1d67e807-019f-4435-9602-7877a3bdc141} {1d67e807-019f-4435-9602-7877a3bdc141} {1d67e807-019f-4435-9602-7877a3bdc141}(DspCmdRemoveVmBackup)
04-24 04:30:00.562 F /disp:897009:623706/ connect to the target dispatcher: host '127.0.0.1' port 64000 {6cd6bd4f-5f86-4812-833e-570f82dc081a} {1d67e807-019f-4435-9602-7877a3bdc141} {1d67e807-019f-4435-9602-7877a3bdc141} {1d67e807-019f-4435-9602-7877a3bdc141}(DspCmdRemoveVmBackup)
04-24 04:50:00.454 F /disp:897009:623706/ Responce receiving failure {6cd6bd4f-5f86-4812-833e-570f82dc081a} {1d67e807-019f-4435-9602-7877a3bdc141} {1d67e807-019f-4435-9602-7877a3bdc141} {1d67e807-019f-4435-9602-7877a3bdc141}(DspCmdRemoveVmBackup)
04-24 04:50:00.551 F /disp:897009:623706/ Failed to get backup tree {6cd6bd4f-5f86-4812-833e-570f82dc081a} {1d67e807-019f-4435-9602-7877a3bdc141} {1d67e807-019f-4435-9602-7877a3bdc141} {1d67e807-019f-4435-9602-7877a3bdc141}(DspCmdRemoveVmBackup)
04-24 04:50:00.674 F /disp:897009:623706/ Task '25Task_RemoveVmBackupSource' with uuid = {c94ca7dc-35ca-4d38-8219-ef1ab0ccc567} was finished with result PRL_ERR_BACKUP_INTERNAL_ERROR (0x80037001) )  {6cd6bd4f-5f86-4812-833e-570f82dc081a} {1d67e807-019f-4435-9602-7877a3bdc141} {1d67e807-019f-4435-9602-7877a3bdc141} {1d67e807-019f-4435-9602-7877a3bdc141}(DspCmdRemoveVmBackup)

Listing the backups manually on the backup node results in error as well:

# prlctl backup-list
Failed to back up the virtual machine due to an internal protocol error. Please try again. If the problem persists, contact your system administrator for assistance.
Failed to get the backup tree: Failed to back up the virtual machine due to an internal protocol error. Please try again. If the problem persists, contact your system administrator for assistance.

Cause

Virtuozzo Dispatcher service takes too long to list the backups and fails with a timeout.

The issue is known to Virtuozzo:

Backups listing fails: Failed to back up the virtual machine due to an internal protocol error

Resolution

Restart the Virtuozzo Dispatcher service to make backup-list executed in shorter time:

# service parallels-server restart

Run the backup-list command manually with increased timeout to make the list cached:

# prlctl backup-list --timeout 108000 >dev/null

At this point the error should be gone.

In case the default 10-minute timeout is still not enough, follow the above KB article to increase the timeout.

Internal content