Symptoms

An OACI VE backup experienced a failure, but the backup got created physically on the hardware node.

/var/log/pa/vps.log on OACI IM node contains these errors:

2019-05-30 09:01:05,813 (f5e0bc3e-a67a-4b23-9f0b-8cdeebd7c5f6) ERROR GenericVm2VfTask [Shared executor thread #14 @1 @BACKGROUND] - timeout()
2019-05-30 09:01:05,813 (f5e0bc3e-a67a-4b23-9f0b-8cdeebd7c5f6) WARN  GenericVm2VfTask [Shared executor thread #14 @1 @BACKGROUND] - VM2VF operation [BACKUP] (reqId=15) finished with rc=-1 (0xFFFFFFFF)

After such backup failures, the used backup space is not reclaimed.

Cause

In case a scheduled backup hits the timeout, the actual backup operation on the Virtuozzo backend continues, completes successfully and uses the disk space on the backup node.

The behavior is planned to be improved in scope of request CCU-20237 by adding a cancellation call to the backup operation after the timeout is reached.

Resolution

Resolve the reason behind the operation timeout. There could be 2 cases:

  • the backup indeed takes place and the default timeout value is not sufficient - follow OACI: Huge VE backup tasks fail by timeout to increase the timeout
  • the backup operation or vm2vf service are stuck without providing response to the Instance Manager - it is needed to find the exact reason and resolve the situation

In order to free up the backup space on the customer's subscription, it is necessary to contact Odin Technical Support and apply the following article for the failed backups:

How to delete a scheduled VE backup manually in OACI?

There is also a possibility to remove all backups missing from OACI database automatically.

Internal content

Link on internal Article