Symptoms
Provider found one or several orphaned backups on OACI storage nodes. Recently OA environment suffered from vm2vf outages on Virtuozzo hosts.
In vps.log of IM one may find that initial backup request (scheduled or manual) failed after several minutes with the following message:
fast failure for backup_ve_cb after 240000 MILLISECONDS
That means that IM tried to connect to vm2vf for 4 minutes and then cancelled the operation due to vm2vf unavailability. However, backup pipeline survived the outage and when vm2vf became available, backup operation was completed with the following message:
Failed to retrieve pipeline #3202, callback backup_done will not be invoked
Cause
Even if vm2vf is not available in some particular moment of time, ActiveMQ stores scheduled messages for later send. When it finds vm2vf 'endpoint' online, it delivers the message and backup procedure is started. Upon its finish, result will not be pushed to IM database since operation timeout already expired, but physically backup will remain on Storage host.
This behavior will be improved in scope of CCU-15182.
Resolution
The issue here is vm2vf service instability on Virtuozzo host, which should be addressed at first priority. Issue with backups will be solved by stable work of vm2vf.
To perform a cleanup of orphaned backups, please contact Odin Technical Support.