Symptoms
Online migration of a ploop container fails with an error :
can not undump CT#1952305 : /usr/sbin/vzctl exited with code 152
The following errors can be found in /var/log/messages on the destination node :
Aug 18 13:15:23 bnt-node41 kernel: [165386.798310] EXT4-fs (ploop55455p1): recovery complete
Aug 18 13:15:23 bnt-node41 kernel: [165386.798322] EXT4-fs (ploop55455p1): mounted filesystem with ordered data mode. Opts:
Aug 18 13:15:23 bnt-node41 kernel: [165386.805435] PFCache: relink 182:887281 to "/vz/pfcache" +0 -0 =0 peers
Aug 18 13:15:23 bnt-node41 kernel: [165386.805472] EXT4-fs (ploop55455p1): Couldn't remount RDWR because of unprocessed orphan inode list. Please umount/remount instead
Aug 18 13:15:23 bnt-node41 vzmdest[929387]: vzctl : Failed to mount image /vz/private/1952305: Can't mount file system dev=/dev/ploop55455p1 target=/vz/root/1952305: Invalid argument
Cause
Online migration fails because of file system errors inside the container.
Resolution
Restart the container.
If restart does not help, perform a file system check manually on the container's ploop image.
Internal content
It's unclear if the issue is in real fs corruption in the container or some problems triggered by migration/suspend/resume procedures.
To save the history the actions with container :
Though, it's unclear who is guilty for the corruption and at which stage it occurs. I can see the error present on CT's first start on node46 :
# grep -i 'container 1952305' /var/log/vzctl.log*
/var/log/vzctl.log:2013-08-16T17:42:15+0200 vzctl : Container 1952305 : Container is mounted
/var/log/vzctl.log:2013-08-16T17:42:15+0200 vzctl : Container 1952305 : Resuming the Container ...
/var/log/vzctl.log:2013-08-16T17:42:15+0200 vzctl : Container 1952305 : undump...
/var/log/vzctl.log:2013-08-16T17:42:15+0200 vzctl : Container 1952305 : CPU limit: 200.0%
/var/log/vzctl.log:2013-08-16T17:42:15+0200 vzctl : Container 1952305 : Set cpus: 2
/var/log/vzctl.log:2013-08-16T17:42:15+0200 vzctl : Container 1952305 : Adding IP addresses: 89.21.49.231 10.201.50.100
/var/log/vzctl.log:2013-08-16T17:42:17+0200 vzctl : Container 1952305 : Starting the Container ...
/var/log/vzctl.log:2013-08-16T17:42:18+0200 vzctl : Container 1952305 : Name i22-jabber is assigned
/var/log/vzctl.log:2013-08-16T17:42:18+0200 vzctl : Container 1952305 : Saved parameters for Container 1952305
/var/log/vzctl.log:2013-08-16T17:42:18+0200 vzctl : Container 1952305 : Resuming the Container...
# grep 'ploop35640' /var/log/messages*
/var/log/messages-20130818:Aug 16 17:42:15 bnt-node46 kernel: [ 8365.221033] ploop35640: p1
/var/log/messages-20130818:Aug 16 17:42:15 bnt-node46 kernel: [ 8365.227752] ploop35640: p1
/var/log/messages-20130818:Aug 16 17:42:15 bnt-node46 kernel: [ 8365.278542] EXT4-fs (ploop35640p1): orphan cleanup on readonly fs
/var/log/messages-20130818:Aug 16 17:42:15 bnt-node46 kernel: [ 8365.310742] EXT4-fs (ploop35640p1): 25 orphan inodes deleted
/var/log/messages-20130818:Aug 16 17:42:15 bnt-node46 kernel: [ 8365.377611] EXT4-fs (ploop35640p1): recovery complete
/var/log/messages-20130818:Aug 16 17:42:15 bnt-node46 kernel: [ 8365.379385] EXT4-fs (ploop35640p1): mounted filesystem with ordered data mode. Opts:
/var/log/messages-20130818:Aug 16 17:42:15 bnt-node46 kernel: [ 8365.394586] EXT4-fs (ploop35640p1): loaded balloon from 12 (0 blocks)
/var/log/messages-20130818:Aug 16 18:21:53 bnt-node46 kernel: [10739.117663] EXT4-fs error (device ploop35640p1): ext4_mb_generate_buddy: EXT4-fs: group 34: 14092 blocks in bitmap, 14083 in gd
I also see that the CT was earlier started on node41 :
# grep -i 'container 1952305' /var/log/vzctl.log*
/var/log/vzctl.log:2013-08-16T17:33:23+0200 vzctl : Container 1952305 : Container is mounted
/var/log/vzctl.log:2013-08-16T17:33:23+0200 vzctl : Container 1952305 : Resuming the Container ...
/var/log/vzctl.log:2013-08-16T17:33:23+0200 vzctl : Container 1952305 : undump...
/var/log/vzctl.log:2013-08-16T17:33:23+0200 vzctl : Container 1952305 : CPU limit: 200.0%
/var/log/vzctl.log:2013-08-16T17:33:23+0200 vzctl : Container 1952305 : Set cpus: 2
/var/log/vzctl.log:2013-08-16T17:33:23+0200 vzctl : Container 1952305 : Adding IP addresses: 89.21.49.231 10.201.50.100
/var/log/vzctl.log:2013-08-16T17:33:26+0200 vzctl : Container 1952305 : Starting the Container ...
/var/log/vzctl.log:2013-08-16T17:33:27+0200 vzctl : Container 1952305 : Name i22-jabber is assigned
/var/log/vzctl.log:2013-08-16T17:33:27+0200 vzctl : Container 1952305 : Saved parameters for Container 1952305
/var/log/vzctl.log:2013-08-16T17:33:27+0200 vzctl : Container 1952305 : Resuming the Container...
Here, no EXT4 errors are reported, only one warning :
# grep -i 'ploop.*1952305' /var/log/vzctl.log*
/var/log/vzctl.log:2013-08-16T17:33:23+0200 : add delta dev=/dev/ploop43050 img=/pstorage/shared/private/1952305/root.hdd/root.hds (rw)
/var/log/vzctl.log:2013-08-16T17:33:23+0200 : Mounting /dev/ploop43050p1 at /vz/root/1952305 fstype=ext4 data='balloon_ino=12,pfcache_csum,pfcache=/vz/pfcache,'
# grep ploop43050 /var/log/messages-20130818
Aug 16 17:33:23 bnt-node41 kernel: [ 8375.335556] ploop43050: p1
Aug 16 17:33:23 bnt-node41 kernel: [ 8375.340903] ploop43050: p1
Aug 16 17:33:23 bnt-node41 kernel: [ 8375.381133] EXT4-fs (ploop43050p1): orphan cleanup on readonly fs
Aug 16 17:33:23 bnt-node41 kernel: [ 8375.390262] EXT4-fs warning (device ploop43050p1): ext4_orphan_get: bad orphan inode 273443! e2fsck was run?
Aug 16 17:33:23 bnt-node41 kernel: [ 8375.390281] EXT4-fs (ploop43050p1): 5 orphan inodes deleted
Aug 16 17:33:23 bnt-node41 kernel: [ 8375.396997] EXT4-fs (ploop43050p1): recovery complete
Aug 16 17:33:23 bnt-node41 kernel: [ 8375.398766] EXT4-fs (ploop43050p1): mounted filesystem with ordered data mode. Opts:
Aug 16 17:33:23 bnt-node41 kernel: [ 8375.400495] EXT4-fs (ploop43050p1): loaded balloon from 12 (0 blocks)