Migration error

[mir]

Hi,

A see the following error message when trying to migrate a VZ container:
Jul 16 03:14:22 starting migration of CT 102 to node 'esx2' (192.168.2.9)
Jul 16 03:14:22 container is running - using online migration
Jul 16 03:14:22 starting rsync phase 1
Jul 16 03:14:22 # /usr/bin/rsync -aH --delete --numeric-ids --sparse /var/lib/vz/private/102 root@192.168.2.9:/var/lib/vz/private
Jul 16 03:22:06 start live migration - suspending container
Jul 16 03:22:06 # vzctl --skiplock chkpnt 102 --suspend
Jul 16 03:22:06 Setting up checkpoint...
Jul 16 03:22:06 suspend...
Jul 16 03:22:06 Can not suspend container: Invalid argument
Jul 16 03:22:06 Error: unsupported fs type nfsd
Jul 16 03:22:06 ERROR: Failed to suspend container: Checkpointing failed
Jul 16 03:22:06 aborting phase 1 - cleanup resources
Jul 16 03:22:06 removing copied files on target node
Jul 16 03:22:16 start final cleanup
Jul 16 03:22:16 ERROR: migration aborted (duration 00:07:54): Failed to suspend container: Checkpointing failed
TASK ERROR: migration aborted

The error I expect to be causing the failure is this: "unsupported fs type nfsd"

The node migrating to:
$ cat /proc/filesystems
nodev sysfs
nodev rootfs
nodev bdev
nodev proc
nodev cgroup
nodev cpuset
nodev tmpfs
nodev devtmpfs
nodev binfmt_misc
nodev debugfs
nodev securityfs
nodev sockfs
nodev usbfs
nodev pipefs
nodev anon_inodefs
nodev inotifyfs
nodev devpts
nodev ramfs
nodev pram
nodev hugetlbfs
iso9660
nodev pstore
nodev mqueue
ext3
nodev fuse
fuseblk
nodev fusectl
nodev configfs
nodev rpc_pipefs
nodev nfs
nodev nfs4
nodev delayfs
nodev simfs

no nfsd!
Could this be avoided or should the nfsd kernel module be inserted manually?

Michael.

 

[RobFantini]

are there any sshfs mounts?

 

[mir]

After a manual modprobe nfsd on target node:
lsmod |grep nfsd
nfsd 307413 0
lockd 77737 3 nfsd,vzrst,nfs
nfs_acl 2695 2 nfsd,nfs
auth_rpcgss 44941 2 nfsd,nfs
sunrpc 248925 18 nfsd,nfs,lockd,nfs_acl,auth_rpcgss


cat /proc/filesystems |grep nfs
nodev nfs
nodev nfs4
nodev nfsd


But the error still persists???
Jul 16 18:46:38 starting migration of CT 102 to node 'esx2' (192.168.2.9)
Jul 16 18:46:38 container is running - using online migration
Jul 16 18:46:38 starting rsync phase 1
Jul 16 18:46:38 # /usr/bin/rsync -aH --delete --numeric-ids --sparse /var/lib/vz/private/102 root@192.168.2.9:/var/lib/vz/private
Jul 16 18:52:52 start live migration - suspending container
Jul 16 18:52:52 # vzctl --skiplock chkpnt 102 --suspend
Jul 16 18:52:52 Setting up checkpoint...
Jul 16 18:52:52 suspend...
Jul 16 18:52:52 Can not suspend container: Invalid argument
Jul 16 18:52:52 Error: unsupported fs type nfsd
Jul 16 18:52:52 ERROR: Failed to suspend container: Checkpointing failed
Jul 16 18:52:52 aborting phase 1 - cleanup resources
Jul 16 18:52:52 removing copied files on target node
Jul 16 18:53:03 start final cleanup
Jul 16 18:53:03 ERROR: migration aborted (duration 00:06:25): Failed to suspend container: Checkpointing failed
TASK ERROR: migration aborted

What do I miss here??

 

[mir]

are there any sshfs mounts?

No, both source and target node gives this response:

root@esx2:~# mount |grep sshfs
root@esx2:~#

 

[dietmar]

What kind of filesystem do you use on /var/lib/vz? Do you mount nfs inside the container?

 

[mir]

What kind of filesystem do you use on /var/lib/vz? Do you mount nfs inside the container?

/var/lib/vz on both nodes is local storage - ext3 above LVM.
source: /dev/mapper/pve-data on /var/lib/vz type ext3 (rw,noatime,barrier=0,data=writeback,nobh)
target: /dev/mapper/pve-data on /var/lib/vz type ext3 (rw)

No nfs mount inside the container. The container provides an nfsshare so maybe the reason is that a nfs mount is not migrateable?

 

[sunnyside]

Does someone has a solution for a NFS server on proxmox that can be live-migrate easily ?

As it's still not possible in 3.1 to live-migrate a CT with nfsd inside, the only possibility is to shutdown the CT before migrating.
But it is really anoying because it has impacts on all the NFS clients...

Thanks (and great work proxmox!!!)

root@hn15:~# pveversion -v
proxmox-ve-2.6.32: 3.1-109 (running kernel: 2.6.32-23-pve)
pve-manager: 3.1-3 (running version: 3.1-3/dc0e9b0e)
pve-kernel-2.6.32-23-pve: 2.6.32-109
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.0-1
pve-cluster: 3.0-7
qemu-server: 3.1-1
pve-firmware: 1.0-23
libpve-common-perl: 3.0-6
libpve-access-control: 3.0-6
libpve-storage-perl: 3.0-10
pve-libspice-server1: 0.12.4-1
vncterm: 1.1-4
vzctl: 4.0-1pve3
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.4-17
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.0-2