Hello I am having problems with live migration
i am currently testing it with scientific linux
i have a 3 node cluster connected to an nfs store
first error i had was i was missing the dump folder in my nfs store
any ideas on where to debug
May 02 14:29:34 starting migration of CT 101 to node 'bn-co-ngn-svi-prox2' (10.1.7.172)
May 02 14:29:34 container is running - using online migration
May 02 14:29:34 container data is on shared storage 'FS2-VM-STORE'
May 02 14:29:34 start live migration - suspending container
May 02 14:29:34 dump container state
May 02 14:29:35 dump 2nd level quota
May 02 14:29:36 initialize container on remote node 'bn-co-ngn-svi-prox2'
May 02 14:29:36 initializing remote quota
May 02 14:29:44 turn on remote quota
May 02 14:29:44 load 2nd level quota
May 02 14:29:44 starting container on remote node 'bn-co-ngn-svi-prox2'
May 02 14:29:44 restore container state
May 02 14:29:45 # /usr/bin/ssh -c blowfish -o 'BatchMode=yes' root@10.1.7.172 vzctl restore 101 --undump --dumpfile /mnt/pve/FS2-VM-STORE/dump/dump.101 --skip_arpdetect
May 02 14:29:44 Restoring container ...
May 02 14:29:44 Starting container ...
May 02 14:29:44 Container is mounted
May 02 14:29:44 undump...
May 02 14:29:44 Setting CPU units: 1000
May 02 14:29:44 Setting CPUs: 1
May 02 14:29:44 Configure veth devices: veth101.0 veth101.1
May 02 14:29:44 Adding interface veth101.0 to bridge vmbr0 on CT0 for CT101
May 02 14:29:44 Adding interface veth101.1 to bridge vmbr666 on CT0 for CT101
May 02 14:29:45 vzquota : (warning) Quota is running for id 101 already
May 02 14:29:45 Error: undump failed: No such file or directory
May 02 14:29:45 Restoring failed:
May 02 14:29:45 Error: _rst_get_object: wrong object type: 1030 (expected: 1045) @162912
May 02 14:29:45 Error: rst_open_file: failed to lookup path '/usr/lib64/.nfs00000000000043c300000014': -2
May 02 14:29:45 Error: can't open file /usr/lib64/.nfs00000000000043c300000014
May 02 14:29:45 Error: do_rst_vma: rst_file: 104864
May 02 14:29:45 Error: do_rst_mm: failed to restore vma: -2
May 02 14:29:45 Error: do_rst_mm 1087520
May 02 14:29:45 Error: rst_mm: -2
May 02 14:29:45 Error: make_baby: -2
May 02 14:29:45 Error: rst_clone_children
May 02 14:29:45 Container is unmounted
May 02 14:29:45 ERROR: online migrate failure - Failed to restore container: Container start failed
May 02 14:29:45 start final cleanup
May 02 14:29:45 ERROR: migration finished with problems (duration 00:00:11)
TASK ERROR: migration problems
Any Thoughts
D
i am currently testing it with scientific linux
i have a 3 node cluster connected to an nfs store
first error i had was i was missing the dump folder in my nfs store
any ideas on where to debug
May 02 14:29:34 starting migration of CT 101 to node 'bn-co-ngn-svi-prox2' (10.1.7.172)
May 02 14:29:34 container is running - using online migration
May 02 14:29:34 container data is on shared storage 'FS2-VM-STORE'
May 02 14:29:34 start live migration - suspending container
May 02 14:29:34 dump container state
May 02 14:29:35 dump 2nd level quota
May 02 14:29:36 initialize container on remote node 'bn-co-ngn-svi-prox2'
May 02 14:29:36 initializing remote quota
May 02 14:29:44 turn on remote quota
May 02 14:29:44 load 2nd level quota
May 02 14:29:44 starting container on remote node 'bn-co-ngn-svi-prox2'
May 02 14:29:44 restore container state
May 02 14:29:45 # /usr/bin/ssh -c blowfish -o 'BatchMode=yes' root@10.1.7.172 vzctl restore 101 --undump --dumpfile /mnt/pve/FS2-VM-STORE/dump/dump.101 --skip_arpdetect
May 02 14:29:44 Restoring container ...
May 02 14:29:44 Starting container ...
May 02 14:29:44 Container is mounted
May 02 14:29:44 undump...
May 02 14:29:44 Setting CPU units: 1000
May 02 14:29:44 Setting CPUs: 1
May 02 14:29:44 Configure veth devices: veth101.0 veth101.1
May 02 14:29:44 Adding interface veth101.0 to bridge vmbr0 on CT0 for CT101
May 02 14:29:44 Adding interface veth101.1 to bridge vmbr666 on CT0 for CT101
May 02 14:29:45 vzquota : (warning) Quota is running for id 101 already
May 02 14:29:45 Error: undump failed: No such file or directory
May 02 14:29:45 Restoring failed:
May 02 14:29:45 Error: _rst_get_object: wrong object type: 1030 (expected: 1045) @162912
May 02 14:29:45 Error: rst_open_file: failed to lookup path '/usr/lib64/.nfs00000000000043c300000014': -2
May 02 14:29:45 Error: can't open file /usr/lib64/.nfs00000000000043c300000014
May 02 14:29:45 Error: do_rst_vma: rst_file: 104864
May 02 14:29:45 Error: do_rst_mm: failed to restore vma: -2
May 02 14:29:45 Error: do_rst_mm 1087520
May 02 14:29:45 Error: rst_mm: -2
May 02 14:29:45 Error: make_baby: -2
May 02 14:29:45 Error: rst_clone_children
May 02 14:29:45 Container is unmounted
May 02 14:29:45 ERROR: online migrate failure - Failed to restore container: Container start failed
May 02 14:29:45 start final cleanup
May 02 14:29:45 ERROR: migration finished with problems (duration 00:00:11)
TASK ERROR: migration problems
Any Thoughts
D