vz migrate from non local storage, possible small bug.

[bread-baker]

migration of openvz works great,

but I found a small bug and did not see which section it should be put to at https://bugzilla.proxmox.com/

this happend two times.

I have Container and other storage set to /data/fbc1-storage on node fbc1 .

after migrating 2 containers to another node , /data/fbc1-storage/private still contains:

root@fbc1 /data/fbc1-storage/private # du -sh /data/fbc1-storage/private/*
1.7G    /data/fbc1-storage/private/10004
615M    /data/fbc1-storage/private/14101

here are the logs from the migrate:

Dec 18 19:52:53 starting migration of CT 10004 to node 'fbc208' (10.100.100.208)
Dec 18 19:52:53 container is running - using online migration
Dec 18 19:52:53 starting rsync phase 1
Dec 18 19:52:53 # /usr/bin/rsync -aH --delete --numeric-ids --sparse /var/lib/vz/private/10004 root@10.100.100.208:/var/lib/vz/private
Dec 18 19:53:32 start live migration - suspending container
Dec 18 19:53:32 dump container state
Dec 18 19:53:32 copy dump file to target node
Dec 18 19:53:33 starting rsync (2nd pass)
Dec 18 19:53:33 # /usr/bin/rsync -aH --delete --numeric-ids /var/lib/vz/private/10004 root@10.100.100.208:/var/lib/vz/private
Dec 18 19:53:41 dump 2nd level quota
Dec 18 19:53:41 copy 2nd level quota to target node
Dec 18 19:53:41 initialize container on remote node 'fbc208'
Dec 18 19:53:41 initializing remote quota
Dec 18 19:53:42 turn on remote quota
Dec 18 19:53:42 load 2nd level quota
Dec 18 19:53:42 starting container on remote node 'fbc208'
Dec 18 19:53:42 restore container state
Dec 18 19:53:44 removing container files on local node
Dec 18 19:53:45 start final cleanup
Dec 18 19:53:45 migration finished successfuly (duration 00:00:52)
TASK OK

Dec 18 19:45:48 starting migration of CT 14101 to node 'fbc208' (10.100.100.208)
Dec 18 19:45:48 container is running - using online migration
Dec 18 19:45:48 starting rsync phase 1
Dec 18 19:45:48 # /usr/bin/rsync -aH --delete --numeric-ids --sparse /var/lib/vz/private/14101 root@10.100.100.208:/var/lib/vz/private
Dec 18 19:46:08 start live migration - suspending container
Dec 18 19:46:08 dump container state
Dec 18 19:46:08 copy dump file to target node
Dec 18 19:46:08 starting rsync (2nd pass)
Dec 18 19:46:08 # /usr/bin/rsync -aH --delete --numeric-ids /var/lib/vz/private/14101 root@10.100.100.208:/var/lib/vz/private
Dec 18 19:46:09 dump 2nd level quota
Dec 18 19:46:09 copy 2nd level quota to target node
Dec 18 19:46:09 initialize container on remote node 'fbc208'
Dec 18 19:46:09 initializing remote quota
Dec 18 19:46:09 turn on remote quota
Dec 18 19:46:10 load 2nd level quota
Dec 18 19:46:10 starting container on remote node 'fbc208'
Dec 18 19:46:10 restore container state
Dec 18 19:46:10 removing container files on local node
Dec 18 19:46:11 start final cleanup
Dec 18 19:46:11 migration finished successfuly (duration 00:00:23)
TASK OK

note the containers are running OK on node fbc208

 

[bread-baker]

also on the original node , note the .mount files from migrated containers still exist:

root@fbc1 /etc/pve # ls /etc/pve/openvz
10001.conf  10001.mount  10002.conf  10002.mount  128.mount  14101.mount  8007.mount

 

[dietmar]

Please can you file a bug report at https://bugzilla.proxmox.com/ ?

Many thank for you help!

 

[dietmar]

Something is strange - you logs indicate that the containers are stored at:

/var/lib/vz/private/

So that containers are not on /data/fbc1-storage/ ?

 

[bread-baker]

I am not certain where they were stored, I just assumed they were at originally /data/fbc1-storage/

fbc1 was reinstalled Saturday at 7PM or so. around 9 I restored those from backup.

I have rsnapshot backups of fbc1 before the reinstall . so check this:

root@fbc208 /data/rsnapshot-pve/daily.5/fbc1/pve/etc/pve/nodes/fbc1/openvz # ls -l
total 40
-rw-r----- 5 root www-data 1465 Dec 11 15:20 10004.conf
-rw-r----- 5 root www-data 1428 Dec 16 17:17 10220.conf
-rwxr----- 5 root www-data  249 Dec 16 17:17 10220.mount
-rw-r----- 5 root www-data 1485 Dec 11 14:53 14101.conf
-rwxr----- 5 root www-data   98 Dec 11 14:51 14101.mount
-rw-r----- 5 root www-data 1446 Dec 11 12:59 148.conf
-rwxr----- 5 root www-data   97 Dec  8 16:29 148.mount
-rw-r----- 5 root www-data 1540 Dec 11 12:59 155.conf
-rwxr----- 5 root www-data   98 Dec  8 16:32 155.mount
-rw-r----- 5 root www-data 1524 Dec 16 16:49 8005.conf

root@fbc208 /data/rsnapshot-pve/daily.5/fbc1/pve/etc/pve/nodes/fbc1/openvz # grep VE_ROOT *conf
10004.conf:VE_ROOT="/var/lib/vz/root/10004"
10220.conf:VE_ROOT="/var/lib/vz/root/10220"
14101.conf:VE_ROOT="/var/lib/vz/root/14101"
148.conf:VE_ROOT="/var/lib/vz/root/148"
155.conf:VE_ROOT="/var/lib/vz/root/155"
8005.conf:VE_ROOT="/var/lib/vz/root/8005"

so on the 16-th the ct's were at the default location.

then on the 17-th:

root@fbc208 /data/rsnapshot-pve/daily.0/fbc1/pve/etc/pve/nodes/fbc1/openvz # ll
total 48
drwxr-x--- 2 root www-data 4096 Dec 17 18:01 ./
drwxr-x--- 5 root www-data 4096 Dec 17 18:01 ../
-rw-r----- 1 root www-data  970 Dec 17 19:47 10001.conf
-rwxr----- 1 root www-data  216 Dec 17 19:47 10001.mount*
-rw-r----- 1 root www-data  917 Dec 17 20:00 10002.conf
-rwxr----- 1 root www-data  120 Dec 17 20:00 10002.mount*
-rw-r----- 1 root www-data 1458 Dec 17 21:35 10004.conf
-rw-r----- 1 root www-data 1911 Dec 17 21:51 128.conf
-rwxr----- 1 root www-data  243 Dec 17 21:51 128.mount*
-rw-r----- 1 root www-data 1478 Dec 17 18:31 14101.conf
-rwxr----- 1 root www-data   98 Dec 17 18:31 14101.mount*
-rwxr----- 1 root www-data  216 Dec 17 19:38 8007.mount*
root@fbc208 /data/rsnapshot-pve/daily.0/fbc1/pve/etc/pve/nodes/fbc1/openvz # grep VE_ROOT *conf
10001.conf:VE_ROOT="/var/lib/vz/root/10001"
10002.conf:VE_ROOT="/var/lib/vz/root/10002"
10004.conf:VE_ROOT="/var/lib/vz/root/10004"
128.conf:VE_ROOT="/var/lib/vz/root/128"
14101.conf:VE_ROOT="/var/lib/vz/root/14101"
root@fbc208 /data/rsnapshot-pve/daily.0/fbc1/pve/etc/pve/nodes/fbc1/openvz #

Saturday I had reinstalled prox2.0 on all our servers to start from scratch due to residual problems from the managed switch issue. I did a lot of moving containers around.