Update Stategy for Proxmox VE?

[lousek]

Hello forum,

I have built up a Proxmox VE cluster using two servers with mirrored storage (DRBD).
Btw: this is only @home for testing etc., so nothing business-critical or such things

Now: Earlier with VMware ESXi / vCenter, we did the updates of the hosts like this:

1. Move all VMs to Host 1 (Live Migration)
2. Set Host 2 to maintenance mode and install updates
3. Move a non-critical VM back to Host 2 and see if everything is working as expected
4. Move all VMs to Host 2
5. Set HOst 1 to maintenance mode and isntall updates
6. Move a non-critical VM back to Host 1 and see if everything is working as expected
7. Move a part of the VMs to Host 1

I have tried to do this using Proxmox VE, but after updating the first host, I cannot move back any VM back to this host using live/online migration: the VM is just powered off (Task failed)!

Now: what would be the correct update strategy in a Proxmox VE cluster when used in a High-Availability environment?

BR,
lousek

 

[nik-weter]

Which version of proxmox? Live Migration is not working between qemu - kvm 1.2 and 1.3. It is written in the wiki.

 

[lousek]

I guess it is 2.1 (see below).
And KVM should be in Version 1.3.

pve-manager: 2.1-14 (pve-manager/2.1/f32f3f46)
running kernel: 2.6.32-14-pve
proxmox-ve-2.6.32: 2.1-73
pve-kernel-2.6.32-14-pve: 2.6.32-74
pve-kernel-2.6.32-7-pve: 2.6.32-60
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.4-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.93-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.9-1
pve-cluster: 1.0-34
qemu-server: 2.0-72
pve-firmware: 1.0-21
libpve-common-perl: 1.0-41
libpve-access-control: 1.0-25
libpve-storage-perl: 2.0-36
vncterm: 1.0-3
vzctl: 3.0.30-2pve5
vzprocps: 2.0.11-2
vzquota: 3.1-1
pve-qemu-kvm: 1.3-10
ksm-control-daemon: 1.1-1

I have the following updates:

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages have been kept back:
ceph-common libpve-storage-perl librados2 librbd1 proxmox-ve-2.6.32 pve-manager vzctl
The following packages will be upgraded:
apache2 apache2-mpm-prefork apache2-utils apache2.2-bin apache2.2-common corosync-pve libcorosync4-pve libiscsi1 libpve-access-control libpve-common-perl pve-cluster pve-qemu-kvm qemu-server
13 upgraded, 0 newly installed, 0 to remove and 7 not upgraded.
Need to get 6,299 kB of archives.
After this operation, 385 kB disk space will be freed.
Do you want to continue [Y/n]?

So it should work as I have written above similiar to ESXi?

BR,
lousek

 

[Jérôme]

I guess it is 2.1 (see below).
And KVM should be in Version 1.3.


I have the following updates:


So it should work as I have written above similiar to ESXi?

BR,
lousek

Hi,

A thing you can try is to make a snapshot (no unavailable time) and restore the vm from the updated node.

Let us know if it works.

 

[lousek]

Hi,

A thing you can try is to make a snapshot (no unavailable time) and restore the vm from the updated node.

Let us know if it works.

Hello

The backup / snapshot is clear for me. But how about restoring?
Then I have the VM twice (once on Host1 and once on Host2) ... ?
Sorry, can you explain a bit more?

BR,
lousek

 

[udo]

I guess it is 2.1 (see below).
And KVM should be in Version 1.3.


I have the following updates:


So it should work as I have written above similiar to ESXi?

BR,
lousek

Hi,
you should use "apt-get update && apt-get dist-upgrade" to get all updates - to version 2.3 (apt-get upgrade is not enough).
For live migration:
Work only between same kvm-versions (sometimes also the same kernel-version is required). Also a short downtime on huge updates are nessesary.

Udo

 

[Jérôme]

Hello

The backup / snapshot is clear for me. But how about restoring?
Then I have the VM twice (once on Host1 and once on Host2) ... ?
Sorry, can you explain a bit more?

BR,
lousek

Yes, it's a "cope" technical.

To "clone" a VM from a backup use, qmrestore with this syntax :

qmrestore /mnt/pve/tcmsrvnfs/dump/vzdump-qemu-102-2013_02_26-09_59_58.tar.gz -storage tcmsrvnfs 220

Where "/mnt/pve/tcmsrvnfs/dump/vzdump-qemu-102-2013_02_26-09_59_58.tar.gz" is the snapshot file.
Where "-storage tcmsrvnfs" is the destination where the VM will be restored.
Where "220" is the new VMID of the VM


You can use local storage, without '-storage tcmsrvnfs' argument. You can see more at "man qmrestore".


After you have restore the VM, you need to change the MAC @, it's very easy, go on the WebGUI and remove the network controller, then, add a new one.


That's all. Have you got others questions?

Regards

 

[lousek]

Yes, it's a "cope" technical.

When I change the MAC-address, then I have also to reconfigure the IP-addresses in the VMs as they are Linux-VMs and there is a relations-ship between MAC-address and the numer of the interface (e.g. eth0).

Tell me if I am right:

1. VMs are running on Host 1
2. I make a live backup (snapshot) of all VMs running on Host 1
3. Then I make a restore to new VMs on Host 2???
4. And then? Turn off the VMs on Host 1, and turn the VMs on on Host 2? Then I have a down-time ...

Of course, I understand that it might not be possible to keep VMs up and running when upgrading from e.g. 2.2 to 2.3.
But what is about minor updates?
Only as soon as KVM and PVE-kernel is not part of the updates?

Thank you all very much for the help!

BR,
lousek

 

[Jérôme]

When I change the MAC-address, then I have also to reconfigure the IP-addresses in the VMs as they are Linux-VMs and there is a relations-ship between MAC-address and the numer of the interface (e.g. eth0).

Tell me if I am right:

1. VMs are running on Host 1
2. I make a live backup (snapshot) of all VMs running on Host 1
3. Then I make a restore to new VMs on Host 2???
4. And then? Turn off the VMs on Host 1, and turn the VMs on on Host 2? Then I have a down-time ...

Of course, I understand that it might not be possible to keep VMs up and running when upgrading from e.g. 2.2 to 2.3.
But what is about minor updates?
Only as soon as KVM and PVE-kernel is not part of the updates?

Thank you all very much for the help!


BR,
lousek

1. Backup VM's snapshot without downtime
2. Reconfigure MAC/IP @
3. Stop the VM from node1 and start quickly the VM from node2 (minimal downtime)
4. Upgrade on the node1
5. See if all is nice.. vms running etc...
6. Do this 1-5 on the node2

It helps you ?


I have do the upgrade on my dedicated server without close any vm's.

Current status: 19 updates [+19], 27936 new [+3].Resolving dependencies...
The following NEW packages will be installed:
  libboost-thread1.42.0{a} libfile-chdir-perl{a} pve-kernel-2.6.32-18-pve{a}
The following packages will be REMOVED:
  pve-kernel-2.6.32-17-pve{u}
The following packages will be upgraded:
  apache2 apache2-mpm-prefork apache2-utils apache2.2-bin apache2.2-common ceph-common corosync-pve libcorosync4-pve libiscsi1 libpve-access-control libpve-common-perl libpve-storage-perl librados2
  librbd1 proxmox-ve-2.6.32 pve-cluster pve-manager pve-qemu-kvm qemu-server
19 packages upgraded, 3 newly installed, 1 to remove and 0 not upgraded.
Need to get 45.2 MB of archives. After unpacking 1,221 kB will be used.
Do you want to continue? [Y/n/?] [B]y[/B]
Get:1 http://ftp.fr.debian.org/debian/ squeeze/main libboost-thread1.42.0 amd64 1.42.0-4 [52.8 kB]
Get:2 http://download.proxmox.com/debian/ squeeze/pve pve-kernel-2.6.32-18-pve amd64 2.6.32-88 [31.9 MB]
Get:3 http://security.debian.org/ squeeze/updates/main apache2 amd64 2.2.16-6+squeeze11 [1,400 B]
Get:4 http://security.debian.org/ squeeze/updates/main apache2-mpm-prefork amd64 2.2.16-6+squeeze11 [2,284 B]
Get:5 http://security.debian.org/ squeeze/updates/main apache2.2-common amd64 2.2.16-6+squeeze11 [302 kB]
Get:6 http://ftp.fr.debian.org/debian/ squeeze/main libfile-chdir-perl all 0.1003-0.1 [13.6 kB]
Get:7 http://security.debian.org/ squeeze/updates/main apache2.2-bin amd64 2.2.16-6+squeeze11 [1,436 kB]
Get:8 http://security.debian.org/ squeeze/updates/main apache2-utils amd64 2.2.16-6+squeeze11 [165 kB]
Get:9 http://download.proxmox.com/debian/ squeeze/pve ceph-common amd64 0.56.3-1~bpo60+1 [4,053 kB]
Get:10 http://download.proxmox.com/debian/ squeeze/pve librbd1 amd64 0.56.3-1~bpo60+1 [255 kB]
Get:11 http://download.proxmox.com/debian/ squeeze/pve librados2 amd64 0.56.3-1~bpo60+1 [1,265 kB]
Get:12 http://download.proxmox.com/debian/ squeeze/pve libiscsi1 amd64 1.8.0-1 [47.9 kB]
Get:13 http://download.proxmox.com/debian/ squeeze/pve pve-qemu-kvm amd64 1.4-6 [3,253 kB]
Get:14 http://download.proxmox.com/debian/ squeeze/pve libpve-common-perl all 1.0-48 [38.3 kB]
Get:15 http://download.proxmox.com/debian/ squeeze/pve libpve-storage-perl all 2.3-6 [41.3 kB]
Get:16 http://download.proxmox.com/debian/ squeeze/pve libcorosync4-pve amd64 1.4.4-4 [309 kB]
Get:17 http://download.proxmox.com/debian/ squeeze/pve pve-cluster amd64 1.0-36 [247 kB]
Get:18 http://download.proxmox.com/debian/ squeeze/pve qemu-server amd64 2.3-17 [102 kB]
Get:19 http://download.proxmox.com/debian/ squeeze/pve libpve-access-control amd64 1.0-26 [30.9 kB]
Get:20 http://download.proxmox.com/debian/ squeeze/pve pve-manager all 2.3-13 [1,256 kB]
Get:21 http://download.proxmox.com/debian/ squeeze/pve proxmox-ve-2.6.32 all 2.3-88 [4,488 B]
Get:22 http://download.proxmox.com/debian/ squeeze/pve corosync-pve amd64 1.4.4-4 [364 kB]
Fetched 45.2 MB in 4s (10.5 MB/s)
Reading changelogs... Done
Preconfiguring packages ...
Selecting previously deselected package pve-kernel-2.6.32-18-pve.
(Reading database ... 40331 files and directories currently installed.)
Unpacking pve-kernel-2.6.32-18-pve (from .../pve-kernel-2.6.32-18-pve_2.6.32-88_amd64.deb) ...
Preparing to replace apache2 2.2.16-6+squeeze10 (using .../apache2_2.2.16-6+squeeze11_amd64.deb) ...
Unpacking replacement apache2 ...
Preparing to replace apache2-mpm-prefork 2.2.16-6+squeeze10 (using .../apache2-mpm-prefork_2.2.16-6+squeeze11_amd64.deb) ...
Stopping web server: apache2 ... waiting .....
Unpacking replacement apache2-mpm-prefork ...
Preparing to replace apache2.2-common 2.2.16-6+squeeze10 (using .../apache2.2-common_2.2.16-6+squeeze11_amd64.deb) ...
Unpacking replacement apache2.2-common ...
Preparing to replace apache2.2-bin 2.2.16-6+squeeze10 (using .../apache2.2-bin_2.2.16-6+squeeze11_amd64.deb) ...
Unpacking replacement apache2.2-bin ...
Preparing to replace apache2-utils 2.2.16-6+squeeze10 (using .../apache2-utils_2.2.16-6+squeeze11_amd64.deb) ...
Unpacking replacement apache2-utils ...
Selecting previously deselected package libboost-thread1.42.0.
Unpacking libboost-thread1.42.0 (from .../libboost-thread1.42.0_1.42.0-4_amd64.deb) ...
Preparing to replace ceph-common 0.48.2argonaut-1~bpo60+1 (using .../ceph-common_0.56.3-1~bpo60+1_amd64.deb) ...
Unpacking replacement ceph-common ...
Preparing to replace librbd1 0.48.2argonaut-1~bpo60+1 (using .../librbd1_0.56.3-1~bpo60+1_amd64.deb) ...
Unpacking replacement librbd1 ...
Preparing to replace librados2 0.48.2argonaut-1~bpo60+1 (using .../librados2_0.56.3-1~bpo60+1_amd64.deb) ...
Unpacking replacement librados2 ...
Preparing to replace libiscsi1 1.7.0-1 (using .../libiscsi1_1.8.0-1_amd64.deb) ...
Unpacking replacement libiscsi1 ...
Preparing to replace pve-qemu-kvm 1.3-10 (using .../pve-qemu-kvm_1.4-6_amd64.deb) ...
Unpacking replacement pve-qemu-kvm ...
Preparing to replace libpve-common-perl 1.0-41 (using .../libpve-common-perl_1.0-48_all.deb) ...
Unpacking replacement libpve-common-perl ...
Selecting previously deselected package libfile-chdir-perl.
Unpacking libfile-chdir-perl (from .../libfile-chdir-perl_0.1003-0.1_all.deb) ...
Preparing to replace libpve-storage-perl 2.0-36 (using .../libpve-storage-perl_2.3-6_all.deb) ...
Unpacking replacement libpve-storage-perl ...
Preparing to replace libcorosync4-pve 1.4.4-1 (using .../libcorosync4-pve_1.4.4-4_amd64.deb) ...
Unpacking replacement libcorosync4-pve ...
Preparing to replace pve-cluster 1.0-34 (using .../pve-cluster_1.0-36_amd64.deb) ...
Unpacking replacement pve-cluster ...
Preparing to replace qemu-server 2.0-72 (using .../qemu-server_2.3-17_amd64.deb) ...
Unpacking replacement qemu-server ...
Preparing to replace libpve-access-control 1.0-25 (using .../libpve-access-control_1.0-26_amd64.deb) ...
Unpacking replacement libpve-access-control ...
Preparing to replace pve-manager 2.2-32 (using .../pve-manager_2.3-13_all.deb) ...
Unpacking replacement pve-manager ...
Preparing to replace proxmox-ve-2.6.32 2.2-83 (using .../proxmox-ve-2.6.32_2.3-88_all.deb) ...
Unpacking replacement proxmox-ve-2.6.32 ...
Processing triggers for man-db ...
(Reading database ... 42772 files and directories currently installed.)
Removing pve-kernel-2.6.32-17-pve ...
(Reading database ... 40348 files and directories currently installed.)
Preparing to replace corosync-pve 1.4.4-1 (using .../corosync-pve_1.4.4-4_amd64.deb) ...
Unpacking replacement corosync-pve ...
Processing triggers for man-db ...
Setting up pve-kernel-2.6.32-18-pve (2.6.32-88) ...
update-initramfs: Generating /boot/initrd.img-2.6.32-18-pve
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-2.6.32-18-pve
Found initrd image: /boot/initrd.img-2.6.32-18-pve
Found linux image: /boot/vmlinuz-2.6.32-16-pve
Found initrd image: /boot/initrd.img-2.6.32-16-pve
Found memtest86+ image: /memtest86+.bin
Found memtest86+ multiboot image: /memtest86+_multiboot.bin
done
Setting up apache2.2-bin (2.2.16-6+squeeze11) ...
Setting up apache2-utils (2.2.16-6+squeeze11) ...
Setting up apache2.2-common (2.2.16-6+squeeze11) ...
Setting up apache2-mpm-prefork (2.2.16-6+squeeze11) ...
Starting web server: apache2.
Setting up apache2 (2.2.16-6+squeeze11) ...
Setting up libboost-thread1.42.0 (1.42.0-4) ...
Setting up librados2 (0.56.3-1~bpo60+1) ...
Setting up librbd1 (0.56.3-1~bpo60+1) ...
Setting up ceph-common (0.56.3-1~bpo60+1) ...
Installing new version of config file /etc/bash_completion.d/rbd ...
Setting up libiscsi1 (1.8.0-1) ...
Setting up pve-qemu-kvm (1.4-6) ...
Setting up libpve-common-perl (1.0-48) ...
Setting up libfile-chdir-perl (0.1003-0.1) ...
Setting up libpve-storage-perl (2.3-6) ...
Setting up libcorosync4-pve (1.4.4-4) ...
Setting up pve-cluster (1.0-36) ...
Restarting pve cluster filesystem: pve-cluster.
Setting up qemu-server (2.3-17) ...
Setting up libpve-access-control (1.0-26) ...
Setting up pve-manager (2.3-13) ...
Restarting PVE Daemon: pvedaemon.
Restarting PVE Status Daemon: pvestatd.
Restarting web server: apache2 ... waiting .
Setting up proxmox-ve-2.6.32 (2.3-88) ...
installing proxmox release key: OK
Setting up corosync-pve (1.4.4-4) ...


Current status: 0 updates [-19].
root@proxmox:~# [B]qm list[/B]
      VMID NAME                 STATUS     MEM(MB)    BOOTDISK(GB) PID
       100 SQL                  running    512               50.00 58293
root@proxmox:~#

root@proxmox:~# pveversion -vpve-manager: 2.3-13 (pve-manager/2.3/7946f1f1)
running kernel: 2.6.32-17-pve
proxmox-ve-2.6.32: 2.3-88
pve-kernel-2.6.32-16-pve: 2.6.32-82
pve-kernel-2.6.32-18-pve: 2.6.32-88
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.4-4
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.93-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.9-1
pve-cluster: 1.0-36
qemu-server: 2.3-17
pve-firmware: 1.0-21
libpve-common-perl: 1.0-48
libpve-access-control: 1.0-26
libpve-storage-perl: 2.3-6
vncterm: 1.0-3
vzctl: 4.0-1pve2
vzprocps: 2.0.11-2
vzquota: 3.1-1
pve-qemu-kvm: 1.4-6
ksm-control-daemon: 1.1-1

But it's probably a bad thing cause it can be broken at the reboot, I have not check if the VM's are running nice on 2.3 doing this.

Regards