Proxmox VE 6.3 available

I'm having an issue with storage migration. I have a VM in local storage and want to migrate it to another local storage on a different host.

I did it offline, the process started, I could see traffic on the network interfaces, no feedback of the trasfer, the traffic stopped after a minute then after 15 more minutes the process failed:

Code: 
2020-12-30 23:57:35 starting migration of VM 5200 to node 'pilu3' (10.0.0.130)
2020-12-30 23:57:35 found local disk 'local-lvm:vm-5200-disk-0' (in current VM config)
2020-12-30 23:57:36 copying local disk images
2020-12-30 23:57:38 Logical volume "vm-5200-disk-0" created.
2020-12-31 00:16:24 packet_write_wait: Connection to 10.0.0.130 port 22: Broken pipe
2020-12-31 00:16:25 command 'dd 'if=/dev/pve/vm-5200-disk-0' 'bs=64k'' failed: got signal 13
send/receive failed, cleaning up snapshot(s)..
2020-12-31 00:16:25 ERROR: Failed to sync data - storage migration for 'local-lvm:vm-5200-disk-0' to storage 'local-lvm' failed - command 'set -o pipefail && pvesm export local-lvm:vm-5200-disk-0 raw+size - -with-snapshots 0 | /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=pilu3' root@10.0.0.130 -- pvesm import local-lvm:vm-5200-disk-0 raw+size - -with-snapshots 0 -allow-rename 1' failed: exit code 255
2020-12-31 00:16:25 aborting phase 1 - cleanup resources
2020-12-31 00:16:25 ERROR: found stale volume copy 'local-lvm:vm-5200-disk-0' on node 'pilu3'
2020-12-31 00:16:25 ERROR: migration aborted (duration 00:18:51): Failed to sync data - storage migration for 'local-lvm:vm-5200-disk-0' to storage 'local-lvm' failed - command 'set -o pipefail && pvesm export local-lvm:vm-5200-disk-0 raw+size - -with-snapshots 0 | /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=pilu3' root@10.0.0.130 -- pvesm import local-lvm:vm-5200-disk-0 raw+size - -with-snapshots 0 -allow-rename 1' failed: exit code 255
TASK ERROR: migration aborted

The hosts are bridged through an eoip tunnel. It should not be a factor as AFAIK the transfer is done via ssh.
 
Ultimately the issue was with networking equipment, so you may disregard this post.

vihai said:
I'm having an issue with storage migration. I have a VM in local storage and want to migrate it to another local storage on a different host.

I did it offline, the process started, I could see traffic on the network interfaces, no feedback of the trasfer, the traffic stopped after a minute then after 15 more minutes the process failed:
Click to expand...
 
I'm having trouble with qm shutdown on PVE 6.3.3, where the client operating system shuts down but the VM does not stop. I then have to wait for the qm shutdown command to timeout (or cancel it via the web GUI) and issue a further qm stop command to switch off the VM. This wasn't happening on PVE 6.2

Where it gets odd, is the problem is only happening on host nodes on Fujitsu Blade servers. I have HP Proliant server hosts on the same cluster and can shutdown VMs fine on those. I can even migrate VMs from the Fujitsu Blade to the HP Proliant and successfully issue a shutdown command, and also the opposite, where a VM migrated from the HP to the Fujitsu won't shutdown. Where it gets even stranger, is I have another PVE 6.3.3 cluster installed across several more Fujitsu Blade servers and problem is not happening on there.

I have compared all configurations between the two PVE cluster installations on the Fujitsu Blades and they look to be the same. Both were installed using the 6.2 ISO some time last year then upgraded to 6.3 in December

The problem is seen on all my VMs, which are a selection of Windows 7, Windows 10 and many different Linux distros. Most VMs are using VirtIO but some are SCSI. All have qemu-guest-agent installed.

If I have the VNC Console open for Ubuntu VMs when I issue qa shutdown it shows "systemd-shutdown[1]: Failed to finalize DM devices, ignoring" while the VM tries to shutdown.
 
Matthew Daniel said:
I'm having trouble with qm shutdown on PVE 6.3.3, where the client operating system shuts down but the VM does not stop. I then have to wait for the qm shutdown command to timeout (or cancel it via the web GUI) and issue a further qm stop command to switch off the VM. This wasn't happening on PVE 6.2

Where it gets odd, is the problem is only happening on host nodes on Fujitsu Blade servers. I have HP Proliant server hosts on the same cluster and can shutdown VMs fine on those. I can even migrate VMs from the Fujitsu Blade to the HP Proliant and successfully issue a shutdown command, and also the opposite, where a VM migrated from the HP to the Fujitsu won't shutdown. Where it gets even stranger, is I have another PVE 6.3.3 cluster installed across several more Fujitsu Blade servers and problem is not happening on there.

I have compared all configurations between the two PVE cluster installations on the Fujitsu Blades and they look to be the same. Both were installed using the 6.2 ISO some time last year then upgraded to 6.3 in December

The problem is seen on all my VMs, which are a selection of Windows 7, Windows 10 and many different Linux distros. Most VMs are using VirtIO but some are SCSI. All have qemu-guest-agent installed.

If I have the VNC Console open for Ubuntu VMs when I issue qa shutdown it shows "systemd-shutdown[1]: Failed to finalize DM devices, ignoring" while the VM tries to shutdown.
Click to expand...

After poking around in Syslog I see thousands of 'qmeventd[962]: accept: Too many open files' entries and also 'qmeventd[962]: error opening /proc/20196/cmdline: Too many open files'.
The upgrade to PVE 6.3.3 also coincided with installation of extra RAM on the servers to allow hosting many more VMs (around 50 in total), which get switched on/off throughout the day as required.

I rebooted the server and no more 'Too many open files' entries were seen, plus I can now correctly shut down VMs using qm shutdown.

I'm now thinking fixing the 'Too many open files' issue should allow me to proceed so I'm guessing default ulimit settings are not adequate for my implementation and am seeking advice on how to make the correct adjustments
 
Hi,
vRod said:
Is there any update on this? Having the cloud-init images located on rbd also breaks cloud-init (no longer gets applied). Moving the cloud-init image back to local storage fixes this.
Click to expand...
there are fixes for the offline EFI disk move and also for cloud-init cloning in qemu-server 6.3-3, which is currently available in the pvetest repository. The online EFI disk move to ZFS/RBD storages requires a more in-depth look, you can follow the bug report to stay informed.
 
Hi everyone,

We've been running Proxmox for less than a year now, planning for upgrading it to the latest version (6.3).

We have a 4 node cluster, with VM local storage.

Any recommendations on how to proceed? Should we shut down all VM's or move them to another node, is it good idea to upgrade one node at a time, etc? Was trying to find some upgrade steps somewhere, but couldn't.

Thanks in advance,
J.
 
  • Like
Reactions: sumsum
jlmorgado said:
Any recommendations on how to proceed? Should we shut down all VM's or move them to another node, is it good idea to upgrade one node at a time, etc? Was trying to find some upgrade steps somewhere, but couldn't.
Click to expand...
If you mean to upgrade from the old, now EOL, 5.X major version please check out our official upgrade documentation:
https://pve.proxmox.com/wiki/Upgrade_from_5.x_to_6.0

For an upgrade from an older state of the 6.X major release it's basically:
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_system_software_updates
and
https://pve.proxmox.com/wiki/Downlo...Proxmox_Virtual_Environment_6.x_to_latest_6.3
This can also be done over the Webinterface (Node -> Updates)

It may pull in a new kernel version, which needs a node reboot. In that case you should clear a node from all important service, reboot, check if everything is up and running and then continue doing the same on the next node, until the whole cluster is running the new kernel.
 
Last edited:
  • Like
Reactions: jlmorgado
t.lamprecht said:
听起来有点像这个线程。这是内核（特定于be2iscsi模块）中的回归，我们发现并在上游修复，在pvetest上有一个更新的kenrel，请参阅：
https://forum.proxmox.com/threads/h...5-4-73-5-4-78-kernel.79907/page-2#post-354354
你可以试试那个吗？
Click to expand...
升级proxmox 6.3.3后，似乎会出现此问题。请帮助解决这个问题。很着急
https://forum.proxmox.com/threads/usb-device-not-working-only-dmesg-can-display-usb-devices.83440/
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom