Proxmox Virtual Environment 9.0 released!
- Thread starter t.lamprecht
- Start date
Upgraded another proxmox-8 node to latest 8.4.2. Logged to it and status of all nodes, both 8 and 9 versions is green.
The solution is clear.
Many thanks and best regards!
No, I removed the partitions and reinitialized the disks. but that may have not been enough?
I was reviewing my RAM uses and the ballooning was not working with 4096MB 50% share so i turned it off. it had pci passthrough but no longer. It shows full ram uses in Windows 11 pro, actual uses is half or even lower around 3GB. PVE is on nvme ext4 default.
That alone does not get rid of file system signatures, those are not part of the partition table, but are in the partition's data. There also is the
wipefs tool that can help to get rid of left-overs.
Hi,
without ballooning, this is expected, as already mentioned multiple times in this thread. See also: https://pve.proxmox.com/wiki/Upgrade_from_8_to_9#VM_Memory_Consumption_Shown_is_Higher
Hi,
the Proxmox VE upgrade guide was extended to also link this specific issue in the Debian guide: https://pve.proxmox.com/wiki/Upgrade_from_8_to_9#General
I've upgraded our production cluster to pve9 and most things are good but there are clearly some problems with virtio networking. Two things stand out (and there may be more).
First, our cluster has compute, storage and hybrid nodes. The storage nodes are all R740XD and the compute nodes are a mix of R740 and R750 and the hybrid node is a R750. The hybrid node is essentially the same as the other R750s in the stack, with a newer bios, and guests with virtio network drivers cannot be successfully migrated to our from this host. When migrating to the host the following error is observed (this is from any other node in the stack)
When migrating from the node to any other node in the stack, the receiving node generates the following error.
guests without the virtio network driver migrate fine to and from.
The second issue took me almost 24 hours to identify because it is so odd. We have a load balancer that talks to three vms that live on a different network segment. The vms all seem to work fine and are on the network without issue. From certain network segments an ssl session can be established but then when the vm starts sending data, it disappears / never makes it to the requester. This does not happen across all network segments. We moved the vms to the same network as the load balancer and it then worked. We initially thought it was a firewall issue and spent a lot of time on that but as a test I changed the interface to e1000 on one of the vms and everything worked on its old network segment. There is clearly some weird problem here.
It seems there is some issue with virtio network in the latest qemu. We are running the following across all hosts:
First, our cluster has compute, storage and hybrid nodes. The storage nodes are all R740XD and the compute nodes are a mix of R740 and R750 and the hybrid node is a R750. The hybrid node is essentially the same as the other R750s in the stack, with a newer bios, and guests with virtio network drivers cannot be successfully migrated to our from this host. When migrating to the host the following error is observed (this is from any other node in the stack)
Code:
Aug 10 18:17:02 hybr-01-prod QEMU[1473513]: kvm: Features 0x130afffaf unsupported. Allowed features: 0x1c0010179bfffe7
Aug 10 18:17:02 hybr-01-prod QEMU[1473513]: kvm: Failed to load virtio-net:virtio
Aug 10 18:17:02 hybr-01-prod QEMU[1473513]: kvm: error while loading state for instance 0x0 of device '0000:00:12.0/virtio-net'
Aug 10 18:17:02 hybr-01-prod QEMU[1473513]: kvm: Putting registers after init: Failed to set XCRs: Invalid argumentWhen migrating from the node to any other node in the stack, the receiving node generates the following error.
Code:
Aug 10 18:51:00 comp-06-prod QEMU[1578269]: kvm: get_pci_config_device: Bad config data: i=0x10 read: a1 device: 1 cmask: ff wmask: c0 w1cmask:0
Aug 10 18:51:00 comp-06-prod QEMU[1578269]: kvm: Failed to load PCIDevice:config
Aug 10 18:51:00 comp-06-prod QEMU[1578269]: kvm: Failed to load virtio-net:virtio
Aug 10 18:51:00 comp-06-prod QEMU[1578269]: kvm: error while loading state for instance 0x0 of device '0000:00:12.0/virtio-net'
Aug 10 18:51:00 comp-06-prod QEMU[1578269]: kvm: load of migration failed: Invalid argumentguests without the virtio network driver migrate fine to and from.
The second issue took me almost 24 hours to identify because it is so odd. We have a load balancer that talks to three vms that live on a different network segment. The vms all seem to work fine and are on the network without issue. From certain network segments an ssl session can be established but then when the vm starts sending data, it disappears / never makes it to the requester. This does not happen across all network segments. We moved the vms to the same network as the load balancer and it then worked. We initially thought it was a firewall issue and spent a lot of time on that but as a test I changed the interface to e1000 on one of the vms and everything worked on its old network segment. There is clearly some weird problem here.
It seems there is some issue with virtio network in the latest qemu. We are running the following across all hosts:
Code:
QEMU emulator version 10.0.2 (pve-qemu-kvm_10.0.2-4)
Copyright (c) 2003-2025 Fabrice Bellard and the QEMU Project developershello proxmox team!
Just wanted to say a big thank you!
Last weekend I updated first pbs in a VM on my NAS without any troubles.
Today I updated my pve from 8 to 9. Just a simple home lab setup, so nothing fancy, but my first update after only a short time using pve. So, was I nervous? Betcha!
So: thank you for your great work, great documentation and great product. With that and a little help from https://forum.heimnetz.de/ I got it done without headaches.
These days it is almost "normal" (sad but true) for major updates to cause major trouble (even in a home lab). So: thumbs up for this work of yours!
Just wanted to say a big thank you!
Last weekend I updated first pbs in a VM on my NAS without any troubles.
Today I updated my pve from 8 to 9. Just a simple home lab setup, so nothing fancy, but my first update after only a short time using pve. So, was I nervous? Betcha!
So: thank you for your great work, great documentation and great product. With that and a little help from https://forum.heimnetz.de/ I got it done without headaches.
These days it is almost "normal" (sad but true) for major updates to cause major trouble (even in a home lab). So: thumbs up for this work of yours!
Last edited:
lol, thank you, so easyHi,
There is also an official way to do thishttps://www.debian.org/releases/trixie/release-notes/upgrading.en.html#obsolete
But please make sure you have all repositories correctly configured and updated before runningapt list '~o'! This will list all locally installed packages that are not available in any configured repository. There are cases where you actually want to keep a local package, but you probably know about the ones you explicitly built/installed/want to keep. Others can be removed.
Thanks for the report! This was already reported in the PBS release thread. The PBS checker script will be updated to align this error (the relevant change unfortunately didn't make it in time for the PBS release).
could you open a new thread and tag me there? please also include the output of "lsblk", thanks!
could you please open a new thread, tag @fiona and include (in addition to the info above):
the VM config of such a failing migration
the full migration task log
the VM start task log on the target node
the system log for both nodes covering the time period of the migration (journalctl --since ... --until ...)
thanks!
Yes, I noticed the machines where I had to do this were performing terribly and had other issues. I ended up migrating the VM to another host then reloading the host with v9 clean and migrating them back. I don't know what happened with these upgrades, I did the same thing as the others. Either way, its resolved.
A fresh installation from the ISO has the
[1] https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#_debian_base_repositories
contrib component for the Debian repositories enabled/included, while the docs: [1] do not mention it.[1] https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#_debian_base_repositories
I had to add
I found this by googling the logs, there was one mention on archlinux forums that it's caused by CONFIG_SATA_PMP enabled in kernel.
ChatGPT provided me this noncq solution and explained this as (what a time, these can be actually really helpful):
libata.force=noncq to /etc/default/grub because I started having lot of SATA link errors, this happened in the log over and over:
Code:
4.209263] ata6: hard resetting link
4.672951] ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
4.676669] ata6.00: configured for UDMA/133 (device error ignored)
4.707234] ata6: EH complete
4.745046] ata6.00: Read log 0x10 page 0x00 failed, Emask 0x1
4.745050] ata6: failed to read log page 10h (errno =- 5)
4.745053] ata6.00: NCQ disabled due to excessive errors
4.745054] ata6.00: exception Emask 0x1 SAct 0x1 SErr 0x0 action 0x6
4.745057] ata6.00: irq_stat 0x40000001
4.745059] ata6.00: failed command: READ FPDMA QUEUED
4.745061] ata6.00: cmd 60/08:00:00:00:00/00:00:00:00:00/40 tag 0 ncq dma 4096 in
res 00/00:00:00:00:00/00:00:00:00:00/00 Emask 0x3 (HSM violation)
4.745071] ata6: hard resetting link
5.208941] ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
5.234558] ata6.00: configured for UDMA/133 (device error ignoredI found this by googling the logs, there was one mention on archlinux forums that it's caused by CONFIG_SATA_PMP enabled in kernel.
ChatGPT provided me this noncq solution and explained this as (what a time, these can be actually really helpful):
Love a new update, but are we going to get a resolution to HTTP proxy & acme issue that was reported back in 2021?
https://bugzilla.proxmox.com/show_bug.cgi?id=3536
https://bugzilla.proxmox.com/show_bug.cgi?id=3536