Proxmox Virtual Environment 9.1 available!

JCNED said:
Several VM's this test round, 120 as the example. I stopped the backup as it got stuck and the VM was frozen.
Click to expand...
The output does not seem to hint at a QEMU-internal deadlock.

Code: 
INFO:  26% (764.0 MiB of 2.8 GiB) in 6s, read: 60.0 MiB/s, write: 58.7 MiB/s
INFO:  29% (856.0 MiB of 2.8 GiB) in 1h 20m 55s, read: 19.4 KiB/s, write: 19.4 KiB/s
At this time it was already extremely slow. How much load was on the network or Ceph?
Do you run the backup at the same time for multiple nodes? If yes, scheduling at different times could help.

Is the network used for backup different than the one for Ceph? If not, it's recommended to separate it.

In general, you can try setting a bandwidth limit for the backup job and also enable backup fleecing, those might help already.

If nothing helps, what exactly are the symptoms of being "frozen" (console/network/...)? Is there anything in the guest's system logs from around the time of the issue?
 
fiona said:
The output does not seem to hint at a QEMU-internal deadlock.

Code: 
INFO:  26% (764.0 MiB of 2.8 GiB) in 6s, read: 60.0 MiB/s, write: 58.7 MiB/s
INFO:  29% (856.0 MiB of 2.8 GiB) in 1h 20m 55s, read: 19.4 KiB/s, write: 19.4 KiB/s
At this time it was already extremely slow. How much load was on the network or Ceph?
Do you run the backup at the same time for multiple nodes? If yes, scheduling at different times could help.

Is the network used for backup different than the one for Ceph? If not, it's recommended to separate it.

In general, you can try setting a bandwidth limit for the backup job and also enable backup fleecing, those might help already.

If nothing helps, what exactly are the symptoms of being "frozen" (console/network/...)? Is there anything in the guest's system logs from around the time of the issue?
Click to expand...

Hi Fiona,

Same behavior on one nod versus all 3. In the past all three backed up at once without issue.
Low network utilization, backup server is on a 20gb lagg as well. Graphs don't show anything about 4 gbps.
seperate networks for cluster, frontend (VM traffic) and ceph.
Error on any frozen vm =
VM 114 qmp command 'set_password' failed - unable to connect to VM 114 qmp socket - timeout after 51 retries
TASK ERROR: Failed to run vncproxy.
CPU (Virtual) is over 100% and/or Guest Ram is peaked at 100%. VM is not responsive at all via network or console (console error above)
Nothing in the guest system logs.

Note, This dev cluster is about 1.5 years old, 100% backup success rate until 9.1.1. Random VM's, everytime. I'll turn fleecing on and see if it changes anything and report back!

Appreciate the help.
 
Last edited:
JCNED said:
Same behavior on one nod versus all 3. In the past all three backed up at once without issue.
Low network utilization, backup server is on a 20gb lagg as well. Graphs don't show anything about 4 gbps.
seperate networks for cluster, frontend (VM traffic) and ceph.
Click to expand...
What about the load on the PBS side? Your log shows the backup was extremely slow at some point even before the freeze. I'd still recommend using fleecing and setting a bandwidth limit for the backup job.
JCNED said:
Error on any frozen vm =
VM 114 qmp command 'set_password' failed - unable to connect to VM 114 qmp socket - timeout after 51 retries
TASK ERROR: Failed to run vncproxy.
CPU (Virtual) is over 100% and/or Guest Ram is peaked at 100%. VM is not responsive at all via network or console (console error above)
Click to expand...
The debug info you posted for 120 showed the VM still responding to other QMP commands. Did you have these errors for VM 120 before obtaining the debug info this time?

Any slow ops messages or similar when you look at the Ceph logs?
 
I wanted to ask because the information on the documentation is not clear and I think it is a 9.1 feature. The host-managed option on network tab what is doing? Is it related to the docker network?
 
Hi,

On Dell PowerEdge R630 PVE 9.1 installer dies at early phase (when populating /dev tree).

After installing PVE 8.4 and upgrade to 9.1 kernel 6.17 resets machine just before leaving kernel boot stage, but booting system with latest 6.8 and 6.14 goes just fine.

On Dell PowerEdge R640 6.17 there is no problem with upgrading to PVE 9.1 with kernel 6.17.
 
  • Like
Reactions: Stoiko Ivanov
fumantsu said:
I wanted to ask because the information on the documentation is not clear and I think it is a 9.1 feature. The host-managed option on network tab what is doing? Is it related to the docker network?
Click to expand...
When enabled, Proxmox VE sets up networking on CT start, opposed to having that done from the CTs network management stack.
It's indeed most useful for OCI images that do not have their own network management stack, i.e. application container ("docker") ones, but can be also useful for system containers.
 
Another question related to docker storage this time. Storage Backed Mount Points: there is option to set size to 0 and this will create directory, however neither the GUI nor the CLI (I tried something) allow me to do this. Plus bind mounts I guess needs to done only from CLI, right?
My use case is that I want to have the main directory of the container available to pass configuration files (since something like docker exec is... not available for now)
 
cweakland said:
I am in the same boat using the Frigate container. I noticed if I change it to a unprivileged LXC it will create the container. Privileged fails for me.

Edit: I just confirmed its the privilege argument:
# pct create 122 /data1/images/template/cache/frigate_stable.tar --hostname Frigate1 --storage data1 --password 'password' --rootfs data1:8 --memory 1024 --swap 512 --net0 name=eth0,bridge=vmbr0,ip=dhcp --features nesting=1 --unprivileged 0
Detected OCI archive
unable to create CT 122 - Invalid argument

It works if I change unprivileged to 1.
Click to expand...

I was having the same issue, but I was able to work around it in theory (container shows unprivileged=false now, stat /proc shows owned by root) by creating the container as unprivileged, then performing the usual unprivileged -> privileged conversion (i.e. doing a backup and changing privilege level on restore). I have not yet tried actually doing anything else that would exercise the privilege level.
 
Hi,
fumantsu said:
Another question related to docker storage this time. Storage Backed Mount Points: there is option to set size to 0 and this will create directory, however neither the GUI nor the CLI (I tried something) allow me to do this.
Click to expand...
What error did you get? Did you use a directory type storage? For example:
Code: 
pct set 107 --mp0 dir:0,mp=/example/path

fumantsu said:
Plus bind mounts I guess needs to done only from CLI, right?
Click to expand...
Yes, this can be done via CLI similarly, specifying a full path on the host:
Code: 
pct set 107 --mp0 /path/on/host/,mp=/example/path

fumantsu said:
My use case is that I want to have the main directory of the container available to pass configuration files (since something like docker exec is... not available for now)
Click to expand...
Feel free to open a feature request (after checking if one already exists): https://bugzilla.proxmox.com/
 
it was using a bit little different the command (pct set 111 -mp0 local-lvm:0=/mosquitto). It wasn't very clear in the documentation so I tried to "improvise"
 
Was this a change in 9.1 for bulk migrate? I swear I saw a thread somewhere on this... Went from 9.0.10 to 9.1 and now get this on bulk migrate.
Code: 
either 'maxworkers' parameter or max_workers in datacenter.cfg must be set! (500)
 
  • Like
Reactions: cfgmgr
Moved my cluster to 9.1 from 8.4. Did the management node first for testing and then the hosts. Had a hiccup on OSD authentication in one of the hosts for some reason but got that cleared up relatively quickly. Only side effect from that was about 1TiB of data needing to go through rebalance but no real downtime for the ceph cluster. Overall a clean upgrade compared to some other platforms I've worked with.
 
  • Like
Reactions: Johannes S and fba
t.lamprecht said:
We're proud to present the next iteration of our Proxmox Virtual Environment platform. This new version 9.1 is the first point release since our major update and is dedicated to refinement.

This release is based on Debian 13.2 "Trixie" but we're using the newer Linux kernel 6.17.2 as new stable default. In addition to the main system enhancements, this update incorporates the latest versions of core technologies, including QEMU 10.1.2, LXC 6.0.5, ZFS 2.3.4, and Ceph Squid 19.2.3, all fully tested and integrated.

Please review the key highlights below, and thank you, as always, for your invaluable support.

Here are some of the highlights in Proxmox VE 9.1:
  • Create LXC containers from OCI images
  • Support for TPM state in qcow2 format
  • New vCPU flag for fine-grained control of nested virtualization
  • Enhanced SDN status reporting
  • and much more
This release incorporates numerous bugfixes and performance improvements across the platform. For a comprehensive list of changes, please see the full release notes.

Release notes
https://pve.proxmox.com/wiki/Roadmap

Press release
https://www.proxmox.com/en/news/press-releases

Video tutorial
https://www.proxmox.com/en/training/video-tutorials/item/what-s-new-in-proxmox-ve-9-1

Download
https://www.proxmox.com/en/downloads
Alternate ISO download:
https://enterprise.proxmox.com/iso

Documentation
https://pve.proxmox.com/pve-docs

Community Forum
https://forum.proxmox.com

Bugtracker
https://bugzilla.proxmox.com

Source code
https://git.proxmox.com

This latest release features enhancements and updates shaped by your ideas and support. A huge thank you to all of our community members and customers reporting bugs, submitting patches and getting involved in testing - THANK YOU!

FAQ
Q: Can I upgrade latest Proxmox VE 8 to 9 with apt?
A: Yes, please follow the upgrade instructions on https://pve.proxmox.com/wiki/Upgrade_from_8_to_9

Q: Can I upgrade an 9.0 installation to the stable 9.1 via apt?
A: A: Yes, upgrading from is possible via apt and GUI. We recommend using the pve-enterprise repository on upgrade for the most stable experience.

Q: How long will Proxmox VE 8.4 receive bug fixes and security support?
A: Proxmox VE 8.4 will receive security updates and critical bug fixes until August 2026. This support window provides an overlap of approximately one year after the release of Proxmox VE 9.0, giving users ample time to plan their upgrade to the new major version.
For more information on the support lifecycle of Proxmox VE releases, please visit:
https://pve.proxmox.com/pve-docs/chapter-pve-faq.html#faq-support-table

Q: Can I install Proxmox VE 9.1 on top of Debian 13 "Trixie"?
A: Yes, see https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_13_Trixie

Q: Can I upgrade my Proxmox VE 8.4 cluster with Ceph Reef to Proxmox VE 9.1 and to Ceph Squid?
A: This is a two-step process. First, upgrade Ceph from Reef to Squid. Then, upgrade Proxmox VE from version 8.4 to version 9.1. Be sure to closely follow the upgrade documentation, see:
https://pve.proxmox.com/wiki/Ceph_Reef_to_Squid
https://pve.proxmox.com/wiki/Upgrade_from_8_to_9

Q: Where can I get more information about feature updates?
A: Check the roadmap, forum, the mailing list, and/or subscribe to our newsletter.
Click to expand...
Congratulations on the Proxmox VE 9.1 release—great to see continued refinement and stability improvements.
The new kernel, updated core components, and features like OCI-based LXC containers look very promising.
Thanks to the Proxmox team and community for the hard work and clear upgrade guidance.
 
You must log in or register to reply here.
Top Bottom