Proxmox VE 8.0 (beta) released!

effgee · Jun 18, 2023

Here is a solution for the lid issue:

Bash:

nano /etc/systemd/logind.conf

[Login]
#NAutoVTs=6
#ReserveVT=6
#KillUserProcesses=no
#KillOnlyUsers=
#KillExcludeUsers=root
#InhibitDelayMaxSec=5
#HandlePowerKey=poweroff
#HandleSuspendKey=suspend
#HandleHibernateKey=hibernate
HandleLidSwitch=ignore
#HandleLidSwitchExternalPower=suspend
HandleLidSwitchDocked=ignore
#PowerKeyIgnoreInhibited=no
#SuspendKeyIgnoreInhibited=no
#HibernateKeyIgnoreInhibited=no
#LidSwitchIgnoreInhibited=yes
#HoldoffTimeoutSec=30s
#IdleAction=ignore
#IdleActionSec=30min
#RuntimeDirectorySize=10%
#RemoveIPC=yes
#InhibitorsMax=8192
#SessionsMax=8192

systemctl restart systemd-logind.service

Hi, thank you for your reply but it appears you are just disabling suspend (via closing the lid)

I am actually suspending manually, not just trying to run with the lid closed.

effgee · Jun 18, 2023

effgee said:
Hello all. So far my experience with PVE 8 is excellent, running it in several places as well as on my laptop for development purposes.
I am encountering an issue though with suspend and bluetooth, on my laptop.

Problems with USB(internal) bluetooth are quite common so it took me awhile to perhaps find where the issue lays.

On boot, bluetooth works great.

If I suspend (to ram) the machine bluetooth adapter disappears and it seems it is because of some missing firmware? I tried to install the firmware-atheros (but this is prohibited due to pve/pve-firmware)

The error is:

Code:

2023-06-18T17:47:30.185838+03:00 fattop-pve kernel: [ 906.266058] bluetooth hci0: Direct firmware load for qca/rampatch_usb_00000302.bin failed with error -2 2023-06-18T17:47:30.185839+03:00 fattop-pve kernel: [ 906.266064] Bluetooth: hci0: failed to request rampatch file: qca/rampatch_usb_00000302.bin (-2)

So I always have to reboot if I suspend my laptop, this is less than optimal.
Perhaps somehow this firmware was excluded?

I have one kernel update to run and will try it and update if something changes.

Ok, so newest kernel did not accomplish anything I needed.
I did end up solving this via these steps.

Code:

apt download firmware-atheros
Extract the firmware-atheros deb pack and copy (and create needed folders, qca did NOT exist.)   the needed files to
/lib/firmware/qca/*
Make sure file permissions are maintained and owned by root:root

Suspend and restore seems to work properly.

Please fix good sirs.

Ramalama · Jun 20, 2023

I've updated today and there is something new!
Can anyone explain what a mapped device is?

spirit · Jun 20, 2023

Ramalama said:
View attachment 51889

I've updated today and there is something new!
Can anyone explain what a mapped device is?

local devices (usb/pci) group, to be able to migrate (offline for now or restart with HA) between nodes, even if devices are on another usb port or pci slot.

(could be extended later for livemigration with vgpu/mdev for example)

ness1602 · Jun 20, 2023

Yeah,really interesting thing, not just for gpus.

fabian · Jun 20, 2023

spirit said:
local devices (usb/pci) group, to be able to migrate (offline for now or restart with HA) between nodes, even if devices are on another usb port or pci slot.

(could be extended later for livemigration with vgpu/mdev for example)

that, and also, this now allows to define host devices using a special privilege, and then delegating actually using it via a different privilege, possibly limited to certain named devices, instead of requiring root for all of that

davemcl · Jun 20, 2023

Upgrade finished successfully but no traffic passes, cant ping anything.
Dell R640 with Broadcom NetXtreme II BCM57800 ethernet in a bond.
Interfaces and bond are up.
Names dont seem to have changed from eno1 & eno2

ip link

lspci -k | grep -A3 -i ethernet

ip a

Code:

auto lo
iface lo inet loopback

#onboard 10Gb
iface eno1 inet manual

#onboard 10Gb
iface eno2 inet manual

#onboard 1Gb
iface eno3 inet manual

#onboard 1Gb
iface eno4 inet manual

#Dual 10GbE LACP
auto bond0
iface bond0 inet manual
      bond-slaves eno1 eno2
      bond-miimon 100
      bond-mode 802.3ad
      bond-xmit-hash-policy layer3+4
#      bond-lacp-rate 0
#      bond-downdelay 200
#      bond-updelay 200
      bond-min-links 1
#      mtu 1500

#Default bridge
auto vmbr0
iface vmbr0 inet static
      bridge-ports bond0
      bridge-stp off
      bridge-fd 0
      #Honor VLAN tags:
      bridge-vlan-aware yes
      bridge-vids 2-128

#Management Interface (VLAN 20)
auto vmbr0.20
iface vmbr0.20 inet static
      address 10.10.20.3/24
      gateway 10.10.20.254
      dns-nameservers 10.10.20.254

davemcl · Jun 20, 2023

Tried a fresh install from the iso - same issue.

foxt · Jun 21, 2023

t.lamprecht said:
Yes, during the beta we only upload to the test repo, to avoid suggesting that this is already a stable release.

Could we get this noted on the wiki pages? Since there's no stable for Bookworm, and there is a pretty big disclaimer on https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_12_Bookworm, it seems appropriate to mention (or change the commands to pvetest)

spirit · Jun 21, 2023

davemcl said:
Upgrade finished successfully but no traffic passes, cant ping anything.
Dell R640 with Broadcom NetXtreme II BCM57800 ethernet in a bond.
Interfaces and bond are up.
Names dont seem to have changed from eno1 & eno2

ip link
View attachment 51901

lspci -k | grep -A3 -i ethernet
View attachment 51902

ip a
View attachment 51912

Code:

auto lo iface lo inet loopback #onboard 10Gb iface eno1 inet manual #onboard 10Gb iface eno2 inet manual #onboard 1Gb iface eno3 inet manual #onboard 1Gb iface eno4 inet manual #Dual 10GbE LACP auto bond0 iface bond0 inet manual bond-slaves eno1 eno2 bond-miimon 100 bond-mode 802.3ad bond-xmit-hash-policy layer3+4 # bond-lacp-rate 0 # bond-downdelay 200 # bond-updelay 200 bond-min-links 1 # mtu 1500 #Default bridge auto vmbr0 iface vmbr0 inet static bridge-ports bond0 bridge-stp off bridge-fd 0 #Honor VLAN tags: bridge-vlan-aware yes bridge-vids 2-128 #Management Interface (VLAN 20) auto vmbr0.20 iface vmbr0.20 inet static address 10.10.20.3/24 gateway 10.10.20.254 dns-nameservers 10.10.20.254

do you have any error if you do a "ifreload -a -d" ?

davemcl · Jun 21, 2023

spirit said:
do you have any error if you do a "ifreload -a -d" ?

I was doing a "systemctl restart networking.service" when I made changes, which would finish without error. Also rebooted countless times.

I ended up reinstalling 7.4 on the node in question, same config posted above. Connectivity not an issue.
I need PV8 to see if it resolves a different issue so Ill need to have another crack at this.

spirit · Jun 21, 2023

davemcl said:
I was doing a "systemctl restart networking.service" when I made changes, which would finish without error. Also rebooted countless times.

I ended up reinstalling 7.4 on the node in question, same config posted above. Connectivity not an issue.
I need PV8 to see if it resolves a different issue so Ill need to have another crack at this.

"ifreload -a -d" can give you the debug log of applying configuration, it could be usefull to see if something wrong happen.

davemcl · Jun 21, 2023

spirit said:
"ifreload -a -d" can give you the debug log of applying configuration, it could be usefull to see if something wrong happen.

Wasnt aware it had a debug flag. Thanks.

Im thinking I could try the networking config from the installer ISO without installing PV8 or maybe an Ubuntu live CD...

mavaf3f · Jun 21, 2023

Hello guys, we're setting up new cluster so we took the chance to play with VE 8.0 and I've run into issue with network, short description - after setting up cluster networking works, after creating VM with iface with 802.1q tag on bridge - the networking on that bridge stop working..

Longer description:
relevant networking configuration:

auto ens3
iface ens3 inet manual
mtu 9028
pre-up /usr/sbin/ethtool -K ens3 gso off gro off tso off
pre-up /usr/sbin/ethtool -G ens3 rx 8192 tx 8192

auto ens3d1
iface ens3d1 inet manual
mtu 9028
pre-up /usr/sbin/ethtool -K ens3d1 gso off gro off tso off
pre-up /usr/sbin/ethtool -G ens3d1 rx 8192 tx 8192

auto eno49
iface eno49 inet manual
mtu 9028
post-up /usr/sbin/ethtool -K eno49 gso off gro off tso off
post-up /usr/sbin/ethtool -G eno49 rx 8192 tx 8192

auto eno49d1
iface eno49d1 inet manual
mtu 9028
post-up /usr/sbin/ethtool -K eno49d1 gso off gro off tso off
post-up /usr/sbin/ethtool -G eno49d1 rx 8192 tx 8192

auto vmbr0
iface vmbr0 inet static
bridge-ports ens3 ens3d1
bridge-stp on
bridge-fd 0
mtu 9028

auto vmbr1
iface vmbr1 inet static
bridge-ports eno49 eno49d1
bridge-stp on
bridge-fd 0
mtu 9028

auto vlan202
iface vlan202 inet static
address 192.168.202.190/24
gateway 192.168.202.1
vlan-raw-device vmbr0
mtu 1500

auto vlan910
iface vlan910 inet static
address 172.31.0.10/24
vlan-raw-device vmbr0
post-up ip ro add 172.16.0.0/12 via 172.31.0.1
post-up ip ro add 192.168.222.0/24 via 172.31.0.1
post-up ip ro add 192.168.248.0/24 via 172.31.0.1
mtu 9000

=> in this setup everything works OK and IP's routing works well..
Next step is creating new VM with networking as fillowing:

I end up with this setup:

root@skbasixhv01:~# ip add sh | grep -A 10 16:
16: tap805i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9028 qdisc pfifo_fast master vmbr0v202 state UNKNOWN group default qlen 1000
link/ether 0a:f4:a6:e2:86:1b brd ff:ff:ff:ff:ff:ff
17: vmbr0v202: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9028 qdisc noqueue state UP group default qlen 1000
link/ether 66:c9:2c:76:9d:56 brd ff:ff:ff:ff:ff:ff
18: ens3.202@ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9028 qdisc noqueue master vmbr0v202 state UP group default qlen 1000
link/ether 50:65:f3:84:dd:b0 brd ff:ff:ff:ff:ff:ff
19: ens3d1.202@ens3d1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9028 qdisc noqueue master vmbr0v202 state UP group default qlen 1000
link/ether 50:65:f3:84:dd:b1 brd ff:ff:ff:ff:ff:ff

root@skbasixhv01:~# brctl show
bridge name bridge id STP enabled interfaces
vmbr0 8000.1a68e9c21344 yes ens3
ens3d1
vmbr0v202 8000.66c92c769d56 yes ens3.202
ens3d1.202
tap805i0
vmbr1 8000.4afcbb10f95d yes eno49
eno49d1
root@skbasixhv01:~#

PVE creates vlan subifaces on physical iface => then create bridge on top of that and => then add VM iface (tap805i0) into it, after this all networking in vlan 202 towards PVE doesn't work, but networking in vlan 202 within VM does work..

Any idea how to be able to configure it that I can communicate in that vlan in VM as well as on HV directly?

Thanks for ideas..

P.S. on PVE 7.4 such network setup works OK..

jsterr · Jun 21, 2023

davemcl said:
Wasnt aware it had a debug flag. Thanks.

Im thinking I could try the networking config from the installer ISO without installing PV8 or maybe an Ubuntu live CD...

Maybe also try to use the newest firmware, we had lots of bugs with broadcom nics regarding different topics, but to use the latest firmware fixed it usually. Didnt use that chipset thought.

RolandK · Jun 21, 2023

there is a new "localnetwork" object in my pve8 after update. you can apply permissions to that.

where can i read what this is/means and how it works?

fabian · Jun 21, 2023

RolandK said:
there is a new "localnetwork" object in my pve8 after update. you can apply permissions to that.

where can i read what this is/means and how it works?

the "localnetwork" is an always-available network zone containing the local vmbrX bridges.

a user needs to have the 'SDN.Use' permission on the bridge (or the whole zone with propagation) in order to use the bridge in a guest (using means adding/modifying/deleting a virtual nic using the bridge, not things like starting the guest with such a nic configured).

RolandK · Jun 21, 2023

can we call this a new "abstraction layer" ?

ok, i read into https://172.20.27.112:8006/pve-docs/chapter-pvesdn.html

fabian · Jun 21, 2023

it's mentioned in the release notes for the beta as well, right at the top

https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_8.0_beta1

Integrate host network bridge and VNet access when configuring virtual guests into Proxmox VE's ACL system.With the new SDN.Use privilege and the new /sdn/zones/<zone>/<bridge-or-vnet>/<vlan-tag> ACL object path, one can give out fine-grained usage permissions for specific networks to users.

if you want to keep the existing approach (user can use any local bridge), just give out PVESDNUser on /sdn/zones/localnetwork with propagation.

spirit · Jun 21, 2023

mavaf3f said:
Hello guys, we're setting up new cluster so we took the chance to play with VE 8.0 and I've run into issue with network, short description - after setting up cluster networking works, after creating VM with iface with 802.1q tag on bridge - the networking on that bridge stop working..

Longer description:
relevant networking configuration:

auto ens3
iface ens3 inet manual
mtu 9028
pre-up /usr/sbin/ethtool -K ens3 gso off gro off tso off
pre-up /usr/sbin/ethtool -G ens3 rx 8192 tx 8192

auto ens3d1
iface ens3d1 inet manual
mtu 9028
pre-up /usr/sbin/ethtool -K ens3d1 gso off gro off tso off
pre-up /usr/sbin/ethtool -G ens3d1 rx 8192 tx 8192

auto eno49
iface eno49 inet manual
mtu 9028
post-up /usr/sbin/ethtool -K eno49 gso off gro off tso off
post-up /usr/sbin/ethtool -G eno49 rx 8192 tx 8192

auto eno49d1
iface eno49d1 inet manual
mtu 9028
post-up /usr/sbin/ethtool -K eno49d1 gso off gro off tso off
post-up /usr/sbin/ethtool -G eno49d1 rx 8192 tx 8192

auto vmbr0
iface vmbr0 inet static
bridge-ports ens3 ens3d1
bridge-stp on
bridge-fd 0
mtu 9028

auto vmbr1
iface vmbr1 inet static
bridge-ports eno49 eno49d1
bridge-stp on
bridge-fd 0
mtu 9028

auto vlan202
iface vlan202 inet static
address 192.168.202.190/24
gateway 192.168.202.1
vlan-raw-device vmbr0
mtu 1500

auto vlan910
iface vlan910 inet static
address 172.31.0.10/24
vlan-raw-device vmbr0
post-up ip ro add 172.16.0.0/12 via 172.31.0.1
post-up ip ro add 192.168.222.0/24 via 172.31.0.1
post-up ip ro add 192.168.248.0/24 via 172.31.0.1
mtu 9000

=> in this setup everything works OK and IP's routing works well..
Next step is creating new VM with networking as fillowing:

View attachment 51944

I end up with this setup:

root@skbasixhv01:~# ip add sh | grep -A 10 16:
16: tap805i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9028 qdisc pfifo_fast master vmbr0v202 state UNKNOWN group default qlen 1000
link/ether 0a:f4:a6:e2:86:1b brd ff:ff:ff:ff:ff:ff
17: vmbr0v202: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9028 qdisc noqueue state UP group default qlen 1000
link/ether 66:c9:2c:76:9d:56 brd ff:ff:ff:ff:ff:ff
18: ens3.202@ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9028 qdisc noqueue master vmbr0v202 state UP group default qlen 1000
link/ether 50:65:f3:84:dd:b0 brd ff:ff:ff:ff:ff:ff
19: ens3d1.202@ens3d1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9028 qdisc noqueue master vmbr0v202 state UP group default qlen 1000
link/ether 50:65:f3:84:dd:b1 brd ff:ff:ff:ff:ff:ff

root@skbasixhv01:~# brctl show
bridge name bridge id STP enabled interfaces
vmbr0 8000.1a68e9c21344 yes ens3
ens3d1
vmbr0v202 8000.66c92c769d56 yes ens3.202
ens3d1.202
tap805i0
vmbr1 8000.4afcbb10f95d yes eno49
eno49d1
root@skbasixhv01:~#

PVE creates vlan subifaces on physical iface => then create bridge on top of that and => then add VM iface (tap805i0) into it, after this all networking in vlan 202 towards PVE doesn't work, but networking in vlan 202 within VM does work..

Any idea how to be able to configure it that I can communicate in that vlan in VM as well as on HV directly?

Thanks for ideas..

P.S. on PVE 7.4 such network setup works OK..

The problem is that if you use vmbr0 + tag 202, proxmox will create a new "vmbr0v202 + ensX.202" interfaces in background, so it'll conflict with your vlan202 device (done at bridge level)

The best/easyway is to enable vlan-aware option on bridge, like this the vm tag is set on bridge port instead physical interfaces.

Proxmox VE 8.0 (beta) released!

Renowned Member

Renowned Member

Well-Known Member

Distinguished Member

Renowned Member

Proxmox Staff Member

Member

Attachments

Member

Member

Distinguished Member

Member

Distinguished Member

Member

New Member

Well-Known Member

Renowned Member

Proxmox Staff Member

Renowned Member

Proxmox Staff Member

Distinguished Member