Proxmox VE nested virtualisation not working after update

[Jellman86]

Hi All.

I have recently updated to the latest version of Proxmox, after which it appears that nested virtualisation no longer works. I noticed this because after the update reboot, docker, which is hosted in a privileged CT, can no longer run any containers.

What ive tried so far:

• Ive tried updating to the test kernel thinking the problem may be related to a kernel update.
• Ive created a fresh CT and attempted to run a docker container with no luck.
• Ive double checked my configs, iommu and the like but nothing has changed.

It seems like nested VT is working because the output of cat /sys/module/kvm_intel/parameters/nested is Y.


Any ideas? What should I be looking at to diagnose this further?

Virtual Environment 8.1.10
6.5.13-5-pve < not working
6.8.4-2-pve < not working

CT Config as an example:

arch: amd64
cores: 12
features: mount=cifs,nesting=1
hostname: dockerhost
memory: 20480
onboot: 1
ostype: fedora
rootfs: storage:subvol-102-disk-0,size=256G
swap: 8192
lxc.cgroup2.devices.allow: c 226:0 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file 0, 0
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net dev/net none bind,create=dir
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

Thanks for any help.

 

[leesteken]

I have recently updated to the latest version of Proxmox, after which it appears that nested virtualisation no longer works. I noticed this because after the update reboot, docker, which is hosted in a privileged CT, can no longer run any containers.

What ive tried so far:

• Ive tried updating to the test kernel thinking the problem may be related to a kernel update.
• Ive created a fresh CT and attempted to run a docker container with no luck.
• Ive double checked my configs, iommu and the like but nothing has changed.

It seems like nested VT is working because the output of cat /sys/module/kvm_intel/parameters/nested is Y.

Those things are for nested VMs (like running Proxmox inside a VM) but you are using a container.

Any ideas? What should I be looking at to diagnose this further?

It's probably because an update of the LXC/container technology and you'll need to find new workaround for your ill-advised setup. Running a privileged container is not recommended by Proxmox. And running Docker in a container is also warned against: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_pct

 

[Jellman86]

Those things are for nested VMs (like running Proxmox inside a VM) but you are using a container.

It's probably because an update of the LXC/container technology and you'll need to find new workaround for your ill-advised setup. Running a privileged container is not recommended by Proxmox. And running Docker in a container is also warned against: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_pct

Hi leeteken, thanks for coming back so quickly.

My understanding was that the CTs also support nesting (as there is a nesting option in the configuration).

I think my configuration is quite common, I understand that running the CT privileged is unadvised but my environment is trusted. Im happy with the extra exposure (im running in a home lab).

Are you aware of any changes to the CT system that would break this?

I get that running a vm then running docker inside that is recommended, i did attempt this in the past but could not for the life of me get the intel igpu passed through to the vm for use sadly.

Thanks
Scott

 

[leesteken]

My understanding was that the CTs also support nesting (as there is a nesting option in the configuration).

I just wanted to let you know that KVM nested virtualization is not related to your problem and you don't need to spend more time on it.

I think my configuration is quite common, I understand that running the CT privileged is unadvised but my environment is trusted. Im happy with the extra exposure (im running in a home lab).

Are you aware of any changes to the CT system that would break this?

I get that running a vm then running docker inside that is recommended, i did attempt this in the past but could not for the life of me get the intel igpu passed through to the vm for use sadly.

I don't have experience with those things, sorry.

 

[keeka]

As @leesteken says, you may be confusing nested kvm with the nesting feature of CTs. The latter I believe allows the CT access to the host's procfs and sysfs. Something I think you need in order to run docker in an unprivileged CT. I have a couple of CTs successfully running docker and PVE is up to date. I have not encountered any problems. I don't however have any hardware access configured in my docker host CTs. Though I do in other CTs. Check logs on the host and CT for some clues.

 

[Jellman86]

As @leesteken says, you may be confusing nested kvm with the nesting feature of CTs. The latter I believe allows the CT access to the host's procfs and sysfs. Something I think you need in order to run docker in an unprivileged CT. I have a couple of CTs successfully running docker and PVE is up to date. I have not encountered any problems. I don't however have any hardware access configured in my docker host CTs. Though I do in other CTs. Check logs on the host and CT for some clues.

Oh i see what you mean now, sorry i appear to have been abit tin eared. Yes i was confusing kvm nested vt with lxc nested vt as you and leesteken stated.

Good to hear that it’s working for you in the latest update.

From my reading i should be looking at /var/log/syslog is that correct?

Thanks

 

[keeka]

If the CT's OS is recent debian, use journalctl command. Same with the PVE host.
What I would do is tail the journal journalctl -f on the PVE host, then start the CT.
Perhaps do the same in the CT guest, tail the journal prior to starting docker and any containers.
journalctl will, if terminal supports it, colour the output according to severity/priority. That sometimes helps spot problems quicker!