problem with installation of Qubes OS on Proxmox server

[qubes-user]

Hello everyone,
I lost a lot of time installing the Qubes system in vProxmox.
I want to ask for your help in this post.

My Hardware as the following

Motherboard
ASUS TUF GAMING B650-PLUS WIFI

CPU
AMD Ryzen 9 7900 12-Core, 24-Thread Desktop Processor with AMD Wraith Prism Cooler, up to 5.4GHz

I had enabled every think in the Bios
IOMMU enabled

The output of the command
kvm-ok

is the following

INFO: /dev/kvm exists
KVM acceleration can be used

and the command lscpu
root@pve:~# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 45 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Vendor ID: AuthenticAMD
BIOS Vendor ID: AuthenticAMD
Model name: AMD Ryzen 9 7900 12-Core Processor
BIOS Model name: AMD Ryzen 9 7900 12-Core Processor CPU @ 3.5GHz
BIOS CPU family: 2
CPU family: 25
Model: 97
Thread(s) per core: 1
Core(s) per socket: 2
Socket(s): 4
Stepping: 2
BogoMIPS: 7386.13
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1
gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid tsc_known_freq pni pclmulqdq monitor ssse3 fma cx1
6 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalign
sse 3dnowprefetch osvw topoext ssbd ibrs ibpb ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed
adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx512_bf16 clzero wbnoinvd ara
t npt svm_lock nrip_save vmcb_clean flushbyasid decodeassists avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni
avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor fsrm flush_l1d
Virtualization features:
Virtualization: AMD-V
Hypervisor vendor: VMware
Virtualization type: full
Caches (sum of all):
L1d: 256 KiB (8 instances)
L1i: 256 KiB (8 instances)
L2: 8 MiB (8 instances)
L3: 128 MiB (4 instances)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0-7
Vulnerabilities:
Gather data sampling: Not affected
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Not affected
Retbleed: Not affected
Spec rstack overflow: Mitigation; safe RET, no microcode
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Spectre v2: Mitigation; Enhanced / Automatic IBRS, IBPB conditional, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Srbds: Not affected
Tsx async abort: Not affected
root@pve:~# ^C
root@pve:~# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 45 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Vendor ID: AuthenticAMD
BIOS Vendor ID: AuthenticAMD
Model name: AMD Ryzen 9 7900 12-Core Processor
BIOS Model name: AMD Ryzen 9 7900 12-Core Processor CPU @ 3.5GHz
BIOS CPU family: 2
CPU family: 25
Model: 97
Thread(s) per core: 1
Core(s) per socket: 2
Socket(s): 4
Stepping: 2
BogoMIPS: 7386.13
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1
gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid tsc_known_freq pni pclmulqdq monitor ssse3 fma cx1
6 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalign
sse 3dnowprefetch osvw topoext ssbd ibrs ibpb ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed
adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx512_bf16 clzero wbnoinvd ara
t npt svm_lock nrip_save vmcb_clean flushbyasid decodeassists avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni
avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor fsrm flush_l1d
Virtualization features:
Virtualization: AMD-V
Hypervisor vendor: VMware
Virtualization type: full
Caches (sum of all):
L1d: 256 KiB (8 instances)
L1i: 256 KiB (8 instances)
L2: 8 MiB (8 instances)
L3: 128 MiB (4 instances)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0-7
Vulnerabilities:
Gather data sampling: Not affected
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Not affected
Retbleed: Not affected
Spec rstack overflow: Mitigation; safe RET, no microcode
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Spectre v2: Mitigation; Enhanced / Automatic IBRS, IBPB conditional, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Srbds: Not affected
Tsx async abort: Not affected


QEMU: Checking for hardware virtualization : PASS

QEMU: Checking if device /dev/kvm exists : PASS

QEMU: Checking if device /dev/kvm is accessible : PASS

QEMU: Checking if device /dev/vhost-net exists : PASS

QEMU: Checking if device /dev/net/tun exists : PASS

QEMU: Checking for cgroup 'cpu' controller support : PASS

QEMU: Checking for cgroup 'cpuacct' controller support : PASS

QEMU: Checking for cgroup 'cpuset' controller support : PASS

QEMU: Checking for cgroup 'memory' controller support : PASS

QEMU: Checking for cgroup 'devices' controller support : PASS

QEMU: Checking for cgroup 'blkio' controller support : PASS

QEMU: Checking for device assignment IOMMU support : WARN (No ACPI IVRS table found, IOMMU either disabled in BIOS or not supported by this hardware platform)

QEMU: Checking for secure guest support : WARN (Unknown if this platform has Secure Guest support)

LXC: Checking for Linux >= 2.6.26 : PASS

LXC: Checking for namespace ipc : PASS

LXC: Checking for namespace mnt : PASS

LXC: Checking for namespace pid : PASS

LXC: Checking for namespace uts : PASS

LXC: Checking for namespace net : PASS

LXC: Checking for namespace user : PASS

LXC: Checking for cgroup 'cpu' controller support : PASS

LXC: Checking for cgroup 'cpuacct' controller support : PASS

LXC: Checking for cgroup 'cpuset' controller support : PASS

LXC: Checking for cgroup 'memory' controller support : PASS

LXC: Checking for cgroup 'devices' controller support : PASS

LXC: Checking for cgroup 'freezer' controller support : FAIL (Enable 'freezer' in kernel Kconfig file or mount/enable cgroup controller in your system)

LXC: Checking for cgroup 'blkio' controller support : PASS

LXC: Checking if device /sys/fs/fuse/connections exists : PASS


The output of the command
cat /proc/cmdline
is
BOOT_IMAGE=/boot/vmlinuz-6.5.11-8-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafbff,efifbff

The output of the command

dmesg | grep -i -e DMAR -e IOMMU

[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.5.11-8-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafbff,efifbff [ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA [ 0.075681] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.5.11-8-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafbff,efifbff [ 0.242980] iommu: Default domain type: Passthrough (set via kernel command line)

When I try to install Qubes Os I become the error message Unsupported Hardware detected (in the attachment)

When I accept this warning and after the reboot

the installation failed in in Networking setup, I got the warning popup

After that when I login into Qubes os I can’t start the Network and the usb Qubes

Hardware does not support IOMMU VT-d AMD-Vi (in the attachment)

The reason is, there are no IOMMU and no Passthrough for the PCIs.

I changed the cpu type on my vm on Vmware workstation but the same Problem

The same situation on my Proxmox server on the same computer Can you please help me.

 

[leesteken]

While Proxmox/QEMU/KVM does support nested virtualization (VT-x/AMD-V), you'll need a virtual IOMMU inside the VM (for nested IOMMU/VT-d/AMD-Vi). QEMU is working on that but I don't think it's in Proxmox yet or soon.

 

[qubes-user]

While Proxmox/QEMU/KVM does support nested virtualization (VT-x/AMD-V), you'll need a virtual IOMMU inside the VM (for nested IOMMU/VT-d/AMD-Vi). QEMU is working on that but I don't think it's in Proxmox yet or soon.

Hi leesteken, thank you for your Answer.
Did you mean that QEMU is a Hypervisor as Proxmox and VMware ESXi and XCP-ng and it is better for this case?

​

 

[gfngfn256]

Qubes (which are technically VM's) inside Qubes OS, inside vProxmox, inside VMware ........
Goodness - "nested" to a new level!

 

[leesteken]

Did you mean that QEMU is a Hypervisor as Proxmox and VMware ESXi and XCP-ng and it is better for this case?

No, is just QEMU/KVM is the technology that Proxmox is build upon.
I just meant to say that QubesOS (whihc is build upon Xen) requires an IOMMU (because it uses passthrough) and cannot run inside a VM.
The details behind all this are more complicated but I fear I cannot convey them correctly.

 

[PapaGigas]

For enabling IOMMU in a nested environment, you can add the following argument for the guest VM:

args: -device intel-iommu,caching-mode=on

Explanation:

• -device intel-iommu: This parameter adds an emulated Intel IOMMU (Input-Output Memory Management Unit) device to the guest virtual machine. The IOMMU is responsible for managing and remapping addresses of DMA (Direct Memory Access) transactions, which allows virtual machines to access hardware devices directly and securely.
  
  
• caching-mode=on: This option enables caching mode for the IOMMU. When caching mode is enabled, the IOMMU caches address translations. This can improve performance by reducing the overhead associated with address translation operations.

Note: This worked for my setup with an AMD EPYC Rome processor and a nested Proxmox VE installation. It should work for you as well!

Additionally, to check the IOMMU groups, you can use this command:

for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done

This command will list all devices in their respective IOMMU groups, which can be helpful for troubleshooting and configuration.

 

[leesteken]

For enabling IOMMU in a nested environment, you can add the following argument for the guest VM:

args: -device intel-iommu,caching-mode=on

You can also do this in the Proxmox web GUI (VM, Machine and select Advanced) in recent Proxmox versions: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_pci_viommu

 

[PapaGigas]

You can also do this in the Proxmox web GUI (VM, Machine and select Advanced) in recent Proxmox versions: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_pci_viommu

I didn't know that, great news! Thanks for the tip!