problem with installation of Qubes OS on Proxmox server

[qubes-user]

Hello everyone,
I lost a lot of time installing the Qubes system in vProxmox.
I want to ask for your help in this post.

My Hardware as the following

Motherboard
ASUS TUF GAMING B650-PLUS WIFI

CPU
AMD Ryzen 9 7900 12-Core, 24-Thread Desktop Processor with AMD Wraith Prism Cooler, up to 5.4GHz

I had enabled every think in the Bios
IOMMU enabled

The output of the command
kvm-ok

is the following

INFO: /dev/kvm exists
KVM acceleration can be used

and the command lscpu
root@pve:~# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 45 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Vendor ID: AuthenticAMD
BIOS Vendor ID: AuthenticAMD
Model name: AMD Ryzen 9 7900 12-Core Processor
BIOS Model name: AMD Ryzen 9 7900 12-Core Processor CPU @ 3.5GHz
BIOS CPU family: 2
CPU family: 25
Model: 97
Thread(s) per core: 1
Core(s) per socket: 2
Socket(s): 4
Stepping: 2
BogoMIPS: 7386.13
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1
gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid tsc_known_freq pni pclmulqdq monitor ssse3 fma cx1
6 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalign
sse 3dnowprefetch osvw topoext ssbd ibrs ibpb ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed
adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx512_bf16 clzero wbnoinvd ara
t npt svm_lock nrip_save vmcb_clean flushbyasid decodeassists avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni
avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor fsrm flush_l1d
Virtualization features:
Virtualization: AMD-V
Hypervisor vendor: VMware
Virtualization type: full
Caches (sum of all):
L1d: 256 KiB (8 instances)
L1i: 256 KiB (8 instances)
L2: 8 MiB (8 instances)
L3: 128 MiB (4 instances)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0-7
Vulnerabilities:
Gather data sampling: Not affected
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Not affected
Retbleed: Not affected
Spec rstack overflow: Mitigation; safe RET, no microcode
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Spectre v2: Mitigation; Enhanced / Automatic IBRS, IBPB conditional, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Srbds: Not affected
Tsx async abort: Not affected
root@pve:~# ^C
root@pve:~# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 45 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Vendor ID: AuthenticAMD
BIOS Vendor ID: AuthenticAMD
Model name: AMD Ryzen 9 7900 12-Core Processor
BIOS Model name: AMD Ryzen 9 7900 12-Core Processor CPU @ 3.5GHz
BIOS CPU family: 2
CPU family: 25
Model: 97
Thread(s) per core: 1
Core(s) per socket: 2
Socket(s): 4
Stepping: 2
BogoMIPS: 7386.13
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1
gb rdtscp lm constant_tsc rep_good nopl tsc_reliable nonstop_tsc cpuid extd_apicid tsc_known_freq pni pclmulqdq monitor ssse3 fma cx1
6 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalign
sse 3dnowprefetch osvw topoext ssbd ibrs ibpb ibrs_enhanced vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed
adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx512_bf16 clzero wbnoinvd ara
t npt svm_lock nrip_save vmcb_clean flushbyasid decodeassists avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni
avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor fsrm flush_l1d
Virtualization features:
Virtualization: AMD-V
Hypervisor vendor: VMware
Virtualization type: full
Caches (sum of all):
L1d: 256 KiB (8 instances)
L1i: 256 KiB (8 instances)
L2: 8 MiB (8 instances)
L3: 128 MiB (4 instances)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0-7
Vulnerabilities:
Gather data sampling: Not affected
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Not affected
Retbleed: Not affected
Spec rstack overflow: Mitigation; safe RET, no microcode
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Spectre v2: Mitigation; Enhanced / Automatic IBRS, IBPB conditional, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Srbds: Not affected
Tsx async abort: Not affected


QEMU: Checking for hardware virtualization : PASS

QEMU: Checking if device /dev/kvm exists : PASS

QEMU: Checking if device /dev/kvm is accessible : PASS

QEMU: Checking if device /dev/vhost-net exists : PASS

QEMU: Checking if device /dev/net/tun exists : PASS

QEMU: Checking for cgroup 'cpu' controller support : PASS

QEMU: Checking for cgroup 'cpuacct' controller support : PASS

QEMU: Checking for cgroup 'cpuset' controller support : PASS

QEMU: Checking for cgroup 'memory' controller support : PASS

QEMU: Checking for cgroup 'devices' controller support : PASS

QEMU: Checking for cgroup 'blkio' controller support : PASS

QEMU: Checking for device assignment IOMMU support : WARN (No ACPI IVRS table found, IOMMU either disabled in BIOS or not supported by this hardware platform)

QEMU: Checking for secure guest support : WARN (Unknown if this platform has Secure Guest support)

LXC: Checking for Linux >= 2.6.26 : PASS

LXC: Checking for namespace ipc : PASS

LXC: Checking for namespace mnt : PASS

LXC: Checking for namespace pid : PASS

LXC: Checking for namespace uts : PASS

LXC: Checking for namespace net : PASS

LXC: Checking for namespace user : PASS

LXC: Checking for cgroup 'cpu' controller support : PASS

LXC: Checking for cgroup 'cpuacct' controller support : PASS

LXC: Checking for cgroup 'cpuset' controller support : PASS

LXC: Checking for cgroup 'memory' controller support : PASS

LXC: Checking for cgroup 'devices' controller support : PASS

LXC: Checking for cgroup 'freezer' controller support : FAIL (Enable 'freezer' in kernel Kconfig file or mount/enable cgroup controller in your system)

LXC: Checking for cgroup 'blkio' controller support : PASS

LXC: Checking if device /sys/fs/fuse/connections exists : PASS


The output of the command
cat /proc/cmdline
is
BOOT_IMAGE=/boot/vmlinuz-6.5.11-8-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafbff,efifbff

The output of the command

dmesg | grep -i -e DMAR -e IOMMU

[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-6.5.11-8-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafbff,efifbff [ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA [ 0.075681] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.5.11-8-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafbff,efifbff [ 0.242980] iommu: Default domain type: Passthrough (set via kernel command line)

When I try to install Qubes Os I become the error message Unsupported Hardware detected (in the attachment)

When I accept this warning and after the reboot

the installation failed in in Networking setup, I got the warning popup

After that when I login into Qubes os I can’t start the Network and the usb Qubes

Hardware does not support IOMMU VT-d AMD-Vi (in the attachment)

The reason is, there are no IOMMU and no Passthrough for the PCIs.

I changed the cpu type on my vm on Vmware workstation but the same Problem

The same situation on my Proxmox server on the same computer Can you please help me.

 

[leesteken]

While Proxmox/QEMU/KVM does support nested virtualization (VT-x/AMD-V), you'll need a virtual IOMMU inside the VM (for nested IOMMU/VT-d/AMD-Vi). QEMU is working on that but I don't think it's in Proxmox yet or soon.

 

[qubes-user]

While Proxmox/QEMU/KVM does support nested virtualization (VT-x/AMD-V), you'll need a virtual IOMMU inside the VM (for nested IOMMU/VT-d/AMD-Vi). QEMU is working on that but I don't think it's in Proxmox yet or soon.

Hi leesteken, thank you for your Answer.
Did you mean that QEMU is a Hypervisor as Proxmox and VMware ESXi and XCP-ng and it is better for this case?

​

 

[gfngfn256]

Qubes (which are technically VM's) inside Qubes OS, inside vProxmox, inside VMware ........
Goodness - "nested" to a new level!

 

[leesteken]

Did you mean that QEMU is a Hypervisor as Proxmox and VMware ESXi and XCP-ng and it is better for this case?

No, is just QEMU/KVM is the technology that Proxmox is build upon.
I just meant to say that QubesOS (whihc is build upon Xen) requires an IOMMU (because it uses passthrough) and cannot run inside a VM.
The details behind all this are more complicated but I fear I cannot convey them correctly.