HotSwap not working with a controller PCIE passthrough

Aug 16, 2023
2
0
1
Italy
cis-spedizioni.com
Hi i’m trying to setup in Proxmox with 2 U.2 SSD drives in a Windows Server 2022 Stantard VM to been used as disaster recovery offsite backup to be taken away once a week (one installed in a detachable tray to be swapped with a second drive) and i’m having problem trying to make hotswap work on this drives.

I’ve tryed to passthrough both disks as SCSI and VirtIO but i was having trouble formatting the drive in SCSI and just writing on it in both SCSI and VirtIo mode, also passing through 2 drive but having just one of them at the time plugged in was giving me trouble booting the vm because one drive was offline for him naturally.
I had the best result passing through the NVME channel where the hdd tray is installed but the problem is that when i unplug the drive the controller is dismounted and when i plug back in the same drive or the otherone the VM don’t see the new drive plugged in (been only one drive on the nvme channel make it so that when unplug the drive there is no more disk on it and proxmox remove dismount the controller from the vm).

I’ve notice that on the VM option u can set hotswap for Disk, network, usb, memory and cpu but not PCIE device…

Is it possible to make it works someway ? I will use VirtIo passthrough but the VM won’t boot if one of the 2 drive is not online and i need to pass both of them to be seen and mounted when i swap the drive.

My configuration is the follow:

-CPU Epyc 9224
-Motherboard supermicro H13SSL-NT updated with the latest bios 1.4
-128GB ram
-n2 drive m.2 samsung enterprise 4TB in ZFS where proxmox is installed and the rest of the space is for the VMs OS.
-1 HBA AOC-S3808L-L8IT-O where are connected 4 sas drive for truenas
-n4 10TB enterprise sas drive passthrough with virtio in a truenas VM used in raid 6 for company file shares
-n2 U.2 4TB INNODISK U.2 SSD 4TG2-P installed in 1 removable tray for backup purpose (will be swapped every week for disaster recovery) installed on the NVMe0/1 PCIe1A JMCIO1 motherboard channel, i passed through the nvme port to the windows vm as shown in the images.

In proxmox i’ve the following VMs:

-a 150GB VM with ubuntu server (150GB for the os is from the 4TB m.2 pool where proxmox is installed)

-a 300GB Truenas Scale VM from the zfs pool drive as the ubuntu vm and with the 4 sas drives (installed on the HBA pcie controller) passed trhough to be used as a ZFS raid 6 for the company shares.

-a 10GB ubuntu VM hosting an FTP service, 10GB from the m.2 ZFS pool as the other VMs

-a 300GB Windows Server 2022 Standard ed. OS drive is the usual ZFS pool as the other VMs from the m.2 drive and i passthrough the PCIE port where is connected the U.2 drive that i need to be hotswappable.
In the Proxmox VM option i’ve enabled the hotswap for cpu, drive and memory has it is by default and when i manually unplug the drive it disappear from the VM but if plugged back will not be detected until i shutdown and power back on the VM.

The idea is that in this Windows server VM there will be a script that using VeraCrypt mount the drive mapped with a letter and during the night will backup data on it and will be unmounted after the copy end, after that a person the day after will swap the drive with a second one and the thing will repeat.

I’ve already prepared all the script and it is all working exept that the drive once swapped will not be detected neither the new drive or the same one.

Any help or tip will be super appreciated this problem is make me going crazy.

I will post some images to help understand my problem.
 

Attachments

  • bios1.png
    bios1.png
    603.8 KB · Views: 6
  • bios2.png
    bios2.png
    257.5 KB · Views: 7
  • IMG_1304.jpg
    IMG_1304.jpg
    332.7 KB · Views: 7
  • IMG_1305.jpg
    IMG_1305.jpg
    306.5 KB · Views: 7
  • IPMI.png
    IPMI.png
    104.1 KB · Views: 5
  • IPMI2.png
    IPMI2.png
    81.9 KB · Views: 5
  • proxmox.png
    proxmox.png
    134.5 KB · Views: 5
  • proxmox1.png
    proxmox1.png
    144.1 KB · Views: 5
  • vm.png
    vm.png
    104.6 KB · Views: 5
  • win-vm.png
    win-vm.png
    247.6 KB · Views: 6

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!