ZFS, NVME, USB & HDD passthrough, Email notifications

_Dejan_

New Member
Aug 5, 2022
26
1
3
Slovenia
Hi everyone,

I’m new here and I’m already read few threads about ZFS and NVME disks and know that I need to use Enterprise SSD’s with ZFS otherwise they will wear out very soon…

Currently I use QNAP TS-873 with 32GB RAM, 2x1TB M.2 SATA SSD(RAID1), 2x 8TB HDD(RAID1), 2x 3TB HDD(RAID1) and Mellanox MCX354A-FCBT network card. This QNAP is used for file shares and to run some virtual machines. There are 5 virtual machines and they all together have assigned 16GB RAM and around 200GB HDD.

I also use Supermicro server with Xeon E3-1220 v3 CPU, 8GB RAM, Intel X520-SR2 network card as Sophos XG Firewall which license allow me to use 4cores and 6GB RAM.

I would like replace both devices and virtualize all with Proxmox on used server which I’m start to build:
MB: Supermicro H11DSi
CPU: 1x AMD EPYC 7601
RAM: 8x 16GB DDR4 ECC 2666MHz
NVME: I have 2 cards 4x nvme card PCI-e 3.0 x16

This parts I already have. Because I have available only 4 SATA ports on MB and don’t want buy another card to connect more SATA drives I would like use this 4 ports to connect old 4 HDD’s. I prefer to use NVME disks for OS and VM storage. I would like use ZFS.

So questions:

1.) I need to buy 2 SSD’s(Mirror and if possible NVME) for OS but don’t know which ones will not wear out quick and do not cost a lot… If I’m read correctly 240-250GB will be enough for OS, logs and some ISO images. I will use one PCI-e to 4x nvme card in PCI-e 3.0 x16 slot for this. I think 2x Samsung PM983 960GB will be overkill for that… Which NVME models I can search and will be ok?

2.) For VM storage I will use 4x Samsung PM983 960GB NVME in RAID10(stripped mirrors). I will use another 4x nvme card in PCI-e 3.0 x16 slot. Do I need to be careful for some setting when I will create ZFS RAID10?

3.) Can Proxmox host send email notifications of some event like HDD issue(S.M.A.R.T., Failure, VM reset, Host reboot…)?

4.) One VM(HomeAssistant) need access to 2 USB Devices. One of them is UPS and another is ZWave module. As I’m read this is not problem right?

5.) I also need some Free Virtual NAS. I don’t need anything special only simple shares(With different user permissions) for windows PC and media player(KODI) on TV and sync some folders to Cloud(Google Drive & Dropbox). If offer me backup of phone(Android) pictures/videos to NAS it will be ok. I’m right now decided which one I will use maybe openmediavault, because I’m read that TrueNAS will not work if I don’t passthrough controller which I can’t do because I will need 3TB Array for VM backups… If I passthrough only 2x 8TB HDD performance will be probably terrible?

I have also a lot questions about networking and VLAN’s. I use 5 vlans and some VM need all 5 to be tagged(Firewall), one VM(UNIFI Controller) need one untagged and 3 tagged(One VLAN is forbidden), other have only one untagged VLAN… But this is question for another section in forum…

Thanks for help and answers.

Best regards, Dejan
 
1.) I need to buy 2 SSD’s(Mirror and if possible NVME) for OS but don’t know which ones will not wear out quick and do not cost a lot… If I’m read correctly 240-250GB will be enough for OS, logs and some ISO images. I will use one PCI-e to 4x nvme card in PCI-e 3.0 x16 slot for this. I think 2x Samsung PM983 960GB will be overkill for that… Which NVME models I can search and will be ok?
Just the OS without stuff like ISOs will only need like 16-32GB. And performance isn't important. Even crappy old 2.5" HDDs should be fine or a 32GB SATA DOM. PVE will write alot but not that much. Actual data is near to nothing but the problem is the write amplification of SSDs that causes all the writes. Any enterprise/datacenter grade SSD should be fine as a PVE system disk and last for alot of years. I would simply get the cheapest enterprise SSDs you can get with 32GB or more. I for example just bought two second hand 100GB enterprise SATA SSDs for 10€ each.
2.) For VM storage I will use 4x Samsung PM983 960GB NVME in RAID10(stripped mirrors). I will use another 4x nvme card in PCI-e 3.0 x16 slot. Do I need to be careful for some setting when I will create ZFS RAID10?
When using more than 4 disks in a striped mirror (raid10) you might want to increase the volblocksize. And I would enable relatime for the pool to save some wear.
3.) Can Proxmox host send email notifications of some event like HDD issue(S.M.A.R.T., Failure, VM reset, Host reboot…)?
You need to setup postfix yourself. Then you can use zfs-zed for alert mails in case your ZFS pool will degrade. You can also send emails on finished/failed backup jobs. I don't think there is a way for warnings after a VM reset or host reboot. For SMART there are ways to configure the smartd to send mails.
But I would recommend to setup a dedicated monitoring tool like Zabbix for all of that. Makes it way easier to keep an eye on your hardware and guests and is way more powerful and versatile.
4.) One VM(HomeAssistant) need access to 2 USB Devices. One of them is UPS and another is ZWave module. As I’m read this is not problem right?
I guess you could use USB Passthrough. Maybe someone running HomeAssistent can tell you how good that works. USB passthrough isn't that fast and can cause some troubles. Some people buy USB-controller-PCIe-cards and use PCI passthrough to bring the whole controller with all ports into the VM.
That way the VM can directly access the physical USB devices without emulation or an additional virtualization layer causing troubles.
5.) I also need some Free Virtual NAS. I don’t need anything special only simple shares(With different user permissions) for windows PC and media player(KODI) on TV and sync some folders to Cloud(Google Drive & Dropbox). If offer me backup of phone(Android) pictures/videos to NAS it will be ok. I’m right now decided which one I will use maybe openmediavault, because I’m read that TrueNAS will not work if I don’t passthrough controller which I can’t do because I will need 3TB Array for VM backups… If I passthrough only 2x 8TB HDD performance will be probably terrible?
You don't have to passthrough the whole controller. You could passthrough individual disks but that way the VM will only see virtual disks and not the real physical disks. If you want your VM to access the real disks you need to passthrough the whole controller.
"terrible" depends on how you define performance. Throughput while for example straming media should be enough to saturate a Gbit NIC. But IOPS performance should be bad, as HDDs are terrible at that and only way to increase the IOPS performance is to stripe more mirrors of HDDs.
I have also a lot questions about networking and VLAN’s. I use 5 vlans and some VM need all 5 to be tagged(Firewall), one VM(UNIFI Controller) need one untagged and 3 tagged(One VLAN is forbidden), other have only one untagged VLAN… But this is question for another section in forum…
PVE got "vlan aware" bridges that should be able to do all that.
 
Hi @Dunuin and thanks for answers.

1.) I can't use SATA DOM's because it is on second CPU lanes(Which I don't have installed)... Also 4 SATA ports which are on CPU1 will be used with SATA HDD. Using another HBA + 2x SATA SSD for example H310 + 2x SM883 240GB will cost me on ebay around 170-180€ which is pretty close as 2x PM983 960GB(NVME M.2) but that will be overkill for OS so this is reason for looking small used nvme enterprise models which will be cheap… But I don't know what to look because until now I’m work only with consumer nvme’s like Samsung 970/980(Evo, Evo Plus, PRO) series... On one QNAP with ZFS we use 970 Evo Plus and after 9 months of usage(OS + 1x VM with small database) usage health fall down to 95%... I don’t want to risk SSD wear out on private server…

2.) Thanks for hint I still need read a lot about ZFS so I will also put eye on this parameters/settings. On that RAID10 ZFS pool I can do changes also later(I migrate VM to HDD, make changes and move VM’s back)… On OS zfs pool I can’t do that so I need to make it right on beginning… Probably something like this will work OK:

zpool create fastpool \ -o ashift=12 \ -o autoexpand=on \ -o autotrim=on \ -O compression=lz4 \ -O dedup=off \ -O xattr=sa \ -O acltype=posixacl \ -O atime=on \ -O relatime=on \ -O sync=standard mirror nvme01 nvme02

3.) Thanks. For host reboot probably I can make some script which send email on reboot… I will check Zabbix, until now I’m only work with Check_MK …

4.) Ok. I will try first to passthrough only USB device and see if there are some issues and If I will have them then I will put some pcie USB card and passthrough all controller… If info that on QNAP USB devices work normally without issues and I attach them(Only USB device and not all USB controller) to VM using “virsh attach-device …” command. Does that info help to predict that it will probably work ok without pcie USB controller?

5.) Ok I will try passthrough only 2 HDD’s to TrueNAS VM and check how it work... Do you maybe know how % performance drop I can expect to have with passthrough HDD instead passthrough controller? Is this 5%, 10%, 20%, 50% ? Will using ZFS in TrueNAS VM be ok when I passthrough HDD instead of controller?

I have new questions

6.) I’m read install instructions but didn’t find if Proxmox Installation support UEFI on host? Because it is based on Debian 11 this is supported correct? Or is better to use BIOS?

7.) I will need migrate VM’s from QNAP to Proxmox. As I know QNAP use QEMU and I can export them as .qvm(Or .ovf if I shutdown VM). All VM’s use UEFI BIOS and VirtIO HDD and VirtIO Network card(With some MAC address). Disk images on production folder is saved as .img files.
Do I need to be careful about some setting after import that VM will boot normally? Or I just use "Qemu/KVM" method under Migration WiKi and before VM boot change in Proxmox BIOS to “OVMF (UEFI)”, HDD to “VirtIO SCSI” and network model to “VirtIO (paravirtualized)” ? Or I will have problems with OS because I use UEFI?

Thanks.
 
usb to m2 sata can be used for pve os. ext4 / lvm + swap to 0 then make swap manually on a partition of one of a nvme drive.

of course uefi is supported.
grub2 is used as bootloader except if pve os is on zfs in uefi mode (in that case systemd-boot will be used)
 
Last edited:
Hi @_gabriel Thanks for reply. I don't like your idea about using USB. MB have only USB 2.0 headers so I will need to use rear USB or additional pcie card. In that case it will probably again cost more than NVME...

Does exist low capacity(Around 240GB) low cost enterprise nvme's? I know for intel optane but they cost more than 1TB(960GB) Samsungs ...
 
no problem about pve os on an usb 2.0 port, pve os doesn't need performance itself.
consumer ssd m2 sata on usb is ok if install is done with ext4/lvm format and without swap (swap set to 0 during install).
i have two systems with 64gb usb key (with mlc chips) since 1,5 years now , no downtime or crash.
datastorage for VM can be ZFS on your NVMe drives without wear pve os storage.
 
Last edited:
Today Im try passthrough onboard sata controller to TrueNAS VM and imediately when I do this Proxmox host reboot and I finish in reboot loop until I disable iommu in emergency boot and then remove pci-e passthrough from vm...
Before that Im normaly passthrough USB controller...

Im passthrough this device:
25:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

Im check iommu groups and if I understand this correctly SATA controller is in 40 iommu group and no other device is in this group:
:~# find /sys/kernel/iommu_groups/ -type l /sys/kernel/iommu_groups/55/devices/0000:60:01.0 /sys/kernel/iommu_groups/17/devices/0000:03:00.0 /sys/kernel/iommu_groups/45/devices/0000:40:04.0 /sys/kernel/iommu_groups/35/devices/0000:24:00.0 /sys/kernel/iommu_groups/7/devices/0000:00:07.1 /sys/kernel/iommu_groups/63/devices/0000:60:08.1 /sys/kernel/iommu_groups/25/devices/0000:20:01.3 /sys/kernel/iommu_groups/53/devices/0000:43:00.0 /sys/kernel/iommu_groups/15/devices/0000:01:00.0 /sys/kernel/iommu_groups/43/devices/0000:40:03.0 /sys/kernel/iommu_groups/33/devices/0000:22:00.0 /sys/kernel/iommu_groups/33/devices/0000:21:00.0 /sys/kernel/iommu_groups/5/devices/0000:00:04.0 /sys/kernel/iommu_groups/61/devices/0000:60:07.1 /sys/kernel/iommu_groups/23/devices/0000:20:01.0 /sys/kernel/iommu_groups/51/devices/0000:42:00.0 /sys/kernel/iommu_groups/13/devices/0000:00:1a.1 /sys/kernel/iommu_groups/13/devices/0000:00:1a.6 /sys/kernel/iommu_groups/13/devices/0000:00:1a.4 /sys/kernel/iommu_groups/13/devices/0000:00:1a.2 /sys/kernel/iommu_groups/13/devices/0000:00:1a.0 /sys/kernel/iommu_groups/13/devices/0000:00:1a.7 /sys/kernel/iommu_groups/13/devices/0000:00:1a.5 /sys/kernel/iommu_groups/13/devices/0000:00:1a.3 /sys/kernel/iommu_groups/41/devices/0000:40:01.0 /sys/kernel/iommu_groups/31/devices/0000:20:08.0 /sys/kernel/iommu_groups/3/devices/0000:00:02.0 /sys/kernel/iommu_groups/21/devices/0000:05:00.0 /sys/kernel/iommu_groups/11/devices/0000:00:18.3 /sys/kernel/iommu_groups/11/devices/0000:00:18.1 /sys/kernel/iommu_groups/11/devices/0000:00:18.6 /sys/kernel/iommu_groups/11/devices/0000:00:18.4 /sys/kernel/iommu_groups/11/devices/0000:00:18.2 /sys/kernel/iommu_groups/11/devices/0000:00:18.0 /sys/kernel/iommu_groups/11/devices/0000:00:18.7 /sys/kernel/iommu_groups/11/devices/0000:00:18.5 /sys/kernel/iommu_groups/68/devices/0000:63:00.1 /sys/kernel/iommu_groups/1/devices/0000:00:01.1 /sys/kernel/iommu_groups/58/devices/0000:60:03.4 /sys/kernel/iommu_groups/48/devices/0000:40:08.0 /sys/kernel/iommu_groups/38/devices/0000:25:00.0 /sys/kernel/iommu_groups/66/devices/0000:62:00.2 /sys/kernel/iommu_groups/28/devices/0000:20:04.0 /sys/kernel/iommu_groups/56/devices/0000:60:02.0 /sys/kernel/iommu_groups/18/devices/0000:04:00.0 /sys/kernel/iommu_groups/46/devices/0000:40:07.0 /sys/kernel/iommu_groups/36/devices/0000:24:00.2 /sys/kernel/iommu_groups/8/devices/0000:00:08.0 /sys/kernel/iommu_groups/64/devices/0000:61:00.0 /sys/kernel/iommu_groups/26/devices/0000:20:02.0 /sys/kernel/iommu_groups/54/devices/0000:43:00.1 /sys/kernel/iommu_groups/16/devices/0000:01:00.1 /sys/kernel/iommu_groups/44/devices/0000:40:03.4 /sys/kernel/iommu_groups/34/devices/0000:23:00.0 /sys/kernel/iommu_groups/6/devices/0000:00:07.0 /sys/kernel/iommu_groups/62/devices/0000:60:08.0 /sys/kernel/iommu_groups/24/devices/0000:20:01.1 /sys/kernel/iommu_groups/52/devices/0000:42:00.2 /sys/kernel/iommu_groups/14/devices/0000:00:1b.6 /sys/kernel/iommu_groups/14/devices/0000:00:1b.4 /sys/kernel/iommu_groups/14/devices/0000:00:1b.2 /sys/kernel/iommu_groups/14/devices/0000:00:1b.0 /sys/kernel/iommu_groups/14/devices/0000:00:1b.7 /sys/kernel/iommu_groups/14/devices/0000:00:1b.5 /sys/kernel/iommu_groups/14/devices/0000:00:1b.3 /sys/kernel/iommu_groups/14/devices/0000:00:1b.1 /sys/kernel/iommu_groups/42/devices/0000:40:02.0 /sys/kernel/iommu_groups/32/devices/0000:20:08.1 /sys/kernel/iommu_groups/4/devices/0000:00:03.0 /sys/kernel/iommu_groups/60/devices/0000:60:07.0 /sys/kernel/iommu_groups/22/devices/0000:05:00.1 /sys/kernel/iommu_groups/50/devices/0000:41:00.0 /sys/kernel/iommu_groups/12/devices/0000:00:19.6 /sys/kernel/iommu_groups/12/devices/0000:00:19.4 /sys/kernel/iommu_groups/12/devices/0000:00:19.2 /sys/kernel/iommu_groups/12/devices/0000:00:19.0 /sys/kernel/iommu_groups/12/devices/0000:00:19.7 /sys/kernel/iommu_groups/12/devices/0000:00:19.5 /sys/kernel/iommu_groups/12/devices/0000:00:19.3 /sys/kernel/iommu_groups/12/devices/0000:00:19.1 /sys/kernel/iommu_groups/40/devices/0000:25:00.2 /sys/kernel/iommu_groups/30/devices/0000:20:07.1 /sys/kernel/iommu_groups/2/devices/0000:00:01.2 /sys/kernel/iommu_groups/59/devices/0000:60:04.0 /sys/kernel/iommu_groups/20/devices/0000:04:00.3 /sys/kernel/iommu_groups/49/devices/0000:40:08.1 /sys/kernel/iommu_groups/10/devices/0000:00:14.3 /sys/kernel/iommu_groups/10/devices/0000:00:14.0 /sys/kernel/iommu_groups/39/devices/0000:25:00.1 /sys/kernel/iommu_groups/67/devices/0000:63:00.0 /sys/kernel/iommu_groups/29/devices/0000:20:07.0 /sys/kernel/iommu_groups/0/devices/0000:00:01.0 /sys/kernel/iommu_groups/57/devices/0000:60:03.0 /sys/kernel/iommu_groups/19/devices/0000:04:00.2 /sys/kernel/iommu_groups/47/devices/0000:40:07.1 /sys/kernel/iommu_groups/37/devices/0000:24:00.3 /sys/kernel/iommu_groups/9/devices/0000:00:08.1 /sys/kernel/iommu_groups/65/devices/0000:62:00.0 /sys/kernel/iommu_groups/27/devices/0000:20:03.0

On onboard controller are only two test SATA drives which are not used for anything. OS is on nvme disks in pcie cards so this can't be problem...

What Im miss?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!