Hello everyone,
My problem lays in a combination of possibilities, to explain I will share here my setup to let you know what I am facing:
Switch
8 port 10Gbe Ethernet
4 Port 10Gbe SFP+
All Fiber connected
PVE Server 8.4.11
Lenovo ThinkStation P720
Dual Xeon Gold 6138 (20 Cores each)
256Gb DDR4
GPU Single Lane Nvidia GK208B
256Gb Sata SSD - PVE
Dual 8Tb HDD Sata - Dedicated to TrueNas Scale VM
Dual NVMe 2Tb Gen3 - VMs Datastore
PCIe Asus Hyper M.2 Gen4 QUAD NVME 2Tb
Dedicated to TrueNas Scale VM
Dual 10GbE SFP+ Broadcom NetXtreme II BCM57810 10 Gigabit Ethernet
both dedicated to PFSense VM as Internet link (passthrough)
Intel Corporation 82599 10 GbE SFP+
enp94s0 = vmbr0
VMs
PfSense 101 (4 Cores - 16Gb ram - 50Gb)
Dual 10GbE SFP+ Broadcom NetXtreme II BCM57810 10 Gigabit Ethernet (passthrough)
vmbr0
Truenas 103 (8 Cores - 32Gb ram - 50Gb)
DUAL 8Tb HDD VirtIO Mapping
QUAD NVMe PCIe passthrough
vmbr0
root@pve:/home/juliano# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.8.12-13-pve root=/dev/mapper/pve-root ro quiet
What is happening is, when I start an rsync from my desktop that is also 10GbE fiber connected to switch, after around 120Gb transferred, the server freezes, literally.
No message of kernel panic, ou anything on dmesg, or logs. I need to force reboot, and start over again.
The backup process is close to 2Tb of various data (photos, videos, ISOs, documents), and worked rsync perfectly to USB4 4Tb NVMe Case.
Once I am planning to use my $HOME via NFS from the server I need to copy all data to Truenas that will store in QUAD 2Tb NVME Raid 0+1, and internally on Truenas will replicate all Dataset to DUAL 8Tb Raid1 HDDs.
Actually I am trying using 2 NVME as Datastores in PVE, and created two 1.7Tb Datasets on Truenas to test using another combination. but the issue persists.
I tried to disable Offloading on network card configuration at /etc/network/interfaces
I tried to disable swap in PVE host
I did a full update/upgrade
Linux pve 6.8.12-13-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-13 (2025-07-22T10:00Z) x86_64 GNU/Linux
I reduced the NVMe Raid for only 2, I tried to move NVMEs to PVE host to share via NFS to avoid passthrough to a vm.
I replaced PCIe NVMe board twice, between 2 slots, and other one with 4 slots.
I also tried:
/etc/sysctl.conf
vm.dirty_ratio = 5
vm.dirty_background_ ratio = 1
vm.swappiness=1
vm.min_free_kbytes=524288
Nothing worked until now
My problem lays in a combination of possibilities, to explain I will share here my setup to let you know what I am facing:
Switch
8 port 10Gbe Ethernet
4 Port 10Gbe SFP+
All Fiber connected
PVE Server 8.4.11
Lenovo ThinkStation P720
Dual Xeon Gold 6138 (20 Cores each)
256Gb DDR4
GPU Single Lane Nvidia GK208B
256Gb Sata SSD - PVE
Dual 8Tb HDD Sata - Dedicated to TrueNas Scale VM
Dual NVMe 2Tb Gen3 - VMs Datastore
PCIe Asus Hyper M.2 Gen4 QUAD NVME 2Tb
Dedicated to TrueNas Scale VM
Dual 10GbE SFP+ Broadcom NetXtreme II BCM57810 10 Gigabit Ethernet
both dedicated to PFSense VM as Internet link (passthrough)
Intel Corporation 82599 10 GbE SFP+
enp94s0 = vmbr0
VMs
PfSense 101 (4 Cores - 16Gb ram - 50Gb)
Dual 10GbE SFP+ Broadcom NetXtreme II BCM57810 10 Gigabit Ethernet (passthrough)
vmbr0
Truenas 103 (8 Cores - 32Gb ram - 50Gb)
DUAL 8Tb HDD VirtIO Mapping
QUAD NVMe PCIe passthrough
vmbr0
root@pve:/home/juliano# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.8.12-13-pve root=/dev/mapper/pve-root ro quiet
What is happening is, when I start an rsync from my desktop that is also 10GbE fiber connected to switch, after around 120Gb transferred, the server freezes, literally.
No message of kernel panic, ou anything on dmesg, or logs. I need to force reboot, and start over again.
The backup process is close to 2Tb of various data (photos, videos, ISOs, documents), and worked rsync perfectly to USB4 4Tb NVMe Case.
Once I am planning to use my $HOME via NFS from the server I need to copy all data to Truenas that will store in QUAD 2Tb NVME Raid 0+1, and internally on Truenas will replicate all Dataset to DUAL 8Tb Raid1 HDDs.
Actually I am trying using 2 NVME as Datastores in PVE, and created two 1.7Tb Datasets on Truenas to test using another combination. but the issue persists.
I tried to disable Offloading on network card configuration at /etc/network/interfaces
I tried to disable swap in PVE host
I did a full update/upgrade
Linux pve 6.8.12-13-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-13 (2025-07-22T10:00Z) x86_64 GNU/Linux
I reduced the NVMe Raid for only 2, I tried to move NVMEs to PVE host to share via NFS to avoid passthrough to a vm.
I replaced PCIe NVMe board twice, between 2 slots, and other one with 4 slots.
I also tried:
/etc/sysctl.conf
vm.dirty_ratio = 5
vm.dirty_background_ ratio = 1
vm.swappiness=1
vm.min_free_kbytes=524288
Nothing worked until now