For the purposes of this post, my environment consists of the following:
3x Dell R630 PVE 8.3.4 Hosts with 768GB RAM each
1x Dell R730XD with 256GB RAM running TrueNAS core (8x WD Black 2TB NVME in striped mirrors)
1x Dell R730XD With 64GB RAM running PBS 3.3.3 (4x WD Black 2TB NVME -- no RAID)
Each server has a dual port 10-GB NIC plugged into a Mikrotik 10GB switch using LACP
I have always had slow PBS restore speeds with this setup. I know that that generation's Xeons are not great in single core, but I always thought it should be better.
I originally had hard drives -- this was slow (as expected)
I upgraded to SSD -- there was some small improvement, but it was disappointing
I then upgraded to the WD NVME drives, and performance was the same as with the SSD's
I tried EVERYTHING I could think of, and I was struggling to get over 100MB/s backing up.
Finally, I figured the PBS is single threaded and since the old Xeons are slow, I went out and bought a new "gaming" PC without a GPU (Ryzen 9600X w/32GB RAM). I loaded it up with the same 10GB card and added an Asus PCIE --> NVME card to run 4x WD Black NVME's off the PCEI x16 slot.
After setting everything up and running some test, I was getting around 310MB/sec. This is a noticeable improvement, but still far lower than I expected -- but that's ok....I am willing to accept this speed, and this is NOT my problem.
My problem is that this 310MB/sec only occurs when I restore to Host #1. If I restore to Host #2 or #3, I get around 180MB/sec. Here is some information:
-All 3 hosts are identical Dell R630 servers
-All BIOS/firmware in the servers is up to date and is the same between all servers
-The Proxmox configuration was validated to be the same
-I triple-checked the network switch config -- all ports are configured the same
-All servers are connected to the switch using 1m DAC cables @ 10Gbps full duplex
-No errors on the switch
-The servers have identical hardware and software configuration (except IP addresses of course)
-When monitoring the switch ports during a restore, they barely go above 1.5Gbps...sometimes they go higher but only for a split-second
How is it possible that I can restore to Host #1 @ 300MB/s but the other two drop to 180MB/s?
It's driving me mad.
What am I missing?
3x Dell R630 PVE 8.3.4 Hosts with 768GB RAM each
1x Dell R730XD with 256GB RAM running TrueNAS core (8x WD Black 2TB NVME in striped mirrors)
1x Dell R730XD With 64GB RAM running PBS 3.3.3 (4x WD Black 2TB NVME -- no RAID)
Each server has a dual port 10-GB NIC plugged into a Mikrotik 10GB switch using LACP
I have always had slow PBS restore speeds with this setup. I know that that generation's Xeons are not great in single core, but I always thought it should be better.
I originally had hard drives -- this was slow (as expected)
I upgraded to SSD -- there was some small improvement, but it was disappointing
I then upgraded to the WD NVME drives, and performance was the same as with the SSD's
I tried EVERYTHING I could think of, and I was struggling to get over 100MB/s backing up.
Finally, I figured the PBS is single threaded and since the old Xeons are slow, I went out and bought a new "gaming" PC without a GPU (Ryzen 9600X w/32GB RAM). I loaded it up with the same 10GB card and added an Asus PCIE --> NVME card to run 4x WD Black NVME's off the PCEI x16 slot.
After setting everything up and running some test, I was getting around 310MB/sec. This is a noticeable improvement, but still far lower than I expected -- but that's ok....I am willing to accept this speed, and this is NOT my problem.
My problem is that this 310MB/sec only occurs when I restore to Host #1. If I restore to Host #2 or #3, I get around 180MB/sec. Here is some information:
-All 3 hosts are identical Dell R630 servers
-All BIOS/firmware in the servers is up to date and is the same between all servers
-The Proxmox configuration was validated to be the same
-I triple-checked the network switch config -- all ports are configured the same
-All servers are connected to the switch using 1m DAC cables @ 10Gbps full duplex
-No errors on the switch
-The servers have identical hardware and software configuration (except IP addresses of course)
-When monitoring the switch ports during a restore, they barely go above 1.5Gbps...sometimes they go higher but only for a split-second
How is it possible that I can restore to Host #1 @ 300MB/s but the other two drop to 180MB/s?
It's driving me mad.
What am I missing?
