Proxmox Storage, Networking, Network Share Questions

[chinyongcy]

Current Setup:

Networking: Nothing Complex, No VLAN

Node 1- Lenovo M910q:
- i5-7500T, 32GB RAM
- 970 Evo Plus 1TB Nvme for VM
- 860 Evo 2.5" SATA SSD for Proxmox OS
- Onboard NIC: 1Gbps connected to Router Directly
- Add On NICE: 2.5Gbps Connected to 2.5Gbps Switch

Node 2 - Dell Optiplex (To Be Added, coming soon):
- i5-8500T, 32GB RAM
- VM SSD TBC- OS SSD TBC
- Onboard NIC: 1Gbps
- Ordered another 2.5Gbps NIC


Questions:

1. SSD Recommendation:
- Have not order the SSD for the 2nd Node. Other than 970 Evo Plus is there any other SSD other there that does not break the bank? For example, would it better if I go for 980 Pro? Understand that it shouldn't make any difference, since both of my nodes are using PCIe Gen 3.
---
2. Storage Type For Proxmox:
- Still torn between LVM, LVM-Thin and ZFS even after reading quite a bit.- I don't think I will have a lot of replications going it. Maybe just 1 or 2 crucial containers like Wireguard and Nginx
- Is it true that if I use LVM, I will not be able to migrate my VM/Containers across different Nodes and Proxmox HA will not be supported?
- Will be moving all my stuff from 7500t to 8500t. Primarily running some simple Homelab stuff, Home Assistant, Nginx Proxmox, Jellyfin etc. The 7500t will then be used for my build agent and experimenting stuff, so not much replication is needed, but good to have if possible.
---
3. Networking:
- For my Node 1, 1Gbps NIC is connected directly to my router with 1Gbps link speed for internet. 2.5Gbps NIC is connected to my 2.5Gbps switch for local connection. Does this make sense or, should I just use 1 x 2.5G link would be fine already. Also left the 1Gbps NIC connected for AMT as needed. But I think in long run I don't need to use this, only need it for troubleshooting.
---
4. Samba/NFS Share:
- I have two USB drive that I am currently passing through directly to a VM which runs samba share, Jellyfin and Transmission is also in the same machine so that it can directly write to the disk.
- I am thinking should I run a TrueNas or any other apps with a GUI on a VM to manage my network share storage?
---
5. Network Share Speed Up:
- Currently as I am already using the 2.5Gbps to read/write to my samba share is it possible to utilize some of the SSD as cache so I can saturate the 2.5Gbps when using it as a Network Drive?

 

[LnxBil]

Is it true that if I use LVM, I will not be able to migrate my VM/Containers across different Nodes and Proxmox HA will not be supported?

That is only possible with ZFS.

Have not order the SSD for the 2nd Node. Other than 970 Evo Plus is there any other SSD other there that does not break the bank? For example, would it better if I go for 980 Pro? Understand that it shouldn't make any difference, since both of my nodes are using PCIe Gen 3.

I would not use ZFS on consumer SSDs, just search the forum. Just enterprise SSDs.

For my Node 1, 1Gbps NIC is connected directly to my router with 1Gbps link speed for internet. 2.5Gbps NIC is connected to my 2.5Gbps switch for local connection. Does this make sense or, should I just use 1 x 2.5G link would be fine already. Also left the 1Gbps NIC connected for AMT as needed. But I think in long run I don't need to use this, only need it for troubleshooting.

Do you build a loop with this or are there disjunct network segments?

 

[Dunuin]

1. SSD Recommendation:
- Have not order the SSD for the 2nd Node. Other than 970 Evo Plus is there any other SSD other there that does not break the bank? For example, would it better if I go for 980 Pro? Understand that it shouldn't make any difference, since both of my nodes are using PCIe Gen 3.

Don't waste your money on expensive prosumer stuff. If you want something better than a consumer TLC with DRAM cache, get an enterprise SSD with PLP which would offer an actual difference boosting sync write performance by several magnitudes and without the heavy performance drop after a few seconds of writes.

 

[chinyongcy]

That is only possible with ZFS.

Understood, guess will have to stick with ZFS then.

I would not use ZFS on consumer SSDs, just search the forum. Just enterprise SSDs.

Got it, will do more research on this. Only afraid the cost will be very steep.

Do you build a loop with this or are there disjunct network segments?

Erm, I don't really get it. I just use different IP. 1Gbps NIC - 10.0.0.11 (Directly to Router), 2.5Gbps NIC: 10.0.0.12 (To my 2.5Gbps switch which is connected to the router).
I just realised something even though I can reach Proxmox via both NIC ip addresses. when I do an iperf test, it is still getting only 1Gbps. Until i changed the IP route to give NIC 2 a higher metric. Guess it didn't exactly work how I thought it would be.

 

[chinyongcy]

Don't waste your money on expensive prosumer stuff. If you want something better than a consumer TLC with DRAM cache, get an enterprise SSD with PLP which would offer an actual difference boosting sync write performance by several magnitudes and without the heavy performance drop after a few seconds of writes.

Got it, it doesn't matter if it is an older model right? Did some research, some of the threads are quite old.

 

[LnxBil]

Got it, will do more research on this. Only afraid the cost will be very steep.

It's really not that steep. I buy at home and in the office often used enterprise SSDs for internal stuff and of course RAID/mirror them and the price is lower than new prosumer drives. I recently bought a 960 GB Samsung Enterprise for less than 40 euros.

Erm, I don't really get it. I just use different IP. 1Gbps NIC - 10.0.0.11 (Directly to Router), 2.5Gbps NIC: 10.0.0.12 (To my 2.5Gbps switch which is connected to the router).

Different IPs is fine. That's what I meant with disjunct network segments.

 

[chinyongcy]

It's really not that steep. I buy at home and in the office often used enterprise SSDs for internal stuff and of course RAID/mirror them and the price is lower than new prosumer drives. I recently bought a 960 GB Samsung Enterprise for less than 40 euros.

Wow 40 euros as in it is second-hand? I always thought it's going to be the range of 700 Euros upwards.
I just saw some of the enterprise SSD. Do you use it with a adapter or something? Seems like enterprise m2 nvme drive all are U.2.
What's your model btw, can i take reference from there?

 

[LnxBil]

Wow 40 euros as in it is second-hand? I always thought it's going to be the range of 700 Euros upwards.
I just saw some of the enterprise SSD. Do you use it with a adapter or something? Seems like enterprise m2 nvme drive all are U.2.
What's your model btw, can i take reference from there?

Mine were SATA. The used NVMe are more expensive because they haven't been on the market for so long and not so many for sale. In my NVMe slot, I went with Optane for ZFS ZIL. Was a better fit for my desired system.

 

[Felix.]

Only afraid the cost will be very steep.

Checkout ebay, there are tons of affordable Enterprise Drives that are in great shape, most of them haven't seen a fraction of the lifetime writes they could handle.

Good examples for SATA are Samsung SM863/SM863a.
Just saw a SM863a 1.92 TB for 129€
Using those with Proxmox on ZFS, very happy.

For NVMe, there are excellent Samsung Enterprise Drives, too.
For PCIe 3.0 take a look at PM983, for PCIe 4.0 PM9A3.
2TB are available for around 100-150€

Beware that you need an 22110 slot if you want M.2 drives, most consumer boards only provide 2280 (shorter) slots.
Otherwise you could use an U.2 adapter for the M.2 slot, those are pretty affordable as well.

I'm using the PM9A3 3.84TB drives for about 2 years now with Proxmox on ZFS as well, no issues and stupid fast.

 

[Dunuin]

Seems like enterprise m2 nvme drive all are U.2.

Because M.2 is a terrible form factor. There is not enough footprint for a lot of NAND chips for more capacity or higher quality NAND. Not enough space for supercaps for power-loss protection. Usually hard to swap out, in case you need to replace it, because directly on the mainboard with maybe PCIe cards blocking access. Hard to cool, especially with chips on both sides, so easily thermal throttling because there isn't space for sufficient heatsinks (especially if you don't want to block access to PCIe slots above it).
It was designed for laptops so all that mattered was that it is small as possible and without cables...

There are a few enterprise M.2 SSDs (Samsung PM9A3/983, Micron 7300/7400/7450 Pro/MAX, Seagate Nytro 5000, Intel P4801X/P4511, Kingston DC1000B...thats it...with only ther Microns and Kingtson fitting in an 2280 slot) but there are also M.2 to U.2 adapters.