Hi All,,
We have a R7525 Dell server (2 sockets, 64 cores/socket, 2 Ghz base clk, 3.2 Ghz boost clk, 128 GB RAM, 96TB [8TB x 12] 7200 rpm SAS HDD, Dell BOSS S1 SSD 240G RAID1 boot, H745 PERC RAID controller with PCIe 4.0 bus. 2 x 2 raiser slots are available, one is having H745 RAID controller in one slot and the other having Dell BOSS S1 [240G SSD RAID-1], and the other two slots are free.
Ideally we needed this for high performance computing. But for some reason, the h/w selection got flawed due to lack of expertise. So the configuration is not suited for GPGPU such as A2, T4, A100 etc... due to lack for right slots and power budget. Also the choice of our CPU core also seem like flawed, (base clk 2 Ghz) for high performance loads (poor single core performance).
For the worst, there are no other SSDs other than the dell BOSS S1 where proxmox HOST is installed. We are attempting for Proxmox VE with ZFS software RAIDs for storage. It would have been ideal if we could get two NVMe PCIe cards (one with RAID-1 for VM hosting and the other for ZFS-LOG and ZFS L2ARC). Probably we could use two free PCIe slot for this after recommendations from dell. We fear warranty will be void if we select any arbitrary PCIe card and NVMe drives for this. So we are now quite pessimistic in using this for HPC needs. However the current aim is to make the best use of it, given the limitations. We are a small research group and do not have much budget to employ external professional help at this point.
Any recommendations for the above puzzle is welcome.
Further we are quite new to proxmox and and such virtualized setup. Our requirements are as follows.. Kindly help us to arrive at a configuration, for our current h/w setup and below requirements
We are using Proxmox 7.2 as the hypervisor installed on Dell BOSS S1 SSD ext4. Further we also plan to use ZFS as the storage and RAID. Hence we have put H745 in eHBA mode. Our workloads are of the following types.
a) A few linux VDI desktops with matlab, ROS and some other scientific computing software.
b) A few Windows 10 desktops with solidworks, Ansys kind of simulation / CAD tools
c) One intranet Web server along with an LDAP server for intranet wide authentication for all services
d) One intranet linux server for command line based embedded development (HOST git server and other toolchains)
e) A mirror RAID-1 storage (ZFS) for hosting VMs
f) A shared RAID-6 storage (ZFS RAIDz2) to host home folders / shared folders of VMs. This should also store git server backend etc.
g) A ZFS mirror storage for VM backups.
All access permissions are authenticated through LDAP server (for shared storage and also VM logins). Should the LDAP server be installed in Proxmox HOST or as a separate VM ?
I am specifically at loss regarding shared storage requirement implementation. My confusions are given below.
a) we just need a shared storage across vms and other services, that are protected using RAID6.
i) Should we go for a complete NAS solution such as FREENAS/TRUENAS or should we just install NFS or Samba servers in HOST or VM if the requirement is a just shared file storage ?
ii) Can ZFS storage in proxmox host in any way server this purpose ? IF yes how ?
iii) Is it possible to share a RAIDz2 dataset (instantiated in proxmox host) accross VMs ? if yes, how ?
iv) If going in NFS / CIFS (samba) route, what is the best practice ? Install it in proxmox HOST or as a proxmox VM ? Can we use ZFS RAIDz2 dataset instantiated in proxmox HOST for its storage ? Is NFS preferred over Samba in our kind of setup ??
v) If going in TrueNAS route, I see no way to passthrough the individual bare drives / disk controllers to the VM as the proxmox host has already acquired them. Is it possible to passthrough the virtual drives to the TrueNAS VMs ? Is there any way in which TrueNAS can use a RAIDz2 Zvol or dataset rather than its own ZFS implementation?
vi) Is their any solution using ZFS volomes Zvol in RAIDz2 for shared storage as envisioned above.
vii) Suggested configuration for our 12x 8TB HDD for the above requirements ??
I am doing this setup for the first time and hence you help is very much appreciated. Any stupid ideas in the above proposal shall be forgiven
. Please guide.
-regards
Tachionic
We have a R7525 Dell server (2 sockets, 64 cores/socket, 2 Ghz base clk, 3.2 Ghz boost clk, 128 GB RAM, 96TB [8TB x 12] 7200 rpm SAS HDD, Dell BOSS S1 SSD 240G RAID1 boot, H745 PERC RAID controller with PCIe 4.0 bus. 2 x 2 raiser slots are available, one is having H745 RAID controller in one slot and the other having Dell BOSS S1 [240G SSD RAID-1], and the other two slots are free.
Ideally we needed this for high performance computing. But for some reason, the h/w selection got flawed due to lack of expertise. So the configuration is not suited for GPGPU such as A2, T4, A100 etc... due to lack for right slots and power budget. Also the choice of our CPU core also seem like flawed, (base clk 2 Ghz) for high performance loads (poor single core performance).
For the worst, there are no other SSDs other than the dell BOSS S1 where proxmox HOST is installed. We are attempting for Proxmox VE with ZFS software RAIDs for storage. It would have been ideal if we could get two NVMe PCIe cards (one with RAID-1 for VM hosting and the other for ZFS-LOG and ZFS L2ARC). Probably we could use two free PCIe slot for this after recommendations from dell. We fear warranty will be void if we select any arbitrary PCIe card and NVMe drives for this. So we are now quite pessimistic in using this for HPC needs. However the current aim is to make the best use of it, given the limitations. We are a small research group and do not have much budget to employ external professional help at this point.
Any recommendations for the above puzzle is welcome.
Further we are quite new to proxmox and and such virtualized setup. Our requirements are as follows.. Kindly help us to arrive at a configuration, for our current h/w setup and below requirements
We are using Proxmox 7.2 as the hypervisor installed on Dell BOSS S1 SSD ext4. Further we also plan to use ZFS as the storage and RAID. Hence we have put H745 in eHBA mode. Our workloads are of the following types.
a) A few linux VDI desktops with matlab, ROS and some other scientific computing software.
b) A few Windows 10 desktops with solidworks, Ansys kind of simulation / CAD tools
c) One intranet Web server along with an LDAP server for intranet wide authentication for all services
d) One intranet linux server for command line based embedded development (HOST git server and other toolchains)
e) A mirror RAID-1 storage (ZFS) for hosting VMs
f) A shared RAID-6 storage (ZFS RAIDz2) to host home folders / shared folders of VMs. This should also store git server backend etc.
g) A ZFS mirror storage for VM backups.
All access permissions are authenticated through LDAP server (for shared storage and also VM logins). Should the LDAP server be installed in Proxmox HOST or as a separate VM ?
I am specifically at loss regarding shared storage requirement implementation. My confusions are given below.
a) we just need a shared storage across vms and other services, that are protected using RAID6.
i) Should we go for a complete NAS solution such as FREENAS/TRUENAS or should we just install NFS or Samba servers in HOST or VM if the requirement is a just shared file storage ?
ii) Can ZFS storage in proxmox host in any way server this purpose ? IF yes how ?
iii) Is it possible to share a RAIDz2 dataset (instantiated in proxmox host) accross VMs ? if yes, how ?
iv) If going in NFS / CIFS (samba) route, what is the best practice ? Install it in proxmox HOST or as a proxmox VM ? Can we use ZFS RAIDz2 dataset instantiated in proxmox HOST for its storage ? Is NFS preferred over Samba in our kind of setup ??
v) If going in TrueNAS route, I see no way to passthrough the individual bare drives / disk controllers to the VM as the proxmox host has already acquired them. Is it possible to passthrough the virtual drives to the TrueNAS VMs ? Is there any way in which TrueNAS can use a RAIDz2 Zvol or dataset rather than its own ZFS implementation?
vi) Is their any solution using ZFS volomes Zvol in RAIDz2 for shared storage as envisioned above.
vii) Suggested configuration for our 12x 8TB HDD for the above requirements ??
I am doing this setup for the first time and hence you help is very much appreciated. Any stupid ideas in the above proposal shall be forgiven
. Please guide.-regards
Tachionic
