ZFS RAID and VM Build

[strix]

Hello everyone, we are new in proxmox community and we have some questions about that.

First of all our setup is 2 Servers with 2x240GB SSD, 4x6TB HDD, 128GB RAM and 2x Ten Core 2.20Ghz.

The 2 SSD Disks is set it up for Proxmox only on ZFS (RAID1) after i successful the installation.

root@srv2:~# zpool status
  pool: rpool
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sdf2    ONLINE       0     0     0
            sde2    ONLINE       0     0     0

errors: No known data errors
root@srv2:~# zpool list
NAME    SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
rpool   222G   904M   221G         -     0%     0%  1.00x  ONLINE  -

I think ZFS RAID1 is ok ? If one of the drives in the array fails the other disk should work propertly right?

root@srv2:~# zfs list
NAME               USED  AVAIL  REFER  MOUNTPOINT
rpool             9.38G   206G   104K  /rpool
rpool/ROOT         900M   206G    96K  /rpool/ROOT
rpool/ROOT/pve-1   899M   206G   899M  /
rpool/data          96K   206G    96K  /rpool/data
rpool/swap        8.50G   214G    56K  -

In web interface of proxmox on "Datacenter" -- > "Summary" --> Storage says 412.24 GiB and not 222GB why?

And on "Local" --> "Summary" says storage 206.56GB and on "local-zfs" --> "Summary" says storage 205.68GB. But in node "Srv2" "Summary" says 206.56GB.


First server will be used for different of types of VM's on local-zfs.

In our second Server the primary use of this will be a VM with a webserver instalation included databases and ± 2000 domains on that and we are thinking to use CentOS with 96GB RAM 30CPU and 10TB HDD. Is that ok? If not what is the best solution for that build?

And on "create VM", "Network" what is the best model to use? Because we gonna have multiple ip address on that VM. Also what Bus/Device on hard disk do you advice ?

And one last question what is the best solution for our storage 4x6TB HDD zfs RAID10 or RAID5 or RAID6? How should the RAIDs be setup for optimal safety and performance?

Hopefully you guys can point me in the right direction. Thank you.

 

[wolfgang]

I think ZFS RAID1 is ok ? If one of the drives in the array fails the other disk should work propertly right?

Correct it will work with one SSD.

In web interface of proxmox on "Datacenter" -- > "Summary" --> Storage says 412.24 GiB and not 222GB why?

Because you have storage "local" and "local-zfs" both are the same but datacenter do not know this.
The datacenter count all storages.
You can set which storage should count. This can be done on the gear on the right top side.

In our second Server the primary use of this will be a VM with a webserver instalation included databases and ± 2000 domains on that and we are thinking to use CentOS with 96GB RAM 30CPU and 10TB HDD. Is that ok? If not what is the best solution for that build?

I would use multiple Containers instead of one fat VM.

Because we gonna have multiple ip address on that VM. Also what Bus/Device on hard disk do you advice ?

Use virtio for the best performance.
As Disk bus use SCSI with virtio SCSI Controller.

And one last question what is the best solution for our storage 4x6TB HDD zfs RAID10 or RAID5 or RAID6? How should the RAIDs be setup for optimal safety and performance?

You should always use Raid10 for virtualization.

 

[Klaus Steinberger]

ZFS Raid1 is fine

For the 6TB HHD's, use at least Raid6 (in ZFS: RaidZ2), as the very long rebuild (resilver) times of such disks will get you into risk of data loss with just one drive redundancy. As you have only 4 drives, use Mirror instead, that's faster and you loose capacity of two drives anyway.

 

[strix]

Thanks for your answers, you helped me very much.

We can't create multiple VM's until one fat because we gonna need more than one subscriptions for web hosting control panel.

How about on VM Build which type of CPU is the best?

And how can i made software RAID10 for 4x6TB HDD from inside of web interface --> storage -- > add ? or from command line, and how?

 

[Klaus Steinberger]

Look into the documentation for zpool command.

Something like :

zpool create tank mirror dev1 dev2 mirrot dev3 dev4

 

[strix]

Ok well i create the zfs Raid 10

ls -l /dev/disk/by-id/ | grep sd[a-z]

zpool create -f -o ashift=12 wd-1092G-dc mirror ata-WDC_WD6002FRYZ-01WD5B0_K1JSXNHD ata-WDC_WD6002FRYZ-01WD5B0_K1JSMWXD mirror ata-WDC_WD6002FRYZ-01WD5B1_K1KATHAD ata-WDC_WD6002FRYZ-01WD5B0_K1J6APRD

root@srv1:~# zpool list

NAME          SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
rpool         222G  5.25G   217G         -     0%     2%  1.00x  ONLINE  -
wd-1092G-dc  10.9T  1.06G  10.9T         -     0%     0%  1.00x  ONLINE  -

i think the zfs Raid 10 works ok, after that i create /vmdata

zfs create wd-1092G-dc/vmdata
zfs set compression=on wd-1092G-dc/vmdata
pvesm zfsscan

root@srv1:~# zfs list
NAME                 USED  AVAIL  REFER  MOUNTPOINT
rpool               13.7G   201G   104K  /rpool
rpool/ROOT          5.24G   201G    96K  /rpool/ROOT
rpool/ROOT/pve-1    5.24G   201G  5.24G  /
rpool/data            96K   201G    96K  /rpool/data
rpool/swap          8.50G   210G    56K  -
wd-1092G-dc         1.83G  10.5T   104K  /wd-1092G-dc
wd-1092G-dc/vmdata  1.83G  10.5T  1.83G  /wd-1092G-dc/vmdata

and i go on Proxmox Web interface
Datacenter - Storage - Add - Directory - /wd-1092G-dc/vmdata

Is that the recommended way for VM Build on /vmdata/ folder for zfs Raid10?

 

[Klaus Steinberger]

I think you should add it as "ZFS", then you have ZVOL Images

 

[strix]

Well if i add it as ZFS on Create VM on tab "Hard Disk" i can't select "Format" because is locked. Vmdata folder is created inside zfs raid 10.

Zfs how much RAM needs for 10.5TB? Minimum 8GB and Maximum 16GB is ok or not ?

 

[Klaus Steinberger]

with ZVOL (and CEPH also) Format is of course locked to RAW. It is an image which is created with a ZVOL so only RAW is possible. It is thin provisioned by ZFS means, so QCOW is not necessary anyway.

RAM: you mean the zfs_arc_max parameter? The size should work but more memory for ARC is always good.

 

[devonblzx]

Zfs how much RAM needs for 10.5TB? Minimum 8GB and Maximum 16GB is ok or not ?

The rule of thumb is 1GB of RAM for every 1TB of storage, but it depends on your performance needs. You can run on less, I've put 32TB on a 16GB RAM system for cold storage, but the more you allow for ARC, the better performance you'll see.

 

[strix]

So if i need format QEMU or VMware image format what can i do ?

For Directory i read this https://pve.proxmox.com/wiki/Storage:_ZFS on "Examples" "It is recommended to create an extra ZFS file system to store your VM images."

Yes i mean zfs_arc_max parameter. On this node i'am gonna create one and only big fat VM with 96GB RAM for Webserver use and the Free RAM is 32GB. Total on this node is 128GB. Do you think 8GB min and max 16GB is ok ? or should i need to configure it like 16GB min and max 32GB?