Proxmox 5 and ZFS...

[Dubard]

Hi everybody,

I want to test ZFS local storage on 3 nodes. You can see below my config:

Proxmox version: 5.1-35

Node-1: HP Proliant ML350 G6

• 32 Go RAM
• CPU X 16 (2 thread/core; 4 core/socket ; 2 sockets)
• Disk used for ZFS storage: 2 x 500 Go hdd SAS 15K
  • ZFS journal into a 5Go partition on SSD
    
  • ZFS log into same SSD that ZFS Journal but on another 5 Go partition
• Network port: 1Gbits

Node-2: HP Proliant DL380 Gen9

• 32 Go RAM
• CPU X 16 (2 thread/core; 8 core/socket ; 1 sockets)
• Disk used for ZFS storage: 2 x 500 Go hdd SAS 15K (Array raid 0 on HP Smart array P440ar controller for each disk)
  • ZFS journal into a 5Go partition on SSD ( SSD into HP Smart array P440ar controller)
  • ZFS log into same SSD that ZFS Journal but on another 5 Go partition
• Network port: 1Gbits
  

Node-3: HP Proliant DL380 Gen9

• 32 Go RAM
• CPU X 16 (2 thread/core; 8 core/socket ; 1 sockets)
• Disk used for ZFS storage: 2 x 500 Go hdd SAS 15K (Array raid 0 on HP Smart array P440ar controller for each disk)
  • ZFS journal into a 5Go partition on SSD ( SSD into HP Smart array P440ar controller)
  • ZFS log into same SSD that ZFS Journal but on another 5 Go partition
• Network port: 1Gbits

For my HP Smart array P440ar controller, i didn't found how disable RAID mode and use disk directly on Proxmox for ZFS storage. When set the controller in IT mode there is just a option where we can put one disk to "Array Raid 0" mode but I don't know what that actually means from the disk access point of view. Also, I was able to enable an option that allows a direct access to the disk cache ("Physical Drive Write Cache State" option).

Also, if by some miracle I managed to disable the RAID mode of the HP Smart array P440ar controller and then see the disks when I boot on the Proxmox DVD, I couldn't from the setup, configure a Linux software raid (mdraid) for the "/" because Proxmox doesn't allow it.

So, i ended up use RAID 1 from HP Smart array P440ar controller for "/" on Proxmox cluster and for all ZFS storage disk, i use "Array RAID 0 on one disk" mode...I don't think I have any other choice with this damn HP Smart array P440ar controller !

I will also have different questions about ZFS:

1. I have set 5GB as the partition size on my SSD for the ZFS journal and another 5GB partition as ZFS log on the same SSD .Is that enought for 500 GB of ZFS storage space ?
   How i can calculate the good value ?
2. I want to enable a replication every 5min of VM on the 3 ZFS storage of my 3 nodes. Does the HA function correctly in case of a node failure: recovery of the VM since the last synchronization + migration of the VM to another available node ?

Thanks !

 

[Dubard]

Re,

Also i try to restore a CT into my local ZFS storage and I've a error:

Formatting '/myzfspool/images/101/vm-101-disk-2.raw', fmt=raw size=26843545600
mke2fs 1.43.4 (31-Jan-2017)
Discarding device blocks: 4096/6553600 done
Creating filesystem with 6553600 4k blocks and 1638400 inodes
Filesystem UUID: 15653b2e-504d-4730-86d3-0fd78cd20805
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000

Allocating group tables: 0/200 done
Writing inode tables: 0/200 done
Creating journal (32768 blocks): done
Multiple mount protection is enabled with update interval 5 seconds.
Writing superblocks and filesystem accounting information: 0/200
Warning, had trouble writing out superblocks.
TASK ERROR: command 'mkfs.ext4 -O mmp -E 'root_owner=0:0' /myzfspool/images/101/vm-101-disk-2.raw' failed: exit code 144
​

Does anyone have any ideas ?
Thanks !

 

[guletz]

Re,

Also i try to restore a CT into my local ZFS storage and I've a error:

Formatting '/myzfspool/images/101/vm-101-disk-2.raw', fmt=raw size=26843545600
mke2fs 1.43.4 (31-Jan-2017)
Discarding device blocks: 4096/6553600 done
Creating filesystem with 6553600 4k blocks and 1638400 inodes
Filesystem UUID: 15653b2e-504d-4730-86d3-0fd78cd20805
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000

Allocating group tables: 0/200 done
Writing inode tables: 0/200 done
Creating journal (32768 blocks): done
Multiple mount protection is enabled with update interval 5 seconds.
Writing superblocks and filesystem accounting information: 0/200
Warning, had trouble writing out superblocks.
TASK ERROR: command 'mkfs.ext4 -O mmp -E 'root_owner=0:0' /myzfspool/images/101/vm-101-disk-2.raw' failed: exit code 144
​

Does anyone have any ideas ?
Thanks !

Hi,

Please verify if on the storage location if you have enable both type(DataCenter-Sorage-YourZFSname-Content) : Disk Image and Container

 

[Dubard]

Hi,

Please verify if on the storage location if you have enable both type(DataCenter-Sorage-YourZFSname-Content) : Disk Image and Container

Thanks @guletz for your reply.
Yeah... I'd already checked that out before I posted. See below my "/etc/pve/storage.cfg"

...
dir: zfs-storage
path /myzfspool
content rootdir,images
maxfiles 1
shared 1

Thanks.

 

[tom]

Thanks @guletz for your reply.
Yeah... I'd already checked that out before I posted. See below my "/etc/pve/storage.cfg"

...
dir: zfs-storage
path /myzfspool
content rootdir,images
maxfiles 1
shared 1

Thanks.

I see two problems.

1. You do not use ZFS storage, you use directory storage on ZFS, this is dangerous and not recommend. Use ZFS pool, see https://pve.proxmox.com/wiki/Storage:_ZFS
2. You marked a local storage as shared. But its not shared as its local storage.

 

[guletz]

dir: zfs-storage

I guess dir is not OK. Try to create a dedicate dataset for this:

- from comand line as root:
zfs create {your-zpool-name}/CT
- create a new zfs storage element(CT for example), and add as "{your-zpool-name}/CT" using whatever name/id you want
- then try to restore your CT in this new storage

 

[Dubard]

Hi everybody,

Thanks @guletz for your reply.

Before posting, I had tried to create the storage via GUI but I couldn't realize this. See below:


I don't see my ZFS storage coming from my node !
That's why I went through the creation of a directory... I didn't know it was not advisable !
Here's what I executed as commands from my server where the ZFS storage is located:

    root@node-1:/# fdisk /dev/sdb
    root@node-1:/# fdisk /dev/sdc
    root@node-1:/# fdisk /dev/sdd
    root@node-1:/# zpool create -f -o ashift=12 cadzfs mirror /dev/sdb /dev/sdc cache /dev/sdd1 log /dev/sdd2
    root@node-1:/# zpool status
      pool: cadzfs
     state: ONLINE
      scan: none requested
    config:

       NAME        STATE     READ WRITE CKSUM
       cadzfs      ONLINE       0     0     0
         mirror-0  ONLINE       0     0     0
           sdb     ONLINE       0     0     0
           sdc     ONLINE       0     0     0
       logs
         sdd2      ONLINE       0     0     0
       cache
         sdd1      ONLINE       0     0     0

    errors: No known data errors

    root@node-1:/#
    root@node-1:/# zfs create cadzfs/CT
    root@node-1:/# pvesm zfsscan
    cadzfs
    cadzfs/CT
    root@node-1:/#

Did I forget something ?

Thanks !

 

[guletz]

Take a look here in /etc/storage.cfg:

zfspool: only-for-cad
pool cadzfs/CT
blocksize 16k
content rootdir,images
sparse 0

And use blocksize 16k or 32k if you will use large files for CAD.

 

[guletz]

Hi everybody,

    root@node-1:/# fdisk /dev/sdb
    root@node-1:/# fdisk /dev/sdc
    root@node-1:/# fdisk /dev/sdd
    root@node-1:/# zpool create -f -o ashift=12 cadzfs mirror /dev/sdb /dev/sdc cache /dev/sdd1 log /dev/sdd2

Do not use /dev/sdX when you create a zfs pool. Instead use SN, see this for help:

ls -l /dev/disk/by-id

 

[Dubard]

thanks @guletz ! It's ok now !

• I don't understand why i' can't do that to GUI ?

• For Blocksize option, What value do you recommend ? 16k or 32k ? I want use ZFS storage for VM and CT only.

Thanks !

 

[Dubard]

Do not use /dev/sdX when you create a zfs pool. Instead use SN, see this for help:

ls -l /dev/disk/by-id

Thanks for this recommend !

 

[guletz]

thanks @guletz ! It's ok now !

• I don't understand why i' can't do that to GUI ?

• For Blocksize option, What value do you recommend ? 16k or 32k ? I want use ZFS storage for VM and CT only.

Thanks !

I know it is working. Try to click on the arrow on the left of box where you see "v"(I do not know how it is the name - my english is bad)

For block size value it depends what you will have in this VM/CT. I use different datasets with different block size(16 k for DBs, 32k for WinOs only, 32-64 k for some linux CT, and so on). In some cases I use 2 vDisk, with the proper and different block size.You must test to see what is OK for you.