Proxmox v5 and Ceph + ZFS

[starnetwork]

Hi,
I have 3x nodes with last version of Proxmox 5.x
this 3 nodes has 3 local hard drives for each nodes
1 hard drive use for boot and Proxmox OS
2x Samsung SSD SM863 using for ZFS pool with RAID1
now, I try to Install Ceph and:
1. during the installation I got this error:

root@server203:~# pveceph createmon
unable to find local address within network '10.10.10.0/24'

but the network as with good configurations:

        up   ip addr add 10.10.10.3/24 dev $IFACE label $IFACE:1
        down ip addr del 10.10.10.3/24 dev $IFACE label $IFACE:1

anyway, I just added the follow text:

iface bond0:1 inet static
address 10.10.10.1
netmask 255.255.255.0

I don't need it in: /etc/network/interfaces, but I leave it just make the Ceph validation "happy"

2. after that, click: pveceph install --version jewel
as I see at: https://pve.proxmox.com/wiki/Ceph_Server#Installation_of_Ceph_packages
show:

400 Parameter verification failed.
version: value 'jewel' does not have a value in the enumeration 'luminous'

error, enter: pveceph install --version luminous
solve that issue:

3. now, after the Ceph installed and also the monitor, I try to add the ZFS local pool or even drive from the pool to the Ceph OSD with: https://pve.proxmox.com/wiki/Ceph_Server#using_a_disk_that_was_part_of_a_zfs_pool

ceph-disk zap /dev/sda

and got this error:

/bin/dd: error writing '/dev/sda9': No space left on device
9+0 records in
8+0 records out
8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0209574 s, 400 MB/s
ceph-disk: Error: Command '['/bin/dd', 'if=/dev/zero', 'of=/dev/sda9', 'bs=1M', 'count=10']' returned non-zero exit status 1

any suggestions to solve that issue?
Regards,

 

[Alwin]

3. now, after the Ceph installed and also the monitor, I try to add the ZFS local pool or even drive from the pool to the Ceph OSD with:

Ceph OSDs are running independent of ZFS, Ceph wants raw disk to work with, as does ZFS. So, use either but not both technologies mixed on the same disks.

ceph-disk zap /dev/sda

I assume, that sda hosts your PVE system, then you just killed it.

If you want to use Ceph as a shared storage, then it needs raw disks for its OSD to work properly. Configure your system to have spare disks for use by Ceph.

 

[arnaudd]

Ceph OSDs are running independent of ZFS, Ceph wants raw disk to work with, as does ZFS. So, use either but not both technologies mixed on the same disks.

not true it's much depend if you use bluestore or filestore
ceph dot com/community/new-luminous-bluestore/

I assume, that sda hosts your PVE system, then you just killed it.

If you want to use Ceph as a shared storage, then it needs raw disks for its OSD to work properly. Configure your system to have spare disks for use by Ceph.

lab.piszki dot pl/installing-a-ceph-jewel-cluster-on-ubuntu-lts-16-using-zfs/
kernelpanik dot net/running-ceph-on-zfs/

 

[starnetwork]

Hi Alwin,
1. thanks, so I can use only Ceph only and get Data Redundancy?

2. Proxmox OS installed on sdc
we have total of 3 Drives on each server (microblades)
2x PM863a 960DB SSD for data
1x SATA DOM 128GB for Proxmox OS

 

[arnaudd]

Hi Alwin,
1. thanks, so I can use only Ceph only and get Data Redundancy?

2. Proxmox OS installed on sdc
we have total of 3 Drives on each server (microblades)
2x PM863a 960DB SSD for data
1x SATA DOM 128GB for Proxmox OS

Hello,

On proxmox 4 it's work easy to add osd on ZFS
On proxmox 5 we still have issue due of pveceph not pass correctly paramater (seems)

On your setup you will just have 1 disk for boot, i think you need at least Raid 1.

Advantage to use ZFS with CEPH :
. When you need replication on 2 sites you need less replications and gain speed due half of replication will be local raid (rather than network latency)
. Use all disk of all servers, even more important when you have just 4 disks.
. Deduplication

Others ZFS features are less important for me like clones/snapshot due of Ceph.
Haven't yet test caching in Ceph as well but ZFS is very good at it

 

[starnetwork]

Thanks Arnaudd,
I need Redundancy, so if Chph can provide it, I can remove the ZFS, no?
I have only 2 hard drives on each node and I want to work with Shared storage for HA and Redundancy.

Regards,

 

[Alwin]

ZFS is not shared, when not used with some other means of network storage (eg. zfs over iSCSI, ...), with Ceph you have a shared storage, that survives a node failure (auto healing).

As a basic setup, I would not recommend the use of DOM devices, as those have very low DWPD and a PVE host does a lot of logging, ~10GB per day. In a three node cluster a node can die and easily be setup freshly in a couple of hours, so it is not necessary to use a RAID1 for your OS but makes things easier.

You also need to consider if you have enough disk for Ceph, as a pool with size 3 (recommended) will store three copies of your data. With your setup, roughly 1.8TB of storage (consider also storage growth).

Your PVE servers need enough CPU/memory to host your VMs, OSDs and enough left for caching + other system tasks.

 

[Magneto]

Has anyone done any comparisons of CEPH on top of ZFS? I am particularly interested to know if it would add any performance improvements?

 

[jeffwadsworth]

Just like Alwin wrote, use either Ceph or ZFS; do not mix them. There is no need. I have no idea how ZFS is related to Bluestore/Filestore in regards to Arnaudd's post. Twilight Zone. Bluestore has its own filesystem and Filestore uses XFS.

 

[Magneto]

Just like Alwin wrote, use either Ceph or ZFS; do not mix them. There is no need. I have no idea how ZFS is related to Bluestore/Filestore in regards to Arnaudd's post. Twilight Zone. Bluestore has its own filesystem and Filestore uses XFS.

Yes I read what he said. Still doesn't answer the question.
With CEPH I now have 4 individual spinning disks in each machine. ZFS would add a bit of performance boost in most situations, so I am wondering if it would have added a performance boost in this case as well.

 

[Alwin]

There may be some perf charts on the internet. But as by Ceph recommendation and mostly used in the field, XFS is for Ceph the most tested and supportedFS. On the Ceph mailling lists hardly anyone is writing about zfs as FS for filestore OSDs. Also with Bluestore being the default in Luminous, the "which FS for my OSD question" has lesser importance.