Clustered storage status (ceph & sheepdog)

[spirit]

What is the output of

# pvesm status

Is the storage in that list?

It should be available without refresh. Are you sure to use the pvetest repository ?

 

[cristianm]

file /etc/pve/storage.cfg line 22 (skip section 'web01'): unsupported type 'rbd'
backup nfs 1 1457861632 955720704 429176832 69.51%
images nfs 1 1457861632 955720704 429176832 69.51%
local dir 1 349130060 199400 348930660 0.56%

But I have the last pvetest. Any idea?

 

[cristianm]

I run a new full-upgrade and now works.

pvesm status
backup nfs 1 1457861632 955720704 429176832 69.51%
images nfs 1 1457861632 955720704 429176832 69.51%
local dir 1 349130060 199400 348930660 0.56%
web01 rbd 1 0 0 0 100.00%

But I can´t view this in the GUI.

 

[cristianm]

Can I add the same rbd pool in all the nodes of the cluster?

 

[dietmar]

Can I add the same rbd pool in all the nodes of the cluster?

yes

 

[cristianm]

Hi.

I'm testing ceph into proxmox. I need to setup the rbd_cache_size, rbd_cache_max_age and type. Is it possible, how? I check low performance and maybe is issue of the cache setup.

 

[spirit]

you can add a /etc/ceph.conf and define values inside.

 

[cristianm]

I need to create /etc/ceph.conf ? In my servers the file /etc/ceph.conf don´t exist.

 

[cristianm]

Thanks. Then add parameters, need to reboot proxmox server or shutdown/start the VM? The setup of disk cache works fine with ceph?

 

[spirit]

just stop/start your vm.
(and use writeback for your disk).

 

[cristianm]

Currently I test ceph for some weeks and this works fine. Have errors when run backups and have selected backup the VM. Need to check "No Backup" for the ceph images. In the next days ceph will release a major stable version with some performance improvements and is a good idea upgrade to this.

 

[cristianm]

Please upgrade ceph to v0.56. I have some issues with 0.48 client and 0.56 servers. Thanks.

 

[kchawk]

I have been testing with ceph also. I noticed the backup issue and wrote a bash script to backup the ceph images. Put the ceph.conf on the proxmox servers /etc/ceph/ceph.conf and run from each node. It works with KVMs. I don't use containers so didn't test it.

#!/bin/bash
#Author: Troy Hockman - Afewteks LLC
#Desc: Backs up ceph images and conf file for kvms.
#      Please test and use at own risk.

baktemp=/mnt/pve/KVMBaks/temp
bakdir=/mnt/pve/KVMBaks/dump
for i in $(qm list | grep -v VMID | awk '{print $1":"$2}');
do
  echo $i
  id=$(echo $i | awk -F ":" '{print $1}')
  name=$(echo $i | awk -F ":" '{print $2}')
  diskimg=$(rbd ls | grep $id)
  if [ "$diskimg" ]; then
    rbd snap create --snap ProxMoxBak $diskimg 
    rbd export $diskimg $baktemp/$diskimg
    rbd snap rm --snap ProxMoxBak $diskimg
    tar --use-compress-program=lzop -cf $bakdir/rbd.$name.$id.`date +%Y_%m_%d_%H_%M`.tar.lzo $baktemp/$diskimg /etc/pve/qemu-server/$id.conf
    rm -f $baktemp/$diskimg
  fi
done

 

[kchawk]

I have also created a restore script for kvms

#!/bin/bash
#Author: Troy Hockman Afewteks LLC
#Desc:   This will do a quick restore for ceph images
bakimage="rbd.TEST.111.2013_01_01_13_09.tar.lzo"
baktemp=/mnt/pve/KVMBaks/temp
bakdir=/mnt/pve/KVMBaks/dump

resimage=$(tar --use-compress-program=lzop -tf $bakdir/$bakimage | grep disk | awk -F "/" '{print $5}')
tar --use-compress-program=lzop -C / -xf $bakdir/$bakimage
rbd import $baktemp/$resimage

 

[spirit]

Please upgrade ceph to v0.56. I have some issues with 0.48 client and 0.56 servers. Thanks.

Hi, this is planned to upgrade proxmox packages when next stable ceph release (bobtail) will be release. (should be in some days,weeks)

 

[Cayuga]

kchawk,

I like your backup script and started using it. However, it bothered me that we had to write and read the uncompressed archive to a local disk first, so I built 7za (which can archive from stdin -- http://sourceforge.net/projects/p7zip/files/latest/download) and modified the script to take the backup from rbd stdout so we could skip all of the extra IO. This approach does have a couple of downsides:

1) it produces a 7z archive rather than a tar archive
2) using bzip2 instead of lzop uses more CPU (using 7z is worse) but we do get a slightly smaller backup (about 10-12% savings here)

Anyway, if anyone is interested, here's my modified script:

#!/bin/bash
#Author: Troy Hockman - Afewteks LLC
#7za mods by Jeff Moskow - Ready-to-Run Software, Inc.
#Desc: Backs up ceph images and conf file for kvms.
#      Please test and use at own risk.


baktemp=/mnt/pve/proxsnaps6/temp
bakdir=/mnt/pve/proxsnaps6/dump
for i in $(qm list | grep -v VMID | awk '{print $1":"$2}');
do
  echo $i
  id=$(echo $i | awk -F ":" '{print $1}')
  name=$(echo $i | awk -F ":" '{print $2}')
  diskimg=$(rbd ls | grep $id)
  if [ "$diskimg" ]; then
    rbd snap create --snap ProxMoxBak $diskimg
    archive="$baktemp/vzdump-qemu-$name-$id-`date +%Y_%m_%d_%H_%M`.7z"
    /usr/local/bin/7za a $archive  -mx=0 -m0=bzip2 -mmt=on /etc/pve/qemu-server/$id.conf
    rbd export $diskimg - | /usr/local/bin/7za -w$baktemp a $archive -si -mx=0 -m0=bzip2 -mmt=on
    rbd snap rm --snap ProxMoxBak $diskimg
    mv $archive $bakdir
  fi
done

I can send a precompiled binary of 7za, for anyone interested.

 

[kchawk]

This is great. I will take a lot at it for sure. I think for me, the main reason I decided to rbd export and then compress was to remove the snap asap. I like getting rid of snaps quickly. Does 7z not support lzop? I haven't used it mayself.
The other thing I am looking at is dumping the rbd export directly to a lessfs file system for dedup and replication. Maybe btrfs also. No need for compression.

 

[Cayuga]

No, there is no lzop option. There is supposed to be a gzip option, but I haven't had much luck with it.

 

[spirit]

Hi Huys, coming proxmox 2.3 release, will support rbd officialy (gui), and proxmox live backup too (without snapshot).

 

[badji]

Hi Huys, coming proxmox 2.3 release, will support rbd officialy (gui), and proxmox live backup too (without snapshot).

Thank you Alexandre, Dietmar and The Proxmox Team.