can't make zfs-zsync working

[jeanlau]

Hello !

I just installed a two proxmox cluster with last iso 3.4 and have subscription so i made all the updates. The pve-zsync fonction is really great and I really really would like to succeed make it work.


If someone already managed to make it work would you like please tell me how, I followed all the instructions on the page :
https://pve.proxmox.com/wiki/PVE-zsync


Not working for me


First, a little bit more on my configuration, i made a completely zfs install, i created some zfs datasets but I could not add them in the storage configuration with the gui so had to create the directly into storage.cfg :


dir: local
path /var/lib/vz
content images,iso,vztmpl,backup,rootdir
maxfiles 3


zfspool: Disks
pool rpool/disks
content images
sparse


zfspool: BKP_24H
pool rpool/BKP_24H
content images
sparse


zfspool: rpool
pool rpool
content images
sparse



I show you what my zfs config look like :


NAME USED AVAIL REFER MOUNTPOINT
rpool 199G 6,83T 96K /rpool
rpool/BKP_24H 96K 6,83T 96K /rpool/BKP_24H
rpool/ROOT 73,1G 6,83T 96K /rpool/ROOT
rpool/ROOT/pve-1 73,1G 6,83T 73,1G /
rpool/disks 92,5G 6,83T 96K /rpool/disks
rpool/disks/vm-100-disk-1 2,40G 6,83T 2,40G -
rpool/disks/vm-106-disk-1 963M 6,83T 963M -
rpool/disks/vm-107-disk-1 3,61G 6,83T 3,61G -
rpool/disks/vm-108-disk-1 9,29G 6,83T 9,29G -
rpool/disks/vm-110-disk-1 62,9G 6,83T 62,9G -
rpool/disks/vm-204-disk-1 13,4G 6,83T 13,4G -
rpool/swap 33,0G 6,86T 64K -



and for the exemple the configuration of my test machine 106.conf :


balloon: 256
bootdisk: virtio0
cores: 1
ide0: none,media=cdrom
memory: 1024
name: Deb-Test
net0: virtio=52:a5:C1:5C:3F:61,bridge=vmbr1
ostype: l26
scsihw: virtio-scsi-pci
sockets: 1
virtio0: Disks:vm-106-disk-1,cache=writeback,size=5G



So now the result when i try the command given in the wiki :


COMMAND:
zfs list -r -t snapshot -Ho name, -S creation rpool/vm-106-disk-1
GET ERROR:
cannot open 'rpool/vm-106-disk-1': dataset does not exist



I understand the command expect the vm disk to be at the root of the pool so I change the disk place and i try either to send directly to the pool to the other side, but with no more luck :


send from @ to rpool/vm-106-disk-1@rep_default_2015-09-24_12:30:49 estimated size is 1,26G
total estimated size is 1,26G
TIME SENT SNAPSHOT
warning: cannot send 'rpool/vm-106-disk-1@rep_default_2015-09-24_12:30:49': Relais brisé (pipe)
COMMAND:
zfs send -v rpool/vm-106-disk-1@rep_default_2015-09-24_12:30:49 | zfs recv ouragan:rpool/vm-106-disk-1@rep_default_2015-09-24_12:30:49
GET ERROR:
cannot open 'ouragan:rpool/vm-106-disk-1': dataset does not exist
cannot receive new filesystem stream: dataset does not exist





Now it seem to work from the sender side but from the receiver side I get the error « dataset does not exist » of course, it’s supposed to be created no ?


I am completely new to ZFS so surely i’m doing something wrong… for exemple I don’t understand the difference between volume and dataset i searched a lot but nothing helped me understand clearly on the web and i suspect it could be it.


Is there a way to tell the command the vm disk is not at the root of the pool ?


Thank very much if someone can help me
P.S. I’m posting in the ML too (not sure what the best place to ask)


Best regards,

 

[jeanlau]

hi,

I continue my investigations (never abandoning) if i do the command according to the wiki :

root@cyclone ~ # pve-zsync sync --source 106 --dest ouragan:rpool/BKP_24H --verbose
COMMAND:
zfs list -r -t snapshot -Ho name, -S creation rpool/vm-106-disk-1
GET ERROR:
cannot open 'rpool/vm-106-disk-1': dataset does not exist

I assume this is because my vm disk is in not at the root of the rpool... so i tried specifying the disk i want to sync :

root@cyclone ~ # pve-zsync sync --source rpool/disks/vm-106-disk-1 --dest ouragan:rpool/BKP_24H --verbose
send from @ to rpool/disks/vm-106-disk-1@rep_default_2015-09-25_16:55:51 estimated size is 1,26G
total estimated size is 1,26G
TIME SENT SNAPSHOT
warning: cannot send 'rpool/disks/vm-106-disk-1@rep_default_2015-09-25_16:55:51': Relais brisé (pipe)
COMMAND:
zfs send -v rpool/disks/vm-106-disk-1@rep_default_2015-09-25_16:55:51 | zfs recv ouragan:rpool/BKP_24H/vm-106-disk-1@rep_default_2015-09-25_16:55:51
GET ERROR:
cannot open 'ouragan:rpool/BKP_24H/vm-106-disk-1': dataset does not exist
cannot receive new filesystem stream: dataset does not exist

Always the same error from the remote side : dataset does not exist

Although if I try to create a snapshot and sending it by myself it seem to work :

root@cyclone ~ # zfs send rpool/disks/vm-106-disk-1@25-09-2015_16h58m14s | ssh -p 2223 ouragan zfs receive rpool/BKP_24H/vm-106-disk-1
root@cyclone ~ #

no error... and I can see it from the other side (even tried to boot from it and it works)

root@ouragan ~ # zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 3,39T 3,63T 96K /rpool
rpool/BKP_24H 964M 3,63T 96K /rpool/BKP_24H
rpool/BKP_24H/vm-106-disk-1 963M 3,63T 963M -
rpool/ROOT 2,37T 3,63T 96K /rpool/ROOT

root@ouragan ~ # zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
rpool/BKP_24H/vm-106-disk-1@25-09-2015_16h58m14s 0 - 963M -

but can't do it for a second snapshot :

root@cyclone ~ # zfs send rpool/disks/vm-106-disk-1@25-09-2015_17h03m07s | ssh -p 2223 ouragan zfs receive rpool/BKP_24H/vm-106-disk-1
cannot receive new filesystem stream: destination 'rpool/BKP_24H/vm-106-disk-1' exists
must specify -F to overwrite it


I can try to find how to send multiples snapshot and end up with a script made by myself but seem a little bit silly to me not to use the tool already provided...

FYI I changed the default ssh port for security reasons but in case it would be the problem i tried go back to the standard one but it changes nothing.

I remarked there is a difference between the command used by the pve-zsync script and the command that works (at least for the first snapshot)
pve-zsync :
zfs send -v rpool/disks/vm-106-disk-1@rep_default_2015-09-25_16:55:51 | zfs recv ouragan:rpool/BKP_24H/vm-106-disk-1@rep_default_2015-09-25_16:55:51
my command :
zfs send rpool/disks/vm-106-disk-1@25-09-2015_16h58m14s | ssh -p 2223 ouragan zfs receive rpool/BKP_24H/vm-106-disk-1

Tried to have a look in the pve-zsync script since it seem to be a perl script but it didn't helped me (I only practice bash shell)

If someone came by and had an idea to help me to progress I would be very grateful

Best regards,

 

[redmop]

Proxmox 3.4 does not support disk images anywhere every the root of the zpool. That is why you could not add them to the gui. I would suggest moving the zbols to the root, fixing your storage.cfg, then trying pve-zsync.

 

[jeanlau]

hello thanks for your answer I already tried a test from a vm witch has its disk directly on the rpool and it was the same but to be sure i did exactly what you said, go back to strictly standard configuration, I moved all my zvols, changed my storage.cfg, change all my vm XXX.conf, reboot all the VMs, and even restart the proxmox services, just created a dataset called backup to store the backups :

root@cyclone /localpveconf/qemu-server # cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content images,iso,vztmpl,backup,rootdir
maxfiles 3


zfspool: rpool
pool rpool
content images
sparse

root@cyclone /localpveconf/qemu-server # zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 280G 6,75T 96K /rpool
rpool/ROOT 155G 6,75T 96K /rpool/ROOT
rpool/ROOT/pve-1 155G 6,75T 155G /
rpool/backup 96K 6,75T 96K /rpool/backup
rpool/swap 33,0G 6,78T 4,64M -
rpool/vm-100-disk-1 635M 6,75T 635M -
rpool/vm-106-disk-1 965M 6,75T 963M -
rpool/vm-107-disk-1 3,61G 6,75T 3,61G -
rpool/vm-108-disk-1 9,29G 6,75T 9,29G -
rpool/vm-110-disk-1 63,9G 6,75T 63,9G -
rpool/vm-204-disk-1 13,6G 6,75T 13,6G -



root@ouragan /localpveconf/qemu-server # zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 3,39T 3,63T 96K /rpool
rpool/ROOT 2,37T 3,63T 96K /rpool/ROOT
rpool/ROOT/pve-1 2,37T 3,63T 2,37T /
rpool/backup 96K 3,63T 96K /rpool/backup
rpool/base-202-disk-1 6,37G 3,63T 6,37G -
rpool/base-203-disk-1 11,8G 3,63T 11,8G -
rpool/swap 33,0G 3,66T 9,57M -
rpool/vm-101-disk-1 621M 3,63T 621M -
rpool/vm-102-disk-1 73,9G 3,63T 73,9G -
rpool/vm-103-disk-1 5,63G 3,63T 5,63G -
rpool/vm-104-disk-1 72,9G 3,63T 72,9G -
rpool/vm-105-disk-1 776G 3,63T 776G -
rpool/vm-200-disk-1 7,64G 3,63T 7,64G -
rpool/vm-201-disk-1 46,3G 3,63T 46,3G -
rpool/vm-205-disk-1 13,2G 3,63T 13,2G -
rpool/vm-300-disk-1 617M 3,63T 617M -
rpool/vm-400-disk-1 818M 3,63T 818M -
rpool/vm-400-disk-2 135M 3,63T 135M -


root@cyclone ~ # cat /etc/pve/nodes/cyclone/qemu-server/106.conf
balloon: 256
bootdisk: virtio0
cores: 1
ide0: none,media=cdrom
memory: 1024
name: Deb-Test
net0: virtio=525:C1:5C:3F:61,bridge=vmbr1
ostype: l26
scsihw: virtio-scsi-pci
sockets: 1
virtio0: rpool:vm-106-disk-1,cache=writeback,size=5G

/etc/init.d/cman restart
/etc/init.d/pvedaemon restart
/etc/init.d/pvestatd restart
/etc/init.d/pve-manager restart
/etc/init.d/pve-cluster restart
/etc/init.d/pveproxy restart

Sadly, as expected it changed nothing :

root@cyclone ~ # pve-zsync sync --source 106 --dest ouragan:rpool/backup --verbose
send from @ to rpool/vm-106-disk-1@rep_default_2015-09-25_18:38:25 estimated size is 1,26G
total estimated size is 1,26G
TIME SENT SNAPSHOT
warning: cannot send 'rpool/vm-106-disk-1@rep_default_2015-09-25_18:38:25': Relais brisé (pipe)
COMMAND:
zfs send -v rpool/vm-106-disk-1@rep_default_2015-09-25_18:38:25 | zfs recv ouragan:rpool/backup/vm-106-disk-1@rep_default_2015-09-25_18:38:25
GET ERROR:
cannot open 'ouragan:rpool/backup/vm-106-disk-1': dataset does not exist
cannot receive new filesystem stream: dataset does not exist

Even tried directly on the root rpool...


root@cyclone ~ # pve-zsync sync --source 106 --dest ouragan:rpool --verbose
send from @ to rpool/vm-106-disk-1@rep_default_2015-09-25_18:38:32 estimated size is 1,26G
total estimated size is 1,26G
TIME SENT SNAPSHOT
warning: cannot send 'rpool/vm-106-disk-1@rep_default_2015-09-25_18:38:32': Relais brisé (pipe)
COMMAND:
zfs send -v rpool/vm-106-disk-1@rep_default_2015-09-25_18:38:32 | zfs recv ouragan:rpool/vm-106-disk-1@rep_default_2015-09-25_18:38:32
GET ERROR:
cannot open 'ouragan:rpool/vm-106-disk-1': dataset does not exist
cannot receive new filesystem stream: dataset does not exist


Maybe you have other suggestion ?

 

[redmop]

ouragan:rpool/backup/ Does this dataset exist?

 

[jeanlau]

I gave you the output of the command zfs list in my previous posts... Can't you see by yourself that it exists ?
But i'm not really sure that it's a dataset and I suspect it could be the problem because as I said I don't really understand the difference between a volume and a dataset

 

[redmop]

First. I am trying to help you. I don't actually use pve-zsync, I use another tool for this job. "Can't you see by yourself that it exists?" was unnecessary. I could easily respond "The answer lies here: https://www.google.com"



This is info on a filesystem: https://pthree.org/2012/12/17/zfs-administration-part-x-creating-filesystems/



This is info on volumes: https://pthree.org/2012/12/21/zfs-administration-part-xiv-zvols/



All in all, that site in general is very good for zfs.



Something to note: The instructions here: https://pve.proxmox.com/wiki/PVE-zsync do not use hostnames, but IP address. I am quite certain that the zfs filesystem 'ouragan:rpool/backup/vm-106-disk-1' does not exist, but instead 'rpool/backup/vm-106-disk-1' on the host ouragan.

Try this change:

from: "pve-zsync sync --source 106 --dest ouragan:rpool/backup --verbose"

to: "pve-zsync sync --source 106 --dest <ouragan ip address>:rpool/backup --verbose"



If that fixes your issue, Please file a bug report here: https://bugzilla.proxmox.com/ so they can fix it hostname resolution.

 

[jeanlau]

sorry wasn't mean to hurt your feelings... I appreciate your hep

The website you give me i already saw it but if you search, the is nowhere where you can find precisely how to create a dataset, the volume creation you can find it everywhere, I already found websites that explained that a dataset is basically a volume that contains others so I assumed that you just create the same way you create a volume with "zfs create" command but that i'm not really sure but I have searched hours and haven't found a specific command to create a dataset.

for you other suggestion, the servers are supposed to know each other, both their names with IP are in the hosts file so, when i use the name the system knows perfectly what address it is but I tried you suggestion even if it's supposed to be the exact same thing and it worked !!!

this is not normal that the system do not use resolution but it is the way it works !!! so a BIG thank you, you made my day (or evening now)

thank you again !

If a dev come by, it should be corrected and at least the wiki should precise that only ip addresses can be used to avoid other like me to search days and days for a silly thing like this...

anyway thank you i'm thrilled that it works

 

[redmop]

(Note, on my phone)

This makes a dataset/subvol/filesystem: zfs create <pool name>/<dataset/subvolume>

Datasets are formatted filesystems they agree also used for organization

This makes a zvol (this is that you are looking at with vm-106-disk-1)
zfs create -V <size> <pool name>/<dataset/subvolume>/<zvol>

Zvols are like partitions. They are just raw data. You can partition them, format them, etc. They can be created under datasets.

 

[jeanlau]

Hello,

If you look at the man you will see that the -F option is only used to assign a size to the volume but nothing says that it creates a dataset rather than a volume.
What i found during my researches is that a dataset seem to basically be a volume that contains others. So, like you say, I assume that you can just create datasets the same way you create a volumes with "zfs create" the only difference is that it is under another volume....

Do you know how to close the tread and say it's resolved ?

P.S. I updated the wiki to avoid other people to struggle like i have and i really don't know how you had the idea to use ip addresses instead of hostnames... personally i had no clues...

 

[jeanlau]

Hello

If someone is interested i made cool script. What the script make is to keep in sync in both the hosts, create a new log file everyday to log all the sync and send you an email containing the log file if something bad happens. In fact it's a loop and the goal is to always have the most recent copies of the vm disk on both sides. Almost as interesting than DRBD but without the split-brain complications
In my case for exemple i have approximately 15 KVM VM witch are not to much solicited and the script need 1 minute to make a loop, I think during solicited period maybe 2 or 3 minutes, surely less than 5... It's all new so i have not experience with it, if someone use it i would be very happy if he let me know how it works for him.

It's made to work almost "out of the box"" in a full ZFS Proxmox installation in a two hosts cluster only, if your configuration is different you will have to adapt it...

You just have to verify that you have following packages installed : pve-zsync and screen, you will have to put your mail address in the var monmail at the beginning of the script.

Sorry all the comments in the script are in french, hope you will understand

#!/bin/bash


monmail="admin@mydomain.com"


gosync() {
##On commence la boucle
while true;do
## Creation du log (on verifie que le dossier des logs est cree avant)
if [ ! -d "/var/log/syncro" ];then
mkdir -p /var/log/syncro
fi
logfic="/var/log/syncro/syncro-`date '+%d-%m-%Y'`.log"
##On detecte sur quelle machine on se trouve et quelle est la machine distante
loc=`hostname`
dist=`ls /etc/pve/nodes/ | grep -v $loc`
###On recupere les ID des VM qui utilisent zfs locales puis de VM distantes
vmloc=`grep rpool /etc/pve/nodes/$loc/qemu-server/*.conf | cut -d / -f 7 | cut -d . -f 1`
vmdist=`grep rpool /etc/pve/nodes/$dist/qemu-server/*.conf | cut -d / -f 7 | cut -d . -f 1`
###On recupere l'IP de l'hote distant
ipdist=$(ping -c 1 $dist | gawk -F'[()]' '/PING/{print $2}')
##On vérifie la présence du répertoire des hotes du cluster
if [ ! -d "/etc/pve/nodes/" ]; then
echo "PB avec le cluster a `date '+%d-%m-%Y_%Hh%Mm%Ss'`" >> $logfic
##On laisse une trace d'envoi de mail et on l'envoie
if [ $logfic != `cat /tmp/mail.tmp` ];then
echo $logfic > /tmp/mail.tmp
cat $logfic | mail -s "PB Syncro ZFS" $monmail;
fi
fi


echo "syncro des machines de $loc vers $dist" >> $logfic
for n in $vmloc
do
if test -f "/tmp/stopsync.req"
then
rm /tmp/stopsync.req
touch /tmp/stopsync.ok
exit 0
else
echo "debut syncro de la machine $n a `date '+%d-%m-%Y_%Hh%Mm%Ss'`" >> $logfic
pve-zsync sync --source $n --dest $ipdist:rpool/lastsync --maxsnap 1 --verbose >> $logfic
if test ${?} -eq 0 ; then
echo "syncro de la machine $n finie a `date '+%d-%m-%Y_%Hh%Mm%Ss'`" >> $logfic
else
##On laisse une trace d'envoi de mail et on l'envoie
if [ $logfic != `cat /tmp/mail.tmp` ];then
echo $logfic > /tmp/mail.tmp
cat $logfic | mail -s "PB Syncro ZFS" $monmail;
fi
fi
fi
done


echo "syncro des machines de $dist vers $loc" >> $logfic
for n in $vmdist
do
if test -f "/tmp/stopsync.req"
then
rm /tmp/stopsync.req
touch /tmp/stopsync.ok
exit 0

else
echo "debut syncro de la machine $n a `date '+%d-%m-%Y_%Hh%Mm%Ss'`" >> $logfic
pve-zsync sync --source $ipdist:$n --dest rpool/lastsync --maxsnap 1 --verbose >> $logfic
if test ${?} -eq 0 ; then
echo "syncro de la machine $n finie a `date '+%d-%m-%Y_%Hh%Mm%Ss'`" >> $logfic
else
##On laisse une trace d'envoi de mail et on l'envoie
if [ $logfic != `cat /tmp/mail.tmp` ];then
echo $logfic > /tmp/mail.tmp
cat $logfic | mail -s "PB Syncro ZFS" $monmail;
fi
fi
fi


done


done
}


stop() {
touch /tmp/stopsync.req
##On commence une nouvelle boucle pour attendre que la syncro en cours soit finie
while true;do
if test -f "/tmp/stopsync.ok"
then
echo "Arret de la syncro : OK"
##Et l'arret du script en lui meme
rm /tmp/stopsync.ok
kill $$
exit 0
else
echo "Arret en cours..."
echo "la syncronisation en cours se finit, cela peut durer un peu..."
sleep 3
fi
done
}


case "$1" in
gosync)
gosync
;;
start)
screen -d -m -S syncro-zfs bash -c '/root/scripts/syncro-zfs gosync'
echo "Lancement de la syncronisation : OK"
echo "taper la commande 'screen -r syncro-zfs' pour voir la sortie standard."
;;
stop)
stop
;;
*)
echo "Usage: $0 {start|stop}" >&2
exit 1
;;
esac

Hope you will like it, please let me know.

Best regards,