HA cluster with two node and qDevice + CEPH: don't work HA: why?

C

cybermod

Active Member
Sep 21, 2019
5
1
43
46
Hi everyone, I'm here because I think I'm missing something important.I have read and watched many guides and videos, but I can't get the HA to work when I "kill" one of the two nodes.The quorum remains respected, I set the number of replies to the minimum. Yet I get an error that I can't interpret.Maybe I'm just getting the basics wrong?Thank you
 
cybermod said:
Yet I get an error that I can't interpret
Click to expand...
Letting us know what error that is would be helpful.

How many Ceph MONs do you have? Because Ceph needs to also have Quorum on the MONs (similar to PVE), you need a majority of MONs present. In a 2-node cluster.... One of the reasons why Ceph is only useful once you have at least 3 full nodes!
 
dear, tnx for your reply!
my configurations is:
2 node + qDevice on Ubuntu Server 22.04.
I have two monitor.....
So.... i can't use CEPH for HA because i have not 3 full node but only 2 full node + 1 qDevice....
i'm confused....
what is the point of having qdevice?
Can i have HA on 2 nodes?
maybe...i need a shared external storage (like nas) and like vmware.

TASK ERROR: start failed: command '/usr/bin/kvm -id 100 -name 'ubusrvHAlabtest,debug-threads=on' -no-shutdown -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server=on,wait=off' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=f0349f09-e1c4-47ef-a938-5c15b2464175' -smp '4,sockets=1,cores=4,maxcpus=4' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc 'unix:/var/run/qemu-server/100.vnc,password=on' -cpu qemu64,+aes,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+pni,+popcnt,+sse4.1,+sse4.2,+ssse3 -m 2048 -object 'iothread,id=iothread-virtioscsi0' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.3,chassis_nr=3,bus=pci.0,addr=0x5' -device 'vmgenid,guid=39c0cf2e-d3c2-413a-84f4-cfee196311df' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'VGA,id=vga,bus=pci.0,addr=0x2' -chardev 'socket,path=/var/run/qemu-server/100.qga,server=on,wait=off,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:8cdf10601a72' -drive 'if=none,id=drive-ide2,media=cdrom,aio=io_uring' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=101' -device 'virtio-scsi-pci,id=virtioscsi0,bus=pci.3,addr=0x1,iothread=iothread-virtioscsi0' -drive 'file=rbd:ceph01/vm-100-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/ceph01.keyring,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' -device 'scsi-hd,bus=virtioscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=BC:24:11:9C:33:41,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,bootindex=102' -machine 'type=pc+pve0'' failed: got timeout
 
Last edited:
cybermod said:
what is the point of having qdevice?
Click to expand...
You need a quorum for the PVE cluster, this where you can use a qdevice.
Ceph needs a separate, its own quorum. The Ceph monitors are the voters in this case.

So currently you've got a quorate PVE cluster (2/3), but Ceph is without quorum (1/2).
 
  • Like
Reactions: cybermod
cybermod said:
what is the point of having qdevice?
Can i have HA on 2 nodes?
Click to expand...
sure, but not with Ceph. Either use a network share, or alternatively local ZFS + the guest replication feature. The downside of the ZFS + Replication approach is, that it is async. In a worst case scenario, you might lose some data (since the last successful replication). Though it is possible to run the replication as short as each minute.
 
  • Like
Reactions: cybermod
aaron said:
sure, but not with Ceph. Either use a network share, or alternatively local ZFS + the guest replication feature. The downside of the ZFS + Replication approach is, that it is async. In a worst case scenario, you might lose some data (since the last successful replication). Though it is possible to run the replication as short as each minute.
Click to expand...
tnx, i tried it and now i feel myselfe moore confidence.
tnx for your clarify me
really appreciate
 
  • Like
Reactions: aaron
hi Gilberto, sorry for my delay.
So, you say to use GlusterFs. Is GlusterFS an external storage?
Best regards

PVT for more info.
Click to expand...

ops, sorry. I write you
 
Last edited:
kotakomputer said:
Is there something like QDevice for ceph?
Click to expand...
No, you could add another node to your cluster just for quorum ( so no vms and no storage on it) though. Whether this is a good idea I don't know, maybe the Cephperts here can say something about it?
Edit: You will still need at least three storage nodes, my proposal is more for cases where you have like forum storage nodes and want to avoid a split-brain scenario. Still not sure whether this is actually a good idea
 
Last edited:
  • Like
Reactions: UdoB
You must log in or register to reply here.
Top Bottom