Now that I pretty sure it is watchdog. Basically all previous event also same root cause
root@pve1:/var/log# journalctl -u watchdog-mux | grep expired
Dec 04 06:10:31 pve1 watchdog-mux[1827]: client watchdog expired - disable watchdog updates
Dec 27 05:50:50 pve1 watchdog-mux[1824]: client...
Pretty much watchdog root cause. I had this issue multiple times. it all random, When other node return from poweroff, pve1 reloaded.
I also check replicated all completed before Sep 01 15:59:59.
Sep 01 15:57:15 pve1 corosync[2987]: [KNET ] rx: host: 2 link: 0 is up
Sep 01 15:57:15 pve1...
Hi.
Thanks you for your reply
Unfortunately in this case, 3 node is running and pve1 suddenly reloaded. Is there a way i can check the log confirm it is reloaded due to fencing?
If you see the log, there are 3 node
Sep 01 15:57:24 pve1 corosync[2987]: [QUORUM] Sync joined[1]: 3
Nothing...
Pve1 was running pvecm e 1.
pve2 and pve3 were shutdown and brought up together.
How do check logs PVE1 reloaded due to losing quorum ?
I brought up pve2 and pve3
15:57:15 -- pve2 started
15:57:24 -- pve3 started
16:09:07 -- Pve1 lost watchdog and reloaded(Losing 3 HA VM)
PVE1
Aug 31...
I have some question regarding fencing and node force reboot which recently cause outage to my cluster.
Having 3(let call 1,2,3) node cluster, all same version 8.0.3
node 3 having some issue thus was shutdown. No vm is running
node 2 is out for maintenance few days ago
node 1 is running...
I edit the datastore.cfg
I can see the datastore in PBS.
I can create new backup to datastore from Proxmox
however i cant see all the previous backup data.
Update:
I notice datastore use some kind of encoding as it was mount from samba. Rename it seems to fix the issue .
I am sharing incase...
My machine has 16GB ram(Virtual Environment 7.2-7).
VM1 given 2GB and 6GB ballooning enabled(qemu agent enabled).
During midnight VM1 will run backup which consuming all 6GB and mostly cache
total used free shared buff/cache available
Mem: 5035...
To answer to my own question
Create a copy of /proc/cmdline to /etc/kernel/cmdline
add to end of the line
rootflags=rw,degraded
eg.
initrd=\EFI\proxmox\5.13.19-2-pve\initrd.img-5.13.19-2-pve BOOT_IMAGE=/boot/vmlinuz-5.13.19-2-pve root=UUID=e115f582-3078-4d22-a8e3-xxxxxxxxxxx ro quiet...
I simulate disk fail on the raid1 btrfs which is also my root for proxmox ve 7.1-7.
When one of the disk failed, the system will not boot.
It stuck at initramfs, i am unable to mount the system as the initramfs does not have mount.btrfs
The system will boot normally with both disk exits...
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.