Hello,
I have upgrade my nodes 2 weeks ago and today, after rebooting, one of my 3 nodes have a serious problem. The VMs can't reboot because they disappeared of the storage. I'm using DRBD.
First, I notice that the volumes wasn't sync. I had this message on all of the nodes.
root@px3:~# drbdmanage list-resources
Waiting for server: ...............
Error: Startup not successful (no quorum? not *both* nodes up in a 2 node cluster?)
I try this command: drbdmanage reelect
Now, I can see only the VM from node 1 and not all of them.
root@px3:~# drbdmanage list-volumes
+--------------------------------------------------------------------------------------------+
| Name | Vol ID | Size | Minor | | State |
|--------------------------------------------------------------------------------------------|
| vm-100-disk-1 | 0 | 20 GiB | 102 | | ok |
| vm-100-disk-2 | 0 | 400 GiB | 119 | | ok |
| vm-113-disk-1 | 0 | 32 GiB | 131 | | ok |
| vm-114-disk-1 | 0 | 32 GiB | 132 | | ok |
| vm-115-disk-1 | 0 | 40 GiB | 107 | | ok |
| vm-120-disk-1 | 0 | 80 GiB | 135 | | ok |
| vm-121-disk-1 | 0 | 32 GiB | 136 | | ok |
| vm-122-disk-1 | 0 | 32 GiB | 137 | | ok |
| vm-123-disk-1 | 0 | 32 GiB | 139 | | ok |
| vm-126-disk-1 | 0 | 50 GiB | 138 | | ok |
| vm-128-disk-1 | 0 | 8 GiB | 141 | | ok |
| vm-129-disk-1 | 0 | 80 GiB | 142 | | ok |
| vm-130-disk-1 | 0 | 300 GiB | 143 | | ok |
| vm-130-disk-2 | 0 | 300 GiB | 144 | | ok |
| vm-131-disk-1 | 0 | 50 GiB | 146 | | ok |
| vm-201-disk-1 | 0 | 5 GiB | 109 | | ok |
| vm-226-disk-1 | 0 | 100 GiB | 147 | | ok |
+--------------------------------------------------------------------------------------------+
root@px1:~# drbdmanage list-volumes
+----------------------------------------------------------------------------------------------+
| Name | Vol ID | Size | Minor | | State |
|----------------------------------------------------------------------------------------------|
| vm-100-disk-1 | 0 | 20 GiB | 102 | | ok |
| vm-100-disk-2 | 0 | 400 GiB | 119 | | ok |
| vm-113-disk-1 | 0 | 32 GiB | 131 | | ok |
| vm-114-disk-1 | 0 | 32 GiB | 132 | | ok |
| vm-115-disk-1 | 0 | 40 GiB | 107 | | ok |
| vm-120-disk-1 | 0 | 80 GiB | 135 | | ok |
| vm-121-disk-1 | 0 | 32 GiB | 136 | | ok |
| vm-122-disk-1 | 0 | 32 GiB | 137 | | ok |
| vm-123-disk-1 | 0 | 32 GiB | 139 | | ok |
| vm-126-disk-1 | 0 | 50 GiB | 138 | | ok |
| vm-128-disk-1 | 0 | 8 GiB | 141 | | ok |
| vm-129-disk-1 | 0 | 80 GiB | 142 | | ok |
| vm-130-disk-1 | 0 | 300 GiB | 143 | | ok |
| vm-130-disk-2 | 0 | 300 GiB | 144 | | ok |
| vm-131-disk-1 | 0 | 50 GiB | 146 | | ok |
| vm-201-disk-1 | 0 | 5 GiB | 109 | | ok |
| vm-226-disk-1 | 0 | 100 GiB | 147 | | ok |
+----------------------------------------------------------------------------------------------+
From proxmox I see that the storage are full(2To) but it's not regardind to amount of data from above. So I have the hope that the others VMs are still there. But I dont know how to proceed.
I specify that the support has just ended. So I just have you to help me.
Have a good day.
I have upgrade my nodes 2 weeks ago and today, after rebooting, one of my 3 nodes have a serious problem. The VMs can't reboot because they disappeared of the storage. I'm using DRBD.
First, I notice that the volumes wasn't sync. I had this message on all of the nodes.
root@px3:~# drbdmanage list-resources
Waiting for server: ...............
Error: Startup not successful (no quorum? not *both* nodes up in a 2 node cluster?)
I try this command: drbdmanage reelect
Now, I can see only the VM from node 1 and not all of them.
root@px3:~# drbdmanage list-volumes
+--------------------------------------------------------------------------------------------+
| Name | Vol ID | Size | Minor | | State |
|--------------------------------------------------------------------------------------------|
| vm-100-disk-1 | 0 | 20 GiB | 102 | | ok |
| vm-100-disk-2 | 0 | 400 GiB | 119 | | ok |
| vm-113-disk-1 | 0 | 32 GiB | 131 | | ok |
| vm-114-disk-1 | 0 | 32 GiB | 132 | | ok |
| vm-115-disk-1 | 0 | 40 GiB | 107 | | ok |
| vm-120-disk-1 | 0 | 80 GiB | 135 | | ok |
| vm-121-disk-1 | 0 | 32 GiB | 136 | | ok |
| vm-122-disk-1 | 0 | 32 GiB | 137 | | ok |
| vm-123-disk-1 | 0 | 32 GiB | 139 | | ok |
| vm-126-disk-1 | 0 | 50 GiB | 138 | | ok |
| vm-128-disk-1 | 0 | 8 GiB | 141 | | ok |
| vm-129-disk-1 | 0 | 80 GiB | 142 | | ok |
| vm-130-disk-1 | 0 | 300 GiB | 143 | | ok |
| vm-130-disk-2 | 0 | 300 GiB | 144 | | ok |
| vm-131-disk-1 | 0 | 50 GiB | 146 | | ok |
| vm-201-disk-1 | 0 | 5 GiB | 109 | | ok |
| vm-226-disk-1 | 0 | 100 GiB | 147 | | ok |
+--------------------------------------------------------------------------------------------+
root@px1:~# drbdmanage list-volumes
+----------------------------------------------------------------------------------------------+
| Name | Vol ID | Size | Minor | | State |
|----------------------------------------------------------------------------------------------|
| vm-100-disk-1 | 0 | 20 GiB | 102 | | ok |
| vm-100-disk-2 | 0 | 400 GiB | 119 | | ok |
| vm-113-disk-1 | 0 | 32 GiB | 131 | | ok |
| vm-114-disk-1 | 0 | 32 GiB | 132 | | ok |
| vm-115-disk-1 | 0 | 40 GiB | 107 | | ok |
| vm-120-disk-1 | 0 | 80 GiB | 135 | | ok |
| vm-121-disk-1 | 0 | 32 GiB | 136 | | ok |
| vm-122-disk-1 | 0 | 32 GiB | 137 | | ok |
| vm-123-disk-1 | 0 | 32 GiB | 139 | | ok |
| vm-126-disk-1 | 0 | 50 GiB | 138 | | ok |
| vm-128-disk-1 | 0 | 8 GiB | 141 | | ok |
| vm-129-disk-1 | 0 | 80 GiB | 142 | | ok |
| vm-130-disk-1 | 0 | 300 GiB | 143 | | ok |
| vm-130-disk-2 | 0 | 300 GiB | 144 | | ok |
| vm-131-disk-1 | 0 | 50 GiB | 146 | | ok |
| vm-201-disk-1 | 0 | 5 GiB | 109 | | ok |
| vm-226-disk-1 | 0 | 100 GiB | 147 | | ok |
+----------------------------------------------------------------------------------------------+
From proxmox I see that the storage are full(2To) but it's not regardind to amount of data from above. So I have the hope that the others VMs are still there. But I dont know how to proceed.
I specify that the support has just ended. So I just have you to help me.
Have a good day.
