How to extract data from LVM (RAW), there is a cluster not working.

[ogankvik]

Hello. I do not understand virtualization, but I have to solve the problem.

We have a blade server running two servers clustered together. The network suddenly stopped working on the server and now the server do not see each other. The cluster is down, the server cannot be started. There is important data on the server that us need to get.

Can you tell me how can I extract the data? I can physically connect to each server individually and have sudo rights.

Thanks!

 

[Dunuin]

We have a blade server running two servers clustered together. The network suddenly stopped working on the server and now the server do not see each other. The cluster is down, the server cannot be started. There is important data on the server that us need to get.

You need atleast 3 nodes for a cluster. Otherwise both nodes will stop working when any of the two nodes got a problem, as there is no quorum.

Can you tell me how can I extract the data?

Make sure the VM isn't running. Then you can mount the LV on the host like any other block storage. Something like mount -t whateverFilesystemThatLVisFormatedWith /dev/YourVG/YourLV /path/to/mount. And don't forget to unmount it before starting a VM, so you don't corrupt your data.

 

[VictorSTS]

If you have no quorum VM should still work. You will just be unable to most actions in the cluster.

You can issue pvecm expected 1 to regain quorum and be able to start your VM. Also, post the output of command pvecm status on the working node so we can check cluster configuration.

 

[ogankvik]

pvecm expected 1

Executed the command. When starting the server, an error appears:

TASK ERROR: volume 'local:101/vm-101-disk-1.qcow2' does not exist

[root@svh-01 ~]# pvecm status
Quorum information
------------------
Date:             Mon Feb 27 08:18:51 2023
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          0x00000001
Ring ID:          1/964
Quorate:          No


Votequorum information
----------------------
Expected votes:   2
Highest expected: 2
Total votes:      1
Quorum:           2 Activity blocked
Flags:           


Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 10.1.1.1 (local)

It needs to be clarified. The network interface in the server is damaged, it does not see the network, the servers inside the blade also do not see each other over the network. I can physically connect the keyboard and monitor to the server, but the network does not work at all.

 

[VictorSTS]

Ok, you have a 2 node cluster. You can run pvecm expected 1 on each node so each one gets quorum. After this, pvecm status should return "Quorate". If you have a backup of the VM you an restore it now.

If you dont... TASK ERROR: volume 'local:101/vm-101-disk-1.qcow2' does not exist is telling you that VM101 on server shv-01 can't find a disk in a storage called "local", which has nothing to do with the LVM storage "LVM_BD".

Are all disks local to the blades? I mean, they are not some kind of iSCSI disks which use the broken nics?

Run this on each node:

pveversion
pvesm status
pvecm list local
pvecm list LVM_BD
pvdisplay
lsblk

Also run pvereport on each node and attach the output here.

 

[ogankvik]

Thanks all! Problem different. Network card is brocken, after repair server again online.