One of clustered node down, storage is clustered , how can we retrieve the VM on the down machine to another node

[joseph.john]

Dear All,
Good morning
We have a PVE 8 , ceph based clustered setup. One of the node is being down
Now we have some VM images created on the node which has been down, This VM has storage stored on the CEPH clustered
Like to request guidance on how to retrieved the instance from the CEPH storage and deply to the other nodes
thanks
Joseph John

 

[ubu]

With ceph you data is NOT local but distributed on all ceph nodes. As log as your ceph is working and has all the data blocks available due to redundancy you can access it.

Please post the output of
pvecm status
ceph -s
qm config ID

 

[LnxBil]

Like to request guidance on how to retrieved the instance from the CEPH storage and deply to the other nodes

If you set everything up correctly, just start the VMs on another node. Normally you would setup HA-VMs so that they automatically failover to another node. This is what clustering is about.

 

[joseph.john]

pvecm status

root@pve-1:~#

Cluster information ------------------- Name: rnd-cluster Config Version: 8 Transport: knet Secure auth: on Quorum information ------------------ Date: Mon Oct 2 10:28:53 2023 Quorum provider: corosync_votequorum Nodes: 7 Node ID: 0x00000001 Ring ID: 1.1447 Quorate: Yes Votequorum information ---------------------- Expected votes: 8 Highest expected: 8 Total votes: 7 Quorum: 5 Flags: Quorate Membership information ---------------------- Nodeid Votes Name 0x00000001 1 10.115.129.51 (local) 0x00000002 1 10.115.129.52 0x00000003 1 10.115.129.53 0x00000004 1 10.115.129.54 0x00000005 1 10.115.129.56 0x00000006 1 10.115.129.57 0x00000008 1 10.115.129.55

Now for ceph -s

ceph -s
  cluster:
    id:     6346d7b8-713e-4a84-be38-7fd483f49da0
    health: HEALTH_WARN
            7 osds down
            1 host (8 osds) down
            Degraded data redundancy: 1136806/7993830 objects degraded (14.221%), 475 pgs degraded, 475 pgs undersized
 
  services:
    mon: 3 daemons, quorum pve-1,pve-2,pve-3 (age 7w)
    mgr: pve-1(active, since 7w)
    osd: 64 osds: 40 up (since 23h), 47 in (since 23h); 44 remapped pgs
 
  data:
    pools:   2 pools, 1114 pgs
    objects: 2.66M objects, 10 TiB
    usage:   30 TiB used, 39 TiB / 68 TiB avail
    pgs:     1136806/7993830 objects degraded (14.221%)
             113601/7993830 objects misplaced (1.421%)
             595 active+clean
             475 active+undersized+degraded
             44  active+clean+remapped
 
  io:
    client:   944 KiB/s rd, 53 MiB/s wr, 24 op/s rd, 2.65k op/s wr
 
  progress:
    Global Recovery Event (0h)
      [================............] (remaining: 18h)

Now for qm config

qm config 118
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_ADDRESS = "en_GB.UTF-8",
LC_NAME = "en_GB.UTF-8",
LC_MONETARY = "en_GB.UTF-8",
LC_PAPER = "en_GB.UTF-8",
LC_IDENTIFICATION = "en_GB.UTF-8",
LC_TELEPHONE = "en_GB.UTF-8",
LC_MEASUREMENT = "en_GB.UTF-8",
LC_TIME = "en_GB.UTF-8",
LC_NUMERIC = "en_GB.UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
Configuration file 'nodes/pve-1/qemu-server/118.conf' does not exist

Thanks
Joseph John

 

[ubu]

You can do dpkg-reconfigure locales to fix this locale errors, if not found apt install locales first.

Try (i hope i got the path right) to move the qemu config to another node, then start the vm
mv /etc/pve/nodes/FAILED_NODE/qemu/118.conf /etc/pve/nodes/THIS_NODE/qemu/118.conf

 

[joseph.john]

Try (i hope i got the path right) to move the qemu config to another node, then start the vm
mv /etc/pve/nodes/FAILED_NODE/qemu/118.conf /etc/pve/nodes/THIS_NODE/qemu/118.conf

THANKS A LOT
Much appreciated
it worked !!