One of clustered node down, storage is clustered , how can we retrieve the VM on the down machine to another node

Aug 16, 2023
Dear All,
Good morning
We have a PVE 8 , ceph based clustered setup. One of the node is being down
Now we have some VM images created on the node which has been down, This VM has storage stored on the CEPH clustered
Like to request guidance on how to retrieved the instance from the CEPH storage and deply to the other nodes
Joseph John
With ceph you data is NOT local but distributed on all ceph nodes. As log as your ceph is working and has all the data blocks available due to redundancy you can access it.

Please post the output of
pvecm status
ceph -s
qm config ID
Like to request guidance on how to retrieved the instance from the CEPH storage and deply to the other nodes
If you set everything up correctly, just start the VMs on another node. Normally you would setup HA-VMs so that they automatically failover to another node. This is what clustering is about.
pvecm status

Cluster information ------------------- Name: rnd-cluster Config Version: 8 Transport: knet Secure auth: on Quorum information ------------------ Date: Mon Oct 2 10:28:53 2023 Quorum provider: corosync_votequorum Nodes: 7 Node ID: 0x00000001 Ring ID: 1.1447 Quorate: Yes Votequorum information ---------------------- Expected votes: 8 Highest expected: 8 Total votes: 7 Quorum: 5 Flags: Quorate Membership information ---------------------- Nodeid Votes Name 0x00000001 1 (local) 0x00000002 1 0x00000003 1 0x00000004 1 0x00000005 1 0x00000006 1 0x00000008 1

Now for ceph -s

ceph -s
    id:     6346d7b8-713e-4a84-be38-7fd483f49da0
    health: HEALTH_WARN
            7 osds down
            1 host (8 osds) down
            Degraded data redundancy: 1136806/7993830 objects degraded (14.221%), 475 pgs degraded, 475 pgs undersized
    mon: 3 daemons, quorum pve-1,pve-2,pve-3 (age 7w)
    mgr: pve-1(active, since 7w)
    osd: 64 osds: 40 up (since 23h), 47 in (since 23h); 44 remapped pgs
    pools:   2 pools, 1114 pgs
    objects: 2.66M objects, 10 TiB
    usage:   30 TiB used, 39 TiB / 68 TiB avail
    pgs:     1136806/7993830 objects degraded (14.221%)
             113601/7993830 objects misplaced (1.421%)
             595 active+clean
             475 active+undersized+degraded
             44  active+clean+remapped
    client:   944 KiB/s rd, 53 MiB/s wr, 24 op/s rd, 2.65k op/s wr
    Global Recovery Event (0h)
      [================............] (remaining: 18h)

Now for qm config

qm config 118
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_NAME = "en_GB.UTF-8",
LC_PAPER = "en_GB.UTF-8",
LC_TIME = "en_GB.UTF-8",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
Configuration file 'nodes/pve-1/qemu-server/118.conf' does not exist

Joseph John
You can do dpkg-reconfigure locales to fix this locale errors, if not found apt install locales first.

Try (i hope i got the path right) to move the qemu config to another node, then start the vm
mv /etc/pve/nodes/FAILED_NODE/qemu/118.conf /etc/pve/nodes/THIS_NODE/qemu/118.conf
  • Like
Reactions: joseph.john


The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!