Attaching LV to VM fails with "failed to update VM 199: not enough free space (0 < 33554432) (500)"

[lpallard]

Long story short, I tried renaming the hostname of my PVE single host (no cluster) and it turned into a nightmare. If I'd knew I would have NEVER tried this...

Please bear in mind "old" hostname was "proxmox" and new hostname is "srv-vir01"

Basically I followed https://pve.proxmox.com/wiki/Renaming_a_PVE_node

Steps taken to rename hostname:

• nano /etc/hosts
• nano /etc/hostname
• nano /etc/mailname
• nano /etc/postfix/main.cf
• nano /etc/network/interfaces

Then I tried to move the node's files using mv:

mv /etc/pve/nodes/proxmox /etc/pve/nodes/srv-vir01

but it failed (directory not empty)...

I then tried to use rsync:

rsync -av /etc/pve/nodes/proxmox/ /etc/pve/nodes/srv-vir01/

It also failed. There was a bunch of errors like:

mkstemp /etc/pve/nodes//qemu-server/..conf failed: Operation not permitted (1)

Finally I rsync'ed "/etc/pve/nodes/proxmox" to an NFS server, deleted the folder "/etc/pve/nodes/proxmox" then re-rsync'ed it back into "/etc/pve/nodes/"

I then compared initial and final folders, same file count and same disk usage. OK seems good enough for now...

Then I copied the contents of "/var/lib/rrdcached/db/pve2-node/proxmox" to "/var/lib/rrdcached/db/pve2-node/srv-vir01" and deleted the old folder. I did the same for the other folder (storage), then rebooted the server.

The server came back with the new hostname but all VM's were GONE! I can see the LV's and other NFS mounts in the web-interface but all VM's are gone...

I verified that the RAW VM disk files were still in the VM storage LV (in my case "local-lvm") and indeed they all are.

Instead of troubleshooting this, I decided to perform a restore of VM199 to see if it would work which it did. However the issue I have now is that it doesnt seem to re-attach a logical volume (highperf_lv2) I manually created a while ago. The CONF file for this VM explicitly instruct PVE to attach this logical volume to the VM (see virtio1).

199.conf

balloon: 0
boot: order=virtio0;net0
cores: 2
memory: 24576
name: database
net0: virtio=7E:9D:C6:85:75:3C,bridge=vmbr0,firewall=1
numa: 1
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=fcbe1b55-d874-4f36-9d1e-588b60e8ccf3
sockets: 2
startup: order=2,up=60
virtio0: local-lvm:vm-199-disk-0,size=10G
virtio1: /dev/highperf/highperf_lv2,backup=no,size=158.46G
vmgenid: ca1339a5-a0f3-483f-9322-adabd37562e6

I can see the logical volume "highperf_lv2" in the web-interface but it is FULL (299GB of 299GB).

lvdisplay

--- Logical volume ---
  LV Path                /dev/highperf/highperf_lv2
  LV Name                highperf_lv2
  VG Name                highperf
  LV UUID                eghXF5-2V98-Onn6-6Y48-iLSL-ZleB-lYRoQi
  LV Write Access        read/write
  LV Creation host, time proxmox, 2015-07-01 21:19:49 -0400
  LV Status              available
  # open                 0
  LV Size                158.46 GiB
  Current LE             40566
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:12

I dont understand why "highperf_lv2" is not re-attached to the VM as it should... If I try to attach it manually (Hardware > Add > Hard Disk > Storage: highperf_lv2") and put 158.46GB in the disk size, I always get the following error:

failed to update VM 199: not enough free space (0 < 165675008) (500)

I tried with smaller sizes to no avail.

So either I manage to restore all VM's (after all nothing was erased or deleted) or I manage to re-attach "highperf_lv2" to VM199, do a backup of its content then wipe the server clean and restart fresh....

What a mess...

Any help appreciated

 

[lpallard]

OK I'm both stupid and lucky...

When I recopied the /etc/pve/nodes/ I accidentally included the domain in the folder name.... PVE could not find the conf files because the folder name didnt match the node name....

After renaming the folder, and rebooted PVE, everything came back as normal.