Hi,
My VM and CT's stopped working today, after taking a look i found out that my local lvm is full.
I must admit I'm pretty much a noob and quite lost and terrified of losing the data found on three (vm100/ct101/ct105) of the CT's.
The problems seems that I have somehow managed to allocate to much space to vm 105.
Here are the commands I ran so far:
(added a usb drive to try to fix it temporarly /dev/sda)
LSBLK:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 1 115.5G 0 disk
├─USB1-USB1_tmeta 252:4 0 1.2G 0 lvm
│ └─USB1-USB1-tpool 252:6 0 113.1G 0 lvm
│ └─USB1-USB1 252:7 0 113.1G 1 lvm
└─USB1-USB1_tdata 252:5 0 113.1G 0 lvm
└─USB1-USB1-tpool 252:6 0 113.1G 0 lvm
└─USB1-USB1 252:7 0 113.1G 1 lvm
nvme0n1 259:0 0 931.5G 0 disk
├─nvme0n1p1 259:1 0 1007K 0 part
├─nvme0n1p2 259:2 0 1G 0 part /boot/efi
└─nvme0n1p3 259:3 0 930.5G 0 part
├─pve-swap 252:0 0 7.4G 0 lvm [SWAP]
├─pve-root 252:1 0 96G 0 lvm /
├─pve-data_tmeta 252:2 0 8.1G 0 lvm
│ └─pve-data-tpool 252:8 0 794.9G 0 lvm
│ ├─pve-data 252:9 0 794.9G 1 lvm
│ ├─pve-vm--101--disk--0 252:10 0 50G 0 lvm
│ ├─pve-vm--102--disk--0 252:11 0 4G 0 lvm
│ ├─pve-vm--100--disk--0 252:12 0 4M 0 lvm
│ ├─pve-vm--100--disk--1 252:13 0 64G 0 lvm
│ ├─pve-vm--103--disk--0 252:14 0 8G 0 lvm
│ ├─pve-vm--104--disk--0 252:15 0 2G 0 lvm
│ └─pve-vm--105--disk--0 252:16 0 810G 0 lvm
└─pve-data_tdata 252:3 0 794.9G 0 lvm
└─pve-data-tpool 252:8 0 794.9G 0 lvm
├─pve-data 252:9 0 794.9G 1 lvm
├─pve-vm--101--disk--0 252:10 0 50G 0 lvm
├─pve-vm--102--disk--0 252:11 0 4G 0 lvm
├─pve-vm--100--disk--0 252:12 0 4M 0 lvm
├─pve-vm--100--disk--1 252:13 0 64G 0 lvm
├─pve-vm--103--disk--0 252:14 0 8G 0 lvm
├─pve-vm--104--disk--0 252:15 0 2G 0 lvm
└─pve-vm--105--disk--0 252:16 0 810G 0 lvm
LVS:
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
USB1 USB1 twi-aotz-- <113.07g 0.00 1.40
data pve twi-aotzD- <794.93g 100.00 3.36
root pve -wi-ao---- 96.00g
swap pve -wi-ao---- 7.36g
vm-100-disk-0 pve Vwi-a-tz-- 4.00m data 0.00
vm-100-disk-1 pve Vwi-a-tz-- 64.00g data 20.23
vm-101-disk-0 pve Vwi-a-tz-- 50.00g data 9.77
vm-102-disk-0 pve Vwi-a-tz-- 4.00g data 67.33
vm-103-disk-0 pve Vwi-a-tz-- 8.00g data 44.91
vm-104-disk-0 pve Vwi-a-tz-- 2.00g data 76.95
vm-105-disk-0 pve Vwi-a-tz-- 810.00g data 94.97
VGS:
VG #PV #LV #SN Attr VSize VFree
USB1 1 1 0 wz--n- <115.50g 120.00m
pve 1 10 0 wz--n- <930.51g 16.00g
PVS:
PV VG Fmt Attr PSize PFree
/dev/nvme0n1p3 pve lvm2 a-- <930.51g 16.00g
/dev/sda USB1 lvm2 a-- <115.50g 120.00m
I was about to setup a backup but the hardware arrives only the day after tomorrow :/
What I also managed to do: trying to restore a backup of ct102 on the USB drive, which failed and somehow deleted that ct, any way back ?
Also tried to migrate the storage of 2 ct's whitout success, I get the following :
Task viewer: CT 101 - Move Volume
OutputStatus
Stop
Download
Logical volume "vm-101-disk-0" created.
Creating filesystem with 13107200 4k blocks and 3276800 inodes
Filesystem UUID: d5c3728f-372f-4d56-85f1-ebbf4cceff25
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424
mount: /var/lib/lxc/101/.copy-volume-2: wrong fs type, bad option, bad superblock on /dev/mapper/pve-vm--101--disk--0, missing codepage or helper program, or other error.
dmesg(1) may have more information after failed mount system call.
Specified filename /var/lib/lxc/101/.copy-volume-1 does not exist.
Logical volume "vm-101-disk-0" successfully removed.
TASK ERROR: command 'mount -o ro /dev/dm-10 /var/lib/lxc/101/.copy-volume-2//' failed: exit code 32
Thank you for all the help!
---
Edit:
Managed to temporarly move the VM to a USB Flash drive, then backup the remaining CT's to a PBS, then restore existing backups, only lost 6 days of data on my haos and influxdb.
Will now move everything to a new PVE with a 4TB nvme and then create a more permanent backup solution.
My VM and CT's stopped working today, after taking a look i found out that my local lvm is full.
I must admit I'm pretty much a noob and quite lost and terrified of losing the data found on three (vm100/ct101/ct105) of the CT's.
The problems seems that I have somehow managed to allocate to much space to vm 105.
Here are the commands I ran so far:
(added a usb drive to try to fix it temporarly /dev/sda)
LSBLK:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 1 115.5G 0 disk
├─USB1-USB1_tmeta 252:4 0 1.2G 0 lvm
│ └─USB1-USB1-tpool 252:6 0 113.1G 0 lvm
│ └─USB1-USB1 252:7 0 113.1G 1 lvm
└─USB1-USB1_tdata 252:5 0 113.1G 0 lvm
└─USB1-USB1-tpool 252:6 0 113.1G 0 lvm
└─USB1-USB1 252:7 0 113.1G 1 lvm
nvme0n1 259:0 0 931.5G 0 disk
├─nvme0n1p1 259:1 0 1007K 0 part
├─nvme0n1p2 259:2 0 1G 0 part /boot/efi
└─nvme0n1p3 259:3 0 930.5G 0 part
├─pve-swap 252:0 0 7.4G 0 lvm [SWAP]
├─pve-root 252:1 0 96G 0 lvm /
├─pve-data_tmeta 252:2 0 8.1G 0 lvm
│ └─pve-data-tpool 252:8 0 794.9G 0 lvm
│ ├─pve-data 252:9 0 794.9G 1 lvm
│ ├─pve-vm--101--disk--0 252:10 0 50G 0 lvm
│ ├─pve-vm--102--disk--0 252:11 0 4G 0 lvm
│ ├─pve-vm--100--disk--0 252:12 0 4M 0 lvm
│ ├─pve-vm--100--disk--1 252:13 0 64G 0 lvm
│ ├─pve-vm--103--disk--0 252:14 0 8G 0 lvm
│ ├─pve-vm--104--disk--0 252:15 0 2G 0 lvm
│ └─pve-vm--105--disk--0 252:16 0 810G 0 lvm
└─pve-data_tdata 252:3 0 794.9G 0 lvm
└─pve-data-tpool 252:8 0 794.9G 0 lvm
├─pve-data 252:9 0 794.9G 1 lvm
├─pve-vm--101--disk--0 252:10 0 50G 0 lvm
├─pve-vm--102--disk--0 252:11 0 4G 0 lvm
├─pve-vm--100--disk--0 252:12 0 4M 0 lvm
├─pve-vm--100--disk--1 252:13 0 64G 0 lvm
├─pve-vm--103--disk--0 252:14 0 8G 0 lvm
├─pve-vm--104--disk--0 252:15 0 2G 0 lvm
└─pve-vm--105--disk--0 252:16 0 810G 0 lvm
LVS:
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
USB1 USB1 twi-aotz-- <113.07g 0.00 1.40
data pve twi-aotzD- <794.93g 100.00 3.36
root pve -wi-ao---- 96.00g
swap pve -wi-ao---- 7.36g
vm-100-disk-0 pve Vwi-a-tz-- 4.00m data 0.00
vm-100-disk-1 pve Vwi-a-tz-- 64.00g data 20.23
vm-101-disk-0 pve Vwi-a-tz-- 50.00g data 9.77
vm-102-disk-0 pve Vwi-a-tz-- 4.00g data 67.33
vm-103-disk-0 pve Vwi-a-tz-- 8.00g data 44.91
vm-104-disk-0 pve Vwi-a-tz-- 2.00g data 76.95
vm-105-disk-0 pve Vwi-a-tz-- 810.00g data 94.97
VGS:
VG #PV #LV #SN Attr VSize VFree
USB1 1 1 0 wz--n- <115.50g 120.00m
pve 1 10 0 wz--n- <930.51g 16.00g
PVS:
PV VG Fmt Attr PSize PFree
/dev/nvme0n1p3 pve lvm2 a-- <930.51g 16.00g
/dev/sda USB1 lvm2 a-- <115.50g 120.00m
I was about to setup a backup but the hardware arrives only the day after tomorrow :/
What I also managed to do: trying to restore a backup of ct102 on the USB drive, which failed and somehow deleted that ct, any way back ?
Also tried to migrate the storage of 2 ct's whitout success, I get the following :
Task viewer: CT 101 - Move Volume
OutputStatus
Stop
Download
Logical volume "vm-101-disk-0" created.
Creating filesystem with 13107200 4k blocks and 3276800 inodes
Filesystem UUID: d5c3728f-372f-4d56-85f1-ebbf4cceff25
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424
mount: /var/lib/lxc/101/.copy-volume-2: wrong fs type, bad option, bad superblock on /dev/mapper/pve-vm--101--disk--0, missing codepage or helper program, or other error.
dmesg(1) may have more information after failed mount system call.
Specified filename /var/lib/lxc/101/.copy-volume-1 does not exist.
Logical volume "vm-101-disk-0" successfully removed.
TASK ERROR: command 'mount -o ro /dev/dm-10 /var/lib/lxc/101/.copy-volume-2//' failed: exit code 32
Thank you for all the help!
---
Edit:
Managed to temporarly move the VM to a USB Flash drive, then backup the remaining CT's to a PBS, then restore existing backups, only lost 6 days of data on my haos and influxdb.
Will now move everything to a new PVE with a 4TB nvme and then create a more permanent backup solution.
Last edited: