Impossible to boot any VMs or container after Clonezilla backup, from original drive

totoxaraymond · Thursday at 18:09

Hi everyone,

I'm in quite a bad situation here.

I wanted to backup my proxmox installation before upgrading to proxmox 9, so i used CLoneZilla to clone my boot SSD to a spare HDD.

So far, so good but after rebooting to proxmox, i can't run ANY of my VMs or Container.

VMs will return "not a bootable disk" and loop forever.

while CTs will return:

Code:

run_buffer: 571 Script exited with status 32
lxc_init: 845 Failed to run lxc.hook.pre-start for container "107"
__lxc_start: 2034 Failed to initialize container "107"
TASK ERROR: startup for container '107' failed

Any idea of what could have happen?

Impact · Thursday at 19:30

Check pct start 107 --debug. Clonezilla (by default) tends to not clone the LVM-Thin part properly if you use that. I generally clone via dd through the installer's shell.

gfngfn256 · Thursday at 19:58

If you put the original SSD in the host, does it boot correctly? AFAIK Clonezilla will not alter the source drive at all. Your thread title suggests that the ORIGINAL SSD will not boot. (If this is not the case, change the title to reflect this).

How did you perform the Clonezilla clone? Did you correctly shut down the host etc.

(I have never had much luck with Clonezilla on a Proxmox host clone, but always use dd (as Impact has also pointed out above), and have successfully restored an image from it).

Possibly, you physically removed the original SSD & upon replacing it back, the drives are somehow being differently read by the host at bootup (enumeration)?

What disk sizes were the original/clone.

You have not provided any details on the storage config. of the host.

totoxaraymond · Thursday at 20:25

gfngfn256 said:
If you put the original SSD in the host, does it boot correctly? AFAIK Clonezilla will not alter the source drive at all. Your thread title suggests that the ORIGINAL SSD will not boot. (If this is not the case, change the title to reflect this).

Sorry if it's unclear, I made a backup using clonezilla, but i'm booting from the original SSD. Proxmox boots but no VMs or CTs will boot.

gfngfn256 said:
How did you perform the Clonezilla clone? Did you correctly shut down the host etc.

Yes I shut down the host correctly, then boot from a Clonezilla live USB, performed the clone, then shut down, Took out the backup drive and booted back up.

gfngfn256 said:
You have not provided any details on the storage config. of the host.

My storage config is very basic, It's a1TB SSD running Proxmox and all VMs and CTs from that same SSD.

Impact said:
Check pct start 107 --debug. Clonezilla (by default) tends to not clone the LVM-Thin part properly if you use that.

This is the output from

Code:

pct start 107 --debug

:

Code:

run_buffer: 571 Script exited with status 32
lxc_init: 845 Failed to run lxc.hook.pre-start for container "107"
__lxc_start: 2034 Failed to initialize container "107"
0 hostid 100000 range 65536
INFO     lsm - ../src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
INFO     utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "107", config section "lxc"
DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 107 lxc pre-start produced output: mount: /var/lib/lxc/.pve-staged-mounts/rootfs: wrong fs type, bad option, bad superblock on /dev/mapper/pve-vm--107--disk--0, missing codepage or helper program, or other error.
dmesg(1) may have more information after failed mount system call.

DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 107 lxc pre-start produced output: command 'mount /dev/dm-42 /var/lib/lxc/.pve-staged-mounts/rootfs' failed: exit code 32

ERROR    utils - ../src/lxc/utils.c:run_buffer:571 - Script exited with status 32
ERROR    start - ../src/lxc/start.c:lxc_init:845 - Failed to run lxc.hook.pre-start for container "107"
ERROR    start - ../src/lxc/start.c:__lxc_start:2034 - Failed to initialize container "107"
INFO     utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "107", config section "lxc"
startup for container '107' failed

Thank you for your support!

Impact · Thursday at 20:31

Seems like your thin volumes are broken. I can't really explain why CloneZilla would have caused this for the source disk. I'd restore the guests from backups.

gfngfn256 · Thursday at 20:49

totoxaraymond said:
Sorry if it's unclear

Change the title to reflect this. (Possibly: "Impossible to boot any VMs or container after Clonezilla backup, from original drive")

I concur, this is probably not caused by Clonezilla cloning. However there is a chance that an already failing SSD will fail further after the stress of reading every single sector.

When was the last time you successfully rebooted this system (prior to the above cloning).

Can you provide lsblk output of the drive. Can you compare this to an older output prior to this incident.

totoxaraymond said:
Took out the backup drive

Was that drive always in the host, or was it placed there only for the backup?

What does cat /etc/pve/storage.cfg ouput.

totoxaraymond · Thursday at 21:12

gfngfn256 said:
When was the last time you successfully rebooted this system (prior to the above cloning).

I did a few reboot earlier this day with no problems.

gfngfn256 said:
Can you provide lsblk output of the drive. Can you compare this to an older output prior to this incident.

This is the ouput of `lslbk` : Sorry i can't compare it.

Code:

sdg                                                           8:96   0 931.5G  0 disk
├─sdg1                                                        8:97   0  1007K  0 part
├─sdg2                                                        8:98   0     1G  0 part /boot/efi
└─sdg3                                                        8:99   0 930.5G  0 part
  ├─pve-swap                                                252:0    0     8G  0 lvm  [SWAP]
  ├─pve-root                                                252:1    0    96G  0 lvm  /
  ├─pve-data_tmeta                                          252:2    0   8.1G  0 lvm 
  │ └─pve-data-tpool                                        252:4    0 794.3G  0 lvm 
  │   ├─pve-data                                            252:5    0 794.3G  1 lvm 
  │   ├─pve-snap_vm--102--disk--0_NC--26_UB20--04--PHP8--1  252:6    0    16G  0 lvm 
  │   ├─pve-snap_vm--102--disk--0_NC27_UB20--04_PHP8--1     252:7    0    16G  0 lvm 
  │   ├─pve-snap_vm--102--disk--0_NC28_UB20--04             252:8    0    16G  0 lvm 
  │   ├─pve-snap_vm--102--disk--0_NC28_UB20--04--2          252:9    0    16G  0 lvm 
  │   ├─pve-snap_vm--102--disk--0_NC29_UB20--04             252:10   0    16G  0 lvm 
  │   ├─pve-snap_vm--102--disk--0_NC30_UB22--04             252:11   0    16G  0 lvm 
  │   ├─pve-snap_vm--102--disk--0_NC_25--UB20--04--PHP8--1  252:12   0    16G  0 lvm 
  │   ├─pve-snap_vm--102--disk--0_NC_25--UB20--04_PHP8--1_2 252:13   0    16G  0 lvm 
  │   ├─pve-snap_vm--102--disk--0_beforedatamigration       252:14   0    16G  0 lvm 
  │   ├─pve-snap_vm--102--disk--0_v30--0--12                252:15   0    16G  0 lvm 
  │   ├─pve-snap_vm--103--disk--0_ub2204                    252:16   0     8G  0 lvm 
  │   ├─pve-snap_vm--104--disk--0_Inventree--1--2--6        252:17   0    32G  0 lvm 
  │   ├─pve-snap_vm--104--disk--0_UB2204                    252:18   0    32G  0 lvm 
  │   ├─pve-snap_vm--104--disk--0_do--release--OK           252:19   0    32G  0 lvm 
  │   ├─pve-snap_vm--104--disk--0_inventree1--1--6          252:20   0    32G  0 lvm 
  │   ├─pve-snap_vm--104--disk--0_inventree1--2--6--2       252:21   0    32G  0 lvm 
  │   ├─pve-snap_vm--104--disk--0_snapshot--before--upgrade 252:22   0    32G  0 lvm 
  │   ├─pve-snap_vm--104--disk--0_upgrade--done             252:23   0    32G  0 lvm 
  │   ├─pve-snap_vm--104--disk--0_upgradeTo22--04           252:24   0    32G  0 lvm 
  │   ├─pve-snap_vm--104--disk--0_upgraded--22--02          252:25   0    32G  0 lvm 
  │   ├─pve-snap_vm--105--disk--0_R19--UB20--04_php7        252:26   0     8G  0 lvm 
  │   ├─pve-snap_vm--105--disk--0_R19_UB20--04_php8--1      252:27   0     8G  0 lvm 
  │   ├─pve-snap_vm--107--disk--0_added_user_inventree      252:28   0    32G  0 lvm 
  │   ├─pve-snap_vm--107--disk--0_beforeUpdate              252:29   0    32G  0 lvm 
  │   ├─pve-snap_vm--107--disk--0_initial_install           252:30   0    32G  0 lvm 
  │   ├─pve-snap_vm--107--disk--0_inventreeOK               252:31   0    32G  0 lvm 
  │   ├─pve-snap_vm--107--disk--0_sshConfigured             252:32   0    32G  0 lvm 
  │   ├─pve-snap_vm--107--disk--0_userConf                  252:33   0    32G  0 lvm 
  │   ├─pve-snap_vm--108--disk--0_beforeupgradeto22--04     252:34   0     8G  0 lvm 
  │   ├─pve-vm--100--disk--0                                252:35   0    12G  0 lvm 
  │   ├─pve-vm--101--disk--0                                252:36   0    64G  0 lvm 
  │   ├─pve-vm--102--disk--0                                252:37   0    16G  0 lvm 
  │   ├─pve-vm--103--disk--0                                252:38   0     8G  0 lvm 
  │   ├─pve-vm--104--disk--0                                252:39   0    32G  0 lvm 
  │   ├─pve-vm--105--disk--0                                252:40   0     8G  0 lvm 
  │   ├─pve-vm--106--disk--0                                252:41   0    32G  0 lvm 
  │   ├─pve-vm--107--disk--0                                252:42   0    32G  0 lvm 
  │   ├─pve-vm--108--disk--0                                252:43   0     8G  0 lvm 
  │   ├─pve-vm--109--disk--0                                252:44   0    32G  0 lvm 
  │   ├─pve-vm--110--disk--0                                252:45   0    16G  0 lvm 
  │   ├─pve-vm--111--disk--0                                252:46   0    32G  0 lvm 
  │   └─pve-vm--112--disk--0                                252:47   0    32G  0 lvm 
  └─pve-data_tdata                                          252:3    0 794.3G  0 lvm 
    └─pve-data-tpool                                        252:4    0 794.3G  0 lvm 
      ├─pve-data                                            252:5    0 794.3G  1 lvm 
      ├─pve-snap_vm--102--disk--0_NC--26_UB20--04--PHP8--1  252:6    0    16G  0 lvm 
      ├─pve-snap_vm--102--disk--0_NC27_UB20--04_PHP8--1     252:7    0    16G  0 lvm 
      ├─pve-snap_vm--102--disk--0_NC28_UB20--04             252:8    0    16G  0 lvm 
      ├─pve-snap_vm--102--disk--0_NC28_UB20--04--2          252:9    0    16G  0 lvm 
      ├─pve-snap_vm--102--disk--0_NC29_UB20--04             252:10   0    16G  0 lvm 
      ├─pve-snap_vm--102--disk--0_NC30_UB22--04             252:11   0    16G  0 lvm 
      ├─pve-snap_vm--102--disk--0_NC_25--UB20--04--PHP8--1  252:12   0    16G  0 lvm 
      ├─pve-snap_vm--102--disk--0_NC_25--UB20--04_PHP8--1_2 252:13   0    16G  0 lvm 
      ├─pve-snap_vm--102--disk--0_beforedatamigration       252:14   0    16G  0 lvm 
      ├─pve-snap_vm--102--disk--0_v30--0--12                252:15   0    16G  0 lvm 
      ├─pve-snap_vm--103--disk--0_ub2204                    252:16   0     8G  0 lvm 
      ├─pve-snap_vm--104--disk--0_Inventree--1--2--6        252:17   0    32G  0 lvm 
      ├─pve-snap_vm--104--disk--0_UB2204                    252:18   0    32G  0 lvm 
      ├─pve-snap_vm--104--disk--0_do--release--OK           252:19   0    32G  0 lvm 
      ├─pve-snap_vm--104--disk--0_inventree1--1--6          252:20   0    32G  0 lvm 
      ├─pve-snap_vm--104--disk--0_inventree1--2--6--2       252:21   0    32G  0 lvm 
      ├─pve-snap_vm--104--disk--0_snapshot--before--upgrade 252:22   0    32G  0 lvm 
      ├─pve-snap_vm--104--disk--0_upgrade--done             252:23   0    32G  0 lvm 
      ├─pve-snap_vm--104--disk--0_upgradeTo22--04           252:24   0    32G  0 lvm 
      ├─pve-snap_vm--104--disk--0_upgraded--22--02          252:25   0    32G  0 lvm 
      ├─pve-snap_vm--105--disk--0_R19--UB20--04_php7        252:26   0     8G  0 lvm 
      ├─pve-snap_vm--105--disk--0_R19_UB20--04_php8--1      252:27   0     8G  0 lvm 
      ├─pve-snap_vm--107--disk--0_added_user_inventree      252:28   0    32G  0 lvm 
      ├─pve-snap_vm--107--disk--0_beforeUpdate              252:29   0    32G  0 lvm 
      ├─pve-snap_vm--107--disk--0_initial_install           252:30   0    32G  0 lvm 
      ├─pve-snap_vm--107--disk--0_inventreeOK               252:31   0    32G  0 lvm 
      ├─pve-snap_vm--107--disk--0_sshConfigured             252:32   0    32G  0 lvm 
      ├─pve-snap_vm--107--disk--0_userConf                  252:33   0    32G  0 lvm 
      ├─pve-snap_vm--108--disk--0_beforeupgradeto22--04     252:34   0     8G  0 lvm 
      ├─pve-vm--100--disk--0                                252:35   0    12G  0 lvm 
      ├─pve-vm--101--disk--0                                252:36   0    64G  0 lvm 
      ├─pve-vm--102--disk--0                                252:37   0    16G  0 lvm 
      ├─pve-vm--103--disk--0                                252:38   0     8G  0 lvm 
      ├─pve-vm--104--disk--0                                252:39   0    32G  0 lvm 
      ├─pve-vm--105--disk--0                                252:40   0     8G  0 lvm 
      ├─pve-vm--106--disk--0                                252:41   0    32G  0 lvm 
      ├─pve-vm--107--disk--0                                252:42   0    32G  0 lvm 
      ├─pve-vm--108--disk--0                                252:43   0     8G  0 lvm 
      ├─pve-vm--109--disk--0                                252:44   0    32G  0 lvm 
      ├─pve-vm--110--disk--0                                252:45   0    16G  0 lvm 
      ├─pve-vm--111--disk--0                                252:46   0    32G  0 lvm 
      └─pve-vm--112--disk--0                                252:47   0    32G  0 lvm

gfngfn256 said:
Was that drive always in the host, or was it placed there only for the backup?

That drive was already there. It was part of my old RAID5 that i replaced for a ZFS a few months ago.

gfngfn256 said:
What does cat /etc/pve/storage.cfg ouput.

There you go:

Code:

dir: local
        path /var/lib/vz
        content backup,iso,vztmpl

lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images

dir: backups
        path /mnt/backups
        content backup
        prune-backups keep-last=10
        shared 0

Again, thanks for your help!

It seems that restoring from backup would work. My latest backups was on sunday, so I might lose a few days of work... Not the end of the world, but i'd rather not to.

gfngfn256 · Thursday at 21:34

totoxaraymond said:
I did a few reboot earlier this day with no problems.

I assume you mean to include CTs & VMs starting correctly after host boot.

totoxaraymond said:
Sorry i can't compare it.

Do you at least recall, if that drive was always sdg.

totoxaraymond said:
that i replaced for a ZFS a few months ago.

Your above ouput of storage.conf above does not seem to reflect any zfs storage (at least in the Proxmox backend). Nor does you host drive (sdg) install seem to be on zfs. Disregard this comment if this is truly unrelated. Did possibly your VMs/CTs access this zfs storage?

How old is this sdg drive. What is its current health state & history.

totoxaraymond · Thursday at 21:50

gfngfn256 said:
Do you at least recall, if that drive was always sdg.

the letter might have changed when i removed 2 failing drives. But it also changed a few months ago when I added new drives for a ZFS pool.

gfngfn256 said:
Your above ouput of storage.conf above does not seem to reflect any zfs storage (at least in the Proxmox backend). Nor does you host drive (sdg) install seem to be on zfs. Disregard this comment if this is truly unrelated. Did possibly your VMs/CTs access this zfs storage?

This ZFS is not seen by proxmox. One of my VMs is a TrueNAS instance, the disked are passed by Proxmox to TrueNas which manages the ZFS pool and then serves it back as NFS or SMB shares to some of my VMs (but only for data storage, not boot drives).

gfngfn256 said:
How old is this sdg drive. What is its current health state & history.

This drive is about 3 year old. I've bought it new and put inside my proxmox server. Never moved from there.

totoxaraymond · Saturday at 06:03

So In the end, i upgraded to Proxmox 9 by reinstalling and restore backups.

Everything is back to normal, fingers crossed.

thank @gfngfn256 and @Impact for your time and help.

Bests,

Thomas

Impossible to boot any VMs or container after Clonezilla backup, from original drive

totoxaraymond

New Member

Impact

Distinguished Member

gfngfn256

Distinguished Member

totoxaraymond

New Member

Impact

Distinguished Member

gfngfn256

Distinguished Member

totoxaraymond

New Member

gfngfn256

Distinguished Member

totoxaraymond

New Member

totoxaraymond

New Member

We value your privacy