Don't use the rescue boot, it's slightly missleading named for what it does - use the "Debug" mode, and exit to the second prompt it offers you - there you can try some stuff.
Proxmox wont boot. Not sure where to start
- Thread starter TheJM
- Start date
So I got into the command line of the debug screen of the Proxmox 5.4 ISO and the pools all showed as unavailable.
But then I entered debug in the proxmox 6.1 installer and all of my pools appeared in good health - though it said they all need to be upgraded.
What does this mean for my next steps? Are my pools not compatible with proxmox 5.4? Now that I know the pools appear to be in good health (ill believe it when I can actually access the data again though), I assume I should somehow move to proxmox 6.1 if they aren't compatible with 5.4? How do I integrate my old pools into my next steps?
Last edited:
That's great as a starter, it means your data should be well
Well not directly, but you run into some issues which the the ZFS version from Proxmox VE 5.x cannot cope with, but the newer on from 6.x can - so similar to what Behlendorf suggested in his comment https://github.com/openzfs/zfs/issues/6244#issuecomment-422187850
I would first try to see if we still can get the PVE 5.4 ZFS to import this pool, then upgrade to 6.1 would be much easier.
So as you can still boot into the problematic rpool I'd try first from there, we would need the history and the current internal ZFS config.
When booting the real system you could try:
Code:
zdb -h rpool
zdb -C rpoolif this hangs and you need to use the ISO again you need to tell zdb to work on a exported pool:
Code:
zdb -h -e rpool
zdb -C -e rpoolTo get something out as text file if you do not have sane means to connect, e.g., in the ISO debug environment you could use the good ol' "server directory as HTTP with python trick".
To do this from the ISO environment you would need following steps, in the second shell:
Then open a browser and navigate to that IP you got from the "ip" command with port 8000, e.g.: http://192.168.1.42:8000 as example.
Here you should be able to download the generated output files directly from the server.
To do this from the ISO environment you would need following steps, in the second shell:
Code:
# first get some networking (DHCP)
dhclient
# check what IPs we got with
ip -c addr
# run zdb commands but save to file
zdb -h -e rpool > rpool.history
zdb -C -e rpool > rpool.config
# launch our webserver
python3 -m http.serverThen open a browser and navigate to that IP you got from the "ip" command with port 8000, e.g.: http://192.168.1.42:8000 as example.
Here you should be able to download the generated output files directly from the server.
I was able to connect to the proxmox webpage and run the commands fro there. I am posting the outputs of the commands below. Let me know what the next step is. Also, just for your info, im still getting that kernel panic message on my current proxmox instance (the same one i ran these commands on).
this is the zdb -h output:
zdb -C rpool:
this is the zdb -h output:
Code:
2019-04-19.23:31:38 zpool create -f -o cachefile=none -o ashift=12 rpool mirror /dev/sda3 /dev/sdj3
2019-04-19.23:31:38 zfs create rpool/ROOT
2019-04-19.23:31:38 zfs create rpool/data
2019-04-19.23:31:38 zfs create rpool/ROOT/pve-1
2019-04-19.23:31:38 zfs set atime=off rpool
2019-04-19.23:31:38 zfs set compression=on rpool
2019-04-19.23:31:39 zfs set sync=disabled rpool
2019-04-19.23:32:40 zfs set sync=standard rpool
2019-04-19.23:32:40 zfs set mountpoint=/ rpool/ROOT/pve-1
2019-04-19.23:32:40 zpool set bootfs=rpool/ROOT/pve-1 rpool
2019-04-19.23:32:40 zpool export rpool
2019-04-19.23:48:28 zpool import -N rpool
2019-04-19.22:34:37 zpool import -N rpool
2019-04-20.11:05:37 zpool import -N rpool
2019-04-20.11:12:32 zpool import -N rpool
2019-04-20.11:21:01 zpool import -N rpool
2019-04-20.16:25:04 zpool import -N rpool
2019-04-20.16:35:35 zpool import -N rpool
2019-04-20.16:57:19 zpool import -N rpool
2019-04-20.17:13:05 zpool import -N rpool
2019-04-20.17:30:08 zpool import -N rpool
2019-04-21.10:22:49 zpool import -N rpool
2019-04-21.10:32:16 zpool import -N rpool
2019-04-21.10:38:42 zpool import -N rpool
2019-04-30.22:02:17 zpool import -N rpool
2019-04-30.22:51:19 zpool import -N rpool
2019-04-30.23:05:47 zpool import -N rpool
2019-05-02.17:12:17 zpool import -N rpool
2019-05-03.13:43:30 zpool import -N rpool
2019-05-03.18:50:35 zpool import -N rpool
2019-05-03.19:02:33 zpool import -N rpool
2019-05-03.19:07:27 zpool import -N rpool
2019-05-03.19:27:35 zpool import -N rpool
2019-05-03.22:26:09 zpool import -N rpool
2019-05-03.22:36:15 zpool import -N rpool
2019-05-03.22:39:07 zpool import -N rpool
2019-05-04.00:16:52 zpool import -N rpool
2019-05-04.00:27:30 zpool import -N rpool
2019-05-04.00:37:17 zpool import -N rpool
2019-05-04.00:43:20 zpool import -N rpool
2019-05-04.00:50:22 zpool import -N rpool
2019-05-04.00:54:31 zpool import -N rpool
2019-05-04.00:57:35 zpool import -N rpool
2019-05-04.01:05:49 zpool import -N rpool
2019-05-04.01:09:14 zpool import -N rpool
2019-05-04.01:53:49 zpool import -N rpool
2019-05-04.02:01:44 zpool import -N rpool
2019-05-04.10:19:58 zpool import -N rpool
2019-05-04.10:39:45 zpool import -N rpool
2019-05-04.11:17:14 zpool import -N rpool
2019-05-04.11:33:30 zpool import -N rpool
2019-05-04.12:08:22 zpool import -N rpool
2019-05-04.12:39:00 zpool import -N rpool
2019-05-04.12:47:28 zpool import -N rpool
2019-05-04.12:51:12 zpool import -N rpool
2019-05-04.12:58:29 zpool import -N rpool
2019-05-04.13:22:04 zpool import -N rpool
2019-05-04.13:26:24 zpool import -N rpool
2019-05-04.13:43:54 zpool import -N rpool
2019-05-04.13:49:19 zpool import -N rpool
2019-05-04.14:03:24 zpool import -N rpool
2019-05-04.14:06:53 zpool import -N rpool
2019-05-04.15:18:32 zpool import -N rpool
2019-05-04.19:32:48 zpool import -N rpool
2019-05-04.20:21:32 zpool import -N rpool
2019-05-04.20:43:29 zpool import -N rpool
2019-05-04.21:03:59 zpool import -N rpool
2019-05-09.11:32:09 zpool import -N rpool
2019-05-12.00:24:11 zpool scrub rpool
2019-05-13.21:45:59 zpool import -N rpool
2019-05-14.18:52:34 zpool import -N rpool
2019-06-01.18:40:53 zpool import -N rpool
2019-06-09.00:24:11 zpool scrub rpool
2019-06-13.19:39:05 zpool import -N rpool
2019-06-13.19:47:02 zpool import -N rpool
2019-06-13.20:08:16 zpool import -N rpool
2019-06-13.21:00:17 zpool import -N rpool
2019-06-13.21:03:03 zpool import -N rpool
2019-06-13.21:15:04 zpool import -N rpool
2019-06-15.15:29:25 zpool import -N rpool
2019-06-15.15:39:54 zpool import -N rpool
2019-06-15.15:43:03 zpool import -N rpool
2019-06-15.15:50:28 zpool import -N rpool
2019-06-15.16:01:56 zpool import -N rpool
2019-06-16.09:12:43 zpool import -N rpool
2019-06-16.12:45:55 zpool import -N rpool
2019-06-16.12:49:46 zpool import -N rpool
2019-06-16.12:59:43 zpool import -N rpool
2019-06-22.00:31:12 zpool import -N rpool
2019-07-05.12:26:45 zpool import -N rpool
2019-07-05.17:02:04 zpool import -N rpool
2019-07-05.17:43:48 zpool import -N rpool
2019-07-14.00:24:10 zpool scrub rpool
2019-07-14.22:11:44 zpool import -N rpool
2019-07-16.08:34:24 zpool import -N rpool
2019-08-11.00:24:11 zpool scrub rpool
2019-08-13.23:13:26 zpool import -N rpool
2019-08-17.14:19:13 zpool import -N rpool
2019-08-27.08:51:15 zpool import -N rpool
2019-09-08.00:24:11 zpool scrub rpool
2019-10-10.10:32:53 zpool import -N rpool
2019-10-13.00:24:05 zpool scrub rpool
2019-11-10.00:24:11 zpool scrub rpool
2019-12-04.09:11:03 zpool import -N rpool
2019-12-08.00:24:11 zpool scrub rpool
2019-12-21.10:14:36 zpool import -N rpool
2020-01-12.00:24:12 zpool scrub rpool
2020-01-26.10:14:59 zpool import -N rpool
2020-02-09.00:24:11 zpool scrub rpool
2020-03-02.08:49:37 zpool import -N rpool
2020-03-08.00:24:10 zpool scrub rpool
2020-03-09.18:51:30 zpool import -N rpool
2020-03-09.19:09:10 zpool import -N rpool
2020-03-09.19:22:10 zpool import -N rpool
2020-03-09.19:43:34 zpool import -N rpool
2020-03-09.20:13:35 zpool import -N rpool
2020-03-09.20:29:26 zpool import -N rpool
2020-03-09.20:49:42 zpool import -N rpool
2020-03-09.21:03:50 zpool import -N rpool
2020-03-09.21:07:33 zpool import -N rpool
2020-03-09.21:36:40 zpool import -N rpool
2020-03-09.21:38:46 zpool import -N rpool
2020-03-09.21:53:19 zpool import -N rpool
2020-03-09.22:06:42 zpool import -N rpool
2020-03-09.22:44:19 zpool import -N rpool
2020-03-09.23:03:09 zpool import -N rpool
2020-03-09.23:11:21 zpool import -N rpool
2020-03-10.07:06:38 zpool import -N rpool
2020-03-10.07:10:57 zpool import -N rpool
2020-03-10.07:14:54 zpool import -N rpool
2020-03-10.07:34:54 zpool import -N rpool
2020-03-10.07:39:39 zpool import -N rpool
2020-03-10.07:52:34 zpool import -N rpool
2020-03-10.08:12:20 zpool import -N rpool
2020-03-10.19:24:35 zpool import -N rpool
2020-03-10.20:05:12 zpool import -N rpool
2020-03-11.07:46:11 zpool import -N rpoolzdb -C rpool:
Code:
root@pve:~# zdb -C rpool
MOS Configuration:
version: 5000
name: 'rpool'
state: 0
txg: 6967751
pool_guid: 4576389356468900125
errata: 0
hostname: '(none)'
com.delphix:has_per_vdev_zaps
vdev_children: 1
vdev_tree:
type: 'root'
id: 0
guid: 4576389356468900125
children[0]:
type: 'mirror'
id: 0
guid: 6116497428964366396
metaslab_array: 256
metaslab_shift: 33
ashift: 12
asize: 999662026752
is_log: 0
create_txg: 4
com.delphix:vdev_zap_top: 129
children[0]:
type: 'disk'
id: 0
guid: 637619487918091874
path: '/dev/sdj3'
whole_disk: 0
DTL: 261
create_txg: 4
com.delphix:vdev_zap_leaf: 130
children[1]:
type: 'disk'
id: 1
guid: 4237211854874725763
path: '/dev/sde3'
whole_disk: 0
DTL: 260
create_txg: 4
com.delphix:vdev_zap_leaf: 131
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_dataHmm, seems
I'd try to import the pool with newer ZFS from 6.1 ISO (do NOT run any "zpool upgrade" commands, else you won't be able to import the pool with PVE 5.4 anymore).
Then do a full scrub of the pool, maybe it helps to iron out the issues - you could repeat that for the two other pools too.
repeat for other pools, reboot and see if that helps. If that does not helps we need to upgrade you to 6.1, somehow.
I'd try to import the pool with newer ZFS from 6.1 ISO (do NOT run any "zpool upgrade" commands, else you won't be able to import the pool with PVE 5.4 anymore).
Then do a full scrub of the pool, maybe it helps to iron out the issues - you could repeat that for the two other pools too.
Code:
mkdir /zfstmp
zpool import -R /zfstmp rpool
zpool scrup rpool
# scrub runs in background, to check if it's finished use:
zpool statusrepeat for other pools, reboot and see if that helps. If that does not helps we need to upgrade you to 6.1, somehow.
The rpool imported fine (though I did have to use -f as it said it was previously used by another system) started the scrub.
While I was waiting I imported one of my other pools with about 12TB of data on it. Currently waiting for that to import. Once done I will scrub it and move onto the final pool. Then boot back into my existing Proxmox 5.4 and see if this has fixed the issues.
The large Zpool I mentioned above is taking a while to import. will post more when all are done.
I started the import for the large Zpool and it went to this screen about 5 minutes after it started. It has been stuck here for the past 3 hours. Should I be concerned?
I cannot put in any commands as it looks like it’s midway through the import. Just don’t know how long imports usually take of if it’s stuck.
EDIT: my disk activity light on the server is not on. I heard the disks whirring for the first few minutes but now dont hear anything. Im wondering if its hung but dont want to damage the pool by stopping the import.
Last edited:
The below info is done on the Proxmox 6.1 bootable ISO I made:
So I am able to import and scrub 2 of my pools but the scrub found no data errors on those 2 pools.
The third pool is a different story. After starting the import command, it locks up on this screen. The first time I got to that above screen, I left it for 24 hours but saw no disk activity light and the computer locked up (I couldn't cancel the command). I have tried to import it using the "-Fn" tag as well but that yielded the same result. I was able to import it as read only but that isn't helpful.
It should be noted that this troublesome pool is a 13TB data pool. The VMs and Proxmox are both stored in the other 2 pools so AFAIK this shouldn't cause the VMs to not be able to spin up, their data pools just wouldn't be accessible. How do I get this to import?
So I am able to import and scrub 2 of my pools but the scrub found no data errors on those 2 pools.
The third pool is a different story. After starting the import command, it locks up on this screen. The first time I got to that above screen, I left it for 24 hours but saw no disk activity light and the computer locked up (I couldn't cancel the command). I have tried to import it using the "-Fn" tag as well but that yielded the same result. I was able to import it as read only but that isn't helpful.
It should be noted that this troublesome pool is a 13TB data pool. The VMs and Proxmox are both stored in the other 2 pools so AFAIK this shouldn't cause the VMs to not be able to spin up, their data pools just wouldn't be accessible. How do I get this to import?
Now Proxmox won't boot again. This is extremely frustrating. I have offsite backups that are a several months old but im about to lose months of work on several different systems, other data, computer backups, and more.
Screenshot
Does anyone have an idea of how this whole situation could have happened? It seems to coincide with the timing of my update to 'pve-kernel-4.15' just this past February 27, 2020 so it sounds like a Proxmox Bug. I rebooted the server to install that and didnt realize it didnt come back up.
Screenshot
Does anyone have an idea of how this whole situation could have happened? It seems to coincide with the timing of my update to 'pve-kernel-4.15' just this past February 27, 2020 so it sounds like a Proxmox Bug. I rebooted the server to install that and didnt realize it didnt come back up.
Last edited:
looks identical to https://github.com/openzfs/zfs/issues/7002
if you can import the pool read-only, no data should be lost since you can just send/receive it to another pool/system ?
if you can import the pool read-only, no data should be lost since you can just send/receive it to another pool/system ?
You are correct. I will see what data I can pull off of the troublesome pool. I will post the results of that command in a few hours.
One other question, my proxmox 5.4 install was a very generic install so Im not afraid of losing anything on that pool. Also remember that my VMs are stored on a different pool than rpool. Can I just erase that pool, install proxmox 6.1, and import my old VMs into the new proxmox 6.1 install? Would this be a recommended option?
Last edited:
That could be feasible, you'd just need to ensure that you really selected the correct disks for the rpool on install - so that not any data is from the two important pools is overwritten.
I'd first make backups of /etc, especially /etc/pve - so that you have all VM and the storage configs - this allows quicker restoring.
Is there a tutorial or walkthrough that you could point me to, to do this?
After I get Proxmox installed and import the VM pool, will the VMs appear? How do I get the VMs on the new Proxmox after the pool has been imported?
just copy it to a USB pen drive, e.g.:
Code:
mkdir /mnt/usb
mount /dev/sdX /mnt/usb
cp -a /etc/pve /mnt/usb/pve-bak
# to be sure you could also copy over the config.db:
# first stop pmxcfs so that db is consistent:
systemctl stop pve-cluster
cp /var/lib/pve-cluster/config.db /mnt/usb/No, not automagically.
You restore your backed up configs from /etc/pve.
just copy it to a USB pen drive, e.g.:
Code:mkdir /mnt/usb mount /dev/sdX /mnt/usb cp -a /etc/pve /mnt/usb/pve-bak # to be sure you could also copy over the config.db: # first stop pmxcfs so that db is consistent: systemctl stop pve-cluster cp /var/lib/pve-cluster/config.db /mnt/usb/
No, not automagically.
You restore your backed up configs from /etc/pve.
Thank you. I will try this and see what happens. I am not in front of the server right now so I can't look at the file structure. Are there specific VM files from inside /etc/pve that I want to copy to the new proxmox install? or are you suggesting to copy the entire folder to the new proxmox install?
Easiest would be entire folder, ensure's you do not miss anything relevant and also get the storage config, which let's you also start a bit faster after the reinstallation.
For a single standalone node, I'd in all cases save and restore (single node only!)
Code:
/etc/pve/qemu-server/*.conf
/etc/pve/lxc/*.conf
# storage cfg
/etc/pve/storage.cfg
# backup jobs
/etc/pve/vzdump.cron
# firewall rules/options
/etc/pve/firewall/*.fw