[SOLVED] Solaris QEMU ZPools

S

sascha-angeln

New Member
Feb 10, 2024
22
0
1
Hello,

I'm running proxmox 8.1.3 on a ZFS rpool.

I recently added a Solaris11 VM to my Proxmox configuration. The VM has 3 ZPools (rpool for OS, and two others for data). The VM is working perfectly.

The problem occurs, when the proxmox host is rebooted. During the boot sequence proxmox complains:

Feb 01 09:57:21 proxmox kernel: GPT: Primary header thinks Alt. header is not at the end of the disk.
Feb 01 09:57:21 proxmox kernel: GPT:33554303 != 33554431
Feb 01 09:57:21 proxmox kernel: GPT:Alternate GPT header not at the end of the disk.
Feb 01 09:57:21 proxmox kernel: GPT:33554303 != 33554431
Feb 01 09:57:21 proxmox kernel: GPT: Use GNU Parted to correct GPT errors.
Feb 01 09:57:21 proxmox kernel: zd32: p1 p2 p9
Feb 01 09:57:21 proxmox kernel: GPT: Primary header thinks Alt. header is not at the end of the disk.
Feb 01 09:57:21 proxmox kernel: GPT:301988735 != 301989887
Feb 01 09:57:21 proxmox kernel: GPT:Alternate GPT header not at the end of the disk.
Feb 01 09:57:21 proxmox kernel: GPT:301988735 != 301989887
Feb 01 09:57:21 proxmox kernel: GPT: Use GNU Parted to correct GPT errors.
Feb 01 09:57:21 proxmox kernel: zd16: p1 p9
Feb 01 09:57:21 proxmox kernel: GPT: Primary header thinks Alt. header is not at the end of the disk.
Feb 01 09:57:21 proxmox kernel: GPT:12582863 != 12582911
Feb 01 09:57:21 proxmox kernel: GPT:Alternate GPT header not at the end of the disk.
Feb 01 09:57:21 proxmox kernel: GPT:12582863 != 12582911
Feb 01 09:57:21 proxmox kernel: GPT: Use GNU Parted to correct GPT errors.
Feb 01 09:57:21 proxmox kernel: zd96: p1 p9

The devices proxmox is complaining about are exactly the VM disks (ZVol's) mapped for the Solaris VM.

My assumption is: Since Solaris is formatting the 3 VM disks with ZFS - which is slightly diffrent from proxmox's ZFS - proxmox detects the disks at boot time, but is unable to identify them correctly. Unfortunatly booting the proxmox host always ends for that reason in a single user mode, requiring administrative intervention. It's sufficient to enter ^D at the prompt and the boot sequence completes and everything is fine.

Any ideas how to deal with such a challenge?
 
Last edited:
that's weird, as proxmox shouldn't care for the contents of the zvols

there is a volmode setting for zfs zvols, maybe worth a closer look ?

volmode=default|full|geom|dev|none
This property specifies how volumes should be exposed to the OS. Setting it to full exposes volumes as fully fledged block devices, providing maximal functionality. The
value geom is just an alias for full and is kept for compatibility. Setting it to dev hides its partitions. Volumes with property set to none are not exposed outside ZFS,
but can be snapshotted, cloned, replicated, etc, that can be suitable for backup purposes. Value default means that volumes exposition is controlled by system-wide tunable
zvol_volmode, where full, dev and none are encoded as 1, 2 and 3 respectively. The default value is full.

https://forum.proxmox.com/threads/proxmox-zfs-sieht-zfs-pools-von-vms-als-eigene-pools.58459/
 
RolandK said:
that's weird, as proxmox shouldn't care for the contents of the zvols

there is a volmode setting for zfs zvols, maybe worth a closer look ?

volmode=default|full|geom|dev|none
This property specifies how volumes should be exposed to the OS. Setting it to full exposes volumes as fully fledged block devices, providing maximal functionality. The
value geom is just an alias for full and is kept for compatibility. Setting it to dev hides its partitions. Volumes with property set to none are not exposed outside ZFS,
but can be snapshotted, cloned, replicated, etc, that can be suitable for backup purposes. Value default means that volumes exposition is controlled by system-wide tunable
zvol_volmode, where full, dev and none are encoded as 1, 2 and 3 respectively. The default value is full.

https://forum.proxmox.com/threads/proxmox-zfs-sieht-zfs-pools-von-vms-als-eigene-pools.58459/
Click to expand...
I just played with the volmode parameter and set it to "dev" for the three volumes in question. The guest is still fine with this setting, but the boot problem still remains :-(
 
i have searched a little bit in kernel code. whith my limited knowledge, it appears to me that there is no way to disable gpt scanning on blockdevices.
i wonder that we can disable partition scan/device node creation. maybe missing feature in linux kernel !?
 
I don't think those warnings are the cause for the system not booting properly. could you please check for other error messages? if your PVE system doesn't have other pools than your rpool, you could also disable "zfs-import-scan" and "zfs-import-cache" to rule out that either of those tries to import your solaris pools..
 
fabian said:
I don't think those warnings are the cause for the system not booting properly. could you please check for other error messages? if your PVE system doesn't have other pools than your rpool, you could also disable "zfs-import-scan" and "zfs-import-cache" to rule out that either of those tries to import your solaris pools..
Click to expand...
PVE itself has 3 ZPools
- rpool: for PVE itself
- ssd: (RAID1 2 M2 drives) for VMs (CT & QEMU)
- nfs: (RAID1 2 SATA drives) for data, ISO images, backups etc.
I assume disabling zfs-import-scan and zfs-import-cache make the VMs unbootable until one manually import them, correct?
 
sascha-angeln said:
PVE itself has 3 ZPools
- rpool: for PVE itself
- ssd: (RAID1 2 M2 drives) for VMs (CT & QEMU)
- nfs: (RAID1 2 SATA drives) for data, ISO images, backups etc.
I assume disabling zfs-import-scan and zfs-import-cache make the VMs unbootable until one manually import them, correct?
Click to expand...
if they are configured as storages they should be imported anyway, but you can also use the "zfs-import@POOL.service" instances to explicitly import those two, but not anything else. but before you try that, the full log from a failed boot would still be interesting - you don't get dropped in a rescue shell unless there is a reason, and the reason should be visible..
 
fabian said:
if they are configured as storages they should be imported anyway, but you can also use the "zfs-import@POOL.service" instances to explicitly import those two, but not anything else. but before you try that, the full log from a failed boot would still be interesting - you don't get dropped in a rescue shell unless there is a reason, and the reason should be visible..
Click to expand...
Find attached the log from the failed boot. The messages about the GPT header problem apear at Feb 12 15:09:55
 

Attachments

  • pve.log
    194.7 KB · Views: 3
i just wanted to write that a typical problem with zfs boot problems is name clash with existing pool names. i had that some times when attaching old root disk via usb and the system then could not decide which rpool to import

and apparently, that's your problem:

Feb 12 15:09:53 proxmox zpool[1290]: cannot import 'rpool': a pool with that name already exists
Feb 12 15:09:53 proxmox zpool[1290]: use the form 'zpool import <pool | id> <newpool>' to give it a new name
Feb 12 15:09:53 proxmox systemd[1]: zfs-import@nfs.service: Main process exited, code=exited, status=1/FAILURE


your system apparently sees two pools named "rpool", but i'm curious how this can happen when the second instance of rpool is contained within a zvol.

lsblk --fs may tell more
 
RolandK said:
> Of course I can ;-) Which GPT bootsector we're talking about? The one of the PVE host, or the one of the Solaris VM?

the solaris VM one
Click to expand...
OK, here the output of gdisk -l

root@proxmox:~# gdisk -l /dev/zvol/ssd/vm-104-disk-0
GPT fdisk (gdisk) version 1.0.9

Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/zvol/ssd/vm-104-disk-0: 33554432 sectors, 16.0 GiB
Sector size (logical/physical): 512/8192 bytes
Disk identifier (GUID): 2ACAD647-7AAD-1C40-980A-923051CB1C43
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 33554270
Partitions will be aligned on 1-sector boundaries
Total free space is 222 sectors (111.0 KiB)

Number Start (sector) End (sector) Size Code Name
1 256 532735 260.0 MiB EF02
2 532736 33537886 15.7 GiB BF01 zfs
9 33537887 33554270 8.0 MiB BF07

The partition boudaries match perfect with the example you referred to. I attached the image of the GPT extracted with dd
 

Attachments

  • gpt.img.gz
    652 bytes · Views: 1
RolandK said:
i just wanted to write that a typical problem with zfs boot problems is name clash with existing pool names. i had that some times when attaching old root disk via usb and the system then could not decide which rpool to import

and apparently, that's your problem:

Feb 12 15:09:53 proxmox zpool[1290]: cannot import 'rpool': a pool with that name already exists
Feb 12 15:09:53 proxmox zpool[1290]: use the form 'zpool import <pool | id> <newpool>' to give it a new name
Feb 12 15:09:53 proxmox systemd[1]: zfs-import@nfs.service: Main process exited, code=exited, status=1/FAILURE


your system apparently sees two pools named "rpool", but i'm curious how this can happen when the second instance of rpool is contained within a zvol.

lsblk --fs may tell more
Click to expand...
When installing Solairs11 it names the ZPool for OS installation "rpool" - like PVE does. The headers of ZPools created by PVE and Solaris seem to be identical enaugh to be somewhat recognized - but only somwhat. I think that's why PVE is confused.
Here the lsblk output:
root@proxmox:~# lsblk --fs
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
loop0 0 100% /nfs/docs/oreilly
sda
|-sda1 zfs_member 5000 nfs 8948124097204160043
`-sda9
sdb
|-sdb1
|-sdb2 vfat FAT32 E13E-83DC
`-sdb3 zfs_member 5000 rpool 4060493314926062563
sdc
|-sdc1
|-sdc2 vfat FAT32 E140-F5B9
`-sdc3 zfs_member 5000 rpool 4060493314926062563
sdd
|-sdd1 zfs_member 5000 nfs 8948124097204160043
`-sdd9
zd0
|-zd0p1 ntfs System Reserved BA9CD7F59CD7A9E1
|-zd0p2 ntfs 0A76D7F176D7DC0F
`-zd0p3 ntfs 2C18E6E318E6AAD2
zd16
|-zd16p1 zfs_member 44 s11repo 5084459037144786900
`-zd16p9
zd32
|-zd32p1 vfat FAT32 EB2C-9B31
|-zd32p2
`-zd32p5 ext4 1.0 ff1cf506-f2da-4161-8ef9-2628e59f61dc
zd48
|-zd48p1
|-zd48p2 zfs_member 44 rpool 18071598343214744242
`-zd48p9
zd64
|-zd64p1 vfat FAT32 EB2C-9B31
|-zd64p2
`-zd64p5 ext4 1.0 ff1cf506-f2da-4161-8ef9-2628e59f61dc
zd80
|-zd80p1 vfat FAT32 F8CC-8820
|-zd80p2
`-zd80p5 ext4 1.0 ce7a36be-15c8-4740-9641-6a5f6df0e2c0
zd96
|-zd96p1
|-zd96p2 zfs_member 51 rpool 13291557186651320025
|-zd96p4 zfs_member 51 metadb 14289209179245178559
|-zd96p5 zfs_member 51 appl 2755042863140903218
`-zd96p9
zd112
|-zd112p1 ntfs System Reserved 6AEE214AEE210FBD
`-zd112p2 ntfs CEF8215AF82141D7
zd128
|-zd128p1 zfs_member 47 s11ai 4818781621356413986
`-zd128p9
nvme1n1
|-nvme1n1p1 zfs_member 5000 ssd 10903156344271734822
`-nvme1n1p9
nvme0n1
|-nvme0n1p1 zfs_member 5000 ssd 10903156344271734822
`-nvme0n1p9
 
This might help to get the big picture of the setup

root@proxmox:~# zpool status
pool: nfs
state: ONLINE
scan: scrub repaired 0B in 01:46:09 with 0 errors on Sun Feb 11 02:10:10 2024
config:

NAME STATE READ WRITE CKSUM
nfs ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-ST8000NM000A-2KE101_WSD8YRHP ONLINE 0 0 0
ata-ST8000NM000A-2KE101_WSD8Z3Q7 ONLINE 0 0 0

errors: No known data errors

pool: rpool
state: ONLINE
scan: scrub repaired 0B in 00:00:05 with 0 errors on Sun Feb 11 00:24:08 2024
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ata-Samsung_SSD_870_EVO_250GB_S6PENX0T366878K-part3 ONLINE 0 0 0
ata-Samsung_SSD_870_EVO_250GB_S6PENX0T366809E-part3 ONLINE 0 0 0

errors: No known data errors

pool: ssd
state: ONLINE
scan: scrub repaired 0B in 00:01:45 with 0 errors on Sun Feb 11 00:25:49 2024
config:

NAME STATE READ WRITE CKSUM
ssd ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nvme-Samsung_SSD_980_PRO_2TB_S69ENF0W703910X ONLINE 0 0 0
nvme-Samsung_SSD_980_PRO_2TB_S69ENF0W704001D ONLINE 0 0 0

errors: No known data errors

Theses are the ZPools created on PVE itself. They provide the storage for CTs and KVMs.

Running a "zpool import" will usually display a list of importable ZPools. In my setup the command is *really* slow (about 2 minutes) and shows

root@proxmox:~# zpool import
pool: rpool
id: 13291557186651320025
state: UNAVAIL
status: The pool was last accessed by another system.
action: The pool cannot be imported due to damaged devices or data.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
config:

rpool UNAVAIL newer version
zd96p2 ONLINE

pool: metadb
id: 14289209179245178559
state: UNAVAIL
status: The pool was last accessed by another system.
action: The pool cannot be imported due to damaged devices or data.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
config:

metadb UNAVAIL newer version
zd96p4 ONLINE

pool: appl
id: 2755042863140903218
state: UNAVAIL
status: The pool was last accessed by another system.
action: The pool cannot be imported due to damaged devices or data.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
config:

appl UNAVAIL newer version
zd96p5 ONLINE

pool: s11repo
id: 5084459037144786900
state: UNAVAIL
status: The pool was last accessed by another system.
action: The pool cannot be imported due to damaged devices or data.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
config:

s11repo UNAVAIL newer version
zd16 ONLINE

pool: rpool
id: 18071598343214744242
state: UNAVAIL
status: The pool was last accessed by another system.
action: The pool cannot be imported due to damaged devices or data.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
config:

rpool UNAVAIL newer version
zd48 ONLINE

pool: s11ai
id: 4818781621356413986
state: UNAVAIL
status: The pool was last accessed by another system.
action: The pool cannot be imported due to damaged devices or data.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
config:

s11ai UNAVAIL newer version
zd128 ONLINE

All theses ZPools are created from within a Solaris VM
 
i'm curious how the problem can continue to exist ,when you have set volmode=dev on zd48 and zd96 (corresponding) zvols
, as zpool import won't find any pool then

i assume that you probably did not set volmode=dev on both !?
 
RolandK said:
i'm curious how the problem can continue to exist ,when you have set volmode=dev on zd48 and zd96 (corresponding) zvols
, as zpool import won't find any pool then

i assume that you probably did not set volmode=dev on both !?
Click to expand...
I just ran another test reboot

1. set volmode=dev for all Solaris labled zvols
nfs/vm-104-disk-0 volmode dev local
nfs/vm-104-disk-1 volmode dev local
ssd/vm-104-disk-0 volmode dev local
ssd/vm-107-disk-0 volmode dev local
2. reboot PVE

The reboot still hangs and is requesting the admin password.

I attached the logfile

A side effect of setting the volmode to dev is, that the "zpool import" command is running quiet fast and is reporting
root@proxmox:~# zpool import
no pools available to import

The two Solaris VMs are fine with this volmode setting
 

Attachments

  • pve.log
    209.4 KB · Views: 5
now, that's really really weird

Feb 13 14:28:52 proxmox zpool[1275]: cannot import 'rpool': a pool with that name already exists
Feb 13 14:28:52 proxmox zpool[1275]: use the form 'zpool import <pool | id> <newpool>' to give it a new name
Feb 13 14:28:52 proxmox systemd[1]: zfs-import@rpool.service: Main process exited, code=exited, status=1/FAILURE
Feb 13 14:28:52 proxmox systemd[1]: zfs-import@rpool.service: Failed with result 'exit-code'.
Feb 13 14:28:52 proxmox systemd[1]: Failed to start zfs-import@rpool.service - Import ZFS pool rpool.
 
could you exactly point to the line in the logfile where you were dropping to the emergency shell / initramfs? what message do you see there?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back