[SOLVED] rpool don't mount after reboot (random times)

C

chencho

Well-Known Member
Nov 25, 2010
92
8
48
I setup a new server with Proxmox 5 and then upgrade to proxmox 6 (Ovh installer don't have support yet for 6)

The server have 2x1 TB nvme drives in RAID1

After this i set the pool

Code: 
zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
rpool   920G  1,38G   919G        -         -     0%     0%  1.00x    ONLINE  -

Code: 
zfs list
NAME                             USED  AVAIL     REFER  MOUNTPOINT
rpool                           1,37G   890G       96K  /rpool
rpool/backup                      96K   890G       96K  /backup_rpool
rpool/tmp_backup                 160K   890G       96K  /var/lib/vz/tmp_backup
rpool/vmdata                    1,37G   890G      664M  /rpool/vmdata
rpool/vmdata/subvol-801-disk-0   737M  49,3G      736M  /rpool/vmdata/subvol-801-disk-0

After reboot, i cannot open /rpool/vmdata (and the containers cannot start)

Code: 
zpool status
  pool: rpool
state: ONLINE
  scan: none requested
config:

    NAME                                                 STATE     READ WRITE CKSUM
    rpool                                                ONLINE       0     0     0
      mirror-0                                           ONLINE       0     0     0
        nvme-eui.1851af8028610001001b448b441dedcb-part4  ONLINE       0     0     0
        nvme-eui.1851af8061910001001b448b441dc2b6-part4  ONLINE       0     0     0

errors: No known data errors

i need to mount it manually:

Code: 
zfs mount rpool/vmdata

But now i cannot start the ct's

Code: 
sep 12 15:56:48 server7 systemd[1]: Starting PVE LXC Container: 801...
sep 12 15:56:48 server7 lxc-start[4027]: lxc-start: 801: lxccontainer.c: wait_on_daemonized_start: 856 No such file or directory - Failed to receive the container state
sep 12 15:56:48 server7 lxc-start[4027]: lxc-start: 801: tools/lxc_start.c: main: 330 The container failed to start
sep 12 15:56:48 server7 lxc-start[4027]: lxc-start: 801: tools/lxc_start.c: main: 333 To get more details, run the container in foreground mode
sep 12 15:56:48 server7 lxc-start[4027]: lxc-start: 801: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority options
sep 12 15:56:48 server7 systemd[1]: pve-container@801.service: Control process exited, code=exited, status=1/FAILURE
sep 12 15:56:48 server7 systemd[1]: pve-container@801.service: Failed with result 'exit-code'.
sep 12 15:56:48 server7 systemd[1]: Failed to start PVE LXC Container: 801.
 
please post:
* `zpool get all rpool`
* `zfs get all rpool`
(if possible while the pool is not mounted)

quite often this problem is due to a not properly set and updated zfs-cache file - you could try:
`zpool set cachefile=/etc/zfs/zpool.cache`
`update-initramfs -k all -u`
`reboot`

I hope this helps!
 
Hi Stoiko.

* `zpool get all rpool`

Code: 
NAME   PROPERTY                       VALUE                          SOURCE
rpool  size                           920G                           -
rpool  capacity                       0%                             -
rpool  altroot                        -                              default
rpool  health                         ONLINE                         -
rpool  guid                           918375377353186025             -
rpool  version                        -                              default
rpool  bootfs                         -                              default
rpool  delegation                     on                             default
rpool  autoreplace                    off                            default
rpool  cachefile                      none                           local
rpool  failmode                       wait                           default
rpool  listsnapshots                  off                            default
rpool  autoexpand                     off                            default
rpool  dedupditto                     0                              default
rpool  dedupratio                     1.00x                          -
rpool  free                           919G                           -
rpool  allocated                      1,38G                          -
rpool  readonly                       off                            -
rpool  ashift                         12                             local
rpool  comment                        -                              default
rpool  expandsize                     -                              -
rpool  freeing                        0                              -
rpool  fragmentation                  0%                             -
rpool  leaked                         0                              -
rpool  multihost                      off                            default
rpool  checkpoint                     -                              -
rpool  load_guid                      12095906521435290469           -
rpool  autotrim                       off                            default
rpool  feature@async_destroy          enabled                        local
rpool  feature@empty_bpobj            active                         local
rpool  feature@lz4_compress           active                         local
rpool  feature@multi_vdev_crash_dump  enabled                        local
rpool  feature@spacemap_histogram     active                         local
rpool  feature@enabled_txg            active                         local
rpool  feature@hole_birth             active                         local
rpool  feature@extensible_dataset     active                         local
rpool  feature@embedded_data          active                         local
rpool  feature@bookmarks              enabled                        local
rpool  feature@filesystem_limits      enabled                        local
rpool  feature@large_blocks           enabled                        local
rpool  feature@large_dnode            enabled                        local
rpool  feature@sha512                 enabled                        local
rpool  feature@skein                  enabled                        local
rpool  feature@edonr                  enabled                        local
rpool  feature@userobj_accounting     active                         local
rpool  feature@encryption             enabled                        local
rpool  feature@project_quota          active                         local
rpool  feature@device_removal         enabled                        local
rpool  feature@obsolete_counts        enabled                        local
rpool  feature@zpool_checkpoint       enabled                        local
rpool  feature@spacemap_v2            active                         local
rpool  feature@allocation_classes     enabled                        local
rpool  feature@resilver_defer         enabled                        local
rpool  feature@bookmark_v2            enabled                        local

* `zfs get all rpool`

Code: 
zfs get all rpool
NAME   PROPERTY              VALUE                   SOURCE
rpool  type                  filesystem              -
rpool  creation              mié sep 11 13:34 2019  -
rpool  used                  1,38G                   -
rpool  available             890G                    -
rpool  referenced            96K                     -
rpool  compressratio         1.95x                   -
rpool  mounted               no                      -
rpool  quota                 none                    default
rpool  reservation           none                    default
rpool  recordsize            128K                    default
rpool  mountpoint            /rpool                  default
rpool  sharenfs              off                     default
rpool  checksum              on                      local
rpool  compression           lz4                     local
rpool  atime                 off                     local
rpool  devices               on                      default
rpool  exec                  on                      default
rpool  setuid                on                      default
rpool  readonly              off                     default
rpool  zoned                 off                     default
rpool  snapdir               hidden                  default
rpool  aclinherit            restricted              default
rpool  createtxg             1                       -
rpool  canmount              on                      default
rpool  xattr                 sa                      local
rpool  copies                1                       default
rpool  version               5                       -
rpool  utf8only              off                     -
rpool  normalization         none                    -
rpool  casesensitivity       sensitive               -
rpool  vscan                 off                     default
rpool  nbmand                off                     default
rpool  sharesmb              off                     default
rpool  refquota              none                    default
rpool  refreservation        none                    default
rpool  guid                  10837788631471600174    -
rpool  primarycache          all                     local
rpool  secondarycache        all                     default
rpool  usedbysnapshots       0B                      -
rpool  usedbydataset         96K                     -
rpool  usedbychildren        1,38G                   -
rpool  usedbyrefreservation  0B                      -
rpool  logbias               latency                 default
rpool  objsetid              54                      -
rpool  dedup                 off                     local
rpool  mlslabel              none                    default
rpool  sync                  standard                default
rpool  dnodesize             legacy                  default
rpool  refcompressratio      1.00x                   -
rpool  written               0                       -
rpool  logicalused           2,59G                   -
rpool  logicalreferenced     42K                     -
rpool  volmode               default                 default
rpool  filesystem_limit      none                    default
rpool  snapshot_limit        none                    default
rpool  filesystem_count      none                    default
rpool  snapshot_count        none                    default
rpool  snapdev               hidden                  default
rpool  acltype               off                     default
rpool  context               none                    default
rpool  fscontext             none                    default
rpool  defcontext            none                    default
rpool  rootcontext           none                    default
rpool  relatime              off                     local
rpool  redundant_metadata    all                     default
rpool  overlay               off                     default
rpool  encryption            off                     default
rpool  keylocation           none                    default
rpool  keyformat             none                    default
rpool  pbkdf2iters           0                       default
rpool  special_small_blocks  0                       default

Cache file ask about pool. But always get same error, with /rpool/vmdata or rpool/vmdata

Code: 
rpool/vmdata                    1,37G   890G      664M  /rpool/vmdata

zpool set cachefile=/etc/zfs/zpool.cache /rpool/vmdata
cannot open '/rpool/vmdata': invalid character '/' in pool name
 
Here is the error while try to start ct:
Code: 
lxc-start -F -f /etc/pve/lxc/801.conf --name sql1 --logfile /tmp/lxc.log --logpriority TRACE
lxc-start: sql1: utils.c: safe_mount: 1187 No such file or directory - Failed to mount "/dev/pts/1" onto "/dev/console"
lxc-start: sql1: conf.c: lxc_setup_dev_console: 1793 Failed to mount "/dev/pts/1" on "/dev/console"
 
Your right about cache, i think:

Code: 
systemctl -l status zfs*
● zfs-mount.service - Mount ZFS filesystems
   Loaded: loaded (/lib/systemd/system/zfs-mount.service; enabled; vendor preset: enabled)
   Active: active (exited) since Thu 2019-09-12 22:50:18 CEST; 21min ago
     Docs: man:zfs(8)
  Process: 834 ExecStart=/sbin/zfs mount -a (code=exited, status=0/SUCCESS)
Main PID: 834 (code=exited, status=0/SUCCESS)

sep 12 22:50:18 server7 systemd[1]: Starting Mount ZFS filesystems...
sep 12 22:50:18 server7 systemd[1]: Started Mount ZFS filesystems.

● zfs-import-cache.service - Import ZFS pools by cache file
   Loaded: loaded (/lib/systemd/system/zfs-import-cache.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2019-09-12 22:50:18 CEST; 21min ago
     Docs: man:zpool(8)
  Process: 833 ExecStart=/sbin/zpool import -c /etc/zfs/zpool.cache -aN (code=exited, status=1/FAILURE)
Main PID: 833 (code=exited, status=1/FAILURE)

sep 12 22:50:18 server7 systemd[1]: Starting Import ZFS pools by cache file...
sep 12 22:50:18 server7 zpool[833]: invalid or corrupt cache file contents: invalid or missing cache file
sep 12 22:50:18 server7 systemd[1]: zfs-import-cache.service: Main process exited, code=exited, status=1/FAILURE
sep 12 22:50:18 server7 systemd[1]: zfs-import-cache.service: Failed with result 'exit-code'.
sep 12 22:50:18 server7 systemd[1]: Failed to start Import ZFS pools by cache file.

I delete the cache file, and now have no errors after reboot; but zfs don't mount the data and after mount it manually, ct don't start
 
Last edited:
chencho said:
I delete the cache file, and now have no errors after reboot;
Click to expand...
the cache file probably is still in the initramfs...

chencho said:
zpool set cachefile=/etc/zfs/zpool.cache /rpool/vmdata cannot open '/rpool/vmdata': invalid character '/' in pool name
Click to expand...
theres a tiny typo here:
Code: 
zpool set cachefile=/etc/zfs/zpool.cache rpool/vmdata
(no / in the beginning before the poolname)

afterwards:
Code: 
update-initramfs -k all -u

and reboot
 
Finally i reinstall all (it's a fresh server)

Now i can reboot and seems to work fine.

By the way, zpool set cachefile=/etc/zfs/zpool.cache rpool/vmdata give me same error "invalid character '/' in pool name"

Thanks Stoiko
 
chencho said:
By the way, zpool set cachefile=/etc/zfs/zpool.cache rpool/vmdata give me same error "invalid character '/' in pool name"
Click to expand...
sorry - again mistyped - you would have needed to provide only the poolname 'rpool'

But glad it worked out with a reinstall!
 
Emmmm... i put my CT into local storage, not ZFS pool :(

When i move ir to pool-vmdata and reboot... it cannot start again :'(

Same error with cache file

Code: 
● zfs-import-cache.service - Import ZFS pools by cache file
   Loaded: loaded (/lib/systemd/system/zfs-import-cache.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2019-09-13 12:47:22 CEST; 3min 2s ago
     Docs: man:zpool(8)
  Process: 873 ExecStart=/sbin/zpool import -c /etc/zfs/zpool.cache -aN (code=exited, status=1/FAILURE)
Main PID: 873 (code=exited, status=1/FAILURE)

sep 13 12:47:22 server7 systemd[1]: Starting Import ZFS pools by cache file...
sep 13 12:47:22 server7 zpool[873]: invalid or corrupt cache file contents: invalid or missing cache file
sep 13 12:47:22 server7 systemd[1]: zfs-import-cache.service: Main process exited, code=exited, status=1/FAILURE
sep 13 12:47:22 server7 systemd[1]: zfs-import-cache.service: Failed with result 'exit-code'.
sep 13 12:47:22 server7 systemd[1]: Failed to start Import ZFS pools by cache file.

After

Code: 
root@server7:~# zpool set cachefile=/etc/zfs/zpool.cache rpool
root@server7:~# update-initramfs -k all -u
update-initramfs: Generating /boot/initrd.img-5.0.21-2-pve
cp: '/etc/udev/rules.d/70-persistent-net.rules' y '/etc/udev/rules.d/70-persistent-net.rules' son el mismo fichero
cp: '/etc/udev/rules.d/70-persistent-net.rules' y '/etc/udev/rules.d/70-persistent-net.rules' son el mismo fichero
Running hook script 'zz-pve-efiboot'..
Re-executing '/etc/kernel/postinst.d/zz-pve-efiboot' in new private mount namespace..
No /etc/kernel/pve-efiboot-uuids found, skipping ESP sync.
update-initramfs: Generating /boot/initrd.img-4.15.18-21-pve
cp: '/etc/udev/rules.d/70-persistent-net.rules' y '/etc/udev/rules.d/70-persistent-net.rules' son el mismo fichero
cp: '/etc/udev/rules.d/70-persistent-net.rules' y '/etc/udev/rules.d/70-persistent-net.rules' son el mismo fichero
Running hook script 'zz-pve-efiboot'..
Re-executing '/etc/kernel/postinst.d/zz-pve-efiboot' in new private mount namespace..
No /etc/kernel/pve-efiboot-uuids found, skipping ESP sync.

It give me another error:

Code: 
● zfs-mount.service - Mount ZFS filesystems
   Loaded: loaded (/lib/systemd/system/zfs-mount.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2019-09-13 12:53:05 CEST; 38s ago
     Docs: man:zfs(8)
  Process: 1132 ExecStart=/sbin/zfs mount -a (code=exited, status=1/FAILURE)
Main PID: 1132 (code=exited, status=1/FAILURE)

sep 13 12:53:05 server7 systemd[1]: Starting Mount ZFS filesystems...
sep 13 12:53:05 server7 zfs[1132]: cannot mount '/backups': directory is not empty
sep 13 12:53:05 server7 systemd[1]: zfs-mount.service: Main process exited, code=exited, status=1/FAILURE
sep 13 12:53:05 server7 systemd[1]: zfs-mount.service: Failed with result 'exit-code'.
sep 13 12:53:05 server7 systemd[1]: Failed to start Mount ZFS filesystems.

ok, i need to delete /backups content , it's normal.

Now it boot and start my CT !!!!!!!!!!!!!!! :)
 
Last edited:
  • Like
Reactions: Stoiko Ivanov
This is the screen where you select the container-template (which container you want to install (the root disk is the next selection)

in any case you need to select the appropriate storage-type (Container template for templates, Container for an actual container) for the storage (Datacenter->Storage)
see https://pve.proxmox.com/pve-docs/chapter-pvesm.html
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back