[SOLVED] PBS takes ages to boot and shutdown (zfs)

N

nemexx

New Member
Feb 22, 2022
4
2
3
Servus!

1st of all i'm new to proxmox PBS and PVE (and ZFS also).
I have a playground for testing purposal and try to become warmed up with the proxmox stuff e.g. clustering, backup etc.

Currently i am testing how i can use an external usb drive as datastore for an offsite backup of important vm's (i am using a simple usb stick with 256gb capacity).

My goal:
1. simply plug in the usb drive to the pbs
2. run a backup job when i need to make the offsite backup
3. unplug and i am done

What i did:

add long term storage:
1. mounted an NFS share (yes yes, slow etc. i know)
2. created a datastore
3. added to fstab

so far so good, all fine, but that's not the problem

next i wanted to test the offsite usb stuff

1. plugged in usb drive
2. created zfs (single disk)
3. created a datastore
-> worked so far

now the trouble begins

i tested a simple shutdown of the server and unplugged the usb drive

while booting i got:
1645601479907.png

yes, reasonable. (my thoughs were: maybe zfs is not the best idea for a offsite backup storage - try directory instead)

1. ok, so i booted in rescue mode and tried to remove the zfs pool but zpool and zfs couldnt find my pool "zfs_usbstick". i do not remeber what exactly it was saying but i couldn't do a destroy because the pool could not be found. (usb device was not plugged in)
2. okay, power off the server, plugged in the usb device, boot, hope it will recognize the device
3. same message again (see image).
4. used google, and realized pbs started after minutes in background. (webgui told me that "zfs_usbstick" isnt available)
5. okay, i was thinking delete the datastore "zfs_usbstick" in webgui and trouble is gone.
6. that worked surprisingly (but afterwards i think it only deleted the entry in datastore.cfg?)
7. reboot (the shutdown took ages)
8. same message on boot again

okay, lets wait if pbs will boot in a few minutes like before, and it did.

now i was thinking i simply try to (re-)create the same zfs pool and datastore again, with same name and maybe it will work again. (i know it doenst make any sense in production but i was trying to learn troubleshooting)

so same steps again:

1. formatted the usb drive
2. created zfs with same name (single disk)
3. created the datastore

fine, that worked and was visible in webgui with no errors.

now i rebooted pbs again (shutdown took ages again)

and pbs started with the same failure i am facing all the time "Failed to start Import ZFS pool zfs_usbstick"

i think i messed up everything

how can i remove the zfs pool and get rid of this?

Or is there a "Best Practice" for offsite backup with a simple external disk?

My goal is simply

1. plug in device (on running server)
2. start backup
3. unplug usb drive
4. done

and i do not wanna have trouble when the server gets a reboot (or powerloss) and the usb drive is not connected.
 
Last edited:
While i was writing this topic and did a little bit of try and error i finally did a step forward regarding to the problem, i was able to recreate the zfs and all configurations.

It now shows up again and is working, BUT only if PBS boot SLOW.
If PBS boot FAST, it is not working.

My datastore.cfg:
Bash: 
root@prx-backup-srv:~# cat /etc/proxmox-backup/datastore.cfg
datastore: nfs-backup-store
        comment NFS Share || xx.xx.xx.xx:/backup-store gemounted auf /var/backup-store-nfs/
        gc-schedule daily
        path /var/backup-store-nfs/
        prune-schedule daily

datastore: zfs_usbstick
        path /mnt/datastore/zfs_usbstick

The problem i am still facing is the startup of the PBS.
It sometimes take ages (~5-6 minutes)
and
it sometimes boot straight up (but always with this error)

1645610099015.png

I am confused right now...


PBS starts SLOWLY syslog says:

Bash: 
Feb 23 11:20:22 prx-backup-srv systemd[1]: Starting Import ZFS pool zfs_usbstick...
Feb 23 11:20:22 prx-backup-srv systemd[1]: Finished Import ZFS pool zfs_usbstick.
Feb 23 11:20:22 prx-backup-srv zed: eid=8 class=config_sync pool='zfs_usbstick'
Feb 23 11:20:22 prx-backup-srv zed: eid=3 class=config_sync pool='zfs_usbstick'
Feb 23 11:20:22 prx-backup-srv zed: eid=4 class=pool_import pool='zfs_usbstick'
Feb 23 11:20:22 prx-backup-srv zed: eid=6 class=delay pool='zfs_usbstick' vdev=usb-PNY_USB_3.0_FD_070B66A492159936-0:0-part1 size=4096 offset=139590639616 priority=3 err=0 flags=0x184880 delay=30355ms bookmark=0:0:-1:0
Feb 23 11:20:22 prx-backup-srv zed: eid=5 class=delay pool='zfs_usbstick' vdev=usb-PNY_USB_3.0_FD_070B66A492159936-0:0-part1 size=4096 offset=135295668224 priority=3 err=0 flags=0x184880 delay=30456ms bookmark=0:0:1:0
Feb 23 11:20:22 prx-backup-srv zed: eid=1 class=delay pool='zfs_usbstick' vdev=usb-PNY_USB_3.0_FD_070B66A492159936-0:0-part1 size=4096 offset=139590631424 priority=3 err=0 flags=0x184880 delay=30276ms bookmark=0:0:-1:0

WebGUI
1645609183154.png

CLI
Bash: 
root@prx-backup-srv:~# zfs list
NAME           USED  AVAIL     REFER  MOUNTPOINT
zfs_usbstick  24.8M   227G     24.3M  /mnt/datastore/zfs_usbstick
root@prx-backup-srv:~# zpool list
NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zfs_usbstick   234G  24.8M   234G        -         -     0%     0%  1.00x    ONLINE  -


PBS starts FAST syslog says:

Bash: 
Feb 23 11:24:16 prx-backup-srv systemd[1]: Starting Import ZFS pool zfs_usbstick...
Feb 23 11:24:16 prx-backup-srv zpool[438]: cannot import 'zfs_usbstick': no such pool available
Feb 23 11:24:16 prx-backup-srv systemd[1]: zfs-import@zfs_usbstick.service: Main process exited, code=exited, status=1/FAILURE
Feb 23 11:24:16 prx-backup-srv systemd[1]: zfs-import@zfs_usbstick.service: Failed with result 'exit-code'.
Feb 23 11:24:16 prx-backup-srv systemd[1]: Failed to start Import ZFS pool zfs_usbstick.

WebGUI
1645612014386.png
CLI
Bash: 
root@prx-backup-srv:~# zfs list
no datasets available
root@prx-backup-srv:~# zpool list
no pools available
root@prx-backup-srv:~#




I guess there might be some debris in configs files somewhere...
But i can't find where the problem is...

But i am still looking for my goal to use a usb device as plug n play datastore.
 
nemexx said:
While i was writing this topic and did a little bit of try and error i finally did a step forward regarding to the problem, i was able to recreate the zfs and all configurations.

It now shows up again and is working, BUT only if PBS boot SLOW.
If PBS boot FAST, it is not working.

My datastore.cfg:
Bash: 
root@prx-backup-srv:~# cat /etc/proxmox-backup/datastore.cfg
datastore: nfs-backup-store
        comment NFS Share || xx.xx.xx.xx:/backup-store gemounted auf /var/backup-store-nfs/
        gc-schedule daily
        path /var/backup-store-nfs/
        prune-schedule daily

datastore: zfs_usbstick
        path /mnt/datastore/zfs_usbstick

The problem i am still facing is the startup of the PBS.
It sometimes take ages (~5-6 minutes)
and
it sometimes boot straight up (but always with this error)

View attachment 34535

I am confused right now...


PBS starts SLOWLY syslog says:

Bash: 
Feb 23 11:20:22 prx-backup-srv systemd[1]: Starting Import ZFS pool zfs_usbstick...
Feb 23 11:20:22 prx-backup-srv systemd[1]: Finished Import ZFS pool zfs_usbstick.
Feb 23 11:20:22 prx-backup-srv zed: eid=8 class=config_sync pool='zfs_usbstick'
Feb 23 11:20:22 prx-backup-srv zed: eid=3 class=config_sync pool='zfs_usbstick'
Feb 23 11:20:22 prx-backup-srv zed: eid=4 class=pool_import pool='zfs_usbstick'
Feb 23 11:20:22 prx-backup-srv zed: eid=6 class=delay pool='zfs_usbstick' vdev=usb-PNY_USB_3.0_FD_070B66A492159936-0:0-part1 size=4096 offset=139590639616 priority=3 err=0 flags=0x184880 delay=30355ms bookmark=0:0:-1:0
Feb 23 11:20:22 prx-backup-srv zed: eid=5 class=delay pool='zfs_usbstick' vdev=usb-PNY_USB_3.0_FD_070B66A492159936-0:0-part1 size=4096 offset=135295668224 priority=3 err=0 flags=0x184880 delay=30456ms bookmark=0:0:1:0
Feb 23 11:20:22 prx-backup-srv zed: eid=1 class=delay pool='zfs_usbstick' vdev=usb-PNY_USB_3.0_FD_070B66A492159936-0:0-part1 size=4096 offset=139590631424 priority=3 err=0 flags=0x184880 delay=30276ms bookmark=0:0:-1:0

WebGUI
View attachment 34533

CLI
Bash: 
root@prx-backup-srv:~# zfs list
NAME           USED  AVAIL     REFER  MOUNTPOINT
zfs_usbstick  24.8M   227G     24.3M  /mnt/datastore/zfs_usbstick
root@prx-backup-srv:~# zpool list
NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zfs_usbstick   234G  24.8M   234G        -         -     0%     0%  1.00x    ONLINE  -


PBS starts FAST syslog says:

Bash: 
Feb 23 11:24:16 prx-backup-srv systemd[1]: Starting Import ZFS pool zfs_usbstick...
Feb 23 11:24:16 prx-backup-srv zpool[438]: cannot import 'zfs_usbstick': no such pool available
Feb 23 11:24:16 prx-backup-srv systemd[1]: zfs-import@zfs_usbstick.service: Main process exited, code=exited, status=1/FAILURE
Feb 23 11:24:16 prx-backup-srv systemd[1]: zfs-import@zfs_usbstick.service: Failed with result 'exit-code'.
Feb 23 11:24:16 prx-backup-srv systemd[1]: Failed to start Import ZFS pool zfs_usbstick.

WebGUI
View attachment 34540
CLI
Bash: 
root@prx-backup-srv:~# zfs list
no datasets available
root@prx-backup-srv:~# zpool list
no pools available
root@prx-backup-srv:~#




I guess there might be some debris in configs files somewhere...
But i can't find where the problem is...

But i am still looking for my goal to use a usb device as plug n play datastore.
Click to expand...
Same problem too !!! He no longer leaves, he goes in search of what has been removed.
 
nemexx said:
While i was writing this topic and did a little bit of try and error i finally did a step forward regarding to the problem, i was able to recreate the zfs and all configurations.

It now shows up again and is working, BUT only if PBS boot SLOW.
If PBS boot FAST, it is not working.

My datastore.cfg:
Bash: 
root@prx-backup-srv:~# cat /etc/proxmox-backup/datastore.cfg
datastore: nfs-backup-store
        comment NFS Share || xx.xx.xx.xx:/backup-store gemounted auf /var/backup-store-nfs/
        gc-schedule daily
        path /var/backup-store-nfs/
        prune-schedule daily

datastore: zfs_usbstick
        path /mnt/datastore/zfs_usbstick

The problem i am still facing is the startup of the PBS.
It sometimes take ages (~5-6 minutes)
and
it sometimes boot straight up (but always with this error)

View attachment 34535

I am confused right now...


PBS starts SLOWLY syslog says:

Bash: 
Feb 23 11:20:22 prx-backup-srv systemd[1]: Starting Import ZFS pool zfs_usbstick...
Feb 23 11:20:22 prx-backup-srv systemd[1]: Finished Import ZFS pool zfs_usbstick.
Feb 23 11:20:22 prx-backup-srv zed: eid=8 class=config_sync pool='zfs_usbstick'
Feb 23 11:20:22 prx-backup-srv zed: eid=3 class=config_sync pool='zfs_usbstick'
Feb 23 11:20:22 prx-backup-srv zed: eid=4 class=pool_import pool='zfs_usbstick'
Feb 23 11:20:22 prx-backup-srv zed: eid=6 class=delay pool='zfs_usbstick' vdev=usb-PNY_USB_3.0_FD_070B66A492159936-0:0-part1 size=4096 offset=139590639616 priority=3 err=0 flags=0x184880 delay=30355ms bookmark=0:0:-1:0
Feb 23 11:20:22 prx-backup-srv zed: eid=5 class=delay pool='zfs_usbstick' vdev=usb-PNY_USB_3.0_FD_070B66A492159936-0:0-part1 size=4096 offset=135295668224 priority=3 err=0 flags=0x184880 delay=30456ms bookmark=0:0:1:0
Feb 23 11:20:22 prx-backup-srv zed: eid=1 class=delay pool='zfs_usbstick' vdev=usb-PNY_USB_3.0_FD_070B66A492159936-0:0-part1 size=4096 offset=139590631424 priority=3 err=0 flags=0x184880 delay=30276ms bookmark=0:0:-1:0

WebGUI
View attachment 34533

CLI
Bash: 
root@prx-backup-srv:~# zfs list
NAME           USED  AVAIL     REFER  MOUNTPOINT
zfs_usbstick  24.8M   227G     24.3M  /mnt/datastore/zfs_usbstick
root@prx-backup-srv:~# zpool list
NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zfs_usbstick   234G  24.8M   234G        -         -     0%     0%  1.00x    ONLINE  -


PBS starts FAST syslog says:

Bash: 
Feb 23 11:24:16 prx-backup-srv systemd[1]: Starting Import ZFS pool zfs_usbstick...
Feb 23 11:24:16 prx-backup-srv zpool[438]: cannot import 'zfs_usbstick': no such pool available
Feb 23 11:24:16 prx-backup-srv systemd[1]: zfs-import@zfs_usbstick.service: Main process exited, code=exited, status=1/FAILURE
Feb 23 11:24:16 prx-backup-srv systemd[1]: zfs-import@zfs_usbstick.service: Failed with result 'exit-code'.
Feb 23 11:24:16 prx-backup-srv systemd[1]: Failed to start Import ZFS pool zfs_usbstick.

WebGUI
View attachment 34540
CLI
Bash: 
root@prx-backup-srv:~# zfs list
no datasets available
root@prx-backup-srv:~# zpool list
no pools available
root@prx-backup-srv:~#




I guess there might be some debris in configs files somewhere...
But i can't find where the problem is...

But i am still looking for my goal to use a usb device as plug n play datastore.
Click to expand...

Then I solved on my PBS in these terms:

After starting in safe mode and typing the root password, I typed the following path:
Code: 
root@pbs:/mnt/datastore# ls -l
total 17
drwxr-xr-x 5 backup backup 7 Feb 23 23:41 ZFS_STORAGE
drwxr-xr-x 5 backup backup 7 Feb 23 21:09 ZFS_USB

What you see is the assembly of the units, in the example mentioned above, there is no mounting point that gave error so I deleted it, but you should see your old mount point name.

Then the system must be restarted with the reboot command.

At startup it will always give you the same error, as the problem is that systemctl mounts the service related to the assembly of the old unit, so I performed the following from the image:

1.png

So systemctl disables the mounting of that specific ZFS drive that no longer exists in practice.

After restarting my PBS works as before but without blocking errors or alert messages.
 
  • Like
Reactions: nemexx
Dont use ZFS for an USB pen drive. It will kill it quickly and it will be very slow.

With ZFS as a portable backup storage you usually, plug in the drive, manually import the pool (zpool import), mount it (zfs mount -a), do your backups, then unmount it and export the pool (zpool export), and unplug the drive. While the pool is exported ZFS wont complain when the drive is not plugged in.
 
Last edited:
  • Like
Reactions: nemexx
frankz said:
Then I solved on my PBS in these terms:

After starting in safe mode and typing the root password, I typed the following path:
Code: 
root@pbs:/mnt/datastore# ls -l
total 17
drwxr-xr-x 5 backup backup 7 Feb 23 23:41 ZFS_STORAGE
drwxr-xr-x 5 backup backup 7 Feb 23 21:09 ZFS_USB

What you see is the assembly of the units, in the example mentioned above, there is no mounting point that gave error so I deleted it, but you should see your old mount point name.

Then the system must be restarted with the reboot command.

At startup it will always give you the same error, as the problem is that systemctl mounts the service related to the assembly of the old unit, so I performed the following from the image:

View attachment 34592

So systemctl disables the mounting of that specific ZFS drive that no longer exists in practice.

After restarting my PBS works as before but without blocking errors or alert messages.
Click to expand...
Thanks @frankz
solution worked!

Dunuin said:
Dont use ZFS for an USB pen drive. It will kill it quickly and it will be very slow.

With ZFS as a portable backup storage you usually, plug in the drive, manually import the pool (zpool import), mount it (zfs mount -a), do your backups, then unmount it and export the pool (zpool export), and unplug the drive. While the pool is exported ZFS wont complain when the drive is not plugged in.
Click to expand...

USB pen drive was only for testing how it works.

I got it, if you use portable storage with zfs, u always need to export it before unplugging! (and import it to zfs when plugged in)
Thanks man!!

like i said before, first time i am working with zfs. I am learning :)
 
  • Like
Reactions: frankz
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom