[SOLVED] ZED Notifications not working

apoc · Oct 24, 2017

Hello everybody,

I hope someone can shed some light in this as I did not find much in the WWW nor in the forum.
I am in the process completing my proxmox setup (5.0) and want to setup ZED for ZFS monitoring.

ZED is installed and active, but it does not send out any notifications. The postfix MTA itself works fine (tested with mails from various scripts).

As this will be one of the first questions here is my zed.rc file

Code:

#ZED_DEBUG_LOG="/tmp/zed.debug.log"
ZED_EMAIL_ADDR="<mailadress removed>"
ZED_EMAIL_PROG="mail"
ZED_EMAIL_OPTS="-s '@SUBJECT@' @ADDRESS@"
ZED_LOCKDIR="/var/lock"
ZED_NOTIFY_INTERVAL_SECS=3600
ZED_NOTIFY_VERBOSE=1
ZED_NOTIFY_DATA=1
#ZED_PUSHBULLET_ACCESS_TOKEN=""
#ZED_PUSHBULLET_CHANNEL_TAG=""
ZED_RUNDIR="/var/run"
#ZED_USE_ENCLOSURE_LEDS=1
#ZED_SYSLOG_PRIORITY="daemon.notice"
ZED_SYSLOG_TAG="zed"
ZED_SPARE_ON_CHECKSUM_ERRORS=5
ZED_SPARE_ON_IO_ERRORS=2

I tried to execute the scripts in /etc/zfs/zed.d manually and test them, but I get the following error-message:

Code:

./data-notify.sh
./data-notify.sh: 17: .: Can't open /zed-functions.sh

The functions file is within the folder:

Code:

/etc/zfs/zed.d# ls -l
total 24
lrwxrwxrwx 1 root root    49 Oct 12 13:34 all-syslog.sh -> /usr/lib/x86_64-linux-gnu/zfs/zed.d/all-syslog.sh
lrwxrwxrwx 1 root root    50 Oct 12 13:34 data-notify.sh -> /usr/lib/x86_64-linux-gnu/zfs/zed.d/data-notify.sh
lrwxrwxrwx 1 root root    54 Oct 12 13:34 pool_import-led.sh -> /usr/lib/x86_64-linux-gnu/zfs/zed.d/pool_import-led.sh
lrwxrwxrwx 1 root root    61 Oct 12 13:34 resilver_finish-notify.sh -> /usr/lib/x86_64-linux-gnu/zfs/zed.d/resilver_finish-notify.sh
lrwxrwxrwx 1 root root    58 Oct 12 13:34 scrub_finish-notify.sh -> /usr/lib/x86_64-linux-gnu/zfs/zed.d/scrub_finish-notify.sh
lrwxrwxrwx 1 root root    54 Oct 12 13:34 statechange-led.sh -> /usr/lib/x86_64-linux-gnu/zfs/zed.d/statechange-led.sh
lrwxrwxrwx 1 root root    57 Oct 12 13:34 statechange-notify.sh -> /usr/lib/x86_64-linux-gnu/zfs/zed.d/statechange-notify.sh
lrwxrwxrwx 1 root root    54 Oct 12 13:34 vdev_attach-led.sh -> /usr/lib/x86_64-linux-gnu/zfs/zed.d/vdev_attach-led.sh
lrwxrwxrwx 1 root root    53 Oct 12 13:34 vdev_clear-led.sh -> /usr/lib/x86_64-linux-gnu/zfs/zed.d/vdev_clear-led.sh
-rw-r--r-- 1 root root 11447 Oct 12 13:34 zed-functions.sh
-rw-r--r-- 1 root root   464 Oct 23 21:42 zed.rc

I then manually modified one of the scripts to write out the variable "${ZED_ZEDLET_DIR}" which is used to find the functions file and guess what, it is empty.

Cant tell if this test is really valid / valuable but I didnt know what else I could do. I also modified file permissions to 777 to rule out a permission issue, no luck as well.

Has anyone suggestions how to get this flying?
Thanks and with best regards

/edit: corrected a typo in the configuration, but that did not change anything

Alwin · Oct 24, 2017

Is the package 'zfs-zed' installed? 3.8.5. Activate E-Mail Notification -> has the rest of the config.
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_zfs_on_linux

apoc · Oct 24, 2017

Hello Alwin,

yes zfs-zed is installed:

Code:

sudo apt-get install zfs-zed
[sudo] password for administrator:
Reading package lists... Done
Building dependency tree       
Reading state information... Done
zfs-zed is already the newest version (0.7.2-pve1~bpo90).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Why are you mentioning the URL?
The only thing I can find is this:

Code:

ZED_EMAIL_ADDR="root"

My eMail address is in a different format so I am able to deliver it to the correct mailbox:

Code:

username@domain.de

Does this matter?
Thanks

kobuki · Oct 24, 2017

What does "systemctl status zfs-zed.service" say? Don't mind the errors running zedlets standalone. They're supposed to run from the daemon. Also, check your postfix logs.

0xFelix · Oct 25, 2017

I can confirm, I have similiar problems. On PVE 5.0 zed was working fine and reported problems via mail. After the upgrade to 5.1 zed is still running but no mails are sent (or at least received?).

apoc · Oct 25, 2017

Thanks Kobuki and 0xFelix,

Code:

 sudo systemctl status zfs-zed.service
● zfs-zed.service - ZFS Event Daemon (zed)
   Loaded: loaded (/lib/systemd/system/zfs-zed.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2017-10-25 07:57:08 CEST; 47min ago
     Docs: man:zed(8)
 Main PID: 11952 (zed)
    Tasks: 3 (limit: 4915)
   CGroup: /system.slice/zfs-zed.service
           └─11952 /usr/sbin/zed -F

Oct 25 07:57:08 proxmox systemd[1]: Started ZFS Event Daemon (zed).
Oct 25 07:57:08 proxmox zed[11952]: ZFS Event Daemon 0.7.2-pve1~bpo90 (PID 11952)
Oct 25 07:57:08 proxmox zed[11952]: Processing events since eid=0

The postfix logs show the succesful delivery from my testsfrom the CLI, but no errors.

I went into more testing yesterday and noticed that

Code:

zpool events

returned a pretty strang error message. I havent captured it (sadly) but it was somthing like "invalid descriptor" or similar. I found that pretty scary.
Additionally the

Code:

zpool status

returned

Code:

status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 4h45m with 0 errors on Mon Oct 23 13:24:50 2017

That raised the idea that maybe the server needs a reboot, even if that is not stated anywhere (no status file in /var/run, etc.)

So I did this morning and I got something back from the events query

Code:

sudo zpool events
TIME                           CLASS
Oct 25 2017 07:57:05.641503194 sysevent.fs.zfs.history_event
Oct 25 2017 07:57:06.041503198 sysevent.fs.zfs.config_sync
Oct 25 2017 07:57:06.041503198 sysevent.fs.zfs.pool_import
Oct 25 2017 07:57:06.041503198 sysevent.fs.zfs.history_event
Oct 25 2017 07:57:06.341503202 sysevent.fs.zfs.config_sync
Oct 25 2017 07:57:06.541503205 sysevent.fs.zfs.history_event
Oct 25 2017 07:57:06.657503206 sysevent.fs.zfs.config_sync
Oct 25 2017 07:57:06.657503206 sysevent.fs.zfs.pool_import
Oct 25 2017 07:57:06.661503206 sysevent.fs.zfs.history_event
Oct 25 2017 07:57:06.721503207 sysevent.fs.zfs.config_sync

This could indeed explain why the service did not send out notifications, simply because they were not picked up.
The main question for me now is: why do I receive messages about pool upgrades?

If I check for the version it is not returning anything, so how am I supposed to know on which version I am on and to which I should upgrade?

Code:

zpool get version
NAME      PROPERTY  VALUE    SOURCE
HDD-POOL  version   -        default
SSD-POOL  version   -        default

edit/
ok I found according to /var/log/messages I am running version 5

Code:

Oct 25 07:57:08 proxmox kernel: [   17.234115] ZFS: Loaded module v0.7.2-1, ZFS pool version 5000, ZFS filesystem version 5

Is there any option to supress the message?
\edit

Yes I have installed updates, but the dpkg.log from that point in time does only show zed and zfsutils being updated. Is that the reason?

Code:

2017-10-23 08:40:49 startup archives unpack
2017-10-23 08:40:50 upgrade dmsetup:amd64 2:1.02.137-pve3 2:1.02.137-pve6
2017-10-23 08:40:50 status half-configured dmsetup:amd64 2:1.02.137-pve3
2017-10-23 08:40:50 status unpacked dmsetup:amd64 2:1.02.137-pve3
2017-10-23 08:40:50 status half-installed dmsetup:amd64 2:1.02.137-pve3
2017-10-23 08:40:50 status triggers-pending man-db:amd64 2.7.6.1-2
2017-10-23 08:40:50 status half-installed dmsetup:amd64 2:1.02.137-pve3
2017-10-23 08:40:50 status unpacked dmsetup:amd64 2:1.02.137-pve6
2017-10-23 08:40:50 status unpacked dmsetup:amd64 2:1.02.137-pve6
2017-10-23 08:40:50 upgrade libdevmapper1.02.1:amd64 2:1.02.137-pve3 2:1.02.137-pve6
2017-10-23 08:40:50 status triggers-pending libc-bin:amd64 2.24-11+deb9u1
2017-10-23 08:40:50 status half-configured libdevmapper1.02.1:amd64 2:1.02.137-pve3
2017-10-23 08:40:50 status unpacked libdevmapper1.02.1:amd64 2:1.02.137-pve3
2017-10-23 08:40:50 status half-installed libdevmapper1.02.1:amd64 2:1.02.137-pve3
2017-10-23 08:40:50 status half-installed libdevmapper1.02.1:amd64 2:1.02.137-pve3
2017-10-23 08:40:50 status unpacked libdevmapper1.02.1:amd64 2:1.02.137-pve6
2017-10-23 08:40:50 status unpacked libdevmapper1.02.1:amd64 2:1.02.137-pve6
2017-10-23 08:40:50 upgrade libdevmapper-event1.02.1:amd64 2:1.02.137-pve3 2:1.02.137-pve6
2017-10-23 08:40:50 status half-configured libdevmapper-event1.02.1:amd64 2:1.02.137-pve3
2017-10-23 08:40:50 status unpacked libdevmapper-event1.02.1:amd64 2:1.02.137-pve3
2017-10-23 08:40:50 status half-installed libdevmapper-event1.02.1:amd64 2:1.02.137-pve3
2017-10-23 08:40:50 status half-installed libdevmapper-event1.02.1:amd64 2:1.02.137-pve3
2017-10-23 08:40:50 status unpacked libdevmapper-event1.02.1:amd64 2:1.02.137-pve6
2017-10-23 08:40:50 status unpacked libdevmapper-event1.02.1:amd64 2:1.02.137-pve6
2017-10-23 08:40:51 upgrade liblvm2cmd2.02:amd64 2.02.168-pve3 2.02.168-pve6
2017-10-23 08:40:51 status half-configured liblvm2cmd2.02:amd64 2.02.168-pve3
2017-10-23 08:40:51 status unpacked liblvm2cmd2.02:amd64 2.02.168-pve3
2017-10-23 08:40:51 status half-installed liblvm2cmd2.02:amd64 2.02.168-pve3
2017-10-23 08:40:51 status half-installed liblvm2cmd2.02:amd64 2.02.168-pve3
2017-10-23 08:40:51 status unpacked liblvm2cmd2.02:amd64 2.02.168-pve6
2017-10-23 08:40:51 status unpacked liblvm2cmd2.02:amd64 2.02.168-pve6
2017-10-23 08:40:51 upgrade dmeventd:amd64 2:1.02.137-pve3 2:1.02.137-pve6
2017-10-23 08:40:51 status half-configured dmeventd:amd64 2:1.02.137-pve3
2017-10-23 08:40:51 status unpacked dmeventd:amd64 2:1.02.137-pve3
2017-10-23 08:40:51 status half-installed dmeventd:amd64 2:1.02.137-pve3
2017-10-23 08:40:51 status half-installed dmeventd:amd64 2:1.02.137-pve3
2017-10-23 08:40:51 status unpacked dmeventd:amd64 2:1.02.137-pve6
2017-10-23 08:40:51 status unpacked dmeventd:amd64 2:1.02.137-pve6
2017-10-23 08:40:51 upgrade liblvm2app2.2:amd64 2.02.168-pve3 2.02.168-pve6
2017-10-23 08:40:51 status half-configured liblvm2app2.2:amd64 2.02.168-pve3
2017-10-23 08:40:51 status unpacked liblvm2app2.2:amd64 2.02.168-pve3
2017-10-23 08:40:51 status half-installed liblvm2app2.2:amd64 2.02.168-pve3
2017-10-23 08:40:52 status half-installed liblvm2app2.2:amd64 2.02.168-pve3
2017-10-23 08:40:52 status unpacked liblvm2app2.2:amd64 2.02.168-pve6
2017-10-23 08:40:52 status unpacked liblvm2app2.2:amd64 2.02.168-pve6
2017-10-23 08:40:52 upgrade libuutil1linux:amd64 0.6.5.11-pve18~bpo90 0.7.2-pve1~bpo90
2017-10-23 08:40:52 status half-configured libuutil1linux:amd64 0.6.5.11-pve18~bpo90
2017-10-23 08:40:52 status unpacked libuutil1linux:amd64 0.6.5.11-pve18~bpo90
2017-10-23 08:40:52 status half-installed libuutil1linux:amd64 0.6.5.11-pve18~bpo90
2017-10-23 08:40:52 status half-installed libuutil1linux:amd64 0.6.5.11-pve18~bpo90
2017-10-23 08:40:52 status unpacked libuutil1linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:40:52 status unpacked libuutil1linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:40:52 upgrade libnvpair1linux:amd64 0.6.5.11-pve18~bpo90 0.7.2-pve1~bpo90
2017-10-23 08:40:52 status half-configured libnvpair1linux:amd64 0.6.5.11-pve18~bpo90
2017-10-23 08:40:52 status unpacked libnvpair1linux:amd64 0.6.5.11-pve18~bpo90
2017-10-23 08:40:52 status half-installed libnvpair1linux:amd64 0.6.5.11-pve18~bpo90
2017-10-23 08:40:52 status half-installed libnvpair1linux:amd64 0.6.5.11-pve18~bpo90
2017-10-23 08:40:52 status unpacked libnvpair1linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:40:52 status unpacked libnvpair1linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:40:52 upgrade lvm2:amd64 2.02.168-pve3 2.02.168-pve6
2017-10-23 08:40:52 status half-configured lvm2:amd64 2.02.168-pve3
2017-10-23 08:40:52 status unpacked lvm2:amd64 2.02.168-pve3
2017-10-23 08:40:52 status half-installed lvm2:amd64 2.02.168-pve3
2017-10-23 08:40:52 status half-installed lvm2:amd64 2.02.168-pve3
2017-10-23 08:40:53 status unpacked lvm2:amd64 2.02.168-pve6
2017-10-23 08:40:53 status unpacked lvm2:amd64 2.02.168-pve6
2017-10-23 08:40:54 upgrade libpve-storage-perl:all 5.0-15 5.0-16
2017-10-23 08:40:54 status triggers-pending pve-manager:amd64 5.0-32
2017-10-23 08:40:54 status half-configured libpve-storage-perl:all 5.0-15
2017-10-23 08:40:54 status unpacked libpve-storage-perl:all 5.0-15
2017-10-23 08:40:54 status half-installed libpve-storage-perl:all 5.0-15
2017-10-23 08:40:54 status half-installed libpve-storage-perl:all 5.0-15
2017-10-23 08:40:54 status unpacked libpve-storage-perl:all 5.0-16
2017-10-23 08:40:54 status unpacked libpve-storage-perl:all 5.0-16
2017-10-23 08:40:54 upgrade pve-cluster:amd64 5.0-14 5.0-15
2017-10-23 08:40:54 status half-configured pve-cluster:amd64 5.0-14
2017-10-23 08:40:54 status unpacked pve-cluster:amd64 5.0-14
2017-10-23 08:40:54 status half-installed pve-cluster:amd64 5.0-14
2017-10-23 08:40:54 status triggers-pending systemd:amd64 232-25+deb9u1
2017-10-23 08:40:54 status half-installed pve-cluster:amd64 5.0-14
2017-10-23 08:40:54 status unpacked pve-cluster:amd64 5.0-15
2017-10-23 08:40:54 status unpacked pve-cluster:amd64 5.0-15
2017-10-23 08:40:54 upgrade libpve-guest-common-perl:all 2.0-12 2.0-13
2017-10-23 08:40:54 status half-configured libpve-guest-common-perl:all 2.0-12
2017-10-23 08:40:54 status unpacked libpve-guest-common-perl:all 2.0-12
2017-10-23 08:40:54 status half-installed libpve-guest-common-perl:all 2.0-12
2017-10-23 08:40:54 status half-installed libpve-guest-common-perl:all 2.0-12
2017-10-23 08:40:54 status unpacked libpve-guest-common-perl:all 2.0-13
2017-10-23 08:40:54 status unpacked libpve-guest-common-perl:all 2.0-13
2017-10-23 08:40:54 upgrade libpve-common-perl:all 5.0-18 5.0-20
2017-10-23 08:40:54 status half-configured libpve-common-perl:all 5.0-18
2017-10-23 08:40:54 status unpacked libpve-common-perl:all 5.0-18
2017-10-23 08:40:54 status half-installed libpve-common-perl:all 5.0-18
2017-10-23 08:40:55 status half-installed libpve-common-perl:all 5.0-18
2017-10-23 08:40:55 status unpacked libpve-common-perl:all 5.0-20
2017-10-23 08:40:55 status unpacked libpve-common-perl:all 5.0-20
2017-10-23 08:40:55 upgrade libpve-access-control:amd64 5.0-6 5.0-7
2017-10-23 08:40:55 status half-configured libpve-access-control:amd64 5.0-6
2017-10-23 08:40:55 status unpacked libpve-access-control:amd64 5.0-6
2017-10-23 08:40:55 status half-installed libpve-access-control:amd64 5.0-6
2017-10-23 08:40:55 status half-installed libpve-access-control:amd64 5.0-6
2017-10-23 08:40:55 status unpacked libpve-access-control:amd64 5.0-7
2017-10-23 08:40:55 status unpacked libpve-access-control:amd64 5.0-7
2017-10-23 08:40:55 upgrade libzpool2linux:amd64 0.6.5.11-pve18~bpo90 0.7.2-pve1~bpo90
2017-10-23 08:40:55 status half-configured libzpool2linux:amd64 0.6.5.11-pve18~bpo90
2017-10-23 08:40:55 status unpacked libzpool2linux:amd64 0.6.5.11-pve18~bpo90
2017-10-23 08:40:55 status half-installed libzpool2linux:amd64 0.6.5.11-pve18~bpo90
2017-10-23 08:40:55 status half-installed libzpool2linux:amd64 0.6.5.11-pve18~bpo90
2017-10-23 08:40:55 status unpacked libzpool2linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:40:55 status unpacked libzpool2linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:40:55 upgrade libzfs2linux:amd64 0.6.5.11-pve18~bpo90 0.7.2-pve1~bpo90
2017-10-23 08:40:55 status half-configured libzfs2linux:amd64 0.6.5.11-pve18~bpo90
2017-10-23 08:40:55 status unpacked libzfs2linux:amd64 0.6.5.11-pve18~bpo90
2017-10-23 08:40:55 status half-installed libzfs2linux:amd64 0.6.5.11-pve18~bpo90
2017-10-23 08:40:55 status half-installed libzfs2linux:amd64 0.6.5.11-pve18~bpo90
2017-10-23 08:40:55 status unpacked libzfs2linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:40:55 status unpacked libzfs2linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:40:55 upgrade pve-ha-manager:amd64 2.0-2 2.0-3
2017-10-23 08:40:55 status half-configured pve-ha-manager:amd64 2.0-2
2017-10-23 08:40:55 status unpacked pve-ha-manager:amd64 2.0-2
2017-10-23 08:40:55 status half-installed pve-ha-manager:amd64 2.0-2
2017-10-23 08:40:55 status half-installed pve-ha-manager:amd64 2.0-2
2017-10-23 08:40:56 status unpacked pve-ha-manager:amd64 2.0-3
2017-10-23 08:40:56 status unpacked pve-ha-manager:amd64 2.0-3
2017-10-23 08:40:56 upgrade pve-container:all 2.0-16 2.0-17
2017-10-23 08:40:56 status half-configured pve-container:all 2.0-16
2017-10-23 08:40:56 status unpacked pve-container:all 2.0-16
2017-10-23 08:40:56 status half-installed pve-container:all 2.0-16
2017-10-23 08:40:57 status half-installed pve-container:all 2.0-16
2017-10-23 08:40:57 status unpacked pve-container:all 2.0-17
2017-10-23 08:40:57 status unpacked pve-container:all 2.0-17
2017-10-23 08:40:57 upgrade pve-firmware:all 2.0-2 2.0-3
2017-10-23 08:40:57 status half-configured pve-firmware:all 2.0-2
2017-10-23 08:40:57 status unpacked pve-firmware:all 2.0-2
2017-10-23 08:40:57 status half-installed pve-firmware:all 2.0-2
2017-10-23 08:41:08 status half-installed pve-firmware:all 2.0-2
2017-10-23 08:41:08 status unpacked pve-firmware:all 2.0-3
2017-10-23 08:41:08 status unpacked pve-firmware:all 2.0-3
2017-10-23 08:41:09 upgrade pve-docs:all 5.0-9 5.0-10
2017-10-23 08:41:09 status half-configured pve-docs:all 5.0-9
2017-10-23 08:41:09 status unpacked pve-docs:all 5.0-9
2017-10-23 08:41:09 status half-installed pve-docs:all 5.0-9
2017-10-23 08:41:09 status half-installed pve-docs:all 5.0-9
2017-10-23 08:41:10 status unpacked pve-docs:all 5.0-10
2017-10-23 08:41:10 status unpacked pve-docs:all 5.0-10
2017-10-23 08:41:10 upgrade pve-manager:amd64 5.0-32 5.0-34
2017-10-23 08:41:10 status half-configured pve-manager:amd64 5.0-32
2017-10-23 08:41:10 status unpacked pve-manager:amd64 5.0-32
2017-10-23 08:41:10 status half-installed pve-manager:amd64 5.0-32
2017-10-23 08:41:11 status half-installed pve-manager:amd64 5.0-32
2017-10-23 08:41:11 status unpacked pve-manager:amd64 5.0-34
2017-10-23 08:41:11 status unpacked pve-manager:amd64 5.0-34
2017-10-23 08:41:12 upgrade zfsutils-linux:amd64 0.6.5.11-pve18~bpo90 0.7.2-pve1~bpo90
2017-10-23 08:41:12 status half-configured zfsutils-linux:amd64 0.6.5.11-pve18~bpo90
2017-10-23 08:41:12 status unpacked zfsutils-linux:amd64 0.6.5.11-pve18~bpo90
2017-10-23 08:41:12 status half-installed zfsutils-linux:amd64 0.6.5.11-pve18~bpo90
2017-10-23 08:41:12 status half-installed zfsutils-linux:amd64 0.6.5.11-pve18~bpo90
2017-10-23 08:41:12 status unpacked zfsutils-linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:12 status unpacked zfsutils-linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:12 upgrade zfs-zed:amd64 0.6.5.11-pve18~bpo90 0.7.2-pve1~bpo90
2017-10-23 08:41:12 status half-configured zfs-zed:amd64 0.6.5.11-pve18~bpo90
2017-10-23 08:41:12 status unpacked zfs-zed:amd64 0.6.5.11-pve18~bpo90
2017-10-23 08:41:12 status half-installed zfs-zed:amd64 0.6.5.11-pve18~bpo90
2017-10-23 08:41:12 status half-installed zfs-zed:amd64 0.6.5.11-pve18~bpo90
2017-10-23 08:41:13 status unpacked zfs-zed:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:13 status unpacked zfs-zed:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:13 upgrade pve-qemu-kvm:amd64 2.9.1-1 2.9.1-2
2017-10-23 08:41:13 status half-configured pve-qemu-kvm:amd64 2.9.1-1
2017-10-23 08:41:13 status unpacked pve-qemu-kvm:amd64 2.9.1-1
2017-10-23 08:41:13 status half-installed pve-qemu-kvm:amd64 2.9.1-1
2017-10-23 08:41:14 status half-installed pve-qemu-kvm:amd64 2.9.1-1
2017-10-23 08:41:14 status unpacked pve-qemu-kvm:amd64 2.9.1-2
2017-10-23 08:41:14 status unpacked pve-qemu-kvm:amd64 2.9.1-2
2017-10-23 08:41:15 startup packages configure
2017-10-23 08:41:15 configure libpve-common-perl:all 5.0-20 <none>
2017-10-23 08:41:15 status unpacked libpve-common-perl:all 5.0-20
2017-10-23 08:41:15 status half-configured libpve-common-perl:all 5.0-20
2017-10-23 08:41:15 status installed libpve-common-perl:all 5.0-20
2017-10-23 08:41:15 configure libuutil1linux:amd64 0.7.2-pve1~bpo90 <none>
2017-10-23 08:41:15 status unpacked libuutil1linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:15 status half-configured libuutil1linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:15 status installed libuutil1linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:15 configure pve-qemu-kvm:amd64 2.9.1-2 <none>
2017-10-23 08:41:15 status unpacked pve-qemu-kvm:amd64 2.9.1-2
2017-10-23 08:41:15 status unpacked pve-qemu-kvm:amd64 2.9.1-2
2017-10-23 08:41:15 status unpacked pve-qemu-kvm:amd64 2.9.1-2
2017-10-23 08:41:15 status half-configured pve-qemu-kvm:amd64 2.9.1-2
2017-10-23 08:41:15 status installed pve-qemu-kvm:amd64 2.9.1-2
2017-10-23 08:41:15 configure libnvpair1linux:amd64 0.7.2-pve1~bpo90 <none>
2017-10-23 08:41:15 status unpacked libnvpair1linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:15 status half-configured libnvpair1linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:15 status installed libnvpair1linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:15 configure pve-docs:all 5.0-10 <none>
2017-10-23 08:41:15 status unpacked pve-docs:all 5.0-10
2017-10-23 08:41:15 status half-configured pve-docs:all 5.0-10
2017-10-23 08:41:15 status installed pve-docs:all 5.0-10
2017-10-23 08:41:15 trigproc libc-bin:amd64 2.24-11+deb9u1 <none>
2017-10-23 08:41:15 status half-configured libc-bin:amd64 2.24-11+deb9u1
2017-10-23 08:41:16 status installed libc-bin:amd64 2.24-11+deb9u1
2017-10-23 08:41:16 trigproc systemd:amd64 232-25+deb9u1 <none>
2017-10-23 08:41:16 status half-configured systemd:amd64 232-25+deb9u1
2017-10-23 08:41:16 status installed systemd:amd64 232-25+deb9u1
2017-10-23 08:41:16 trigproc man-db:amd64 2.7.6.1-2 <none>
2017-10-23 08:41:16 status half-configured man-db:amd64 2.7.6.1-2
2017-10-23 08:41:18 status installed man-db:amd64 2.7.6.1-2
2017-10-23 08:41:18 configure pve-cluster:amd64 5.0-15 <none>
2017-10-23 08:41:18 status unpacked pve-cluster:amd64 5.0-15
2017-10-23 08:41:18 status unpacked pve-cluster:amd64 5.0-15
2017-10-23 08:41:18 status unpacked pve-cluster:amd64 5.0-15
2017-10-23 08:41:18 status unpacked pve-cluster:amd64 5.0-15
2017-10-23 08:41:18 status half-configured pve-cluster:amd64 5.0-15
2017-10-23 08:41:30 status installed pve-cluster:amd64 5.0-15
2017-10-23 08:41:30 configure pve-firmware:all 2.0-3 <none>
2017-10-23 08:41:30 status unpacked pve-firmware:all 2.0-3
2017-10-23 08:41:30 status half-configured pve-firmware:all 2.0-3
2017-10-23 08:41:30 status installed pve-firmware:all 2.0-3
2017-10-23 08:41:30 configure libzpool2linux:amd64 0.7.2-pve1~bpo90 <none>
2017-10-23 08:41:30 status triggers-pending libc-bin:amd64 2.24-11+deb9u1
2017-10-23 08:41:30 status unpacked libzpool2linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:30 status half-configured libzpool2linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:30 status installed libzpool2linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:30 configure libpve-access-control:amd64 5.0-7 <none>
2017-10-23 08:41:30 status unpacked libpve-access-control:amd64 5.0-7
2017-10-23 08:41:30 status half-configured libpve-access-control:amd64 5.0-7
2017-10-23 08:41:30 status installed libpve-access-control:amd64 5.0-7
2017-10-23 08:41:30 configure libzfs2linux:amd64 0.7.2-pve1~bpo90 <none>
2017-10-23 08:41:30 status unpacked libzfs2linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:30 status half-configured libzfs2linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:30 status installed libzfs2linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:30 configure pve-ha-manager:amd64 2.0-3 <none>
2017-10-23 08:41:30 status unpacked pve-ha-manager:amd64 2.0-3
2017-10-23 08:41:30 status unpacked pve-ha-manager:amd64 2.0-3
2017-10-23 08:41:30 status half-configured pve-ha-manager:amd64 2.0-3
2017-10-23 08:41:38 status installed pve-ha-manager:amd64 2.0-3
2017-10-23 08:41:38 configure zfsutils-linux:amd64 0.7.2-pve1~bpo90 <none>
2017-10-23 08:41:38 status unpacked zfsutils-linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:38 status unpacked zfsutils-linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:38 status unpacked zfsutils-linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:38 status unpacked zfsutils-linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:38 status unpacked zfsutils-linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:38 status half-configured zfsutils-linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:40 status installed zfsutils-linux:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:40 configure zfs-zed:amd64 0.7.2-pve1~bpo90 <none>
2017-10-23 08:41:40 status unpacked zfs-zed:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:40 status unpacked zfs-zed:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:40 status unpacked zfs-zed:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:40 status half-configured zfs-zed:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:40 status installed zfs-zed:amd64 0.7.2-pve1~bpo90
2017-10-23 08:41:40 configure libdevmapper1.02.1:amd64 2:1.02.137-pve6 <none>
2017-10-23 08:41:40 status unpacked libdevmapper1.02.1:amd64 2:1.02.137-pve6
2017-10-23 08:41:40 status half-configured libdevmapper1.02.1:amd64 2:1.02.137-pve6
2017-10-23 08:41:40 status installed libdevmapper1.02.1:amd64 2:1.02.137-pve6
2017-10-23 08:41:40 configure libdevmapper-event1.02.1:amd64 2:1.02.137-pve6 <none>
2017-10-23 08:41:40 status unpacked libdevmapper-event1.02.1:amd64 2:1.02.137-pve6
2017-10-23 08:41:40 status half-configured libdevmapper-event1.02.1:amd64 2:1.02.137-pve6
2017-10-23 08:41:40 status installed libdevmapper-event1.02.1:amd64 2:1.02.137-pve6
2017-10-23 08:41:41 configure liblvm2cmd2.02:amd64 2.02.168-pve6 <none>
2017-10-23 08:41:41 status unpacked liblvm2cmd2.02:amd64 2.02.168-pve6
2017-10-23 08:41:41 status half-configured liblvm2cmd2.02:amd64 2.02.168-pve6
2017-10-23 08:41:41 status installed liblvm2cmd2.02:amd64 2.02.168-pve6
2017-10-23 08:41:41 configure dmsetup:amd64 2:1.02.137-pve6 <none>
2017-10-23 08:41:41 status unpacked dmsetup:amd64 2:1.02.137-pve6
2017-10-23 08:41:41 status half-configured dmsetup:amd64 2:1.02.137-pve6
2017-10-23 08:41:41 status installed dmsetup:amd64 2:1.02.137-pve6
2017-10-23 08:41:41 status triggers-pending initramfs-tools:all 0.130
2017-10-23 08:41:41 configure liblvm2app2.2:amd64 2.02.168-pve6 <none>
2017-10-23 08:41:41 status unpacked liblvm2app2.2:amd64 2.02.168-pve6
2017-10-23 08:41:41 status half-configured liblvm2app2.2:amd64 2.02.168-pve6
2017-10-23 08:41:41 status installed liblvm2app2.2:amd64 2.02.168-pve6
2017-10-23 08:41:41 configure dmeventd:amd64 2:1.02.137-pve6 <none>
2017-10-23 08:41:41 status unpacked dmeventd:amd64 2:1.02.137-pve6
2017-10-23 08:41:41 status half-configured dmeventd:amd64 2:1.02.137-pve6
2017-10-23 08:41:45 status installed dmeventd:amd64 2:1.02.137-pve6
2017-10-23 08:41:45 configure lvm2:amd64 2.02.168-pve6 <none>
2017-10-23 08:41:45 status unpacked lvm2:amd64 2.02.168-pve6
2017-10-23 08:41:45 status unpacked lvm2:amd64 2.02.168-pve6
2017-10-23 08:41:45 status half-configured lvm2:amd64 2.02.168-pve6
2017-10-23 08:41:50 status installed lvm2:amd64 2.02.168-pve6
2017-10-23 08:41:50 configure libpve-storage-perl:all 5.0-16 <none>
2017-10-23 08:41:50 status triggers-pending pve-ha-manager:amd64 2.0-3
2017-10-23 08:41:50 status unpacked libpve-storage-perl:all 5.0-16
2017-10-23 08:41:50 status half-configured libpve-storage-perl:all 5.0-16
2017-10-23 08:41:50 status installed libpve-storage-perl:all 5.0-16
2017-10-23 08:41:50 configure libpve-guest-common-perl:all 2.0-13 <none>
2017-10-23 08:41:50 status unpacked libpve-guest-common-perl:all 2.0-13
2017-10-23 08:41:50 status half-configured libpve-guest-common-perl:all 2.0-13
2017-10-23 08:41:50 status installed libpve-guest-common-perl:all 2.0-13
2017-10-23 08:41:50 configure pve-container:all 2.0-17 <none>
2017-10-23 08:41:50 status triggers-pending systemd:amd64 232-25+deb9u1
2017-10-23 08:41:50 status unpacked pve-container:all 2.0-17
2017-10-23 08:41:50 status half-configured pve-container:all 2.0-17
2017-10-23 08:41:51 status installed pve-container:all 2.0-17
2017-10-23 08:41:51 configure pve-manager:amd64 5.0-34 <none>
2017-10-23 08:41:51 status unpacked pve-manager:amd64 5.0-34
2017-10-23 08:41:51 status unpacked pve-manager:amd64 5.0-34
2017-10-23 08:41:51 status unpacked pve-manager:amd64 5.0-34
2017-10-23 08:41:51 status unpacked pve-manager:amd64 5.0-34
2017-10-23 08:41:51 status unpacked pve-manager:amd64 5.0-34
2017-10-23 08:41:51 status unpacked pve-manager:amd64 5.0-34
2017-10-23 08:41:51 status unpacked pve-manager:amd64 5.0-34
2017-10-23 08:41:51 status unpacked pve-manager:amd64 5.0-34
2017-10-23 08:41:51 status unpacked pve-manager:amd64 5.0-34
2017-10-23 08:41:51 status unpacked pve-manager:amd64 5.0-34
2017-10-23 08:41:51 status unpacked pve-manager:amd64 5.0-34
2017-10-23 08:41:51 status unpacked pve-manager:amd64 5.0-34
2017-10-23 08:41:51 status unpacked pve-manager:amd64 5.0-34
2017-10-23 08:41:51 status unpacked pve-manager:amd64 5.0-34
2017-10-23 08:41:51 status unpacked pve-manager:amd64 5.0-34
2017-10-23 08:41:51 status unpacked pve-manager:amd64 5.0-34
2017-10-23 08:41:51 status half-configured pve-manager:amd64 5.0-34
2017-10-23 08:42:00 status installed pve-manager:amd64 5.0-34
2017-10-23 08:42:00 trigproc libc-bin:amd64 2.24-11+deb9u1 <none>
2017-10-23 08:42:00 status half-configured libc-bin:amd64 2.24-11+deb9u1
2017-10-23 08:42:00 status installed libc-bin:amd64 2.24-11+deb9u1
2017-10-23 08:42:00 trigproc initramfs-tools:all 0.130 <none>
2017-10-23 08:42:00 status half-configured initramfs-tools:all 0.130
2017-10-23 08:42:26 status installed initramfs-tools:all 0.130
2017-10-23 08:42:26 trigproc pve-ha-manager:amd64 2.0-3 <none>
2017-10-23 08:42:26 status half-configured pve-ha-manager:amd64 2.0-3
2017-10-23 08:42:33 status installed pve-ha-manager:amd64 2.0-3
2017-10-23 08:42:33 trigproc systemd:amd64 232-25+deb9u1 <none>
2017-10-23 08:42:33 status half-configured systemd:amd64 232-25+deb9u1
2017-10-23 08:42:33 status installed systemd:amd64 232-25+deb9u1

I have to admit I am new to the system and not 100% into it yet. Apologies if I am starting to mix things up...
Thanks all for your help!

fabian · Oct 25, 2017

you need to reboot after upgrading ZFS (at least when upgrading from one major version to another) because the user space utilities and the kernel module need to be in sync. but you need to reboot for the kernel upgrade anyway

seems like zed is especially picky this time around..

apoc · Oct 25, 2017

Thanks Fabian,
I have not tested the eMail notification yet again.
Could you please give me an indication about the "pool upgrade" question I raised?
Best regards

fabian · Oct 25, 2017

ZFS hides features that change the on-disk format in an incompatible way behind feature flags. when you upgrade, it does not automatically enable them so you can still import your pool using the old version (or even systems running the older version). once you are comfortable with the idea of not being able to go back, you can upgrade the pool and enable features that seem relevant for your use case. the message itself is harmless, it's just a reminder that you are potentially missing out on some newer stuff

the new features introduced with 0.7 should not interfere with booting just by being enabled, although they might if you actually use them for your root dataset (e.g., the new compression algorithms might not be compatible with the ZFS implementation in Grub yet).

DevFx11 · Oct 25, 2017

Try this
https://forum.proxmox.com/threads/p...root-pam-does-not-have-a-email-address.35961/

Fixed my mail issues.
Mail was working fine prior to the latest upgrade, this fixed it.

DevFx11 · Oct 25, 2017

I am also geting an error with zpool events and zfs-zed 100% cpu usage.

root@pve1:~# zpool events
TIME CLASS
internal error: Bad file descriptor
Aborted

It is as described here :
https://github.com/zfsonlinux/zfs/issues/4720

root@pve1:~# dmesg | grep -i zfs
[ 0.000000] Command line: BOOT_IMAGE=/ROOT/pve-1@/boot/vmlinuz-4.10.17-4-pve root=ZFS=rpool/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs
[ 0.000000] Kernel command line: BOOT_IMAGE=/ROOT/pve-1@/boot/vmlinuz-4.10.17-4-pve root=ZFS=rpool/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs
[ 9.312493] SPL: Loaded module v0.6.5.11-1
[ 9.329710] ZFS: Loaded module v0.6.5.11-1, ZFS pool version 5000, ZFS filesystem version 5

I need to use the old kernel since with the new kernel i can't make my system boot properly and load all zfs pools.
Too many problems to list here. Opened a new thread about it.

0xFelix · Oct 25, 2017

On my system the PVE 5.1 update succeeded and the new ZFS module is loaded. All pools are upgraded and "zpool events" does output some events which should have generated a mail. The solution in DevFx11's link is not applicable to my environment since a simple "echo "Test" | mail -s Test root" works. Yet still zed does not send any mails.

DevFx11 · Oct 25, 2017

I since noticed that the fix only fixes ones set of problems, where the error in syslog is abour root@pam having no email.
Now i can send mail to root and i get it on my forwarded address.
Yet ZFS-ZED send no mails, it used to work properly before the upgrade.
So i second that 0xFelix , it does not fix the issue with zfs-zed.

fabian · Oct 25, 2017

there was a bug in pvemailforward / pve-manager, a fixed package has been pushed to all repos. can you check if it solves the mail issue with zed for you?

apoc · Oct 25, 2017

@DevFx11 - That was the message!

And for the rest of all of you: The problem seems to have cleared itself after the reboot.
I wasnt able to check eMails of my administrative mailbox today. But I could see, that the scrub I issued this morning, reported correctly.
It seems the ZFS update somehow screwed the ZED and the reboot cleared all that.

Thanks to you all.

edit/
I just installed the new pve-manager. Will run another test with a scrub and let you know.

Scrub finished - email arrived.
/edit

0xFelix · Oct 26, 2017

Still not working for me, I'm on the newest versions but had no chance to reboot yet.

Alwin · Oct 27, 2017

@0xFelix, do you have pve-manager=5.1-36 installed?

0xFelix · Oct 27, 2017

Yes, "pveversion -v" says:

Code:

proxmox-ve: 5.1-25 (running kernel: 4.13.4-1-pve)
pve-manager: 5.1-36 (running version: 5.1-36/131401db)
pve-kernel-4.13.4-1-pve: 4.13.4-25
libpve-http-server-perl: 2.0-6
lvm2: 2.02.168-pve6
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-15
qemu-server: 5.0-17
pve-firmware: 2.0-3
libpve-common-perl: 5.0-20
libpve-guest-common-perl: 2.0-13
libpve-access-control: 5.0-7
libpve-storage-perl: 5.0-16
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-2
pve-docs: 5.1-12
pve-qemu-kvm: 2.9.1-2
pve-container: 2.0-17
pve-firewall: 3.0-3
pve-ha-manager: 2.0-3
ksm-control-daemon: not correctly installed
glusterfs-client: 3.8.8-1
lxc-pve: 2.1.0-2
lxcfs: 2.0.7-pve4
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.7.2-pve1~bpo90

Alwin · Oct 30, 2017

What do you see in the mail.log?

0xFelix · Oct 30, 2017

Weird... I received a scrub finished mail today, so maybe I'm wrong, ZED is working. However I know I should be receiving a lot more mails, cause I'm affected by this bug: https://github.com/zfsonlinux/zfs/issues/4713. The according "ereport.fs.zfs.io" events are not generated anymore, though "dmesg" still outputs the errors mentioned in the bug reported. Seems like ZFS 0.7.2 changed its behaviour in that regard, but the bug is likely in the kernel not in ZFS, so it should still generate warnings.

[SOLVED] ZED Notifications not working

Famous Member

Proxmox Retired Staff

Famous Member

Renowned Member

Member

Famous Member

Proxmox Staff Member

Famous Member

Proxmox Staff Member

New Member

New Member

Member

New Member

Proxmox Staff Member

Famous Member

Member

Proxmox Retired Staff

Member

Proxmox Retired Staff

Member