"ceph : command not allowed" in syslog

[sidereus]

What is a reason of following error messages in the syslog every night? Proxmox 6.3, Ceph 15.2.

Apr 14 03:00:49 asr2 sudo[181953]:     ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a --json=o /dev/sdb
Apr 14 03:00:49 asr2 sudo[181953]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 14 03:00:49 asr2 sudo[181953]: pam_unix(sudo:session): session closed for user root
Apr 14 03:00:49 asr2 sudo[181973]: pam_unix(sudo:auth): conversation failed
Apr 14 03:00:49 asr2 sudo[181973]: pam_unix(sudo:auth): auth could not identify password for [ceph]
Apr 14 03:00:49 asr2 sudo[181973]:     ceph : command not allowed ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=nvme wdc_wd2000fyyz-01ul1b2 smart-log-add --json /dev/sdb
Apr 14 03:00:50 asr2 sudo[182065]:     ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a --json=o /dev/sdc
Apr 14 03:00:50 asr2 sudo[182065]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 14 03:00:50 asr2 sudo[182065]: pam_unix(sudo:session): session closed for user root
Apr 14 03:00:51 asr2 sudo[182068]: pam_unix(sudo:auth): conversation failed
Apr 14 03:00:51 asr2 sudo[182068]: pam_unix(sudo:auth): auth could not identify password for [ceph]
Apr 14 03:00:51 asr2 sudo[182068]:     ceph : command not allowed ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=nvme wdc_wd2000fyyz-01ul1b2 smart-log-add --json /dev/sdc
Apr 14 03:00:52 asr2 sudo[182070]:     ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a --json=o /dev/sdd
Apr 14 03:00:52 asr2 sudo[182070]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 14 03:00:52 asr2 sudo[182070]: pam_unix(sudo:session): session closed for user root
Apr 14 03:00:52 asr2 sudo[182073]: pam_unix(sudo:auth): conversation failed
Apr 14 03:00:52 asr2 sudo[182073]: pam_unix(sudo:auth): auth could not identify password for [ceph]
Apr 14 03:00:52 asr2 sudo[182073]:     ceph : command not allowed ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=nvme wdc_wd2000fyyz-01ul1b2 smart-log-add --json /dev/sdd
Apr 14 03:00:53 asr2 sudo[182075]:     ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a --json=o /dev/sde
Apr 14 03:00:53 asr2 sudo[182075]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 14 03:00:53 asr2 sudo[182075]: pam_unix(sudo:session): session closed for user root
Apr 14 03:00:54 asr2 sudo[182078]: pam_unix(sudo:auth): conversation failed
Apr 14 03:00:54 asr2 sudo[182078]: pam_unix(sudo:auth): auth could not identify password for [ceph]
Apr 14 03:00:54 asr2 sudo[182078]:     ceph : command not allowed ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=nvme wdc_wd2000fyyz-01ul1b2 smart-log-add --json /dev/sde
Apr 14 03:00:54 asr2 sudo[182080]:     ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a --json=o /dev/sdf
Apr 14 03:00:54 asr2 sudo[182080]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 14 03:00:55 asr2 sudo[182080]: pam_unix(sudo:session): session closed for user root
Apr 14 03:00:55 asr2 sudo[182083]: pam_unix(sudo:auth): conversation failed
Apr 14 03:00:55 asr2 sudo[182083]: pam_unix(sudo:auth): auth could not identify password for [ceph]
Apr 14 03:00:55 asr2 sudo[182083]:     ceph : command not allowed ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=nvme wdc_wd2000fyyz-01ul1b2 smart-log-add --json /dev/sdf
Apr 14 03:00:56 asr2 sudo[182085]:     ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a --json=o /dev/sdg
Apr 14 03:00:56 asr2 sudo[182085]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 14 03:00:56 asr2 sudo[182085]: pam_unix(sudo:session): session closed for user root
Apr 14 03:00:57 asr2 sudo[182094]: pam_unix(sudo:auth): conversation failed
Apr 14 03:00:57 asr2 sudo[182094]: pam_unix(sudo:auth): auth could not identify password for [ceph]
Apr 14 03:00:57 asr2 sudo[182094]:     ceph : command not allowed ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=nvme wdc_wd2000fyyz-01ul1b2 smart-log-add --json /dev/sdg
Apr 14 03:00:57 asr2 sudo[182096]:     ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a --json=o /dev/sdh
Apr 14 03:00:57 asr2 sudo[182096]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 14 03:00:57 asr2 sudo[182096]: pam_unix(sudo:session): session closed for user root
Apr 14 03:00:58 asr2 sudo[182099]: pam_unix(sudo:auth): conversation failed
Apr 14 03:00:58 asr2 sudo[182099]: pam_unix(sudo:auth): auth could not identify password for [ceph]
Apr 14 03:00:58 asr2 sudo[182099]:     ceph : command not allowed ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=nvme wdc_wd2000fyyz-01ul1b2 smart-log-add --json /dev/sdh
Apr 14 03:01:00 asr2 systemd[1]: Starting Proxmox VE replication runner...
Apr 14 03:01:00 asr2 sudo[182187]:     ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -a --json=o /dev/
Apr 14 03:01:00 asr2 sudo[182187]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 14 03:01:00 asr2 sudo[182187]: pam_unix(sudo:session): session closed for user root

 

[RokaKen]

You seem to be missing the file /etc/sudoers.d/ceph-osd-smartctl or it is corrupt on (at least) that node. It should contain:

## allow ceph-osd (which runs as user ceph) to collect device health metrics

ceph ALL=NOPASSWD: /usr/sbin/smartctl -a --json=o /dev/*
ceph ALL=NOPASSWD: /usr/sbin/nvme * smart-log-add --json /dev/*

[as of Nautilus (14.2) -- not aware of a change for Octopus]

 

[hepo]

After recent upgrade to PVE7 and Ceph Pacific we started receiving emails with the same errors.
After checking the logs we have concluded that the issues was definitely there prior to the upgrade.

Jul 10 00:04:30 pve21 sudo: pam_unix(sudo:auth): auth could not identify password for [ceph]
Jul 10 00:04:30 pve21 sudo:     ceph : command not allowed ; PWD=/ ; USER=root ; COMMAND=nvme samsung smart-log-add --json /dev/nvme2n1
Jul 10 00:04:31 pve21 sudo:     ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme3n1
Jul 10 00:04:32 pve21 sudo: pam_unix(sudo:auth): auth could not identify password for [ceph]
Jul 10 00:04:32 pve21 sudo:     ceph : command not allowed ; PWD=/ ; USER=root ; COMMAND=nvme samsung smart-log-add --json /dev/nvme3n1
Jul 10 00:04:44 pve21 sudo:     ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/

The content of /etc/sudoers.d/ceph-osd-smartctl appears correct

## allow ceph-osd (which runs as user ceph) to collect device health metrics

ceph ALL=NOPASSWD: /usr/sbin/smartctl -x --json=o /dev/*
ceph ALL=NOPASSWD: /usr/sbin/nvme * smart-log-add --json /dev/*

It looks that the binaries (/usr/sbin/nvme) are missing and need to be installed i.e. apt install nvme-cli

Anyone with advise?

 

[fiona]

Hi,
yes, installing the nvme-cli package is the recommended solution. New Ceph setups in Proxmox VE 7.0 will install that package by default too, see.

 

[hepo]

Thanks for the response, nvme-cli already installed and can confirm the errors are gone.