Pvestatd won't work after deleting a lvm

[VMaster]

Hello, I need some help please.

Today after deleting a volume from a storage array, and later the lvm from proxmox, the service pvestatd isn't working. If I enter to the web gui, I see everything greyed with question marks.

When I restart the service pvestatd it works for a minute, but later everything is greyed again.

This is what I get when I restarted pvestatd in one node:

May 12 09:35:41 pm1 pvestatd[16994]: received signal TERM
May 12 09:35:41 pm1 pvestatd[16994]: server closing
May 12 09:35:41 pm1 pvestatd[16994]: server stopped
May 12 09:37:12 pm1 systemd[1]: pvestatd.service: State 'stop-sigterm' timed out. Killing.
May 12 09:37:12 pm1 systemd[1]: pvestatd.service: Killing process 17096 (vgs) with signal SIGKILL.
May 12 09:37:12 pm1 systemd[1]: pvestatd.service: Killing process 128614 (vgs) with signal SIGKILL.
May 12 09:37:12 pm1 systemd[1]: pvestatd.service: Killing process 229983 (vgs) with signal SIGKILL.
May 12 09:37:12 pm1 systemd[1]: pvestatd.service: Killing process 284358 (vgs) with signal SIGKILL.
May 12 09:38:42 pm1 systemd[1]: pvestatd.service: Processes still around after SIGKILL. Ignoring.
May 12 09:40:12 pm1 systemd[1]: pvestatd.service: State 'stop-final-sigterm' timed out. Killing.
May 12 09:40:12 pm1 systemd[1]: pvestatd.service: Killing process 17096 (vgs) with signal SIGKILL.
May 12 09:40:12 pm1 systemd[1]: pvestatd.service: Killing process 128614 (vgs) with signal SIGKILL.
May 12 09:40:12 pm1 systemd[1]: pvestatd.service: Killing process 229983 (vgs) with signal SIGKILL.
May 12 09:40:12 pm1 systemd[1]: pvestatd.service: Killing process 284358 (vgs) with signal SIGKILL.
May 12 09:41:42 pm1 systemd[1]: pvestatd.service: Processes still around after final SIGKILL. Entering failed mode.
May 12 09:41:42 pm1 systemd[1]: pvestatd.service: Unit entered failed state.
May 12 09:41:42 pm1 systemd[1]: pvestatd.service: Failed with result 'timeout'.
May 12 09:41:43 pm1 pvestatd[25470]: starting server


Also I tried to execute the command pvesm status, bit it gets stucked.

Thanks in advance!

 

[fabian]

looks like LVM hangs on your system (pvestatd calls vgs). look at the system logs starting from shortly before you removed the volume for any related errors.

 

[VMaster]

looks like LVM hangs on your system (pvestatd calls vgs). look at the system logs starting from shortly before you removed the volume for any related errors.

This is the output that I get about that hour:

May 12 08:40:11 pm1 iscsid[7831]: connect to x.x.x.x:3260 failed (No route to host)
May 12 08:40:13 pm1 iscsid[7831]: connect to x.x.x.x:3260 failed (No route to host)
May 12 08:40:15 pm1 iscsid[7831]: connection1:0 is operational after recovery (79 attempts)
May 12 08:40:16 pm1 iscsid[7831]: connection2:0 is operational after recovery (79 attempts)
May 12 08:40:19 pm1 kernel: sd 18:0:0:0: Power-on or device reset occurred
May 12 08:40:19 pm1 kernel: sd 18:0:0:0: alua: port group 01 state N non-preferred supports tolusna
May 12 08:40:20 pm1 multipathd[179429]: 68:16: reinstated
May 12 08:40:20 pm1 multipathd[179429]: 35000035400808145f000000000000002: remaining active paths: 1
May 12 08:40:20 pm1 kernel: device-mapper: multipath: Reinstating path 68:16.
May 12 08:40:22 pm1 kernel: sd 17:0:0:0: Power-on or device reset occurred
May 12 08:40:22 pm1 kernel: sd 17:0:0:0: alua: port group 01 state N non-preferred supports tolusna
May 12 08:40:23 pm1 multipathd[179429]: 68:32: reinstated
May 12 08:40:23 pm1 multipathd[179429]: 35000035400808145f000000000000002: remaining active paths: 2
May 12 08:40:23 pm1 kernel: device-mapper: multipath: Reinstating path 68:32.
May 12 08:40:23 pm1 kernel: sd 17:0:0:0: alua: port group 01 state N non-preferred supports tolusna
May 12 08:40:23 pm1 kernel: sd 17:0:0:0: alua: port group 01 state N non-preferred supports tolusna
May 12 08:40:28 pm1 systemd-udevd[859]: worker [279298] terminated by signal 9 (Killed)
May 12 08:40:28 pm1 systemd-udevd[859]: worker [279298] failed while handling '/devices/virtual/block/dm-251'


I see that there was some timeouts to the iSCSI port, and also a device mapper failed. Could be affecting the dm to the pvestatd service?

Is there any solution? I deleted the lvm that now is unused from proxmox.

Thanks in advance.

 

[fabian]

I don't know enough about your setup/system to give you any more pointers, but I'd look at the config for any other references to the no longer available LVM storage or its devices, and then either restart services that might still use outdated configs or reboot the whole node.

 

[VMaster]

I don't know enough about your setup/system to give you any more pointers, but I'd look at the config for any other references to the no longer available LVM storage or its devices, and then either restart services that might still use outdated configs or reboot the whole node.

I've been researching and it is a problem with another volume from another storage array that I restarted its controller at the same time that I deleted the volume that I said before.

I'll see if there is any way to recover it, if not I will restart all nodes, and I'll see if it works again.

Thanks!