dead node and phantom tasks

[alexskysilk]

I have an odd situation. I have a node that died during a backup window. No big whoop, I had sufficient compute space, but now there are 2 backup tasks for the dead node in the task log and I cant figure out how to get rid of them. I imagine that deleting the node from the cluster would clear it up but there's got to be a way to do that without resorting to that step.

pvesh is no help, since it fails on any requests involving the dead node.

/etc/pve is no help, since it doesnt contain a task list.

help?

 

[bbgeek17]

The tasks are stored here: /var/log/pve/tasks/

There are individual tasks and index files that reference them. If you really want to get rid of those two you'd need to address the index files (probably! I've never done it!). Backup that folder first!

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[alexskysilk]

yeah, I figured that. the problem is that /var/log/pve/tasks keeps tasks striclty for the node its on. since the node in question is dead, this folder is obviously not available :cry:

 

[fba]

Have you tried restarting pve-cluster.service on the remaining nodes?

 

[alexskysilk]

I was thinking about that, but its making me nervous. If I dont come up with a "safer" solution I will do so.

 

[bbgeek17]

I was thinking about that, but its making me nervous. If I dont come up with a "safer" solution I will do so.

I'd give this a try:
systemctl try-reload-or-restart pvedaemon pveproxy pvestatd pvescheduler pve-ha-lrm
This is a safe operation that we recommend doing on new plugin install.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[alexskysilk]

For reasons I cant quite understand, both phantom tasks (one "running" for 2 months) disappeared. @bbgeek17 if thats your doing I owe you a beer

I didnt actually do anything...

 

[fba]

Did you just install updates?

 

[alexskysilk]

I literally did nothing since this is a production cluster and I didnt want any unintended consequences. I went to try @bbgeek17 suggestion when I noticed the tasks were gone....