Accidentally deleted /var/log, cluster having massive issues

[britmob]

Hi there, I was trying to do routine clearing out of /var/log files, and typo'd my rm command and deleted everything inside /var/log.

Right now I have tried to recreate the directory structure and files, as well as fixed permissions. However, my servers are still turning to the grey question mark in the web UI. Their web UI still works, and is mostly functional. But the issue is that all my storage, VMs, and CTs are greyed out. The storage works on those nodes, but not VM/CTs (their booting tasks time out), and I can't properly see VM/CT names either.

The other issue is that shutdown/start/reboot don't work properly on nodes, tasks complete (or sometimes shutdowns/starts time out despite saying "TASK OK") but it simply will not shut down or reboot. It just sits until I unplug it.

Is there any way I can just restore /var/log to what it was like? I don't know exactly what is causing all these things but it is breaking almost every part of the cluster's functionality.

(I don't back up the host nodes, only VM/CT data.. hopefully PBS supports host backups in the next release..)

EDIT: Worth mentioning that on initial boot, the nodes are green in the UI. It is only after a few minutes (or when I navigate to a VM page) that they go grey. I see nothing in the syslog that would align with the switch to grey. Systemctl for pveproxy, pvedaemon, and pvestatd also look all good.

 

[oguz]

hi,

Hi there, I was trying to do routine clearing out of /var/log files, and typo'd my rm command and deleted everything inside /var/log.

whoops :/

Right now I have tried to recreate the directory structure and files, as well as fixed permissions. However, my servers are still turning to the grey question mark in the web UI. Their web UI still works, and is mostly functional. But the issue is that all my storage, VMs, and CTs are greyed out. The storage works on those nodes, but not VM/CTs (their booting tasks time out), and I can't properly see VM/CT names either.

do you have /var/log/pve and /var/log/pveproxy directories?

it looks like this:

$ tree /var/log/pve -d
/var/log/pve
└── tasks
    ├── 0
    ├── 1
    ├── 2
    ├── 3
    ├── 4
    ├── 5
    ├── 6
    ├── 7
    ├── 8
    ├── 9
    ├── A
    ├── B
    ├── C
    ├── D
    ├── E
    └── F

and /var/log/pveproxy just has access.log files (so you can make an empty directory there).

permissions are as follows

/var/log/pve:
total 0
drwxr-xr-x 1 root root   10 Aug  4 11:32 .
drwxr-xr-x 1 root root 2226 Jan 10 12:04 ..
drwxr-xr-x 1 root root   92 Jan 10 13:48 tasks

/var/log/pveproxy:
total 4080
drwx------ 1 www-data www-data     224 Jan 10 12:04 .
drwxr-xr-x 1 root     root        2226 Jan 10 12:04 ..
-rw-r----- 1 www-data www-data       0 Jan 10 12:04 access.log
-rw-r----- 1 www-data www-data 3983559 Jan  5 17:10 access.log.1
-rw-r----- 1 www-data www-data   10202 Jan  4 15:12 access.log.2.gz

Systemctl for pveproxy, pvedaemon, and pvestatd also look all good.

please also check: journalctl -u pve* -b0

you can also check and compare the journal with other nodes in your cluster.