Accidentally deleted /var/log, cluster having massive issues

britmob

New Member
Jan 6, 2022
3
0
1
24
Hi there, I was trying to do routine clearing out of /var/log files, and typo'd my rm command and deleted everything inside /var/log.

Right now I have tried to recreate the directory structure and files, as well as fixed permissions. However, my servers are still turning to the grey question mark in the web UI. Their web UI still works, and is mostly functional. But the issue is that all my storage, VMs, and CTs are greyed out. The storage works on those nodes, but not VM/CTs (their booting tasks time out), and I can't properly see VM/CT names either.

The other issue is that shutdown/start/reboot don't work properly on nodes, tasks complete (or sometimes shutdowns/starts time out despite saying "TASK OK") but it simply will not shut down or reboot. It just sits until I unplug it.

Is there any way I can just restore /var/log to what it was like? I don't know exactly what is causing all these things but it is breaking almost every part of the cluster's functionality.

(I don't back up the host nodes, only VM/CT data.. hopefully PBS supports host backups in the next release..)

EDIT: Worth mentioning that on initial boot, the nodes are green in the UI. It is only after a few minutes (or when I navigate to a VM page) that they go grey. I see nothing in the syslog that would align with the switch to grey. Systemctl for pveproxy, pvedaemon, and pvestatd also look all good.
 
Last edited:
hi,

Hi there, I was trying to do routine clearing out of /var/log files, and typo'd my rm command and deleted everything inside /var/log.
whoops :/

Right now I have tried to recreate the directory structure and files, as well as fixed permissions. However, my servers are still turning to the grey question mark in the web UI. Their web UI still works, and is mostly functional. But the issue is that all my storage, VMs, and CTs are greyed out. The storage works on those nodes, but not VM/CTs (their booting tasks time out), and I can't properly see VM/CT names either.
do you have /var/log/pve and /var/log/pveproxy directories?

it looks like this:
Code:
$ tree /var/log/pve -d
/var/log/pve
└── tasks
    ├── 0
    ├── 1
    ├── 2
    ├── 3
    ├── 4
    ├── 5
    ├── 6
    ├── 7
    ├── 8
    ├── 9
    ├── A
    ├── B
    ├── C
    ├── D
    ├── E
    └── F
and /var/log/pveproxy just has access.log files (so you can make an empty directory there).

permissions are as follows
Code:
/var/log/pve:
total 0
drwxr-xr-x 1 root root   10 Aug  4 11:32 .
drwxr-xr-x 1 root root 2226 Jan 10 12:04 ..
drwxr-xr-x 1 root root   92 Jan 10 13:48 tasks

/var/log/pveproxy:
total 4080
drwx------ 1 www-data www-data     224 Jan 10 12:04 .
drwxr-xr-x 1 root     root        2226 Jan 10 12:04 ..
-rw-r----- 1 www-data www-data       0 Jan 10 12:04 access.log
-rw-r----- 1 www-data www-data 3983559 Jan  5 17:10 access.log.1
-rw-r----- 1 www-data www-data   10202 Jan  4 15:12 access.log.2.gz

Systemctl for pveproxy, pvedaemon, and pvestatd also look all good.
please also check: journalctl -u pve* -b0

you can also check and compare the journal with other nodes in your cluster.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!