Messed up my hypervisor when attempting to reduce ZFS writes

S

scotrod

New Member
Aug 20, 2025
5
0
1
Hey folks, wanted to share a cautionary tale. I recently tried to reduce ZFS writes on my Proxmox hypervisor because I’m running consumer SSDs and wanted to limit wear. I disabled some HA services, moved logs to tmpfs, and pretty much f*cked big time my `/var/log/` permissions

Long story short: I completely broke my Proxmox WebUI. VM/CT names were greyed out, graphs were gone, and pveproxy couldn’t write logs because /var/log/pveproxy and /var/log/pve directories were missing or mis-permissioned. I had to manually recreate directories, set ownership and permissions back to stock, and restart all Proxmox services to get everything working again.

I wanted to ask if someone could provide me with their default permissions sets of everything which sits under `/var/log`. I know this is a dumb requests, but I'm not sure if I can install Proxmox on top of Proxmox since I'm running with a CPU that doesn't support nested virtualization.
 
leesteken said:
Code: 
# ls -alh /var/log
total 22M
drwxr-xr-x 18 root     root              121 22 okt 01:44 .
drwxr-xr-x 11 root     root               14  7 aug 10:02 ..
-rw-r--r--  1 root     root              15K 22 okt 08:24 alternatives.log
-rw-r--r--  1 root     root              27K  1 okt 01:09 alternatives.log.1
-rw-r--r--  1 root     root              889 31 dec  2024 alternatives.log.10.gz
-rw-r--r--  1 root     root             2,0K 30 nov  2024 alternatives.log.11.gz
-rw-r--r--  1 root     root              925 31 okt  2024 alternatives.log.12.gz
-rw-r--r--  1 root     root             3,1K  1 sep 08:17 alternatives.log.2.gz
-rw-r--r--  1 root     root              896 31 jul 16:05 alternatives.log.3.gz
-rw-r--r--  1 root     root              890 30 jun 06:38 alternatives.log.4.gz
-rw-r--r--  1 root     root             1,1K 31 mei 08:57 alternatives.log.5.gz
-rw-r--r--  1 root     root             1,2K 30 apr 15:59 alternatives.log.6.gz
-rw-r--r--  1 root     root             1,2K 31 mrt  2025 alternatives.log.7.gz
-rw-r--r--  1 root     root              642 28 feb  2025 alternatives.log.8.gz
-rw-r--r--  1 root     root              965 31 jan  2025 alternatives.log.9.gz
drwxr-xr-x  2 root     root               29 19 okt 21:11 apt
-rw-r-----  1 root     adm              619K 29 okt  2024 auth.log
-rw-r-----  1 root     adm              1,5M 27 okt  2024 auth.log.1
-rw-r-----  1 root     adm              152K 20 okt  2024 auth.log.2.gz
-rw-r-----  1 root     adm              179K 13 okt  2024 auth.log.3.gz
-rw-r-----  1 root     adm              153K  6 okt  2024 auth.log.4.gz
-rw-rw----  1 root     utmp                0  1 okt 01:58 btmp
-rw-rw----  1 root     utmp                0  1 sep 08:55 btmp.1
drwxrws--T  2 ceph     ceph                2  5 apr  2019 ceph
drwxr-x---  2 _chrony  _chrony             2 14 okt  2023 chrony
drwxr-xr-x  2 root     root                3 17 jul  2019 corosync
-rw-r-----  1 root     adm              318K 29 okt  2024 cron.log
-rw-r-----  1 root     adm              753K 26 okt  2024 cron.log.1
-rw-r-----  1 root     adm               69K 19 okt  2024 cron.log.2.gz
-rw-r-----  1 root     adm               88K 13 okt  2024 cron.log.3.gz
-rw-r-----  1 root     adm               70K  5 okt  2024 cron.log.4.gz
-rw-r-----  1 root     adm              1,5M 14 okt  2023 daemon.log
-rw-r-----  1 root     adm              1,5M  8 okt  2023 daemon.log.1
-rw-r-----  1 root     adm               71K 14 okt  2023 debug
-rw-r-----  1 root     adm               78K  7 okt  2023 debug.1
-rw-r-----  1 root     adm              4,3K 30 sep  2023 debug.2.gz
-rw-r-----  1 root     adm              4,3K 24 sep  2023 debug.3.gz
-rw-r-----  1 root     adm              4,3K 17 sep  2023 debug.4.gz
-rw-r--r--  1 root     root              16K 19 okt 21:11 dpkg.log
-rw-r--r--  1 root     root              92K 29 sep 17:39 dpkg.log.1
-rw-r--r--  1 root     root             2,2K 29 dec  2024 dpkg.log.10.gz
-rw-r--r--  1 root     root              53K 28 nov  2024 dpkg.log.11.gz
-rw-r--r--  1 root     root             1,1K 29 okt  2024 dpkg.log.12.gz
-rw-r--r--  1 root     root              67K 29 aug 13:28 dpkg.log.2.gz
-rw-r--r--  1 root     root             4,0K 31 jul 11:54 dpkg.log.3.gz
-rw-r--r--  1 root     root             1,5K 30 jun 21:53 dpkg.log.4.gz
-rw-r--r--  1 root     root             5,1K 30 mei 10:06 dpkg.log.5.gz
-rw-r--r--  1 root     root             4,9K 29 apr 14:29 dpkg.log.6.gz
-rw-r--r--  1 root     root             4,7K 31 mrt  2025 dpkg.log.7.gz
-rw-r--r--  1 root     root             3,0K 28 feb  2025 dpkg.log.8.gz
-rw-r--r--  1 root     root             5,5K 30 jan  2025 dpkg.log.9.gz
-rw-r--r--  1 root     root             2,0M  7 aug 09:58 faillog
-rw-r--r--  1 root     root             4,0K  7 aug 09:57 fontconfig.log
drwxr-x---  2 frr      frr                 2  4 aug 14:51 frr
drwxr-xr-x  2 root     root                2 15 apr  2019 glusterfs
drwxr-xr-x 45 root     root               45 22 okt 08:24 ifupdown2
drwxr-sr-x  3 root     systemd-journal     3  4 nov  2020 journal
-rw-r-----  1 root     adm              1,2M 29 okt  2024 kern.log
-rw-r-----  1 root     adm              4,0M 27 okt  2024 kern.log.1
-rw-r-----  1 root     adm              694K 20 okt  2024 kern.log.2.gz
-rw-r-----  1 root     adm              422K 12 okt  2024 kern.log.3.gz
-rw-r-----  1 root     adm              572K  6 okt  2024 kern.log.4.gz
-rw-rw-r--  1 root     utmp              18M 22 okt 22:21 lastlog
drwxr-xr-x  2 root     root                3 17 jul  2019 lxc
-rw-r-----  1 root     adm                 0 10 mei  2020 mail.err
-rw-r-----  1 root     adm               101  9 mei  2020 mail.err.1
-rw-r-----  1 root     adm               112 16 nov  2019 mail.err.2.gz
-rw-r-----  1 root     adm               112  2 sep  2019 mail.err.3.gz
-rw-r-----  1 root     adm               142 31 aug  2019 mail.err.4.gz
-rw-r-----  1 root     adm               17K 14 okt  2023 mail.info
-rw-r-----  1 root     adm               17K  7 okt  2023 mail.info.1
-rw-r-----  1 root     adm              1,8K 30 sep  2023 mail.info.2.gz
-rw-r-----  1 root     adm              2,1K 23 sep  2023 mail.info.3.gz
-rw-r-----  1 root     adm              1,8K 16 sep  2023 mail.info.4.gz
-rw-r-----  1 root     adm              3,9K 29 okt  2024 mail.log
-rw-r-----  1 root     adm               15K 27 okt  2024 mail.log.1
-rw-r-----  1 root     adm              1,8K 20 okt  2024 mail.log.2.gz
-rw-r-----  1 root     adm              1,2K 12 okt  2024 mail.log.3.gz
-rw-r-----  1 root     adm              1,5K  6 okt  2024 mail.log.4.gz
-rw-r-----  1 root     adm              2,6K 14 okt  2023 mail.warn
-rw-r-----  1 root     adm              2,6K  7 okt  2023 mail.warn.1
-rw-r-----  1 root     adm               273 30 sep  2023 mail.warn.2.gz
-rw-r-----  1 root     adm               263 23 sep  2023 mail.warn.3.gz
-rw-r-----  1 root     adm               256 16 sep  2023 mail.warn.4.gz
-rw-r-----  1 root     adm              7,0M 14 okt  2023 messages
-rw-r-----  1 root     adm              7,7M  7 okt  2023 messages.1
-rw-r-----  1 root     adm              642K  1 okt  2023 messages.2.gz
-rw-r-----  1 root     adm              637K 24 sep  2023 messages.3.gz
-rw-r-----  1 root     adm              640K 17 sep  2023 messages.4.gz
drwx------  2 root     root                2 17 jul  2019 private
drwxr-xr-x  3 backup   backup              3 28 apr  2021 proxmox-backup
drwxr-xr-x  3 root     root                3 17 jul  2019 pve
-rw-r--r--  1 root     root             184K 22 okt 18:17 pveam.log
-rw-r--r--  1 root     root             257K 15 jul 13:47 pveam.log.0
-rw-r-----  1 root     adm              6,4M 22 okt 22:21 pve-firewall.log
-rw-r-----  1 root     adm              5,4M 22 okt 01:44 pve-firewall.log.1
-rw-r-----  1 root     adm              452K 21 okt 00:14 pve-firewall.log.2.gz
-rw-r-----  1 root     adm              581K 20 okt 00:56 pve-firewall.log.3.gz
-rw-r-----  1 root     adm              637K 19 okt 03:42 pve-firewall.log.4.gz
-rw-r-----  1 root     adm              118K 18 okt 00:18 pve-firewall.log.5.gz
-rw-r-----  1 root     adm              224K 17 okt 00:14 pve-firewall.log.6.gz
-rw-r-----  1 root     adm              232K 16 okt 01:20 pve-firewall.log.7.gz
drwx------  2 www-data www-data           10 22 okt 01:44 pveproxy
lrwxrwxrwx  1 root     root               39 14 okt  2023 README -> ../../usr/share/doc/systemd/README.logs
drwxr-xr-x  3 root     root                3  6 aug  2021 runit
drwxr-x---  2 root     adm                 2 20 jun  2019 samba
-rw-r-----  1 root     adm              2,7M 29 okt  2024 syslog
-rw-r-----  1 root     adm              7,9M 27 okt  2024 syslog.1
-rw-r-----  1 root     adm              1,3M 20 okt  2024 syslog.2.gz
-rw-r-----  1 root     adm             1016K 13 okt  2024 syslog.3.gz
-rw-r-----  1 root     adm              1,2M  6 okt  2024 syslog.4.gz
-rw-r-----  1 root     adm              131K  1 aug  2021 syslog.6.gz
-rw-r-----  1 root     adm               42K 31 jul  2021 syslog.7.gz
-rw-r-----  1 root     adm              670K 29 okt  2024 user.log
-rw-r-----  1 root     adm              1,6M 27 okt  2024 user.log.1
-rw-r-----  1 root     adm              171K 20 okt  2024 user.log.2.gz
-rw-r-----  1 root     adm              206K 12 okt  2024 user.log.3.gz
-rw-r-----  1 root     adm              167K  6 okt  2024 user.log.4.gz
drwxr-xr-x  2 root     root               29 22 okt 09:55 vzdump
-rw-rw-r--  1 root     utmp             270K 22 okt 22:21 wtmp
-rw-rw-r--  1 root     utmp             1,1M  5 aug 12:56 wtmp.1
Click to expand...
Seems for once I actually did it right - my permissions seem normal. Thanks!
 
P

Thread 'Accidentally deleted /var/log cannot access web interface'

I feel secure backupping my VMs and LXCs but didn't think about a proxmox host backup and started my learning curve finding it easier to reinstall PM, allowing me to try new versions from time to time. Now I would prefer to solve this issue rather than reinstall but manually recreating the directory thru ssh and cli seems to much for me.
I can ssh the server on port 22 but it don't show up the web ui on port 8006.

pveversion -v > seems to be ok

proxmox-ve: 8.2.0 (running kernel: 6.8.4-2-pve)
pve-manager: 8.2.2 (running version: 8.2.2/9355359cd7afbae4)
proxmox-kernel-helper: 8.1.0...


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
bbgeek17 said:
P

Thread 'Accidentally deleted /var/log cannot access web interface'

I feel secure backupping my VMs and LXCs but didn't think about a proxmox host backup and started my learning curve finding it easier to reinstall PM, allowing me to try new versions from time to time. Now I would prefer to solve this issue rather than reinstall but manually recreating the directory thru ssh and cli seems to much for me.
I can ssh the server on port 22 but it don't show up the web ui on port 8006.

pveversion -v > seems to be ok

proxmox-ve: 8.2.0 (running kernel: 6.8.4-2-pve)
pve-manager: 8.2.2 (running version: 8.2.2/9355359cd7afbae4)
proxmox-kernel-helper: 8.1.0...


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...
Thanks. I'm a brave soul so as you can guess I haven't take the measurements to deploy a Proxmox Backup Server yet. I will try to deploy another Proxmox instance on top of my current one (hope it even lets me) to see if I can just re-create everything that should be available under `/var/log`. So far all I have is:

Code: 
root@proxmox:/var/log# ls -la

total 13

drwxr-xr-x  4 root     root     100 Oct 22 23:02 .

drwxr-xr-x 11 root     root      14 Aug 21 17:16 ..

drwxr-xr-x  3 root     root      60 Oct 22 05:02 pve

-rw-r--r--  1 root     root     826 Oct 22 05:02 pveam.log

drwx------  2 www-data www-data  80 Oct 22 23:02 pveproxy

root@proxmox:/var/log#
 
scotrod said:
I disabled some HA services, moved logs to tmpfs, and pretty much f*cked big time my `/var/log/` permissions
Click to expand...
yeah thats a bad way to go. keep logs where they are and mount your ramdisk as a unionfs at boot. I would still do a periodic commit.
scotrod said:
I will try to deploy another Proxmox instance on top of my current one (hope it even lets me) to see if I can just re-create everything that should be available under `/var/log`.
Click to expand...
dont do that. backup /etc/pve/storage.cfg, /etc/pve/lxc, and /etc/pve/qemu-server and reinstall as new- you can just write those back on.
 
alexskysilk said:
yeah thats a bad way to go. keep logs where they are and mount your ramdisk as a unionfs at boot. I would still do a periodic commit.

dont do that. backup /etc/pve/storage.cfg, /etc/pve/lxc, and /etc/pve/qemu-server and reinstall as new- you can just write those back on.
Click to expand...
To be honest that host hosts my pfsense router/firewall VM so I'm really trying to escape the downtime here. I will research the ramdisk and unionfs approach (haven't heard about that until now). I hope in the meantime the host doesn't crash.
 
scotrod said:
To be honest that host hosts my pfsense router/firewall VM so I'm really trying to escape the downtime here.
Click to expand...
Lesson for the future: dont mess with your production machine.

scotrod said:
I hope in the meantime the host doesn't crash.
Click to expand...
It probably wont. you just wont be able to access various services depending on the stuff you broke.
 
  • Like
Reactions: Johannes S
scotrod said:
I recently tried to reduce ZFS writes on my Proxmox hypervisor because I’m running consumer SSDs and wanted to limit wear.
Click to expand...
Another one of those messing with stuff that is totally useless to mess with. Haven't you asked before? I answer this here and on reddit almost weekly: Just don't. It's not worth the hassle, just buy used enterprise ssds and keep on doing stuff that makes you happy and/or does not waste your time. Optimizing this is is just wasting your time.
 
  • Like
Reactions: UdoB and Johannes S
scotrod said:
I know this is a dumb requests, but I'm not sure if I can install Proxmox on top of Proxmox since I'm running with a CPU that doesn't support nested virtualization.
Click to expand...
You can you just won't be able to run any VMs. But just running lxc containers should work without any problems ;)

For your current issue I would however strongly recommend to:
  • Buy used enterprise ssds with power-loss-protection (PLP is important! You can filter for it on geizhalz.at to get an idea for which models to look )
  • Backup as much as possible (maybe after fixing your current issue but you might have messed up more than you realize)
  • Reinstall your system
  • Never every worry about "reducing wear" again, or with other words: What Lnxbil says ;)
 
Last edited:
  • Like
Reactions: UdoB
LnxBil said:
Another one of those messing with stuff that is totally useless to mess with. Haven't you asked before? I answer this here and on reddit almost weekly: Just don't. It's not worth the hassle, just buy used enterprise ssds and keep on doing stuff that makes you happy and/or does not waste your time. Optimizing this is is just wasting your time.
Click to expand...
Lesson learned... but to be honest, on my next reinstall I just won't bother with ZFS. I plugged those SSDs at 6% wear out using them for more than 2 years for 24/7 use on my esxi host (which hosted the same services as my current proxmox host). After not even 2 months use, their wear out went to 38%, and it's not like I use any of ZFS' features anyway.

I realize that was a junkyard 'fix' of a problem that I shouldn't have dealt with, what's done is done and I'm past this now. Is there any way for me to see which services may be failing because of the /var/log "cleanup" and could I just re-create the directories & files? I know reinstalling will be easier, but I am not physically present to the host, and like I said, it host my home router/FW.
 
  • Like
Reactions: Johannes S
You must log in or register to reply here.
Top Bottom