Proxmox 8.0.4: No changes to VMs persis after host reboot

[Creat]

Hello everyone,

I'm having a hell of a time after upgrading my Proxmox 7 to 8 recently. NO changes I make to any VM (resulting in changed .conf files in /etc/pve/...) persist after I reboot the host machine. This includes entire lxc/vm configs just disappearing for any lxc/vm that didn't exist at that time. If I copy in the missing configs from manual backups, they work fine (the storage is of course still there). I don't even know where to start looking into why this might be happening. Any files I copy out of the /etc/pve folder remain wherever I put them (as backups or whatever, so it's not a whole filesystem issue) and I can copy them back manually, but obviously this is rather cumbersome and not really a "solution".

It's an installation on a AM4-platform server mainboard (ASRock Rack X470D4U), on ZFS with uefi-boot.

There are 2 anomalies I can think of that might have some influence on this:
A) I recently replaced both mirrored ZFS boot drives with new ones, as the old SSDs were worn out. This went fine and as far as I can tell caused no direct issues.
B) at one point in the past (2ish years ago), I had created a cluster and joined a 2nd node as a test (which I had then removed just a few days later). I recently noticed due to SMART errors about wearout on one of my boot drives that there were excessive writes to the disks and that it being a 1-node-cluster might be a contributing factor (after some googling). I followed this post to revert to a clusterless state. It seems to have worked, as the corosync service is enabled but doesn't start with "condition failed", which should mean it's all fine (as it has no config in /etc/pve/corosync.conf after the changes). Did I miss a step here that isn't in these (old-ish) instructions?

Thanks for any help!

 

[Chris]

Hi,
another user reported a similar behaviour recently, all his configs got reverted on boot [0]. In his case it turned out, that he mounted the whole of /var into a tmpfs in ram, resulting in this behaviour as the sqlite database backing the pmxcfs, which is mounted on /etc/pve is also residing in a subfolder under /var and therefore was never persisted to disk.

So please check and post the output of mount and provide systemctl status pve-cluster.service.

[0] https://forum.proxmox.com/threads/c...les-keep-getting-reverted.134333/#post-593590

 

[Creat]

Thanks so much for the almost instant response. I did search but didn't find that particular post/thread. And yes it at least seems related, finally a point to start searching for a cause/solution!

I do have folder2ram installed like he did, also to reduce wear on SSDs but frankly it seems to have a limited effect. I do not have /var completely in ram, but I do have /var/log mapped like he did. Specifically, I have the following, and pve-cluster and pve-manager seem candidates for trouble. Frankly I had forgotten about those completely.

Edit: I am not sure he even had all of /var in folder2ram, unless I'm misreading his post? He said Ok, I removed the /var/log mount in /etc/fstab and uninstalled the folder2ram service., but I don't see any indication of all of /var being mapped in his case?

Do you recommend against this for reducing SSD wear? I still get a massive amount of writes per day, frankly absurd amounts despite all this effort. Any alternatives or am I just missing a critical folder or something? As often recommended, I have pve-ha-crm and pve-ha-lrm services disabled.

This is my folder2ram config currently:

tmpfs           /var/log
tmpfs           /var/cache
tmpfs           /var/lib/pve-cluster
tmpfs           /var/lib/pve-manager
tmpfs           /var/lib/rrdcached

Additionally here the requested output for both commands:

● pve-cluster.service - The Proxmox VE cluster filesystem
     Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; preset: enabled)
     Active: active (running) since Wed 2023-10-18 14:51:50 CEST; 24min ago
   Main PID: 4369 (pmxcfs)
      Tasks: 7 (limit: 76939)
     Memory: 44.9M
        CPU: 1.585s
     CGroup: /system.slice/pve-cluster.service
             └─4369 /usr/bin/pmxcfs

Oct 18 14:51:49 proxmox systemd[1]: Starting pve-cluster.service - The Proxmox VE cluster filesystem...
Oct 18 14:51:49 proxmox pmxcfs[4285]: [main] notice: resolved node name 'proxmox' to '192.168.1.5' for default node IP address
Oct 18 14:51:49 proxmox pmxcfs[4285]: [main] notice: resolved node name 'proxmox' to '192.168.1.5' for default node IP address

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=32827528k,nr_inodes=8206882,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=6572496k,mode=755,inode64)
rpool/ROOT/pve-1 on / type zfs (rw,relatime,xattr,noacl)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=21495)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /mnt/lxc_shares/video type autofs (rw,relatime,fd=58,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=25061)
rpool on /rpool type zfs (rw,noatime,xattr,noacl)
rpool/data on /rpool/data type zfs (rw,noatime,xattr,noacl)
rpool/ROOT on /rpool/ROOT type zfs (rw,noatime,xattr,noacl)
rpool/data/subvol-512-disk-5 on /rpool/data/subvol-512-disk-5 type zfs (rw,noatime,xattr,posixacl)
rpool/data/subvol-505-disk-0 on /rpool/data/subvol-505-disk-0 type zfs (rw,noatime,xattr,posixacl)
rpool/data/subvol-503-disk-0 on /rpool/data/subvol-503-disk-0 type zfs (rw,noatime,xattr,posixacl)
rpool/data/subvol-512-disk-2 on /rpool/data/subvol-512-disk-2 type zfs (rw,noatime,xattr,posixacl)
rpool/data/subvol-512-disk-3 on /rpool/data/subvol-512-disk-3 type zfs (rw,noatime,xattr,posixacl)
rpool/data/subvol-504-disk-2 on /rpool/data/subvol-504-disk-2 type zfs (rw,noatime,xattr,posixacl)
rpool/data/subvol-510-disk-0 on /rpool/data/subvol-510-disk-0 type zfs (rw,noatime,xattr,posixacl)
rpool/data/subvol-506-disk-0 on /rpool/data/subvol-506-disk-0 type zfs (rw,noatime,xattr,posixacl)
rpool/data/subvol-507-disk-0 on /rpool/data/subvol-507-disk-0 type zfs (rw,noatime,xattr,posixacl)
rpool/data/subvol-504-disk-0 on /rpool/data/subvol-504-disk-0 type zfs (rw,noatime,xattr,posixacl)
rpool/data/subvol-512-disk-0 on /rpool/data/subvol-512-disk-0 type zfs (rw,noatime,xattr,posixacl)
rpool/data/subvol-520-disk-0 on /rpool/data/subvol-520-disk-0 type zfs (rw,noatime,xattr,posixacl)
rpool/data/subvol-512-disk-4 on /rpool/data/subvol-512-disk-4 type zfs (rw,noatime,xattr,posixacl)
rpool/data/subvol-512-disk-1 on /rpool/data/subvol-512-disk-1 type zfs (rw,noatime,xattr,posixacl)
rpool/data/subvol-504-disk-1 on /rpool/data/subvol-504-disk-1 type zfs (rw,noatime,xattr,posixacl)
rpool/data/subvol-511-disk-0 on /rpool/data/subvol-511-disk-0 type zfs (rw,noatime,xattr,posixacl)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
rpool/ROOT/pve-1 on /var/folder2ram/var/log type zfs (rw,relatime,xattr,noacl)
folder2ram on /var/log type tmpfs (rw,relatime,inode64)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
rpool/ROOT/pve-1 on /var/folder2ram/var/cache type zfs (rw,relatime,xattr,noacl)
folder2ram on /var/cache type tmpfs (rw,relatime,inode64)
rpool/ROOT/pve-1 on /var/folder2ram/var/lib/pve-cluster type zfs (rw,relatime,xattr,noacl)
folder2ram on /var/lib/pve-cluster type tmpfs (rw,relatime,inode64)
rpool/ROOT/pve-1 on /var/folder2ram/var/lib/pve-manager type zfs (rw,relatime,xattr,noacl)
folder2ram on /var/lib/pve-manager type tmpfs (rw,relatime,inode64)
rpool/ROOT/pve-1 on /var/folder2ram/var/lib/rrdcached type zfs (rw,relatime,xattr,noacl)
folder2ram on /var/lib/rrdcached type tmpfs (rw,relatime,inode64)
192.168.2.10:/mnt/hdd/pve-backup on /mnt/pve/truenas_nfs_backup type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.2.10,mountvers=3,mountport=48252,mountproto=udp,local_lock=none,addr=192.168.2.10)
//truenas/pve-iso on /mnt/pve/truenas-iso type cifs (rw,relatime,vers=3.1.1,cache=strict,username=proxmox,uid=0,noforceuid,gid=0,noforcegid,addr=192.168.1.10,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,rsize=4194304,wsize=4194304,bsize=1048576,echo_interval=60,actimeo=1,closetimeo=1)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=6572492k,nr_inodes=1643123,mode=700,inode64)

 

[Chris]

/var/log /var/cache /var/lib/pve-cluster /var/lib/pve-manager /var/lib/rrdcached
All these are folders containing files which are expected to be persisted to disk, so a setup like this cannot work correctly.

Do you recommend against this for reducing SSD wear? I still get a massive amount of writes per day, frankly absurd amounts despite all this effort. Any alternatives or am I just missing a critical folder or something?

I recommend strongly against using such solutions on folders you do not exactly know what is stored there. May I ask where do you got the information from that this might help reducing your SSD wear? Well it most certainly does, but at the cost of breaking your Proxmox VE installation.

If you want to avoid writing to your SSD and are not willing to go for enterprise grade SSDs designed to withstand these writes, then I recommend to mount a HDD or any other persistent storage to /var instead.

 

[Creat]

I'll disable/remove folder2ram completely then and reboot, I would hope that would then solve the issue. I will look a bit into what/how this got broken...

Generally the content should still be persistent, as on reboot the ram content is suppsed to be sync'd back to disk. All reboots were graceful, so I'm guessing folder2ram somehow broke when upgrading to V8 and with it the underlying debian? I'm not entirely sure, but it seems the write-back-on-shutdown just doesn't work anymore (the service is still there though).

I don't remember exactly where I got my instructions from, but pretty sure it was here on the forum. From my reverse-research just now, I'm pretty sure this played a role (local forum, but german subsection), but also reddit posts similar to this one. It seems I've enacted this in april, but pretty sure I've just googled "proxmox reduce disk writes" or similar and followed a couple of links.
In essence, folder2ram is just a basic script that wraps memory caching for specified folders into systemd services that sync content from/to disk on boot/shutdown. I don't mind losing logs in case of a power outage or similar, so this seems like a great solution and it just somehow doesn't work anymore.

As a side note: this is obviously a home server. We're running proxmox also in production with enterprise-grade SSDs (reported by proxmox gui as: "Dell Ent NVMe AGN MU U.2 3.2TB") and even there the excessive writes add up quite a bit over time. So having ways to recude that would be appreciated in that context as well. That system is 2ish years old, and obviously in active use, but smart reports 350 TB read and a staggering 7 PB written. Wearout is at ~20% already. I can assure you that at least 90% of that is just from proxmox, if not more! Our applications are not write intensive at all, so that really is rather excessive, even if the drives can (kinda) take it.

 

[Chris]

I'll disable/remove folder2ram completely then and reboot, I would hope that would then solve the issue. I will look a bit into what/how this got broken...

Okay, I hope that will solve your issues, obviously you will have to restore lost configs from backup if not done already.

Generally the content should still be persistent, as on reboot the ram content is suppsed to be sync'd back to disk. All reboots were graceful, so I'm guessing folder2ram somehow broke when upgrading to V8 and with it the underlying debian? I'm not entirely sure, but it seems the write-back-on-shutdown just doesn't work anymore (the service is still there though).

Yes, I have looked this up now since I was curious. Note that while this might be a solution for cases where performance is of more importance than data, this should not be used for important data. A power loss and all the data is gone, but you seem to be already aware of that.

I'm pretty sure this played a role (local forum, but german subsection

Yes, I also found this thread, but at least it was not suggested to put pve-cluster and pve-manager into ram.

reddit posts similar to this one.

Unfortunately it seem that this was ill advised, not being aware of the consequences.

 

[Creat]

Well so far, everything seems back to normal after removing folder2ram. I'm not sure if I could've gotten it to work again (with better or more conservative folder selection, like only log/cache folders), but in any case it seems more fragile than I had assumed/hoped. So it's probably not worth the risk. Also the last update to the script has been made 2ish years ago (original source is here), and it was originally intended for OpenMediaVault, but made for general Debian-based distros. Of course, made quite some time before bookworm.

I still don't know why/how it broke, the script is coneceptually rather basic: on start, create tmpfs, copy folder contentents into it; On stop, rsync the changes back to disk. It's not rocket science. I guess some of the relied on tools or system behavior changed? My knowledge in the area isn't deep enough to find out, honestly.
For reference, it happens at around line 666 in this (only) script file. Respectively, the stop is at 755.

Thanks for the help again, glad I could resolve it rather quickly thanks to you!

I do have to figure out what exactly is causing the excessive writes though, and if it's just the proxmox logging or something else. I have installed new disks on saturday (4 days ago). The installation (mirroring/resilvering of existing partitions) and initial tinkering was ~500 GB in writes. I'm already back to 1.5 TB, which makes it 250 GB/day. That's kinda nuts for a system that was essentially not doing anything since I was sick in bed and not actively using it.

 

[Chris]

Well so far, everything seems back to normal after removing folder2ram. I'm not sure if I could've gotten it to work again (with better or more conservative folder selection, like only log/cache folders), but in any case it seems more fragile than I had assumed/hoped. So it's probably not worth the risk. Also the last update to the script has been made 2ish years ago (original source is here), and it was originally intended for OpenMediaVault, but made for general Debian-based distros. Of course, made quite some time before bookworm.

I still don't know why/how it broke, the script is coneceptually rather basic: on start, create tmpfs, copy folder contentents into it; On stop, rsync the changes back to disk. It's not rocket science. I guess some of the relied on tools or system behavior changed? My knowledge in the area isn't deep enough to find out, honestly.
For reference, it happens at around line 666 in this (only) script file. Respectively, the stop is at 755.

Thanks for the help again, glad I could resolve it rather quickly thanks to you!

I do have to figure out what exactly is causing the excessive writes though, and if it's just the proxmox logging or something else. I have installed new disks on saturday (4 days ago). The installation (mirroring/resilvering of existing partitions) and initial tinkering was ~500 GB in writes. I'm already back to 1.5 TB, which makes it 250 GB/day. That's kinda nuts for a system that was essentially not doing anything since I was sick in bed and not actively using it.

Glad the issue got resolved.

Regarding the excessive writes, 250GB/day seem rather high. I would suggest you try to find out what causes these writes by e.g. letting the following command run for a few days in a screen or tmux session iotop -aoP. This will give you a list of processes with their accumulated disk reads/writes, which could get you an idea on where to look further. Once you have found a suspicions process, you might further investigate using lsof and strace to see what is written to where. Note that you will probably have to install iotop and strace on your system first.