Optimizing pxmcfs Process in Proxmox: Reducing Hard Drive Wear with RAM-Based Storage [Question]

[isasmendi]

Hi everyone,

I have a question regarding the pxmcfs process in Proxmox. I've noticed that it writes to the hard drive constantly, which has me concerned about potential long-term damage to the disk. After looking at the pve-cluster service code (available at here), I discovered that most of the writes are to the /var/lib/pve-cluster folder, specifically in the config.db-wal file.

To address this, I created a service that runs before pve-cluster, which does a bind mount of /dev/shm/pve-cluster-ram to /var/lib/pve-cluster. This results in files that change permanently being written to memory instead of the hard drive.

I should mention that I use Proxmox as a standalone node, and I do not have a cluster configured.

However, I have some questions about this approach. Firstly, what data could I lose in the event of a server restart where the config.db and config.db-wal files are lost? Secondly, would it be risky for my system if I don't make any system modifications like creating containers or changing the firewall? Lastly, how would backups be affected if I create a backup of a container/VM and the system shuts down (with config.db and config.db-wal in memory)? Could the backup list become out of sync with the actual data on my disk?

I would greatly appreciate any advice or suggestions on how to handle this situation. Thank you in advance for your help.

Best regards, Agustin I

 

[LnxBil]

However, I have some questions about this approach. Firstly, what data could I lose in the event of a server restart where the config.db and config.db-wal files are lost?

Your PVE configuration.

Secondly, would it be risky for my system if I don't make any system modifications like creating containers or changing the firewall?

IF you don't change anything, it MAY work

Lastly, how would backups be affected if I create a backup of a container/VM and the system shuts down (with config.db and config.db-wal in memory)? Could the backup list become out of sync with the actual data on my disk?

No, backups are not stored in your /etc/pve directory. A backup is just a compressed file (vma or tar) on your backup storage.

To address this, I created a service that runs before pve-cluster, which does a bind mount of /dev/shm/pve-cluster-ram to /var/lib/pve-cluster. This results in files that change permanently being written to memory instead of the hard drive.

Writing to a harddisk is what it was made for. I don't see that it'll harm it. For consumer SSDs, this IS another matter and the main culprit for early loss of device. Therefore, buy enterprise SSDs, which don't have the huge write amplification that will result in early wear.

 

[isasmendi]

Hello all

Thanks @LnxBil for your response. It helped me to solve the problem.

If you've made it here, it's because you're looking for a way to eliminate I/O operations from the pmxcfs process.
I've created a repository where you'll find a service that 'forces' pmxcfs to use RAM instead of HDD/SDD. You will find the installation instructions and how it works in the repository itself.

Link to repo

If you're going to use it, I recommend that you create a virtual machine, install Proxmox, and perform the necessary tests on that VM. Once you're satisfied, you can install it on your real Proxmox installation.

I'm currently using the service in my home lab and it works perfectly.

 

[mircolino]

Since you mentioned you are running a single node, did you disable all the clustering/replication stuff?

https://forum.proxmox.com/threads/m...-pve-configuration-changes.89104/#post-391299

I have PVE installed on a single node Intel NUC with a ZFS formatted 2TB Samsung 980 Pro. I'm monitoring wear smartctl -a /dev/nvme0n1 and I find it very reasonable.

 

[esi_y]

Hello all

Thanks @LnxBil for your response. It helped me to solve the problem.

If you've made it here, it's because you're looking for a way to eliminate I/O operations from the pmxcfs process.
I've created a repository where you'll find a service that 'forces' pmxcfs to use RAM instead of HDD/SDD. You will find the installation instructions and how it works in the repository itself.

Link to repo

If you're going to use it, I recommend that you create a virtual machine, install Proxmox, and perform the necessary tests on that VM. Once you're satisfied, you can install it on your real Proxmox installation.

I'm currently using the service in my home lab and it works perfectly.

Are you still on the forum? I believe there's a problem with the way it's flushing periodically - the .db* files are not guaranteed to copy all at once, so may end up with mismatching WAL or corrupt .db (if operation caught middle of checkpointing taking place).

 

[esi_y]

Writing to a harddisk is what it was made for. I don't see that it'll harm it. For consumer SSDs, this IS another matter and the main culprit for early loss of device. Therefore, buy enterprise SSDs, which don't have the huge write amplification that will result in early wear.

Well:

https://forum.proxmox.com/threads/etc-pve-500k-600m-amplification.154074/#post-701223