Constant writing on my proxmox main drive

S

sysrq

New Member
Apr 25, 2025
16
0
1
Hello,

I have my homelab for about 1 month, maybe 2 and i noticed that the os is constantly writing, i think in 1 month or so it already writted between 5 to 10 GBs. At first i thought this is temporary logs/files. In this pace, i will run out of memory in less than a year. Can you tell me, please, if i can somehow to reduce this constant writting when my node is effectively iddle?

I'm planning to add another node and make a cluster, this means the constantly writting data will be doubled?

Thank you!
 
Last edited:
sysrq said:
Can you tell me, please, if i can somehow to reduce this constant writting when my node is effectively iddle?
Click to expand...
This cannot be removed completely. It's the internal database and rrd graphs that write constantly.
In order to have fun with your system, buy a used (better two) small enterprise SSDs and install PVE on there.

sysrq said:
I'm planning to add another node and make a cluster, this means the constantly writting data will be doubled?
Click to expand...
Good question, yet I don't think it'll be doubled, rrdgraphs are only written to the node on which the VM is running, yet the internal database will be writte more than on a single node.
 
  • Like
Reactions: Johannes S and sysrq
Already bought a 1tb crucial ssd (not enterprise tho), and i m planning to buy another one for the second one.

In conclusion, there isn't a way to reduce this constant writing?
 
sysrq said:
i will run out of memory in less than a year
Click to expand...
Do you mean disk space? All our cluster nodes are using around 2 GB for each OS drive. You're seeing 5-10 GB of space actually used on disk? Or just that amount written to disk during that time?

Based on a discussion elsewhere, if you are using ZFS for boot then you can maybe raise zfs_txg_timeout to lengthen the time before disk writes are actually written to disk.
https://www.reddit.com/r/zfs/comments/mgibzy/where_can_i_set_the_value_of_zfs_txg_timeout/
A

Thread 'how to change ZFS sync interval in PVE'

In FreeNAS , we can change the sync interval by change the value of vfs.zfs.txg.timeout. That is relative to sysctl.conf or sysctl tool .

But in PVE , I can not find the way to change the sync interval of ZFS in PVE from the internet. By default , it sync every 5 seconds, I don't like that way.

Anyone has any idea ?
 
  • Like
Reactions: Kingneutron and sysrq
sysrq said:
Already bought a 1tb crucial ssd (not enterprise tho), and i m planning to buy another one for the second one.
Click to expand...
Please make sure not to buy drives with QLC flash memory (search for QLC on this forum).
sysrq said:
In conclusion, there isn't a way to reduce this constant writing?
Click to expand...
There are threads on this forum that describe things you can do. But they are not supported, can break things and will make troubleshooting much more difficult. Maybe use KVM/QEMU directly via libvirt instead of a clustered enterprise hypervisor like Proxmox?
EDIT: I bought an enterprise SDD with PLP and stopped worrying about this at all. I would say it's worth it (especially if it's a second-hand enterprise SSD).
 
Last edited:
  • Like
Reactions: Kingneutron, UdoB, sysrq and 1 other person
The logfiles will be rotated (meaning that old files will be removed) after some time, so your disk won't run out of space normally. This doesn't help with the SSD wearout though, for this you can go with these options:
  • Get used enterprise-ssds with powerloss protection
  • Rotating HDDs shouldn't be used as storage for VMs (the performance won't make it fun), but are actually fine for the ProxmoxVE operating system, so you could install the OS on HDDs and the VMs on the SSDs
  • Be prepared, that you might need to replace your non-enterprise SSDs from time to time.
All of this get's a lot easier if you build a ZFS or HW raid mirror out of two disks for the operating system. Why? Because then in case of a failing/broken storage disk you can replace it, rebuilding the ZFS pool/raid 1 without needing to resetup everything.

I myself do something mixed: My homelab consists mainly of used mini-pcs who only has two slots for storage media: One for a M2 NVME and one for SATA 2.5. Since NVME-M2 enterprise SSDs with powerloss protection won't fit this slot and are way to expensive for my budget I use normal consumer NVME for the M2 slot. The SATA-Slot gets a used SATA enterprise-SSD. Both are together in a ZFS mirror for the operating system and as VM storage. I expect that one of the devices will fail sooner or later (propably the NVME, it's at 16% wearout at the moment, the enterprise SATA SSD has just 3% wearout) so I will then replace it. The neccesary steps are documented in the manual: https://pve.proxmox.com/wiki/ZFS_on_Linux#_zfs_administration -> Replace failed bootable device
 
Last edited:
  • Like
Reactions: UdoB and sysrq
What do you mean from time to time?

I am not using ZFS for boot, i have ZFS on my second 1 tb SSD, but the OS is on the main drive (a 128 GB M2 SATA III).

Yea, i know what QLC is and my Crucial BX500 is indeed QLC. That's why i want 3 nodes in a cluster with a Raid 5 (3 ZFS pools for each secondary 1 TB SSD from each node). To have redundancy when one of the ssds/nodes fails.
I will keep that in mind when i buy the next 2 SSDs, i will look for TLC or used enterprise SSDs.

I will probably use RAID 1 until i get my third node.
 
RAID1 or ZFS mirror is a good idea, even with multiple nodes, I always have it on all my Proxmox nodes ;) The biggest benefit with ZFS: With multiple devices in a mirror or RAID ZFS has a function to auto-correct bit errors with.

From time to time means exactly this: Although we don't know the exact date, every storage media will fail at some time, some less, some later. There might be a higher propability for a certain type (which is the reason why I expect my consumer SSD to fail earlier than my enterprise SSD->The first one has a higher wearout rate, thus it's more likely to fail) but this doesn't change, that all of them will fail somehow. That's the reason why it's good to be aware of it and plan/design your storage accordingly.
 
Some tips.

reroute pveproxy log to /dev/null. (each graph update is a new logged web request, when logged in) (unsupported configuration)
use nginx as reverse proxy and use buffered writes for log or also route to /dev/null. (each graph update is a new logged web request, when logged in) (unsupported configuration)
can configure zfs to make consumer SSDs behave like enterprise and not do force sync internally on drive (if doing this should have power safety), can still do this without disabling zil, so in the OS sync is still sync. This is a performance boost and large drop in write amplification. (note this isnt a supported configuration, why I didnt say how to do it so easily, but its in the open zfs documentation)
if using consumer drives make sure they have dram and are at least 3D TLC. Latest firmware. Keep at least 20% space free.
 
Last edited:
I will increase the storage when i will add the second node.

I will research a little bit all of your suggestions, maybe i can reduce that somehow.

Thank you for your time!

PS: if i want some tips for securing my homelab. Can i ask here or do i need to make another dedicated thread for each question?
 
Johannes S said:
While they might work none of that is supported and might lead to different issues/problems.
Click to expand...
That is true, when doing this kind of stuff, its a risk and takes you outside of what is supported. Sort of thing can do in a homelab, but in a commercial setting, the proper thing to do is use actual enterprise SSDs.
The excessive logging isnt important in a homelab, and probably not even important in a proper secure IP restricted environment either.
 
Last edited:
You must log in or register to reply here.
Top Bottom