ZFS pool -> activity every 5 seconds

[fuzzybeanbag]

Hi all,

I am running Proxmox off a NVME (boot device) with a single ZFS pool consisting of RAID10 equivalent setup (i.e. mirrored vdevs) on 4 SSDs connected to a SAS3 HBA. I've noticed that the SSDs activity lights are constantly lit up. Checking iostat i've noticed 150 writes to sda, sdb and 110 to sdc and sdd every 5 seconds. I had a similar issue in TrueNAS Scale which ended up being the system dataset being set to use the ZFS pool rather than the boot device.

Is there a similar setting that can be toggled in proxmox? Anything else that could be causing these writes?

Thanks!

 

[Dunuin]

ZFS by default will cache all writes in RAM and then will write them from RAM to disk every 5 seconds. And PVE will continue write stuff to the disks. Logs, metrics, PMXCFS by the cluster services, ... so thats totally normal. PVE will write some GBs of very small writes to the system pool.

 

[fuzzybeanbag]

ZFS by default will cache all writes in RAM and then will write them from RAM to disk every 5 seconds. And PVE will continue write stuff to the disks. Logs, metrics, PMXCFS by the cluster services, ... so thats totally normal. PVE will write some GBs of very small writes to the system pool.

Right - so can I change system pool (i.e. system dataset) to the boot nvme instead of the ssd pool?

 

[Neobin]

Right - so can I change system pool (i.e. system dataset) to the boot nvme instead of the ssd pool?

There is no system dataset that you can move around like it is on TrueNAS. If you installed PVE on the NVMe, the whole installation is only there.
This leads to the question: For what is the SSD-pool actually used for? And is it configured as a storage (under "Datacenter") in PVE?

 

[fuzzybeanbag]

There is no system dataset that you can move around like it is on TrueNAS. If you installed PVE on the NVMe, the whole installation is only there.
This leads to the question: For what is the SSD-pool actually used for? And is it configured as a storage (under "Datacenter") in PVE?

The pool is used for hosting VMs and yes - it is configured under the storage section in Datacenter.

 

[Neobin]

You should give more information on the whole situation and what you are exactly expecting, respectively trying to accomplish.
I mean, if those VMs are running, it can be totally expected, that there is regular activity on the storage they are on.
So, do you see regular activity even without the VMs running, or?
PVE also checks the status of the configured storages every few seconds. So this produces also activity.
Reads/Access can also lead to writes, because of the access time (atime) being updated.
And so on...

 

[fuzzybeanbag]

You should give more information on the whole situation and what you are exactly expecting, respectively trying to accomplish.
I mean, if those VMs are running, it can be totally expected, that there is regular activity on the storage they are on.
So, do you see regular activity even without the VMs running, or?
PVE also checks the status of the configured storages every few seconds. So this produces also activity.
Reads/Access can also lead to writes, because of the access time (atime) being updated.
And so on...

The writes occur regardless of whether there is one, many or no VMs running. I suppose it is what it is then; if that is expected behavior out of PVE. Guess I thought that there'd be similar ZFS management behavior between TrueNas Scale and PVE (since they're both debian based) but, perhaps I assumed wrong