Caching mode for Proxmox VE and Ceph.

[Vasilisc]

With colleagues, we have been using Proxmox VE for a long time. We want to implement Hyper-Converged Infrastructure (HCI).
I read about caching modes in the official documentation
https://pve.proxmox.com/wiki/Performance_Tweaks

I would like to hear an authoritative opinion. What caching mode would you recommend?

Technical information.
pve-manager / 6.1-5 / 9bf06119 (running kernel: 5.3.13-1-pve)
ceph 14.2.6-pve1
CPU Intel (R) Xeon (R) Silver 4110 CPU @ 2.10GHz
RAM 32 Gb
HDD 4 x 4 Tb (TOSHIBA MG04ACA400E with Bluestore)

 

[Alwin]

I would like to hear an authoritative opinion. What caching mode would you recommend?

None. Why? See below.

HDD 4 x 4 Tb (TOSHIBA MG04ACA400E with Bluestore)

The default memory limit of an OSD is 4 GB. 4 x 4 = 16 GB.

Then the MON will also need a couple of GB. Don't forget the OS and possibly you may run backups, live migrations. The last but not least, you will run VM/CT on them too.

TL;TR; you do not have enough memory for extra caching.

 

[Vasilisc]

Thanks a lot. My experiments also confirmed "cache = none"