Hello.
So we have 3 nodes, each with 2 Samsung sm631a 480GB in each.
Ceph is connected via a 10Gbit switch, and the only thing i have tweaked is:
osd_memory_target = 1073741824
So each OSD only uses 1GB, instead of 4GB.
The pool uses size=3, min size=2.
Now what is so frustrating before the reinstall, this is the results i got in the VM:
![photo_2019-06-17_15-03-49.jpg photo_2019-06-17_15-03-49.jpg](https://forum.proxmox.com/data/attachments/10/10260-f481cb755c1a6b1aef52d2da2ff367ff.jpg)
Very good results for the lower range, and very responsive VM's.
After the install: (its like it don't even want to cache anymore)
![Selection_019.png Selection_019.png](https://forum.proxmox.com/data/attachments/10/10261-e6ddc4af2c0c13e0d53224681d823362.jpg)
Now i don't remember if used the non subscriber repo for the testing or not.
But with the new system i have bought licenses, and updated using the subscriber repos.
I dont have screenshot of the KRDB enabled on the old system, but on the new system:
Still low response on the smaller files.
I have enabled jumbo frames on the Switch and servers.
I have checked with iperf3, the network works great.
rados -p ceph-pool bench 10 write --no-cleanup
rados -p ceph-pool bench 10 seq
1 SSD pr. server.
Write: 350-370MB/sek
Read:1380MB/sek
2 SSD pr.server
Write 550MB/SEK
Read: 1700MB/SEK
So we have 3 nodes, each with 2 Samsung sm631a 480GB in each.
Ceph is connected via a 10Gbit switch, and the only thing i have tweaked is:
osd_memory_target = 1073741824
So each OSD only uses 1GB, instead of 4GB.
The pool uses size=3, min size=2.
Now what is so frustrating before the reinstall, this is the results i got in the VM:
![photo_2019-06-17_15-03-49.jpg photo_2019-06-17_15-03-49.jpg](https://forum.proxmox.com/data/attachments/10/10260-f481cb755c1a6b1aef52d2da2ff367ff.jpg)
Very good results for the lower range, and very responsive VM's.
After the install: (its like it don't even want to cache anymore)
![Selection_019.png Selection_019.png](https://forum.proxmox.com/data/attachments/10/10261-e6ddc4af2c0c13e0d53224681d823362.jpg)
Now i don't remember if used the non subscriber repo for the testing or not.
But with the new system i have bought licenses, and updated using the subscriber repos.
I dont have screenshot of the KRDB enabled on the old system, but on the new system:
![Selection_018.png Selection_018.png](https://forum.proxmox.com/data/attachments/10/10262-b8101a0c206eda719e622046af269d01.jpg)
Still low response on the smaller files.
I have enabled jumbo frames on the Switch and servers.
I have checked with iperf3, the network works great.
rados -p ceph-pool bench 10 write --no-cleanup
rados -p ceph-pool bench 10 seq
1 SSD pr. server.
Write: 350-370MB/sek
Read:1380MB/sek
2 SSD pr.server
Write 550MB/SEK
Read: 1700MB/SEK