Recent content by Byron

You can make ceph automatically handle this, it nicely create partitions accordingly.

A common recommendation is to have 1 SSD per 4-6 HDDs since it can speedup writes dramatically. If I understand correctly, it is indeed so that if the SSD would fail, all HDDs it holds data for will need rebuilding. In CEPH docs, I've seen 4% recommended, I think the recommended minimum is...

It looks like the amount of OSDs per NVMe does not influence the results too much then? I'm looking to run similar drives at 1 OSD per NVMe to save CPU power (64C/128T for 20-24 drives).

We'd put 24 drives in a chassis with 128 CPU threads, giving a ratio of 5.33 CPU threads per NVMe. I've read before that 4 threads per NVMe is the recommended minimum, with 8+ seeming standard practice. It looks like we'll run in to a CPU bottleneck as far as I can find, however it would be not...

Does this depend on the type of drive or the CPU architecture or something else? It would be super helpful to see CPU load during benchmarking, I'm considering using a 64C/128T single socket for 24 U.2 drives, at 5.3 threads per drive, would it be able to keep up?

Thanks for your extended reply, greatly appreciated! When going over 50-60% we intend to start adding drives/servers. Sequential write of a single disk would be around ~200-250MB/s, our drives (HC550) have fairly low IOPS/TB so I would think the performance will suffer. I've read a case where...

Hi people! I'm planning a CEPH cluster which will go in production at some point but first will serve as a testing setup. We need 125TB usable storage initially, with a cap of about 2PB. The cluster will feed 10 intensive users initially, up to 100 later on. The loads are generally read heavy...

I'd like to clear my drive from old leftovers which remained when updating my install. There's several drives/partitions listed like this: Disk /dev/mapper/pve--OLD--C576DBC9-vm--104--disk--0: 200 GiB, 214748364800 bytes, 419430400 sectors Units: sectors of 1 * 512 = 512 bytes Sector size...

That's what I suspected, I also found it difficult to find the correct words to describe it. :D No idea. It appears it doesn't handle Numa nodes without memory properly. This thread seems to be abandoned/ignored though so I suggest starting a new thread. My problem with the normal settings...

memory: 102400 cores: 32 numa0: cpus=0-7;32-39,hostnodes=0,memory=51200,policy=bind numa2: cpus=8-15;40-47,hostnodes=2,memory=51200,policy=bind This fails due to what I explained earlier: You give your VM 32 cores, you can't select/pin the cores of the host, you can only choose from what...

I think you made a mistake here: numa0: cpus=0;1;2;3;4;5;6;7;32;33;34;35;36;37;38;39,hostnodes=0,memory=65536,policy=preferred The cpu's listed after CPU should be numbered sequentially with the core id from the VM, not the host. In other words, you should define that as: numa0...

Bump Problem still exists in latest 5.4-5

Is there something I can do so I don't need to set every VM manually? It makes automated deployment impossible, some kind of workaround would be convenient.

Another issue found with my configuration: When starting 4 VM's taking 30000MB from every numa (machine has 32GB of ram per numa node on an EPYC system), all seems to work fine. When I start the 4VM's with hugepages enabled it fails every time at some point. This snapshot is taken when the 3rd...

This is how it looks when starting 4 VM's with 20GB RAM each: root@ProxS01:~# numactl -H available: 4 nodes (0-3) node 0 cpus: 0 1 2 3 4 5 24 25 26 27 28 29 node 0 size: 32140 MB node 0 free: 11273 MB node 1 cpus: 6 7 8 9 10 11 30 31 32 33 34 35 node 1 size: 32232 MB node 1 free: 31516 MB node 2...