Slow VM on external CEPH Cluster

[fabilau]

Logfile is full of:

2021-06-11 14:37:05.515 7f25f228d700 1 mon.XX@2(peon).osd e584 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 71303168 full_alloc: 75497472 kv_alloc: 868220928
2021-06-11 14:37:10.515 7f25f228d700 1 mon.XX@2(peon).osd e584 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 71303168 full_alloc: 75497472 kv_alloc: 868220928
2021-06-11 14:37:15.514 7f25f228d700 1 mon.XX@2(peon).osd e584 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 71303168 full_alloc: 75497472 kv_alloc: 868220928
2021-06-11 14:37:20.515 7f25f228d700 1 mon.XX@2(peon).osd e584 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 71303168 full_alloc: 75497472 kv_alloc: 868220928
2021-06-11 14:37:25.515 7f25f228d700 1 mon.XX@2(peon).osd e584 _set_new_cache_sizes cache_size:1020054731 inc_alloc: 71303168 full_alloc: 75497472 kv_alloc: 868220928

And:
2021-06-11 14:33:58.253 7f25eb27f700 4 rocksdb: [db/db_impl_compaction_flush.cc:1397] [default] Manual compaction waiting for all other scheduled background compactions to finish
2021-06-11 14:33:58.253 7f25eb27f700 4 rocksdb: [db/db_impl_compaction_flush.cc:1403] [default] Manual compaction starting
2021-06-11 14:33:58.253 7f25eb27f700 4 rocksdb: [db/db_impl_compaction_flush.cc:1403] [default] Manual compaction starting
2021-06-11 14:33:58.253 7f25eb27f700 4 rocksdb: [db/db_impl_compaction_flush.cc:1403] [default] Manual compaction starting
2021-06-11 14:33:58.253 7f25eb27f700 4 rocksdb: [db/db_impl_compaction_flush.cc:1403] [default] Manual compaction starting
2021-06-11 14:33:58.253 7f25eb27f700 4 rocksdb: [db/db_impl_compaction_flush.cc:1403] [default] Manual compaction starting
2021-06-11 14:33:58.253 7f25eb27f700 4 rocksdb: [db/db_impl_compaction_flush.cc:1403] [default] Manual compaction starting
2021-06-11 14:33:58.253 7f25eb27f700 4 rocksdb: [db/db_impl_compaction_flush.cc:1403] [default] Manual compaction starting
2021-06-11 14:33:58.253 7f25eb27f700 4 rocksdb: [db/db_impl_compaction_flush.cc:1403] [default] Manual compaction starting
2021-06-11 14:33:58.253 7f25eb27f700 4 rocksdb: [db/db_impl_compaction_flush.cc:1403] [default] Manual compaction starting
2021-06-11 14:33:58.253 7f25eb27f700 4 rocksdb: [db/db_impl_compaction_flush.cc:1403] [default] Manual compaction starting
2021-06-11 14:33:58.253 7f25eb27f700 4 rocksdb: [db/db_impl_compaction_flush.cc:1403] [default] Manual compaction starting


Does that mean that cache is enabled?

 

[ph0x]

I really don't know. If I had an answer that would only mean I'm better at googling ...

 

[fabilau]

I really don't know. If I had an answer that would only mean I'm better at googling ...

Okay!
Thanks for your help anyway!

 

[fabilau]

Just an update:

Also tried to add:
[client]
rbd_cache = false

[global]
rbd_non_blocking_aio = false

to the config file.

VM still freezes.

 

[zaphyre]

The Windows VM does not hang, but the disk usage remains constant at 100%.

Hi and sorry for adding in here with a question and no answer. Did you finally found a solution to this issue? Or did you stay on iSCSI?
I am currently experiencing the exact same behavior on a somewhat similar setup. Windows disk usage nearly always at around 100% on an all Flash cluster.

Thanks abd best regards
zaph

 

[jvanderstelt]

I am having the exact same problem all my windows VM are slow and disk usage is 100%. Did anyone find a solution?

 

[zaphyre]

I am having the exact same problem all my windows VM are slow and disk usage is 100%. Did anyone find a solution?

Our IO was degraded due to two bad disks affecting the cluster. Swapped disks. Fine.