[SOLVED] Slow LVM (over dm-crypt, over iSCSI) metadata operations

[danielb]

Hi.

I'm running a 3 nodes cluster, running latest PVE 5.2. As storage, I use iSCSI, but, as I don't fully trust the storage operators, I've added a dm-crypt layer. And on top of this dm-crypt, I've created an LVM VG which is shared on all node, and used as LVM storage. So, for every node, it's like

iscsi <-> dm-crypt <-> LVM VG

(on every boot of a node I simply manually unlock the storage using dm-crypt before LVM is seen as available in PVE)

Everything works nicely, I can live migrate etc...

Except that, sometime, LVM metadata operations are painfuly slow on some of the nodes (not always the same node). Eg, if I move a disk from one storage to another, it can takes several minutes to create the LV on the dest storage. Once this is done though, the transfert speed is OK. Same for live migration, starting the VM on the dest node can take several minutes (I guess it's hanging on activating the LV). And once migration is over, performance in the VM is normal. How could I debug this ? Anyone else has this kind of problem ?

 

[danielb]

BTW, without dm-crypt, with just LVM on top of iSCSI, I have the same behaviour, so dm-crypt is not in fault here

 

[guletz]

Hello Daniel,

Some thoughts; )

- you must have the same block size for all storage path (iscsi, dmcrypt, lvm) - you can check with gdisk
- you must check if you have hardware cpu AES support activated on each PMX node
- also check if your partitions/lvm are proper align using a size that is integer multiple of the same block size


Good luck.

 

[danielb]

Thanks for your tips. Looks my issue is on the network. After disabling jumbo frames the random latency seems to disapear. I'll ask my hoster to have a look at this.

 

[guletz]

Ok. Good to know. Be kindly and rename your post as (resolved) after the initial post name!