Fsync Questions

[adamb]

Hey all I have a cluster with 2 identical x3550 M3's.

Yestarday we had an issue with a VM running very hard. For the life of me I couldn't figure it out.

I decided to do a pveperf on each node and discovered that the fsync's on one node was very very low.

Node1: 100-500 fsync/second
Node2: 2400-2800 fsync/second

I updated both nodes and rebooted. Once it came back up Node1 was back on par with node2 for fsync's. The VM in turn was running like it should.

Any ideas on what the cause could be? Possibly a failing drive? Anything in proxmox which could have caused this?

Another thing that threw me off was this. On the VM IO wait was very high 50%+. On the host though IO wait was always 0. I would think if a VM had high IO wait it should be reflected into the host?

 

[udo]

Hey all I have a cluster with 2 identical x3550 M3's.

Yestarday we had an issue with a VM running very hard. For the life of me I couldn't figure it out.

I decided to do a pveperf on each node and discovered that the fsync's on one node was very very low.

Node1: 100-500 fsync/second
Node2: 2400-2800 fsync/second

I updated both nodes and rebooted. Once it came back up Node1 was back on par with node2 for fsync's. The VM in turn was running like it should.

Any ideas on what the cause could be? Possibly a failing drive? Anything in proxmox which could have caused this?

Another thing that threw me off was this. On the VM IO wait was very high 50%+. On the host though IO wait was always 0. I would think if a VM had high IO wait it should be reflected into the host?

Hi,
I guess that the IOs from a VM (or more VMs) drop down the fsyncs during pveperf. For comparision it's only usefull on a calm host.

Udo

 

[adamb]

When the issue was happening I moved all the VM's to the node not having issues. Then I ran pveperf which was also resulting in low numbers. It wasn't until I updated and rebooted that the numbers are back.

 

[udo]

When the issue was happening I moved all the VM's to the node not having issues. Then I ran pveperf which was also resulting in low numbers. It wasn't until I updated and rebooted that the numbers are back.

Hmm,
any info at the events from the raid-controller?

Udo

 

[adamb]

Nothing at all that jumped out, that was my first suspect.