Extensive swapping

[holgihero]

I found an odd situation where ProxMox swapps allmost all VMs out to disk while copying some files. And i can not find an explanation.

In detail: I copy some files(about 8G) from within a VM (Debian, ISCSI-Disk via VIRTIO ) via Samba-Share to an external machine. So it is only reading from disk and some transmitting I would assume.
Transmission is very slow on the one hand (3MB/s) and I/O-delay goes up to 70%.
On the other side allmost all memory of running VMs is being stored to the swap space. This leads to a very poor performance on all VMs afterwards.
I dont understand why this happens while I am only reading some files...

I try to prevent that with swappyness = 0 but no effect.
How would Proxmox behave without swapspace at all? I have no testing machine at the moment, only production. Has anybody tested that?

Regards Holgi

 

[udo]

I found an odd situation where ProxMox swapps allmost all VMs out to disk while copying some files. And i can not find an explanation.

In detail: I copy some files(about 8G) from within a VM (Debian, ISCSI-Disk via VIRTIO ) via Samba-Share to an external machine. So it is only reading from disk and some transmitting I would assume.
Transmission is very slow on the one hand (3MB/s) and I/O-delay goes up to 70%.
On the other side allmost all memory of running VMs is being stored to the swap space. This leads to a very poor performance on all VMs afterwards.
I dont understand why this happens while I am only reading some files...

I try to prevent that with swappyness = 0 but no effect.
How would Proxmox behave without swapspace at all? I have no testing machine at the moment, only production. Has anybody tested that?

Regards Holgi

Hi Holgi,
you mean the swap space of the pve-host, not inside the vm?!
Do you have overcommit your memory, or leave not enough free for the os?

Without swap space is AFAIK a little bit dangerous - if the kernel urgendly needs space they will begin to kill processes... not the best thing for an vm (but good agains swapping ).

Perhaps you can use less memory for the VMs, or buy more memory or use a faster io-system?

Udo

 

[holgihero]

Hi Udo,

yes i speak about memory of the host.
all VMs together leave enough memory for the host system and some gigs of buffer. So there would be no need to swap except linux allways wants to...

For example:
I have on real quick machine with fast raid, 12GB memory and one single VM running.
This VM contains a Win2008 Server and he is running on 6GB memory happyly. So debian should be happy with the remaining 6GB?
No.
After some days it fills about another 4GB of swap space on the disk. And that leads definitely to noticeable effects in regard of response time from some windows services beeing unluckyly banned to disk.
And I bet this would happen again when I would double that memory.

So I play around with the idea to switch off that damned swap space at all...

Regards Holgi

 

[udo]

Hi Udo,

yes i speak about memory of the host.
all VMs together leave enough memory for the host system and some gigs of buffer. So there would be no need to swap except linux allways wants to...

For example:
I have on real quick machine with fast raid, 12GB memory and one single VM running.
This VM contains a Win2008 Server and he is running on 6GB memory happyly. So debian should be happy with the remaining 6GB?

Yes - i agree 6 GB is more than enough

No.
After some days it fills about another 4GB of swap space on the disk. And that leads definitely to noticeable effects in regard of response time from some windows services beeing unluckyly banned to disk.
...

Hmm, strange.
I've look on the pve-server here and the swapspace will only use a little bit (and i haven't notice any performance issues):

Server1:
Mem:  16436584k total, 13893448k used,  2543136k free,   130704k buffers
Swap:  4194300k total,    75240k used,  4119060k free,   990756k cached

Server2:
Mem:  16436584k total, 16329296k used,   107288k free,   244948k buffers
Swap:  4194300k total,    52168k used,  4142132k free,  6597192k cached

Server3:
Mem:  65975692k total, 49972228k used, 16003464k free,   154564k buffers
Swap: 16777212k total,    32756k used, 16744456k free, 26963132k cached

Server4:
Mem:  65975692k total, 56833452k used,  9142240k free,   156424k buffers
Swap: 16777212k total,    44244k used, 16732968k free, 26605820k cached

all Servers above runs with an 2.6.35er kernel. But also on an old 2.6.18 (pve 1.6) looks similiar (only 2800k swap space used). The uptime varied between 140 and 10 days.

Do you have running "special" software on the node, which use sometimes a lot of memory?

Udo

 

[holgihero]

Hi Udo,
on this specific server: Plain Debian w. Proxmox and sshd running for remote access.

I checked the server today and it seems that after setting swappyness to zero it behaves like it should w. very little swapping.

It might have been the case that this extensive buffering occured while running a second VM and copying from one VM to the other. I can not check that now because the server is productive.

But this would be similar behaviour as found on my first mentioned (less powerfull) machine which swapps like crazy when accessing larger amounts of files on disk(in a VM).
And that really is crazy, it fills up every single byte of disk swap space with precious VM memory only to buffer disk reads...
I will try to dive deeper into that when i find some time (so maybe never ).