Strange boot behaviour with q35 VM

[clusty]

I will start by saying I am a bit of a Proxmox noob
Once installed i created a VM using the default params to virtualize my truenas instance. After realizing that passing disks is both tedious and not recommended, I redid the machine as q35 with EFI boot and passed the whole PCI-E Sata controller to the VM. The VM has 190GB of RAM allocated (ballooning from a min 32GB )

In this new configuration, the new machine seems to go though a strange "limbo phase" every time it boots up: the host starts allocating the whole 190GB of ram in 1-2 GB/s increments (this might take a few minutes). Once it allocates everything, the VM starts booting but the host memory usage never drops from the max 190GB, even though the VM reports using only 8-10GB of RAM. Almost as if the balooning never worked and it allocates the max on the host side upfront.

How could this be explained? Is this the normal behaviour with q35 machines ?

 

[Daniel_Dog]

It might be a couple of things but here are the common ones:

1. Qemu-guest-agent is not installed or running on the VM. (This is required to have ballooning work.)
2. The host has enough RAM free. The host will only claim RAM back if the RAM usage is above 80% on the host itself.

 

[clusty]

It might be a couple of things but here are the common ones:

1. Qemu-guest-agent is not installed or running on the VM. (This is required to have ballooning work.)
2. The host has enough RAM free. The host will only claim RAM back if the RAM usage is above 80% on the host itself.

1. Confirmed that the agent is running ( can see all net network interfaces in the proxmox summary of the VM). In fact in the first version, I did not even enable the Agent when i created the machine and the host was claiming ram when other VMs needed the ram.
2. On my first batch of VMs the behaviour was different: ram was allocated on the host only as the VM was asking for it. There seems to be a RAM usage spike now as the VM boots up (before or during the EFI boost screen ), maybe this is making the host allocate that crazy amount of mem.

Is there any connection with the PCI-E device I passed ? ( its a Broadcom SAS-2308 motherbopard device )

 

[leesteken]

1. Confirmed that the agent is running ( can see all net network interfaces in the proxmox summary of the VM). In fact in the first version, I did not even enable the Agent when i created the machine and the host was claiming ram when other VMs needed the ram.

I think the ballooning device (and driver) is important here not the QEMU Guest Agent.

2. On my first batch of VMs the behaviour was different: ram was allocated on the host only as the VM was asking for it. There seems to be a RAM usage spike now as the VM boots up (before or during the EFI boost screen ), maybe this is making the host allocate that crazy amount of mem.

That might only have looked that way. All memory is allocated (but not necessarily loaded into RAM) at the start of the VM. There are always discussions going on on this forum about what the right way of looking at the memory usage is.

Is there any connection with the PCI-E device I passed ? ( its a Broadcom SAS-2308 motherbopard device )

Yes: when using PCI(e) passthrough all memory must be pinned into actual host RAM because of DMA. Ballooning is not possible on a VM that uses PCI(e) passthrough.

 

[clusty]

Yes: when using PCI(e) passthrough all memory must be pinned into actual host RAM because of DMA. Ballooning is not possible on a VM that uses PCI(e) passthrough.

God, this makes so much sense
This pinning process is so slow that I thought for a few hours that the VM/proxmox is broken (rebooted proxmox 3 times ). Would be nice if this passthrough pinning limitation would be documented in bold.