FreeBSD Guest - wrong ram usage

[onepamopa]

Hello,

Looks like PVE doesn't detect ram usage in FreeBSD properly, example - VM has 700 MB usage, PVE dashboard shows 1924.
qemu-guest-agent is installed, however, looks like PVE doesn't use it to get the RAM usage:

dommemstat is the command currently implemented in the qemu-guest-agent port for FreeBSD

# qm guest cmd 104 dommemstat
command: value 'dommemstat' does not have a value in the enumeration 'fsfreeze-freeze, fsfreeze-status, fsfreeze-thaw, fstrim, get-fsinfo, get-host-name, get-memory-block-info, get-memory-blocks, get-osinfo, get-time, get-timezone, get-users, get-vcpus, info, network-get-interfaces, ping, shutdown, suspend-disk, suspend-hybrid, suspend-ram'

get-memory-block-info and get-memory-blocks both return:
"error" : {
"class" : "CommandNotFound",
"desc" : "The command guest-get-memory-blocks has been disabled for this instance"
}
"error" : {
"class" : "CommandNotFound",
"desc" : "The command guest-get-memory-block-info has been disabled for this instance"
}

I've also opened an issue on this in https://github.com/aborche/qemu-guest-agent

 

[aaron]

PVE is using the ballooning device to communicate with the VM regarding the memory usage. Make sure that the ballooning device is enabled. This is done in the Memory settings if you enable the "Advanced" checkbox.

But, FreeBSD does not provide detailed memory usage info IIRC. Therefore PVE will show the memory usage of the whole virtualization process of the VM.

You can verify this by selecting your VM and going to the "Monitor" Panel. Type in info balloon. If the VM provides detailed information, it will look something like this:

# info balloon
balloon: actual=2048 max_mem=2048 total_mem=1986 free_mem=197 mem_swapped_in=6311936 mem_swapped_out=23953408 major_page_faults=3095 minor_page_faults=588652947 last_update=1618556301

 

[onepamopa]

Looks like FreeBSD doesn't have an "info balloon" command.

 

[juliokele]

Looks like FreeBSD doesn't have an "info balloon" command.

pfsense 2.6 / FreeBSD 12.2:

# info balloon
balloon: actual=3072 max_mem=3072

# qom-get /machine/peripheral/balloon0 guest-stats
{
    "stats": {
        "stat-htlb-pgalloc": 18446744073709551615,
        "stat-swap-out": 18446744073709551615,
        "stat-available-memory": 18446744073709551615,
        "stat-htlb-pgfail": 18446744073709551615,
        "stat-free-memory": 18446744073709551615,
        "stat-minor-faults": 18446744073709551615,
        "stat-major-faults": 18446744073709551615,
        "stat-total-memory": 18446744073709551615,
        "stat-swap-in": 18446744073709551615,
        "stat-disk-caches": 18446744073709551615
    },
    "last-update": 0
}

proxmox:


pfsense Diagnostics / System Activity:

 

[aaron]

Looks like FreeBSD doesn't have an "info balloon" command.

Run that on the PVE GUI, select the VM and then the "Monitor" panel. This will show you what PVE gets reported back from the VM and as @julio12345 showed in the previous answer, it will be something like this:

# info balloon
balloon: actual=3072 max_mem=3072

No detailed insights which is why PVE then is falling back to measure the memory usage of the overall virtualization of that VM.

 

[onepamopa]

pfsense 2.6 / FreeBSD 12.2:

# info balloon
balloon: actual=3072 max_mem=3072

# qom-get /machine/peripheral/balloon0 guest-stats
{
    "stats": {
        "stat-htlb-pgalloc": 18446744073709551615,
        "stat-swap-out": 18446744073709551615,
        "stat-available-memory": 18446744073709551615,
        "stat-htlb-pgfail": 18446744073709551615,
        "stat-free-memory": 18446744073709551615,
        "stat-minor-faults": 18446744073709551615,
        "stat-major-faults": 18446744073709551615,
        "stat-total-memory": 18446744073709551615,
        "stat-swap-in": 18446744073709551615,
        "stat-disk-caches": 18446744073709551615
    },
    "last-update": 0
}

proxmox:
View attachment 25496

pfsense Diagnostics / System Activity:
View attachment 25497

Did you install something additional or not?
Because on FreeBSD 13.0 with qemu-guest-agent (the one from the first post) installed & enabled, I get:
# info balloon
info: No menu item 'balloon' in node '(dir)Top'

FreeBSD mon 13.0-RELEASE FreeBSD 13.0-RELEASE #0 releng/13.0-n244733-ea31abc261f: Fri Apr 9 04:24:09 UTC 2021 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64

 

[juliokele]

Did you install something additional or not?
Because on FreeBSD 13.0 with qemu-guest-agent (the one from the first post) installed & enabled, I get:
# info balloon
info: No menu item 'balloon' in node '(dir)Top'

FreeBSD mon 13.0-RELEASE FreeBSD 13.0-RELEASE #0 releng/13.0-n244733-ea31abc261f: Fri Apr 9 04:24:09 UTC 2021 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64

"Did you install something additional or not?"
no, i'm not...

 

[onepamopa]

"Did you install something additional or not?"
no, i'm not...

Then... I don't know why I don't have this on the "stock" 13.0 ...

 

[DynFi User]

We have the same issue with memory report and FreeBSD.
We are using 13.1-STABLE and the report is showing:

Type 'help' for help.
# info balloon
balloon: actual=16384 max_mem=16384

When we do a top system shows Mem: 5405M Active, 1266M Inact, 803M Wired, 541M Buf, 8402M Free

Where the PVE is displaying:



And what is more annoying, the memory reported in the system will drop when in fact it is using way more…

 

[Dunuin]

Really a shame that the FreeBSDs qemu-guest-agent implementation is that bad. Fsfreeze for example isn't implemented at all, so data integrity isn't ensured when doing snapshot-mode backups.
This prevents here that I can regularily backup my TrueNAS Core VM, as stop mode backups aren't an option, as then all the other guests, relying on SMB/NFS shares, would stop working too.

 

[DynFi User]

Really a shame that the FreeBSDs qemu-guest-agent implementation is that bad. Fsfreeze for example isn't implemented at all, so data integrity isn't ensured when doing snapshot-mode backups.
This prevents here that I can regularily backup my TrueNAS Core VM, as stop mode backups aren't an option, as then all the other guests, relying on SMB/NFS shares, would stop working too.

Well you can upgrade your FreeNAS to TrueNAS-Scale ; migration is included in the system and you'll end-up with a Linux based system where your Qemu-guest-agent is well supported.

 

[Dunuin]

Well you can upgrade your FreeNAS to TrueNAS-Scale ; migration is included in the system and you'll end-up with a Linux based system where your Qemu-guest-agent is well supported.

Yes, but updates in TrueNAS Core always break functionalities here and that resulted in situations like once, where I wasn't able to do backups for months or another time where I had to wipe my entire pool. If a well matured TrueNAS Core causes so much problems, then I don't really want to switch to TrueNAS Scale which only recently left beta phase and is now more in "early access" where core features are still missing and being added later, potentially causing a lot of new bugs. So I would like to switch to Scale, but they probably needs some more years until it is matured enoug.

For example the new Scale 22.12.1 update broke the replication again. You can't edit existing replication jobs any longer and you can't replicate an encrypted dataset to an unencrypted pool. Such stuff really shouldn't happen if you rely on replication for redundancy/backups. And because of the slow release cycle you will have to live with these known issues for at least a month.