Slow TrueNas Core VM boot

[andrema2]

Hi

I have a TrueNas Core 13.0-U3.1 VM running under Proxmox 7.3.6.

The boot time for that VM is really high. It doesn't happen for other VMs on the same server. This is what I see: after the start of the VM it takes up to two or more minutes to be able to see anything at the console. The whole process of boot up did not occur during this time and the console fails until the boot process starts.

Once it starts the process runs normally. The VM is on a NVME drive

HW settings

Options Settings

Any idea how to make it boot faster ?

 

[Neobin]

My guess would be, that it is because of the 128 GB RAM which, in combination with the PCIe-passthrough, have all to be allocated on the VM-startup.

Do you already use 1 GB-hugepages for this VM?
If not, you can try if those help, by adding: hugepages: 1024 in a new line to your: /etc/pve/qemu-server/100.conf.
What I read a while ago, for this you also need NUMA enabled for this VM, which you usually want anyway with your two sockets.

 

[andrema2]

My guess would be, that it is because of the 128 GB RAM which, in combination with the PCIe-passthrough, have all to be allocated on the VM-startup.

Do you already use 1 GB-hugepages for this VM?
If not, you can try if those help, by adding: hugepages: 1024 in a new line to your: /etc/pve/qemu-server/100.conf.
What I read a while ago, for this you also need NUMA enabled for this VM, which you usually want anyway with your two sockets.

I didn't nor the NUMA was enabled. I enabled both. The boot time still the same. Anyway, it was a good tip the NUMA

 

[Neobin]

You could try with a good amount less RAM, e.g.: 16 GB, to see if my guess is correct at all.

 

[andrema2]

You could try with a good amount less RAM, e.g.: 16 GB, to see if my guess is correct at all.

I did a test with 32 Gb and it took 28s instead of the 3 min it normally takes.
On the other hand I have another VM with 256Gb of RAM that takes 5s to start.
So my guess is, is the memory plus the HBA passthrough.
I think this is the time it takes and I just have to get over it... I don't restart it that often.

 

[cwt]

HBA and RAID controllers usually need a lot more time (compared to normal BIOS times) to initialize. So I won‘t expect a fast boot like on systems without such cards.

 

[Dunuin]

HBA and RAID controllers usually need a lot more time (compared to normal BIOS times) to initialize. So I won‘t expect a fast boot like on systems without such cards.

Bigger problem should be the RAM allocation. To explain a bit more what Neobin already said...:
Normally a VM, without PCI passthrough, will get RAM dynamically allocated. A 128GB RAM VM that only needs 1GB RAM when booting will start fast, as only 1GB will have to be allocated. Because of PCI passthrough this RAM allocation is fixed and the VM will use the full 128GB RAM all the time right from the start, if that VM actually uses it or not, as a PCI device might use DMA and that requires to be able to access every bit of RAM.