[TTM] Buffer eviction failed

[tanolis]

I am using following Proxmox version.

# pveversion --verboseproxmox-ve: 8.2.0 (running kernel: 6.8.12-1-pve)
pve-manager: 8.2.4 (running version: 8.2.4)
proxmox-kernel-helper: 8.1.0pve-kernel-5.15: 7.4-13
proxmox-kernel-6.8: 6.8.12-1
proxmox-kernel-6.8.12-1-pve-signed: 6.8.12-1

In my System Log, I keep getting these errors;
Aug 12 19:07:27 pvetower kernel: [TTM] Buffer eviction failed
Aug 12 19:07:27 pvetower kernel: qxl 0000:00:1e.0: object_init failed for (4096, 0x00000001)
Aug 12 19:07:27 pvetower kernel: [drm:qxl_alloc_bo_reserved [qxl]] *ERROR* failed to allocate VRAM BO

For debugging and resolution, Can someone point me to the right direction?

 

[THX1138]

I get exactly the same error (except for the precise address, which I expect will be machine specific). In my case, it's not actually ProxMox, but rather a qemu/kvm virtual machine (on Linux). ProxMox uses the kvm hypervisor, suggesting that it's a kvm problem. QXL is the default VM video driver for qemu/kvm, and the error includes reference to 'VRAM", so I guess it's related to that. I'm also running the 6.8 series kernel (6.8.0-40).

This issue report from RedHat seems to support the theory that the problem is in kvm or one of its drivers:
https://access.redhat.com/solutions/7004385

RedHat says the status is Solution in Progress.

 

[jebbam]

I have seen a similar error occassionally for years... My setup and how I see it:

The KVMs are running X with the qxl driver: "SPICE (qxl, memory=128)" is selected under the "Hardware" tab. I'll also note that the max "memory=" I can get it to boot with is 128. Anything larger than that and it doesn't even get to the Proxmox BIOS when powered on. So I always run them with "memory=128".

I heavily use many SPICE KVMs, all day long.

It almost always happens when I am actively using the SPICE exported display on my workstation. In other words, I don't come back to see a crash, it seems to happen more when I'm switching back and forth between windows or just created a new Firefox window or something like that. It isn't happening because of a sleep or suspend or anything like that (which I've seen mentioned elsewhere).

I'll have KVMs run for days, weeks, months without a problem, then it crashes X. Note, the KVM itself keeps running, and I can log into it, run `sync` etc ok. But doing a `poweroff` will never shut down the KVM. I have to do a hard "Stop" of the KVM from Proxmox web gui.

Example crash:

2024-08-30T14:49:08.981384-06:00 muh-kvm kernel: [191594.195388] [TTM] Buffer eviction failed
2024-08-30T14:49:08.981415-06:00 muh-kvm kernel: [191594.195435] qxl 0000:00:02.0: object_init failed for (262144, 0x00000001)
2024-08-30T14:49:08.981417-06:00 muh-kvm kernel: [191594.195440] [drm:qxl_gem_object_create [qxl]] *ERROR* Failed to allocate GEM object (260868, 1, 4096, -12)
2024-08-30T14:49:08.981432-06:00 muh-kvm kernel: [191594.195465] [drm:qxl_alloc_ioctl [qxl]] *ERROR* qxl_alloc_ioctl: failed to create gem ret=-12

I'll also note, I usually enable allowing multiple SPICE connections, but I think the error happens whether I do that or not. I don't think it is related, but I mention it, just in case. I edit `/usr/share/perl5/PVE/Tools.pm` and add this line so I can have SPICE connections to the same KVM from multiple workstations (which is awesome, btw):

local $ENV{SPICE_DEBUG_ALLOW_MC} = "1";

I run Debian on the KVMs. It has happened with Debian bullseye, bookworm, trixie, and sid...

I would love to see a solution to this, it has bugged me for years!

 

[Valombre]

Hi,
same problem with some of my VMs ubuntu mate 24.04 LTS, and with graphical crash, i have to reconnect.
It happens a lot with firefox.
Here some logs :

2024-09-02T15:07:55.319113+02:00 vmtest kernel: [TTM] Buffer eviction failed
2024-09-02T15:07:55.319130+02:00 vmtest kernel: qxl 0000:00:02.0: object_init failed for (258048, 0x00000001)
2024-09-02T15:07:55.319131+02:00 vmtest kernel: [drm:qxl_gem_object_create [qxl]] *ERROR* Failed to allocate GEM object (256020, 1, 4096, -12)
2024-09-02T15:07:55.319132+02:00 vmtest kernel: [drm:qxl_alloc_ioctl [qxl]] *ERROR* qxl_alloc_ioctl: failed to create gem ret=-12
2024-09-02T15:08:00.179099+02:00 vmtest kernel: Lockdown: Xorg: raw io port access is restricted; see man kernel_lockdown.7
2024-09-02T15:08:00.941091+02:00 vmtest kernel: Lockdown: systemd-logind: hibernation is restricted; see man kernel_lockdown.7
2024-09-02T15:08:00.951091+02:00 vmtest kernel: Lockdown: systemd-logind: hibernation is restricted; see man kernel_lockdown.7
2024-09-02T15:08:01.986153+02:00 vmtest kernel: workqueue: page_reporting_process hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND

Regards

 

[jebbam]

It happens a lot with firefox.

Yes, I see it most often with Firefox too, especially when creating a new window or switching windows. It also seems to happen more when I'm "going fast", bouncing between windows, etc. I just had it happen with Thunderbird when creating a new task (which creates a new window).

In the past, I saw it more and I was blaming urxvt, which was my main terminal back in the day. I switched to using lxterminal as my main terminal, and it seems to happen less with that, but that's just a guess.

 

[Valombre]

Hi,
after some VM OS/kernel updates still problems, when i'm away trying to reconnect via SPICE a black screen, i tried a ssh connection (as i installed a ssh server) and i can connect and retrieve some logs still same errors :

2024-09-12T02:31:48.917494+02:00 vmtest kernel: [TTM] Buffer eviction failed
2024-09-12T02:31:48.917515+02:00 vmtestkernel: qxl 0000:00:02.0: object_init failed for (262144, 0x00000001)
2024-09-12T02:31:48.917517+02:00 vmtest kernel: [drm:qxl_gem_object_create [qxl]] *ERROR* Failed to allocate GEM object (262100, 1, 4096, -12)
2024-09-12T02:31:48.917518+02:00 vmtest kernel: [drm:qxl_alloc_ioctl [qxl]] *ERROR* qxl_alloc_ioctl: failed to create gem ret=-12

Via syslog more logs with context at the time of the gui/crash

2024-09-12T02:30:35.175376+02:00 vmtest qemu-ga: info: guest-ping called
2024-09-12T02:30:36.076303+02:00 vmtest qemu-ga: info: guest-fsfreeze called
2024-09-12T02:31:48.917494+02:00 vmtest kernel: [TTM] Buffer eviction failed
2024-09-12T02:31:48.917515+02:00 vmtest kernel: qxl 0000:00:02.0: object_init failed for (262144, 0x00000001)
2024-09-12T02:31:48.917517+02:00 vmtest kernel: [drm:qxl_gem_object_create [qxl]] *ERROR* Failed to allocate GEM object (262100, 1, 4096, -12)
2024-09-12T02:31:48.917518+02:00 vmtest kernel: [drm:qxl_alloc_ioctl [qxl]] *ERROR* qxl_alloc_ioctl: failed to create gem ret=-12
2024-09-12T02:31:49.063912+02:00 vmtest systemd[1]: Starting apport-autoreport.service - Process error reports when automatic reporting is enabled...
2024-09-12T02:31:49.067637+02:00 vmtest systemd[1]: Started whoopsie.service - crash report submission.
2024-09-12T02:31:49.079532+02:00 vmtest whoopsie[136303]: [02:31:49] Using lock path: /var/lock/whoopsie/lock
2024-09-12T02:31:49.082334+02:00 vmtest systemd[1]: whoopsie.service: Deactivated successfully.
2024-09-12T02:31:49.178564+02:00 vmtest whoopsie-upload-all[136302]: INFO:root:/var/crash/_usr_lib_mate-indicator-applet_mate-indicator-applet-complete.1000.crash already marked for upload, skipping
2024-09-12T02:31:49.178637+02:00 vmtest whoopsie-upload-all[136302]: INFO:root:Waiting for whoopsie to upload reports (timeout: 20 s)
2024-09-12T02:31:49.178659+02:00 vmtest whoopsie-upload-all[136302]: INFO:root:All reports uploaded successfully
2024-09-12T02:31:49.189608+02:00 vmtest systemd[1]: Started whoopsie.service - crash report submission.
2024-09-12T02:31:49.195291+02:00 vmtest systemd[1]: apport-autoreport.service: Deactivated successfully.
2024-09-12T02:31:49.195550+02:00 vmtest systemd[1]: Finished apport-autoreport.service - Process error reports when automatic reporting is enabled.
2024-09-12T02:31:49.197308+02:00 vmtest whoopsie[136313]: [02:31:49] Using lock path: /var/lock/whoopsie/lock
2024-09-12T02:31:49.199990+02:00 vmtest systemd[1]: whoopsie.service: Deactivated successfully.
2024-09-12T02:31:52.162855+02:00 vmtest systemd[1]: Starting apport-autoreport.service - Process error reports when automatic reporting is enabled...
2024-09-12T02:31:52.163908+02:00 vmtest systemd[1]: Started whoopsie.service - crash report submission.
2024-09-12T02:31:52.178609+02:00 vmtest whoopsie[136329]: [02:31:52] Using lock path: /var/lock/whoopsie/lock
2024-09-12T02:31:52.181362+02:00 vmtest systemd[1]: whoopsie.service: Deactivated successfully.
2024-09-12T02:31:52.293628+02:00 vmtest systemd[1]: Started whoopsie.service - crash report submission.
2024-09-12T02:31:52.299001+02:00 vmtest whoopsie[136333]: [02:31:52] Using lock path: /var/lock/whoopsie/lock
2024-09-12T02:31:52.300661+02:00 vmtest systemd[1]: whoopsie.service: Deactivated successfully.
2024-09-12T02:31:52.313899+02:00 vmtest whoopsie-upload-all[136328]: INFO:root:Collecting info for /var/crash/_usr_lib_xorg_Xorg.0.crash...
2024-09-12T02:31:52.962355+02:00 vmtest systemd[1]: Started whoopsie.service - crash report submission.
2024-09-12T02:31:52.976486+02:00 vmtest whoopsie[136365]: [02:31:52] Using lock path: /var/lock/whoopsie/lock
2024-09-12T02:31:52.978817+02:00 vmtest systemd[1]: whoopsie.service: Deactivated successfully.
2024-09-12T02:31:52.991867+02:00 vmtest systemd[934]: xdg-desktop-portal-gtk.service: Main process exited, code=exited, status=1/FAILURE
2024-09-12T02:31:52.993587+02:00 vmtest systemd[934]: xdg-desktop-portal-gtk.service: Failed with result 'exit-code'.
2024-09-12T02:31:53.000430+02:00 vmtest at-spi-bus-launcher[2080]: X connection to :0 broken (explicit kill or server shutdown).
2024-09-12T02:31:53.000804+02:00 vmtest systemd-udevd[306]: /run/systemd/network/10-netplan-NM-87f0b43a-8043-49e9-9636-f3ac10510783.link: No valid settings found in the [Match] section, ignoring file. To match all interfaces, add OriginalName=* in the [Match] section.
2024-09-12T02:31:53.000859+02:00 vmtest systemd[934]: xdg-desktop-portal-gtk.service: Consumed 18.175s CPU time.
2024-09-12T02:31:53.011284+02:00 vmtest dbus-daemon[1328]: [session uid=1000 pid=1328] Activating via systemd: service name='org.freedesktop.impl.portal.desktop.gtk' unit='xdg-desktop-portal-gtk.service' requested by ':1.54' (uid=1000 pid=2366 comm="/usr/libexec/xdg-desktop-portal" label="unconfined")
2024-09-12T02:31:53.014338+02:00 vmtest spice-vdagentd: closed vdagent virtio channel

I tried to restart the gui =>

# systemctl restart display-manager

Was quite long 2 minutes to achieve the command and the logs still produce at the end the same TTM and drm:qxl_gem_object_create errors.

As i upraged from a 22.04 LTS and i didn't have those problems i m going to try to use an older kernel, i'm tired to reboot this VM ...

 

[Valombre]

Hi,
i tried several things , testing i my VM last 6.11 kernel and using another guest motherboard q35, same problem in any case.
As i have a snapshot before upgrade in my ubuntu mate 24.04 i rollback to 22.04, since no more problem i 'll confirm next week.

Just for confirmation purpose i tried several fresh installs of VMs (default parameters with 4CPU/4Go ram) via iso ubuntu mate 24.04.01 64bits and ubuntu classic 24.04.01 64bits => same drm:qxl errors in logs in live session while installing it's really easy to reproduce.

Of course i upgraded before several tests my proxmox (no subscription) host with last updates in 8.2.4 (upgrade from a 7.4 a month ago)

May be i'll post tomorrow a different thread concerning crash of fresh ubuntu 24.04 install, i'm sure we are not isolated people using ubuntu 24.04 ,or kernel like distribution.

 

[jebbam]

This is killing me. It's happening a bunch of times today.

$5,000.00 USD bounty reward from me for whoever delivers a working fix for this.

Payable via PayPal or credit card, if you accept credit cards.

Sincerely,

-Jeff Moe
Loveland, Colorado, USA

 

[Valombre]

Hi,
@jebbam, i would like to be a skilled proxmox or kernel dev to solve this for the bounty .. but i m just a Linux Admin
I hope a nice result soon.
Regards

 

[ramius1984]

Hi,

@jebbam I cannot solve your problem but at least I can reference a workaround:
https://bbs.archlinux.org/viewtopic.php?id=215839

QXL should be the culprit and the workaround is to switch to VirtIO for the display settings.



VDI Client still works fine. Also feels faster.

Other reference and bonus:
https://www.reddit.com/r/Proxmox/comments/1b01sbx/spice_really_sluggish_with_ubuntu_2204/
https://www.reddit.com/r/Proxmox/comments/1auvdlg/qxl_vs_virtio_gpu_vs_virgl_gpu_trivial_benchmark/

Best regards,
Stefan

 

[jebbam]

QXL should be the culprit and the workaround is to switch to VirtIO for the display settings.

Thanks! I'll give this a test run on some VMs for the next week or so.

-Jeff

 

[jebbam]

@ramius1984

The VirtIO-GPU workaround hasn't crashed, which is a big plus.

One big drawback is it is very blocky when playing videos. I tried the various SPICE Enhancements Video Streaming options, but it didn't help. I do this on a few VMs, so it won't work for those.

One minor drawback/annoyance is that the screen goes to "Display output is not active" and shows a black screen after some timeout. The QXL display doesn't do this.

VirtIO-GPU does work with the same SPICE client as QXL. It can do audio and USB passthrough, which I use frequently. It can also do multiple SPICE connections to the same VM, with the patchlet I mentioned above.

For a lot of the VMs, this is working just as well as QXL/Spice, so it will be my default except for the video machines.

Thanks!

-Jeff

 

[Valombre]

Hi,
i confirm the VirtIO-GPU is working no crash anymore after 2 weeks of use, but indeed display isn't smooth at all, just enough for console work or simple gui applications.
Regards

 

[jebbam]

i confirm the VirtIO-GPU is working no crash anymore after 2 weeks of use, but indeed display isn't smooth at all, just enough for console work or simple gui applications.

Have you found a way to disable the "Display output is not active" that comes on after a few minutes? I haven't poked at it to see why it is coming up. It isn't the worst, but it is kind of annoying. Not as bad as crashing!

 

[atticushp]

Using Ubuntu 22.04 on QEMU/KVM would freeze for 10-15 seconds with the same errors reporting in syslog. Learning it was from QXL lead to being able to find it being caused by too little VRAM allocated. The default display memory size for vgamem was too low if going beyond the default resolution. Keeping QXL but increasing the vgamem value to 65536 in the video section of the xml file fixed it. Of course the vgamem size would need to be increased for very high resolutions.

 

[jebbam]

Keeping QXL but increasing the vgamem value to 65536 in the video section of the xml file fixed it. Of course the vgamem size would need to be increased for very high resolutions.

Which XML file? I wasn't able to find one that had that setting.

I do have this set: "vga: qxl,memory=128"

Thanks!

 

[THX1138]

I recall increasing my vgamem setting a few months back but, just checking, it has reverted to 16384 at some point. I'll increase it to 65536 and see how it goes. Thanks for the suggestion.

My main VM has been crashing randomly throughout - on average about twice per day - but I had been unable to identify a specific cause beyond the original error message (relating to QXL, vram etc.).

 

[THX1138]

Which XML file? I wasn't able to find one that had that setting.

I do have this set: "vga: qxl,memory=128"

Thanks!

It's the combined XML settings file for your particular VM. Via virt-manager, settings are accessible under "Video": Select "QXL", then select the XML tab and edit the vgamem value (having first enabled XML editing in Preferences). The raw XML file location may vary: for me it's at: /var/run/libvirt/qemu/<vm-name>.xml, but it could be in your home directory somewhere if you elected to set up there.

I should mention, I'm not using Proxmox; I'm using kvm-qemu on desktop Linux, but it should work the same.

 

[jebbam]

I should mention, I'm not using Proxmox; I'm using kvm-qemu on desktop Linux, but it should work the same.

Ya, Proxmox doesn't have it there. In the Proxmox GUI, there is a setting, but it won't start if you set memory= over 128. So I have had it maxed at 128 on all the machines, all along, but that still gets the error. There is config settings per machine in /etc/pve/nodes/[server_name]/qemu-server/foo.conf. Those aren't XML. There isn't a vgamem setting there. But there are lines like "vga: qxl,memory=128".

 

[THX1138]

I recall increasing my vgamem setting a few months back but, just checking, it has reverted to 16384 at some point. I'll increase it to 65536 and see how it goes. Thanks for the suggestion.

No joy. With vgamem = 65536, the VM still crashes with the same error.

I also remember now: the vgamem value reverted because I reverted it. The QXL driver includes a vga mode fallback driver that is not used in normal operation (so I read). The vgamem value allocates memory for that vga mode, meanwhile it's the RAM and VRAM settings that apply to the QXL driver - mine are both set at 131072 (i.e. 128MB).

This bug is known (see links below) and was allegedly fixed earlier this year, but obviously not all the triggering conditions have been addressed or the bug fix isn't in the kernel version I'm using (though it ought to be, since I'm on 6.8.0-48 in both guest and host) or this is a related but different bug. The bug is claimed to be related to QXL driver workload - for me it often seems to trigger when memory demands are highest, such as playing a video in the guest, though not always - sometimes the guest isn't doing much at all.

https://ubuntuforums.org/showthread.php?t=2497311
https://lore.kernel.org/lkml/db4c8e74-5c79-49be-9781-a5d8669eccc1@leemhuis.info/T/

I'll play around with RAM and VRAM settings but I don't expect any change since 128MB is more than enough for my 2D-only screen.