File read/write issues in VM - only in proxmox

ecce

New Member
Feb 17, 2020
11
1
3
43
Prepare for a weird and pretty non-specific question. I work with F5 BIGIP devices and use a few of them in a virtual environment for labs and demo. I've been running ESXi for years and it works fine. I've run KVM as well in a few setups, it has worked flawlessly. This is the first time trying out Proxmox.

The BIGIP is a Linux-based app proxy that can run under KVM according to F5. And it does. F5 makes prepared qcow2 files and provides info about the virtual hardware recommendations. But for some people, including me, there are very weird behavior of the VM that seems to be related to reading and writing files. This is within the VM itself. An example: SSH daemon cannot start because if claims it cannot read the ssh key files. But the files are there, the path is right, there is no permission issue and no SELinux issues. And starting sshd manually with the exact same key file works just fine. Another process claims it cannot create a specific folder, but creating it manually works fine - again no obvious permission or SELinux issues (I chmod 777'd the directory and it still could not create the folder). When you activate the BIGIP license it says it cannot write the license file - but it does write info in the license file. It's confusing.

This does not seem to be Proxmox-related since other VMs does not show this behaviour, and the error is within the VM. But is also does seem to be Proxmox-related since it only happens in proxmox and other people experience the same thing. However, some people virtualize BIGIPs in Proxmox without any issue. I've compared settings with them and the only diff is usage of ceph storage and different Proxmox version.

I have troubleshooted the VM itself for a few days now and I running out of ideas. I have tried different versions of the BIGIP image, tried both qcow2 images (that should be the right one) and converted vmdk images aimed for vmware, changed pretty much all the settings I can find in the proxmox GUI for the VM, re-installed, rebooted... I get the same behaviour regardless of what I do. Maybe I'm missing something obvious?

The pretty non-specific question I have is: How do I troubleshoot this? Has anyone experienced the same behaviour in a VM and solved it? If you need any info about the VM please tell me what commands etc to run and I'll post the info.

I'm running Proxmox 6.1-3 on Intel Server hardware, storage is NVMe-disks that uses LVM thin for VM storage.
 
This sounds like one of the weird (eventually timing) issues to me.
Since one can easily get lost in such a situation: Have you tried
  • using different disk-types (SCSI vs. IDE vs. SATA vs. Virt-IO Block)?
  • using different Caching methods (e.g. go to sync, etc.)
Sorry if that has already been part of your troubleshooting process...
 
  • Like
Reactions: ecce
This sounds like one of the weird (eventually timing) issues to me.
Since one can easily get lost in such a situation: Have you tried
  • using different disk-types (SCSI vs. IDE vs. SATA vs. Virt-IO Block)?
  • using different Caching methods (e.g. go to sync, etc.)
Sorry if that has already been part of your troubleshooting process...

No problem at all, any suggestions are welcome. I have tried different disk types and controllers. None of them solves the problem. virtio-SCSI seems to work best and also supports SSD trim as I understand it. Virtio-block does work but you have to manually specify the boot disk by bringing up the boot menu and select it, otherwise it starts PXE'ing and cant find anything to boot from. Another strange behaviour that I just thought of is that the boot process does now show. It's just a black screen for a minute, then a login prompt shows.

I had not tried different cache settings, that's a good idea. I went though them all but I noticed no difference.
 
OK.
Next I would try a different storage type (an alternative to LVM) - could be a USB-drive which you configure to be a ZFS pool.
Could be as simple as some "weird" interop issue between LVM and NVMe for instance.

Also have you tried using a different base chipset (Machine -> Default is 440fx, alternative is q35)? According this setting exposes different other "virtual HW" types to the VM.

I know these "black screens" from physical computers. I always tied this back to a "driver issue" whenever being within the grub menu. After all kernel modules have been loaded - the screen comes up. It is weird though you experience this in a VM.
 
I think we have a clue. If i turn off SELinux sshd starts (setenforce 0, modifying /etc/selinux/config and set SELINUX=permissive to persist). There are still other weird things going on, but that could be other causes for those.

to answer your question: No, I have not tried other chipsets. When I compared to a working VM on another proxmox setup, it was 440fx. I can try it though.
 
Last edited:
Tested q35 chipset - no difference. Also tried turning off ballooning since I don't need it, that made no diff either.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!