Status: io-error on first boot of any new VM

[Cap10]

I'm new to Proxmox, not new to Linux or VMs.

Any time I start a new VM now, it freezes during the initial setup and Proxmox shows "Status: io-error". I only have a single LXC container and a single Ubuntu VM running right now. They definitely are not using the node's entire disk drive space, or CPU or RAM.

I saw somewhere online it could be due to the LVM being full. When looking at the node I see this

but honestly I can't tell if that means I have the entirety of the disk reserved for Proxmox or the whole thing is in use. Looking at the node's local storage and local-lvm storage, they both show only about 4% usage.

My only guess is it has something to do with VMs I have built and later removed through Proxmox. Maybe I didn't remove their storage properly so there's no more room on the disk for a new VM? I really have no clue, taking a shot in the dark.

What can I be looking for and what more info can I provide to help find this issue?

 

[Karyû]

Hi, i'm also new to proxmox, i got the same problem with one of my VMs (Pop!_OS).
This VM can't boot properly, the same VM was all okay last sunday. Today when i got updates for my Proxmox host i wasn't able to boot the VM.
I personally think that one of these updates is to blame for it.

One of these updates was for the LVM if i'm correct.


edit:
Just tried to create a new Linux (MX) VM and got an IO error again, but my Win 10 VM can boot normal, difference between this 3 VMs is just UEFI support (OVMF for Linux VMs ) / the Windows VM (SeaBios)

 

[Cap10]

I don't think I've updated Proxmox since setting it up. One of the VMs is ubuntu-live-server-amd64, and so is the new VM I'm trying to create.

Are you trying to say that a Proxmox update has just broken the ability to make VMs with one of the most common ISOs? Wouldn't that just be breaking the entire function of Proxmox in the first place? There's no way they'd push an update like that.

 

[Karyû]

Yeah you're right about breaking the core function, but actually my migrated machines and also my attempt to install Ubuntu server 22.04 LTS got an IO error all over the day.
I tried Pop!_OS 21.10 live and it worked (tried all machines with UEFI / OVMF support).
So ... i think it could be the Linux guest itself.

edit:
I encountered some strange behavior, some VMs booting and others not ..., but in the end crashes all.

RAM is all Okay (ECC memory) ...
I tried VM-disks on my CIFS Server and direct on the Host, all VMs with outsourced disks got an IO error, reboot of all my machines don't helped.
Screenshot of one VM with disk on the network saved

When did you set up your Proxmox ?
Do you save your VMs on a NAS or Server in your network ?

 

[Cap10]

I set up Proxmox with the current working VM and LXC like 2 weeks ago. I have all my VMs local, but I have tried storing the ISOs I'm using for boot both local and on a NAS.

 

[Karyû]

Okay, i tried the ISOs on the local machine but the VM-disks one time local and another time outsourced.



VM disk local = Function
VM disk in NAS / server storage = IO error

i know that saving a VM disk on network storage isn't the best way, but for me it worked fine until yesterday.

What's the output of syslog of your Proxmox ?

I found a message in the syslogs of my host machine, something like that and this causes me an IO error with outsourced VM disks

 

[Cap10]

So, that's part of the problem and I should I have been more clear in my original post. How do I get syslog? I can't find anything for logs other than under the firewall.

 

[Karyû]

You can access the syslog under Data center > "your Node" (eg. Proxmox) > syslog

 

[Cap10]

I should feel embarassed, I wass looking for it but mostly with the VM selected.
After watching the log I see i am getting the CIFS: VFS: error too.
After going througha nd making a VM again and looking carefully, I see it wasn't the ISO, the "storage" was defaulting tosetting the disk to be on my NAS I have attached. Once I set that to local again it's booting up fine. Of course that makes sense.
It should always default local, but I also should have caught that sooner.

 

[Karyû]

So in the end we both have the same Problem with CIFS / Samba Storage and outsourced VM Disks xDD
After all i think it is really something with an update for Proxmox, but to have IO errors ist just annoying.
I don't know if this is a problem with Proxmox ( to be honest i use the non subscription repository for PVE updates ), but i hope for some fixes if its from an update....

I consider to reinstall proxmox complete on my Host an see if it's a persistent error with the Repo.

 

[Cap10]

Well at least we got it figured out. Good thing my job hired me to be a Docker guy, not a Proxmox guy lol. I'm just doing this at home.

 

[Karyû]

Well yeah i'm an Electrician for all things over PC to Networking and Linux, but working actually as signal technician xD , i'm doing this at home to virtualize multiple machines and splitting some services from my storage server etc.

 

[Karyû]

Well, a friend of mine sends me a screenshot about his Proxmox nodes and upgradable packages an there is a package for Cifs-utils.

 

[Karyû]

Yeah i found the cause ,.... the I/O error over the network storage is based on the newer Kernel

 

[fiona]

Hi,
I can reproduce the issue here and am looking for the root cause. Apart from downgrading the kernel, a workaround seems to be to switch your VM's disk away from io_uring to threads or native (advanced disk settings in the UI).

 

[Karyû]

@Fabian_E Haha nice, thanks, i just tried the old 5.4 kernel just for fun and that's it.
I would like to know why we get kernel messages about CIFS /VFS in the syslog as i described above in thread #6 and i'll try to switch my VM disks away from IO_uring as you mentioned.

 

[fiona]

@Fabian_E Haha nice, thanks, i just tried the old 5.4 kernel just for fun and that's it.
I would like to know why we get kernel messages about CIFS /VFS in the syslog as i described above in thread #6 and i'll try to switch my VM disks away from IO_uring as you mentioned.

I'm still trying to figure out the cause. It might be that the issue is in the CIFS kernel code and using IO uring makes the issue more likely to appear, but it might also be a problem in IO uring itself.

 

[Karyû]

So... i switched the Disk to Nativ as you mentioned earlier, now with the new kernel it seems to work as usually.

 

[Karyû]

So .. today i got again the I/O Error message but this time with nativ VM disk option.

The other VMs working normally without issues, but Deepin and Linux Mint don't wanna go the right way.

 

[fiona]

So .. today i got again the I/O Error message but this time with nativ VM disk option.

The other VMs working normally without issues, but Deepin and Linux Mint don't wanna go the right way.
View attachment 38798

Hmm, as said maybe using io_uring just makes it more likely to trigger. Can you please check /var/log/syslog from around the time the issue happened? Is it the same Error -512 sending data on socket to server error?

I'm currently testing a commit that's potentially fixing the issue suggested by kernel developers.