7.0 BUG? strange disc/network problems on guest, no problem on 6.4

[grl]

[EDIT: See post #3, did a 6.4 install on the same hardware and had no problem]

Hi!

On my testing machine I replaced some v5-version by a brand new v7.0.
Installed a naked debian bullseye, then installed proxmox 7.0 - everything configured manually from scratch, nothing taken over from the old install.
As the machine uses e1000e I disabled offloading on both nics via ethtool

ethtool -K $IFACE tso off gso off gro off

Then I created a guest (debian bullseye) and tried copying over some data via scp (a big DB-dump, single file, 60GB).

Every time I tried that the guest after some seconds fired a bunch of I/O errors and the whole system (host AND guest) became unresponsive. Could only press the rest button.
Tried creating a disc on the LVM the guests drive is on and mounted that directly on the host. Then tried opying the same file directly to the host - works like a charm.

So I'm pretty sure its not the RAID the LVM is on - worked before replacing v5 by v7 and works when mounting directly in the host os.

Next try was replacing the SCSI controller for the guest from virtio to LSI 53C895A. No I/O errors, but the system became unresponsive again after some seconds.

Then tried to stress the I/O system on the guest by dd:

dd if=/dev/zero of=/mnt/test.dat bs=1M count=102400

worked like a charm once again.

I'm suspecting some sort of network problem as dd worked perfectly - but I'm running out of ideas here - anyone a hint where to search next?

regards
Lukas


I/O Errors are:

Jul 15 12:27:09 fb4test kernel: [   71.465417] print_req_error: 22 callbacks suppressed
Jul 15 12:27:09 fb4test kernel: [   71.465426] blk_update_request: I/O error, dev vdb, sector 252330048 op 0x1:(WRITE) flags 0x0 phys_seg 5 prio class 0
Jul 15 12:27:09 fb4test kernel: [   71.465487] EXT4-fs warning (device vdb1): ext4_end_bio:345: I/O error 10 writing to inode 12 starting block 31541504)
Jul 15 12:27:09 fb4test kernel: [   71.465493] blk_update_request: I/O error, dev vdb, sector 252332032 op 0x1:(WRITE) flags 0x4000 phys_seg 4 prio class 0
Jul 15 12:27:09 fb4test kernel: [   71.465535] blk_update_request: I/O error, dev vdb, sector 252334592 op 0x1:(WRITE) flags 0x4000 phys_seg 6 prio class 0
Jul 15 12:27:09 fb4test kernel: [   71.465576] blk_update_request: I/O error, dev vdb, sector 252337152 op 0x1:(WRITE) flags 0x4000 phys_seg 3 prio class 0
Jul 15 12:27:09 fb4test kernel: [   71.466020] blk_update_request: I/O error, dev vdb, sector 252339712 op 0x1:(WRITE) flags 0x4000 phys_seg 7 prio class 0
Jul 15 12:27:09 fb4test kernel: [   71.468512] buffer_io_error: 12702 callbacks suppressed
Jul 15 12:27:09 fb4test kernel: [   71.468514] Buffer I/O error on device vdb1, logical block 31539720
Jul 15 12:27:09 fb4test kernel: [   71.468911] Buffer I/O error on device vdb1, logical block 31539721
Jul 15 12:27:09 fb4test kernel: [   71.469266] Buffer I/O error on device vdb1, logical block 31539722
Jul 15 12:27:09 fb4test kernel: [   71.469589] Buffer I/O error on device vdb1, logical block 31539723
Jul 15 12:27:09 fb4test kernel: [   71.469911] Buffer I/O error on device vdb1, logical block 31539724
Jul 15 12:27:09 fb4test kernel: [   71.470223] Buffer I/O error on device vdb1, logical block 31539725
Jul 15 12:27:09 fb4test kernel: [   71.470515] Buffer I/O error on device vdb1, logical block 31539726
Jul 15 12:27:09 fb4test kernel: [   71.470809] Buffer I/O error on device vdb1, logical block 31539727
Jul 15 12:27:09 fb4test kernel: [   71.471129] Buffer I/O error on device vdb1, logical block 31539728
Jul 15 12:27:09 fb4test kernel: [   71.471420] Buffer I/O error on device vdb1, logical block 31539729
Jul 15 12:27:09 fb4test kernel: [   71.471728] blk_update_request: I/O error, dev vdb, sector 252347392 op 0x1:(WRITE) flags 0x0 phys_seg 2 prio class 0
Jul 15 12:27:09 fb4test kernel: [   71.472073] EXT4-fs warning (device vdb1): ext4_end_bio:345: I/O error 10 writing to inode 12 starting block 31543552)
Jul 15 12:27:09 fb4test kernel: [   71.472082] blk_update_request: I/O error, dev vdb, sector 252348416 op 0x1:(WRITE) flags 0x4000 phys_seg 2 prio class 0
Jul 15 12:27:09 fb4test kernel: [   71.472426] blk_update_request: I/O error, dev vdb, sector 252350976 op 0x1:(WRITE) flags 0x4000 phys_seg 4 prio class 0
Jul 15 12:27:09 fb4test kernel: [   71.472739] blk_update_request: I/O error, dev vdb, sector 252353536 op 0x1:(WRITE) flags 0x4000 phys_seg 2 prio class 0
Jul 15 12:27:09 fb4test kernel: [   71.488869] EXT4-fs warning (device vdb1): ext4_end_bio:345: I/O error 10 writing to inode 12 starting block 31545600)
Jul 15 12:27:09 fb4test kernel: [   71.499433] blk_update_request: I/O error, dev vdb, sector 252380160 op 0x1:(WRITE) flags 0x0 phys_seg 2 prio class 0
Jul 15 12:27:09 fb4test kernel: [   71.499800] EXT4-fs warning (device vdb1): ext4_end_bio:345: I/O error 10 writing to inode 12 starting block 31547648)
Jul 15 12:27:09 fb4test kernel: [   71.516252] EXT4-fs warning (device vdb1): ext4_end_bio:345: I/O error 10 writing to inode 12 starting block 31549696)
Jul 15 12:27:19 fb4test kernel: [   81.961709] print_req_error: 2 callbacks suppressed
Jul 15 12:27:19 fb4test kernel: [   81.961714] blk_update_request: I/O error, dev vdb, sector 253543424 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0
Jul 15 12:27:19 fb4test kernel: [   81.962039] EXT4-fs warning (device vdb1): ext4_end_bio:345: I/O error 10 writing to inode 12 starting block 31693056)
Jul 15 12:27:19 fb4test kernel: [   81.962045] blk_update_request: I/O error, dev vdb, sector 253544448 op 0x1:(WRITE) flags 0x4000 phys_seg 2 prio class 0
Jul 15 12:27:19 fb4test kernel: [   81.962313] blk_update_request: I/O error, dev vdb, sector 253547008 op 0x1:(WRITE) flags 0x4000 phys_seg 1 prio class 0
Jul 15 12:27:19 fb4test kernel: [   81.962588] blk_update_request: I/O error, dev vdb, sector 253549568 op 0x1:(WRITE) flags 0x4000 phys_seg 2 prio class 0
Jul 15 12:27:19 fb4test kernel: [   81.962846] blk_update_request: I/O error, dev vdb, sector 253552128 op 0x1:(WRITE) flags 0x4000 phys_seg 7 prio class 0
Jul 15 12:27:19 fb4test kernel: [   81.963124] blk_update_request: I/O error, dev vdb, sector 253554688 op 0x1:(WRITE) flags 0x4000 phys_seg 25 prio class 0
Jul 15 12:27:19 fb4test kernel: [   81.963421] blk_update_request: I/O error, dev vdb, sector 253557248 op 0x1:(WRITE) flags 0x4000 phys_seg 4 prio class 0
Jul 15 12:27:19 fb4test kernel: [   81.964508] buffer_io_error: 9710 callbacks suppressed

Hardware is:

Supermicro X9SCL+
Xeon E3-1270
Bios Version 2.3a
32GB RAM
The Board has 2 e1000 ethernet controllers onboard:
Intel Corporation 82579LM Gigabit Network Connection (Lewisville)
Intel Corporation 82574L Gigabit Network Connection

LSI-9261-8i with BBU
4 Seagate SAS drives as RAID5

pveversion:

pve-manager/7.0-9/228c9caa (running kernel: 5.11.22-2-pve)

pveperf

CPU BOGOMIPS:      56003.28
REGEX/SECOND:      2261712
HD SIZE:           54.70 GB (/dev/sda3)
BUFFERED READS:    380.27 MB/sec
AVERAGE SEEK TIME: 7.05 ms
FSYNCS/SECOND:     5122.92
DNS EXT:           56.03 ms
DNS INT:           0.54 ms (test.inst)

/etc/network/interfaces

auto eth0
iface eth0 inet manual
    post-up /sbin/ethtool -K $IFACE tso off gso off gro off

auto eth1
iface eth1 inet manual
    post-up /sbin/ethtool -K $IFACE tso off gso off gro off

auto bond0
iface bond0 inet manual
    bond-slaves eth0 eth1
    bond-miimon 100
    bond-mode 802.3ad
    bond-xmit-hash-policy layer2

auto vmbr0
iface vmbr0 inet static
    address 10.10.10.3/24
    gateway 10.10.10.254
    bridge-ports bond0
    bridge-stp off
    bridge-fd 0

100.conf

agent: 1
balloon: 2048
boot: order=virtio0;ide2;net0
cores: 4
ide2: none,media=cdrom
memory: 8172
name: fb4test
net0: virtio=F2:BE:F5:80:7F:CE,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
smbios1: uuid=74280ebc-ca13-4bc7-bf9d-9ec245dc7d79
sockets: 1
virtio0: lvm_raid5:vm-100-disk-0,size=30G
virtio1: lvm_raid5:vm-100-disk-1,size=250G
vmgenid: 701c1031-c77b-4b72-9343-65186c7ef938

 

[grl]

bump

 

[grl]

Created a second partition with a 6.4 fresh install (manual install of buster, then installed 6.4) on it and used the same test-vm as a guest.
Same LVM for the guest drive images, as its just a different boot partition on the same hardware everything else is the same.

pveversion:

pve-manager/6.4-13/9f411e79 (running kernel: 5.4.124-1-pve)

And there the problem disappearded - its only on the 7.0 installation

So it seems to be a problem with proxmox 7.0...

regards
Lukas