Packet loss within Proxmox (even in-between containers)

[zw1d]

Hi,

I had to rewrite this whole post, because it was misleading. I claimed to resolve my lost packets issue, but turns our I haven't.

I get packet loss of ~1-30% to neighbouring guests and the outside network - not always, but in spikes, say once every 15 seconds?

224 packets transmitted, 157 received, 29.9107% packet loss, time 3585ms
rtt min/avg/max/mdev = 0.230/0.327/0.572/0.051 ms

The host does not have any packet loss to outside network, the issue is somewhere in between host<=>guest.

Please offer any suggestions on to how to diagnose this issue.

/etc/network/interfaces

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

auto eno2
iface eno2 inet manual

auto eno3
iface eno3 inet manual

auto eno4
iface eno4 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves eno1 eno2 eno3 eno4
        bond-miimon 100
        bond-mode 802.3ad
        bond-downdelay 200
        bond-updelay 200

auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vmbr0.1000
iface vmbr0.1000 inet static
        address 10.1.0.190/24
        gateway 10.1.0.254

cat /proc/pressure/cpu

some avg10=0.25 avg60=0.31 avg300=0.32 total=158295511551
full avg10=0.00 avg60=0.00 avg300=0.00 total=0

/proc/loadavg

2.47 2.42 2.49 1/2720 2794077

dmesg

[14389385.454906] vmbr0: port 15(veth129i0) entered blocking state
[14389385.454916] vmbr0: port 15(veth129i0) entered forwarding state
[14390880.079554] audit: type=1400 audit(1719835353.421:9948): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-134_</var/lib/lxc>" name="/dev/" pid=456952 comm="(sd-mkdcreds)" flags="rw, rslave"
[14415853.436879] audit: type=1400 audit(1719860327.296:9949): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-109_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=816031 comm="(certbot)" srcname="/" flags="rw, rbind"
[14426727.051931] audit: type=1400 audit(1719871201.138:9950): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-109_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=967214 comm="(ogrotate)" srcname="/" flags="rw, rbind"
[14426754.445519] audit: type=1400 audit(1719871228.534:9951): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-130_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=967784 comm="(ogrotate)" srcname="/" flags="rw, rbind"
[14429832.837187] audit: type=1400 audit(1719874306.986:9952): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-130_</var/lib/lxc>" name="/dev/" pid=1010734 comm="(sd-mkdcreds)" flags="rw, rslave"
[14433968.776092] audit: type=1400 audit(1719878443.013:9953): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-115_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=1069774 comm="(ogrotate)" srcname="/" flags="rw, rbind"
[14433969.007648] audit: type=1400 audit(1719878443.245:9954): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-134_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=1069784 comm="(ogrotate)" srcname="/" flags="rw, rbind"
[14442423.057681] audit: type=1400 audit(1719886897.472:9955): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-109_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=1189155 comm="(certbot)" srcname="/" flags="rw, rbind"
[14456288.540053] audit: type=1400 audit(1719900763.241:9956): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-134_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=1386326 comm="(install)" srcname="/" flags="rw, rbind"
[14477338.295486] audit: type=1400 audit(1719921813.433:9957): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-134_</var/lib/lxc>" name="/dev/" pid=1685306 comm="(sd-mkdcreds)" flags="rw, rslave"
[14498351.696011] audit: type=1400 audit(1719942827.265:9958): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-109_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=1980891 comm="(certbot)" srcname="/" flags="rw, rbind"
[14513125.383813] audit: type=1400 audit(1719957601.253:9959): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-109_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=2187124 comm="(ogrotate)" srcname="/" flags="rw, rbind"
[14513158.515370] audit: type=1400 audit(1719957634.386:9960): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-130_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=2187783 comm="(ogrotate)" srcname="/" flags="rw, rbind"
[14516261.063000] audit: type=1400 audit(1719960736.992:9961): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-130_</var/lib/lxc>" name="/dev/" pid=2230910 comm="(sd-mkdcreds)" flags="rw, rslave"
[14520366.991790] audit: type=1400 audit(1719964843.006:9962): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-115_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=2288877 comm="(ogrotate)" srcname="/" flags="rw, rbind"
[14520367.228581] audit: type=1400 audit(1719964843.242:9963): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-134_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=2288891 comm="(ogrotate)" srcname="/" flags="rw, rbind"
[14525161.355479] audit: type=1400 audit(1719969637.466:9964): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-109_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=2357201 comm="(certbot)" srcname="/" flags="rw, rbind"
[14526427.093254] audit: type=1400 audit(1719970903.226:9965): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-134_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=2374780 comm="(install)" srcname="/" flags="rw, rbind"
[14556275.268303] vmbr0: port 14(veth114i0) entered disabled state
[14556275.391588] vmbr0: port 14(veth114i0) entered disabled state
[14556275.392565] veth114i0 (unregistering): left allmulticast mode
[14556275.392576] veth114i0 (unregistering): left promiscuous mode
[14556275.392581] vmbr0: port 14(veth114i0) entered disabled state
[14556276.121746] audit: type=1400 audit(1720000752.854:9966): apparmor="STATUS" operation="profile_remove" profile="/usr/bin/lxc-start" name="lxc-114_</var/lib/lxc>" pid=2798349 comm="apparmor_parser"
[14556279.486520] audit: type=1400 audit(1720000756.218:9967): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-114_</var/lib/lxc>" pid=2798417 comm="apparmor_parser"
[14556280.810082] vmbr0: port 14(veth114i0) entered blocking state
[14556280.810093] vmbr0: port 14(veth114i0) entered disabled state
[14556280.810138] veth114i0: entered allmulticast mode
[14556280.810236] veth114i0: entered promiscuous mode
[14556280.896283] eth0: renamed from vethgtU9La
[14556282.055527] vmbr0: port 14(veth114i0) entered blocking state
[14556282.055545] vmbr0: port 14(veth114i0) entered forwarding state

 

[zw1d]

Bump (changed whole post)

 

[zw1d]

I failed to notice that the guests that pinged each other with packet loss were in different networks, so the traffic effectively was routed through the main switch. The switch that I had my doubts about in the past (Mikrotik CSS326-24G-2S+). I tried rebooting the switch and the packet loss is gone for now. I am still confused and I don't know why would the host have no packet loss, when the guests did. At this point, I don't feel confident enough to make any conclusions...