A large number of dropped packets

K

kennyisalive222

Active Member
Nov 18, 2019
2
0
41
Hi everyone!

I have:
Proxmox 9.1,
sfp ens13f0np0 Speed: 10000Mb/s, MTU 9000, physical NIC,
Bridge vmbr0 Speed: 10000Mb/s, MTU 9000

Single VM with Windows 2025 and Red Hat VirtIO Ethernet Adapter 10 Gbps. (VirtIO (paravirtualized)). All last Virtio drivers for Win was installed.

Problem: large dropped packets on vmbr0 and win nic. Its critical for damn MS Exchange (ya, i known). Statistics:

ens13f0np0:

Code: 
ens13f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000

    link/ether 6c:83:75:0e:5d:f4 brd ff:ff:ff:ff:ff:ff

    RX:    bytes   packets errors dropped  missed    mcast

    220619603253 233473964      0   85390    2605 11798108

    TX:    bytes   packets errors dropped carrier  collsns

     89745549722  97421350      0       0       0        0

vmbr0:
Code: 
vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 6c:83:75:0e:5d:f4 brd ff:ff:ff:ff:ff:ff
    RX:   bytes  packets errors dropped  missed    mcast
    17500241620 63673589      0  680897       0 11660146
    TX:   bytes  packets errors dropped carrier  collsns
      682621269  2341953      0       0       0        0

windows nic:
Code: 
IPv4 Statistics

  Packets Received                   = 20762904
  Received Header Errors             = 0
  Received Address Errors            = 277016
  Datagrams Forwarded                = 0
  Unknown Protocols Received         = 7
  Received Packets Discarded         = 2361699
  Received Packets Delivered         = 18124984
  Output Requests                    = 14285872
  Routing Discards                   = 0
  Discarded Output Packets           = 7
  Output Packet No Route             = 0
  Reassembly Required                = 1054
  Reassembly Successful              = 286
  Reassembly Failures                = 0
  Datagrams Successfully Fragmented  = 0
  Datagrams Failing Fragmentation    = 0
  Fragments Created                  = 0



As the statistics show, on Windows, 2 million out of 20 million packets were lost, or one in ten.
Uptime: 3 days.

On Proxmox:

Uptime: 19 days, 63 million packets were sent to vmbr0, 680,000 were lost.
On the physical ens13f0np0: 233 million were sent, 85,000 were lost.

It seems like packets from the Windows VM aren't reaching the physical interface and are being dropped somewhere.
 
Hi kennyisalive222,

I suppose this few point to check:
  1. You are using jumbopacket. The packet loss may be connected with this. Does the Win server accept Jumbopacket?
  2. Is the loss connected with UDP or TCP trafic?
  3. The loss may be connected with server overload.
  4. Dropped traffic may be firewall rule. Do you have enabled firewall?
  5. Dropped traffic may be because traffic is routed here but server no accepted it (wrong routing rule). Can you check this?
  6. I heard about some issue with the last virtio driver v0.1.285. Can you try v0.1.271?
Generally I recommend investigate source and type of packet which was lost.

R.
 
kosnar said:
Hi kennyisalive222,

I suppose this few point to check:
  1. You are using jumbopacket. The packet loss may be connected with this. Does the Win server accept Jumbopacket?
  2. Is the loss connected with UDP or TCP trafic?
  3. The loss may be connected with server overload.
  4. Dropped traffic may be firewall rule. Do you have enabled firewall?
  5. Dropped traffic may be because traffic is routed here but server no accepted it (wrong routing rule). Can you check this?
  6. I heard about some issue with the last virtio driver v0.1.285. Can you try v0.1.271?
Generally I recommend investigate source and type of packet which was lost.

R.
Click to expand...
Hi, kosnar! Thanks for your reply.

1. jimbopackets disable in windows nic
2. Full current statistics:
Code: 
IPv4 Statistics

  Packets Received                   = 22265938
  Received Header Errors             = 0
  Received Address Errors            = 298979
  Datagrams Forwarded                = 0
  Unknown Protocols Received         = 26
  Received Packets Discarded         = 2524822
  Received Packets Delivered         = 19443287
  Output Requests                    = 15495709
  Routing Discards                   = 0
  Discarded Output Packets           = 98
  Output Packet No Route             = 350
  Reassembly Required                = 1217
  Reassembly Successful              = 339
  Reassembly Failures                = 0
  Datagrams Successfully Fragmented  = 0
  Datagrams Failing Fragmentation    = 0
  Fragments Created                  = 0

IPv6 Statistics

  Packets Received                   = 694740
  Received Header Errors             = 0
  Received Address Errors            = 0
  Datagrams Forwarded                = 0
  Unknown Protocols Received         = 0
  Received Packets Discarded         = 22
  Received Packets Delivered         = 769621
  Output Requests                    = 1264
  Routing Discards                   = 0
  Discarded Output Packets           = 12
  Output Packet No Route             = 222
  Reassembly Required                = 149461
  Reassembly Successful              = 74712
  Reassembly Failures                = 0
  Datagrams Successfully Fragmented  = 0
  Datagrams Failing Fragmentation    = 0
  Fragments Created                  = 0

ICMPv4 Statistics

                            Received    Sent
  Messages                  34780       34406
  Errors                    0           0
  Destination Unreachable   602         229
  Time Exceeded             0           0
  Parameter Problems        0           0
  Source Quenches           0           0
  Redirects                 0           0
  Echo Replies              3           34174
  Echos                     34175       3
  Timestamps                0           0
  Timestamp Replies         0           0
  Address Masks             0           0
  Address Mask Replies      0           0
  Router Solicitations      0           0
  Router Advertisements     0           0

ICMPv6 Statistics

                            Received    Sent
  Messages                  127822      41
  Errors                    0           0
  Destination Unreachable   3           3
  Packet Too Big            0           0
  Time Exceeded             0           0
  Parameter Problems        0           0
  Echos                     0           0
  Echo Replies              0           0
  MLD Queries               0           0
  MLD Reports               127542      0
  MLD Dones                 0           0
  Router Solicitations      0           12
  Router Advertisements     0           0
  Neighbor Solicitations    4           18
  Neighbor Advertisements   305         8
  Redirects                 0           0
  Router Renumberings       0           0

TCP Statistics for IPv4

  Active Opens                        = 627209
  Passive Opens                       = 190195
  Failed Connection Attempts          = 927
  Reset Connections                   = 42692
  Current Connections                 = 124
  Segments Received                   = 20826152
  Segments Sent                       = 17854235
  Segments Retransmitted              = 15935

TCP Statistics for IPv6

  Active Opens                        = 167338
  Passive Opens                       = 167333
  Failed Connection Attempts          = 16
  Reset Connections                   = 139145
  Current Connections                 = 192
  Segments Received                   = 6646153
  Segments Sent                       = 6646362
  Segments Retransmitted              = 13

UDP Statistics for IPv4

  Datagrams Received    = 896947
  No Ports              = 2524672
  Receive Errors        = 6
  Datagrams Sent        = 39032

UDP Statistics for IPv6

  Datagrams Received    = 492095
  No Ports              = 6
  Receive Errors        = 0
  Datagrams Sent        = 94

As I see, loss tcp traffic (how can I show monitoring UDP traffic, if than does not require confirmation?)

3. Server resources are barely being used. No CPU, no RAM, no disk, no network, sitting in the task manager

4. Win firewall is disable, fw on pve vmbr0 enable, but there no rules

5. I didn't quite understand the question. The Win server is routed like all other gateways on the network, using the same rules and the same router. This includes the PVE host, which has significantly fewer lost packets.


6. Ya, I heard about this too and I downgraded virtio drivers to 127. Not help. Dropped packets count growing at the same rate. I also tried change nic to E100 and same result. By the way, which is better for Win VM, Virtio Net Device or E1000?

Could packets be lost somewhere between the VM and the hypervisor? There's a huge difference in packet loss between vmbr0 and win nic.
 
ad 1) If you have jumbopacket disabled in windows, aren't dropped packet jumbopackets? (f.e. Deskop1 send jumbo to WinServer)
ad 5) I tried to explain by this example:
Desktop1 10.10.15.16 wants to send packet to IP 10.11.12.13 and finds in his route table that this packet should be send via 10.11.12.1 (WinServer). So Desktop1 send packet. But 10.11.12.1 (WinServer) has no routing rule from 10.10.15.16 to 10.11.12.13 and drop packet by firewall.

I tray to simplify the process - vmbr0 is like switch it gets packet for 10.11.12.1 (WinServer) so it put the packet to win nic interface queue.
The dropped packet may be multicast, wrong route (point 5.) and may other your network dependent isue and non issue.

Are you able to analyze traffic and identify the dropped ones? As I wrote source and type of dropped packet will help. Is it traffic from one source or general problem? What this packet loss affect?

R.
 
kennyisalive222 said:
6. Ya, I heard about this too and I downgraded virtio drivers to 127. Not help. Dropped packets count growing at the same rate. I also tried change nic to E100 and same result. By the way, which is better for Win VM, Virtio Net Device or E1000?
Click to expand...
always use virtio. e1000 don't have any acceleration
 
You must log in or register to reply here.
Top Bottom
Back