+10 Gbps peaks RRD traffic not measured correctly

[jmjosebest]

Hello, we have detected a problem with the RRD data generated by Proxmox.

The failure is occurring in a VM where the user is having traffic peaks of 10 Gbps for about 60-90 seconds... during this time a lot of traffic is generated that the Proxmox RRD does not detect. This is +60 TB of traffic over the month, about 300 € of consumption that we are not billing to the customer who consumes them.

In these images you can see the comparison:

1- The graph generated by Proxmox

2- The graph we generated by obtaining the RRD values from Proxmox through API.

3- The graph that we generate by obtaining the traffic received by the VM interface Port fwlnXXXi0 through SNMP.

As you can see, this VM is having peaks of +5 Gbps.
However, Proxmox is only showing 50 Mbps or less.

We have verified this data with our edge router, where we see that the peaks are 10 Gbps ! And yes, the Proxmox node is connected to 10G SFP+.

So far we are using the Proxmox RRD data per API to account for the traffic consumed by our customers, but this is messing up our economy.

I understand that the value Proxmox is showing is not 100% real, but it is practically showing a null traffic when this client is consuming us +2TB of traffic per day !!!!

Any suggestions?

Thanks!!!

 

[dcsapak]

please check the aggregation? which did you use ? e.g. daily (average)?
also note that you have to multiply the value with the number of seconds in the timeframe (since each point is basically for the complete timeframe until the next)

so if the graph shows e.g. 10M/s at 00:00:00 and 0 at the next (e.g. 00:01:00) it means 10M/s * 60s => 600M for that timeframe

graphing throughput always depends on the interval in which it's measured/displayed

 

[jmjosebest]

Hello,

I don't think you understood me... I am perfectly aware how to get traffic consumption.

What I am reporting is the failure of Proxmox to identify the SPEED.

This VM port is generating a value of 5 Gbps
And Proxmox is only detecting 50 Mbps

 

[dcsapak]

can you post the raw rrd data and which exact request you do?

 

[jmjosebest]

This is the graph reported by Proxmox GUI (Daily Maximum format)

This is the graph reported by LibreNMS (A SNMP monitoring tool) wich monitors the interface of the VPS: fwlnXXXi0

Why Proxmox is showing just 50 Mbps
And LibreNMS is showing +5 Gbps

?

 

[dcsapak]

is that graph from proxmox from the vm ? or the host ?

 

[jmjosebest]

Both are the VM

 

[dcsapak]

ok, so first, the pve graph is bytes not bit, and second when you choose daily max, note that
the graph will (for each data point) show the maximum of the one minute intervals of the rrd (because it gets consolidated)

so if you had a 10gbit burst for 1 second, it'll only show as a 10 Gbit / 8 / 60s ~ 20 M etc.

so to see a 10gbit "spike" it'll have to last at least 60s to show up as that
(and then it'll show as byte, not bit)

i tested here by letting an iperf test run for ~120s, (host <-> guest) and had a throughput of around 80Gbit/s.
my graphs after that show a 10G throuput for that period when selecting 'Hour' and for 'Daily (max)' i also see the 10G

i cannot speak to the resolution of the graph of the snmp (and how often it takes the measurement),
so i cannot say how they calculate the graph

also maybe this helps for understanding:
http://rrdtool.vandenbogaerdt.nl/min-avg-max.php

if you want the most 'raw' data available, i'd recommend to either use our built-in metrics export (influxdb/graphite) or
use another monitoring tool that captures the values itself

 

[jmjosebest]

i tested here by letting an iperf test run for ~120s, (host <-> guest) and had a throughput of around 80Gbit/s.
my graphs after that show a 10G throuput for that period when selecting 'Hour' and for 'Daily (max)' i also see the 10G

Ok, good to know that you also see the problem. The server is running 80 Gbps but your graph has only shown 10 Gbps... Ok.

i cannot speak to the resolution of the graph of the snmp (and how often it takes the measurement),
so i cannot say how they calculate the graph

Well, with SNMP we can confirm that the data is much more reliable when it comes to monitoring the VM network interface.

if you want the most 'raw' data available, i'd recommend to either use our built-in metrics export (influxdb/graphite) or
use another monitoring tool that captures the values itself

As I mentioned in my first post, we generate our charts using the RRD data returned by Proxmox via API.

Are you sure there is not an option for Proxmox to generate more reliable RRD data like SNMP?

 

[dcsapak]

Ok, good to know that you also see the problem. The server is running 80 Gbps but your graph has only shown 10 Gbps... Ok.

no, the graph shows 10 Gigabytes per second which is exactly 80 Gigabits per second...

 

[jmjosebest]

Ah, ok...

Well, in our case we have a customer sending 10 Gbps (1 GB/s) bursts and the graph (RRD) of Proxmox only shows 50 Mbps (5 MB/s)...

 

[dcsapak]

then they are probably very short bursts (which your other tool catches probably). in the rrd data we only collect the data the kernel gives us and put in in an rrd database, there is not really much possible where we'd put wrong data in. aside from that, as i said, i tested it and it works
here. maybe you can test a bit with a test vm and e.g. iperf and see how it behaves

 

[jmjosebest]

This case can be closed, it is not a bug.

In Ticket #7745382 we have identified that the traffic posted on the VM is "post-firewall".

The client is receiving syn-flood attacks and the firewall rules generated in Proxmox block packets with invalid state.

-A PVEFW-FORWARD -m conntrack --ctstate INVALID -j DROP
-A PVEFW-FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A PVEFW-FORWARD -m physdev --physdev-in fwln+ --physdev-is-bridged -j PVEFW-FWBR-IN
-A PVEFW-FORWARD -m physdev --physdev-out fwln+ --physdev-is-bridged -j PVEFW-FWBR-OUT

That is, the traffic reaches the host but not the VM.