apt-get update results in waiting for headers and does not update

[novacoresystems]

Alright, so I've had this problem for about a week now and have tried to research it myself and figure it out but it just doesn't make much sense. So here is all I've done to diagnose the issue...

So here is what results when I try to update:


root@proxmox:/etc/apt# apt-get update
0% [Connecting to ftp.us.debian.org] [Connecting to download.proxmox.com] [Connecting to security.debian.org]




It hangs like that for awhile and then:

root@proxmox:/etc/apt# apt-get update
Err http://download.proxmox.com squeeze Release.gpg
Connection failed
Err http://security.debian.org squeeze/updates Release.gpg
Connection failed [IP: 128.31.0.36 80]
Err http://download.proxmox.com/debian/ squeeze/pve Translation-en
Connection failed
Err http://security.debian.org/ squeeze/updates/contrib Translation-en
Connection failed [IP: 128.101.240.212 80]
0% [Waiting for headers] [Waiting for headers] [Waiting for headers]

It keeps doing that and fails to connect to each IP. When I run a telnet or wget, the server halfway responds but no accepted connection:

root@proxmox:/etc/apt# telnet 128.101.240.212 80
Trying 128.101.240.212...
Connected to 128.101.240.212.
Escape character is '^]'.

root@proxmox:/etc/apt# wget http://128.101.240.212
--2013-03-27 04:30:30-- http://128.101.240.212/
Connecting to 128.101.240.212:80... connected.
HTTP request sent, awaiting response... No data received.
Retrying.


--2013-03-27 04:30:56-- (try: 2) http://128.101.240.212/
Connecting to 128.101.240.212:80... connected.
HTTP request sent, awaiting response... No data received.
Retrying.




From a machine on the same network it works fine:


root@webserver:~# wget http://128.101.240.212
--2013-03-27 04:30:57-- http://128.101.240.212/
Connecting to 128.101.240.212:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 896 [text/html]
Saving to: `index.html.1'


100%[======================================>] 896 --.-K/s in 0s


2013-03-27 04:30:58 (127 MB/s) - `index.html.1' saved [896/896]










This is my /etc/apt/sources.list file:


deb http://ftp.us.debian.org/debian squeeze main contrib


# PVE packages provided by proxmox.com
deb http://download.proxmox.com/debian squeeze pve


# security updates
deb http://security.debian.org/ squeeze/updates main contrib



this is my interfaces file:

auto lo
iface lo inet loopback


auto vmbr0
iface vmbr0 inet static
address 192.168.1.30
netmask 255.255.255.0
gateway 192.168.1.2
bridge_ports eth0
bridge_stp off
bridge_fd 0


auto vmbr1
iface vmbr1 inet manual
bridge_ports eth1
bridge_stp off
bridge_fd 0


resolv.conf:


search ***********.com
nameserver 192.168.1.2
nameserver 8.8.8.8
nameserver 4.2.2.1



ifconfig output:



root@proxmox:/etc/apt# ifconfig
eth0 Link encap:Ethernet HWaddr 00:26:b9:67:98:00
inet6 addr: fe80::226:b9ff:fe67:9800/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:42975 errors:0 dropped:0 overruns:0 frame:0
TX packets:56157 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:8095625 (7.7 MiB) TX bytes:11800964 (11.2 MiB)
Interrupt:17


eth1 Link encap:Ethernet HWaddr 00:14:d1:1e:cb:cb
inet6 addr: fe80::214:d1ff:fe1e:cbcb/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:35873 errors:0 dropped:0 overruns:0 frame:0
TX packets:41499 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3349507 (3.1 MiB) TX bytes:47507268 (45.3 MiB)
Interrupt:18 Base address:0xf00


lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:8908 errors:0 dropped:0 overruns:0 frame:0
TX packets:8908 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2879104 (2.7 MiB) TX bytes:2879104 (2.7 MiB)


tap100i0 Link encap:Ethernet HWaddr fa:9b:1d:7c:27:d4
inet6 addr: fe80::f89b:1dff:fe7c:27d4/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:252 errors:0 dropped:0 overruns:0 frame:0
TX packets:7682 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:28375 (27.7 KiB) TX bytes:565495 (552.2 KiB)


tap101i0 Link encap:Ethernet HWaddr 0a:ee:a1:e7:e0:8b
inet6 addr: fe80::8ee:a1ff:fee7:e08b/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:166 errors:0 dropped:0 overruns:0 frame:0
TX packets:7527 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:11852 (11.5 KiB) TX bytes:531103 (518.6 KiB)


tap102i0 Link encap:Ethernet HWaddr 4a:0f:57:d2:61:a4
inet6 addr: fe80::480f:57ff:fed2:61a4/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:35803 errors:0 dropped:0 overruns:0 frame:0
TX packets:37448 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:45921026 (43.7 MiB) TX bytes:2643824 (2.5 MiB)


tap104i2 Link encap:Ethernet HWaddr ce:b3:4e:89:01:cc
inet6 addr: fe80::ccb3:4eff:fe89:1cc/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:35826 errors:0 dropped:0 overruns:0 frame:0
TX packets:49648 errors:0 dropped:0 overruns:510 carrier:0
collisions:0 txqueuelen:500
RX bytes:3456479 (3.2 MiB) TX bytes:48137197 (45.9 MiB)


tap104i3 Link encap:Ethernet HWaddr 36:ee:22:24:8a:3b
inet6 addr: fe80::34ee:22ff:fe24:8a3b/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:41472 errors:0 dropped:0 overruns:0 frame:0
TX packets:35901 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:47505498 (45.3 MiB) TX bytes:3351282 (3.1 MiB)


tap105i1 Link encap:Ethernet HWaddr a2:e2:64:90:ca:d3
inet6 addr: fe80::a0e2:64ff:fe90:cad3/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:79 errors:0 dropped:0 overruns:0 frame:0
TX packets:6973 errors:0 dropped:0 overruns:501 carrier:0
collisions:0 txqueuelen:500
RX bytes:12572 (12.2 KiB) TX bytes:482238 (470.9 KiB)


venet0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet6 addr: fe80::1/128 Scope:Link
UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:3 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)


vmbr0 Link encap:Ethernet HWaddr 00:26:b9:67:98:00
inet addr:192.168.1.30 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::226:b9ff:fe67:9800/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:37478 errors:0 dropped:0 overruns:0 frame:0
TX packets:49398 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5751631 (5.4 MiB) TX bytes:10140918 (9.6 MiB)


vmbr1 Link encap:Ethernet HWaddr 00:14:d1:1e:cb:cb
inet6 addr: fe80::214:d1ff:fe1e:cbcb/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:120 (120.0 B) TX bytes:468 (468.0 B)


Routes:


root@proxmox:/etc/apt# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 vmbr0
0.0.0.0 192.168.1.2 0.0.0.0 UG 0 0 0 vmbr0




So anyone have any clue what is going on? I believe it's some sort of networking problem on the proxmox machine. And keep in mind that from my webserver in the same network everything connects fine to the same IPs. So it's not my firewall or any rules/filtering. I've checked all that and turned off security and such.. it's not my firewall.

 

[novacoresystems]

I believe I have an idea of what causes the issue. I created the interfaces file manually as per instructions online to get pfsense working with a WAN/LAN port with two different network cards. I created manually in the interface file the two bridges needed. I wonder if I could have used the proxmox GUI to do it would I be having this issue. Along with creating the interfaces needed, there are extra settings in the interfaces file that I just put in because the guy said to - but I suspect issues with that or with the routing of traffic in the proxmox box. The virtual machines that run on the proxmox have no routing issues, but the host itself I suspect does. Again here is the interfaces file and routes

interfaces file:

auto lo
iface lo inet loopback


auto vmbr0
iface vmbr0 inet static
address 192.168.1.30
netmask 255.255.255.0
gateway 192.168.1.2
bridge_ports eth0
bridge_stp off
bridge_fd 0


auto vmbr1
iface vmbr1 inet manual
bridge_ports eth1
bridge_stp off
bridge_fd 0


Routes:

root@proxmox:/etc/apt# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 vmbr0
0.0.0.0 192.168.1.2 0.0.0.0 UG 0 0 0 vmbr0


One thing I may try it setting the network interfaces back to default, rebooting, then creating the extra bridge interface in the GUI. But I don't know if that will fix it..

 

[novacoresystems]

Alright so it seems like it's just the way this machine seems to handle ANY traffic. When I do a telnet SMTP command to charter.net's mail server:


root@proxmox:~# telnet ib1.charter.net 25
Trying 216.33.127.20...
Connected to ib1.charter.net.
Escape character is '^]'.
220 imp08 charter.net ?? ESMTP server ready 20130328 052404
EHLO test.com

Nothing happens! But when I do it on my webserver box it works properly.


root@webserver:~# telnet ib1.charter.net 25
Trying 216.33.127.20...
Connected to ib1.charter.net.
Escape character is '^]'.
220 imp05 charter.net ?? ESMTP server ready 20130328 052914
EHLO test.com
250-imp05 hello [75.140.40.74], ?? pleased to meet you
250-HELP
250-SIZE 15728640
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 OK





Maybe some kind of routing issue? How strange... both of these machines are using the same firewall, gateway, switch, same subnet... everything! There's just something about this install of proxmox that is unique somehow. Anyone have a clue?

 

[novacoresystems]

So there is yet another clue.. I've done further testing and it's not really ANY traffic that this happens to.. it's only external traffic. I tested a wget to my internal webserver:

root@proxmox:~# wget http://www.novacoresystems.com
--2013-03-30 02:22:27-- http://www.novacoresystems.com/
Resolving www.novacoresystems.com... 192.168.1.6
Connecting to www.novacoresystems.com|192.168.1.6|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8745 (8.5K) [text/html]
Saving to: âindex.htmlâ


100%[======================================>] 8,745 --.-K/s in 0s


2013-03-30 02:22:27 (355 MB/s) - âindex.htmlâ


And then I tested it with an external webserver such as google.com:

root@proxmox:~# wget http://www.google.com
--2013-03-30 02:22:36-- http://www.google.com/
Resolving www.google.com... 74.125.224.112, 74.125.224.113, 74.125.224.115, ...
Connecting to www.google.com|74.125.224.112|:80... connected.
HTTP request sent, awaiting response...


So it appears to be this unit has issues receiving packets back to itself from external IPs. It will initiate the connection, but receives no response, while the webserver and all other computers/VMs in the same subnet have no issue. They are all on the same switch, virtual switches.

Anyone have a clue what is going on here?

 

[novacoresystems]

Alright, so the problem is solved. I figured it out myself eventually... : / Just in case someone else has this problem, the issue was because my firewall is virtual, and on the same box as the box I was trying to run apt-get on or any other external requests. So since my gateway was using the same NIC card/bridge I suspect this is where the problem lies. To fix the issue, I set the gateway of the proxmox interface IP to my wireless access point. Which is the only other gateway I have right now in my network besides my pfsense firewall VM.

My wireless AP uses the pfsense VM as it's gateway, so the traffic eventually ends up going to the same place, just in a indirect way.

The path it takes now:

Proxmox IP - 192.168.1.30
Linksys wireless AP - 192.168.1.3
Pfsense firewall - 192.168.1.2
WAN IP

So traffic gets pushed to my wireless AP's gateway IP, then back to the pfsense firewall on the same interface, then to my WAN. Then the incoming packets that respond to my external traffic requests go back through the same path. It adds an extra hop to all my Proxmox interface traffic, but it's only used for the web interface and updating... so whatever.

I hope this helps someone else! And since so many people looked at this post and had no clue how to fix it I hope it is educational at the least.

 

[geoffsim]

I hope this helps someone else! And since so many people looked at this post and had no clue how to fix it I hope it is educational at the least.

Thank you very much! I have been pulling my hair out with this one.
I think this thread should be pinned.

 

[novacoresystems]

No problem Took me days to figure out what was going on. But I think setting up a virtual firewall with proxmox must be rare, because seems like no one else knew what the issue was. The same sort of setup in ESXi works fine, so I had no idea that was the issue. Glad to help!

 

[kamiller42]

Can someone tell me if there is another way to fix this, i.e. without using another router?

I have pfSense setup in an OpenVZ and want all VZ and KVM's and Proxmox host on one machine with 2 NICs. I want all to go through pfSense. They all work except the Proxmox host. apt-get update, etc fails.

 

[novacoresystems]

Can someone tell me if there is another way to fix this, i.e. without using another router?

I have pfSense setup in an OpenVZ and want all VZ and KVM's and Proxmox host on one machine with 2 NICs. I want all to go through pfSense. They all work except the Proxmox host. apt-get update, etc fails.

I don't think there is any other solution. It seems the proxmox OS is unable to use a VM as a gateway. I'd say this is a bug to be honest.. but I myself have not filed a bug on this. I'll look into filing it today and see if we can get it fixed.

 

[kamiller42]

Thanks nova. If pfSense 192.168.1.1 and my wireless AP is 192.168.1.2, is this how my interfaces file should look?

# network interface settings
auto lo
iface lo inet loopback


iface eth0 inet manual


iface eth1 inet manual


auto vmbr0
iface vmbr0 inet static
address 192.168.1.5
netmask 255.255.255.0
gateway 192.168.1.2
bridge_ports eth1
bridge_stp off
bridge_fd 0
dns-nameservers 192.168.1.1
pre-up ifconfig eth1 mtu 9000


auto vmbr1
iface vmbr1 inet manual
bridge_ports eth0
bridge_stp off
bridge_fd 0

 

[novacoresystems]

Yes. Then set your wireless AP's gateway to the pfsense LAN gateway IP ( 192.168.1.1 )

It creates one more hop network-wise.. but hey it works. BTW I created a bug on this, but I'm not sure if they will be able to replicate the issue.. I'll see if we can get this resolved but for now this is a decent work around provided you have an external gateway.

 

[c0mputerking]

I was wondering if there was any information about this bug, as i do not think it is a bug at all, but rather a hardware/software issue with virtio and pfsense. I had this problem before and solved it took days, but could not remember exactly how i solved it. However in had the same problem again today and below is the fix i found

From within the pfsense gui
system >> advanced >> networking >> CHECK Disable Hardware Checksum Offloading

Obviously when they talk about broken network cards they should include virtio cards in the list too.

 

[kamiller42]

From within the pfsense gui
system >> advanced >> networking >> CHECK Disable Hardware Checksum Offloading

Obviously when they talk about broken network cards they should include virtio cards in the list too.

Genius! It worked for me. Was able to apply 2.2GB pending updates.

 

[Virtualz]

Genius! It worked for me. Was able to apply 2.2GB pending updates.

I was wondering if there was any information about this bug, as i do not think it is a bug at all, but rather a hardware/software issue with virtio and pfsense. I had this problem before and solved it took days, but could not remember exactly how i solved it. However in had the same problem again today and below is the fix i found

From within the pfsense gui
system >> advanced >> networking >> CHECK Disable Hardware Checksum Offloading

Obviously when they talk about broken network cards they should include virtio cards in the list too.

3 Years later.. Saved me after a whole morning of troubleshooting! Thanks!

 

[Dallas1978]

And another year later this also saved me !!!! I've been having this exact problem and this exact fix worked for me as well!!!

Thank you.



From within the pfsense gui
system >> advanced >> networking >> CHECK Disable Hardware Checksum Offloading

Obviously when they talk about broken network cards they should include virtio cards in the list too.

 

[macrowe]

I was wondering if there was any information about this bug, as i do not think it is a bug at all, but rather a hardware/software issue with virtio and pfsense. I had this problem before and solved it took days, but could not remember exactly how i solved it. However in had the same problem again today and below is the fix i found

From within the pfsense gui
system >> advanced >> networking >> CHECK Disable Hardware Checksum Offloading

Obviously when they talk about broken network cards they should include virtio cards in the list too.

Its times like these that I feel every forum/reddit account should have an attached donation button, 3hrs wasted this morning and you sir deserve a good portion of pennies thrown in your general direction.

 

[Mustafa57]

I was wondering if there was any information about this bug, as i do not think it is a bug at all, but rather a hardware/software issue with virtio and pfsense. I had this problem before and solved it took days, but could not remember exactly how i solved it. However in had the same problem again today and below is the fix i found

From within the pfsense gui
system >> advanced >> networking >> CHECK Disable Hardware Checksum Offloading

Obviously when they talk about broken network cards they should include virtio cards in the list too.

Awesome man, well done...working smoothly...thanks

 

[CyberKiller_MDS]

I have a similar problem on this configuration:
Router (192.168.0.1)
VirtualBox (192.168.0.100) > Proxmox (192.168.0.110) > Ubuntu/Debian (192.168.0.115), in proxmox all is ok, but in VM/Container (Debian or Ubuntu) i can ping, but can't do apt-get update, it stuck on 0% like in this case.
I use bridged mode on Virtualbox, and same on Proxmox...
Can anybody help me with this issue?