Connectivity Problem

jcasanova · Mar 16, 2011

Greetings,
I have 2 Proxmox servers installed, each one, in a dell 2950. One is a Master and one is Cluster. On Monday, all the VM working at the Master Proxmox failed and our services went down. All the VM's where migrated to the cluster and worked fine.
Now I want to fix the problem in the Master server. When I create a new VM, everything work fine, I'm actually installed CentOs and have all the service but one, NETWORK!
with ethtool I can see that "eth0" is connected. I have modify a hundred times the network IP, gateway, DNS and routes and I've recheck all the parameters.
If I ping to my gateway or any other device in the network segment, it won't answer. It just answer the loopback ip and the Master proxmox server IP (In wich the VM resides). Also, in the web interface of the Master proxmox, if you click the "Hardware" tab of the VM, it returns the following message:
Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator, root and inform them of the time the error occurred, and anything you might have done that may have caused the error.
[2959]ERR: 24: Error in Perl code: mkdir /mnt/pve/VM_Backup: File exists at /usr/share/perl5/PVE/Storage.pm line 2141

Apache Embperl 2.2.0 [Wed Mar 16 15:12:50 2011]

What can I do? what logs do you need to check this problem?
Thank you for your time,

Jose Casanova
Netnovation

dietmar · Mar 17, 2011

Please remove the file "/mnt/pve/VM_Backup" (whatever that is?).

Or change you storage configuration and set the path to a valid directory (instead of a file).

jcasanova · Mar 17, 2011

Thanks for the answer dietmar,
However, the solution didn't work. Which logs do you need to see?

dietmar · Mar 18, 2011

jcasanova said:
Thanks for the answer dietmar,
However, the solution didn't work. Which logs do you need to see?

what is the output of

# file /mnt/pve/

and

# file /mnt/pve/VM_Backup

dgamez · Mar 18, 2011

Hi dietmar,

I'm working with jcasanova on the issue as well.

The output you ask for is:

file /mnt/pve
/mnt/pve: directory

file /mnt/pve/VM_Backup
/mnt/pve/VM_Backup: ERROR: cannot open `/mnt/pve/VM_Backup' (No such file or directory)

/mnt/pve/VM_Backup is not available anymore because of your recommendation (Please remove the file "/mnt/pve/VM_Backup")

(whatever that is?)
R: This is a NFS Volumen mounted for backups.

Our /var/log/messages shows a repetitive:

kernel: IN=vmbr0 OUT=vmbr0 PHYSIN=eth0 PHYSOUT=vmtab108i0d0 SRC=172.17.17.10 DST=172.17.17.255 LEN=211 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=138 DPT=138 LEN=191

Not only with SRC=172.17.17.1, but with any host that interact with this server on the same network segment.

It seems to be a network problem, but no idea how to fix this.

udo · Mar 18, 2011

dgamez said:
...
Our /var/log/messages shows a repetitive:

kernel: IN=vmbr0 OUT=vmbr0 PHYSIN=eth0 PHYSOUT=vmtab108i0d0 SRC=172.17.17.10 DST=172.17.17.255 LEN=211 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=138 DPT=138 LEN=191

Not only with SRC=172.17.17.1, but with any host that interact with this server on the same network segment.

It seems to be a network problem, but no idea how to fix this.

Hi,
your ping isn't working, but you receive broadcast messages from a windows-host (netbios)?!
That you receive this packets it's normal (send to .255). But why you get an entry in messages? Do you have install an firewall?

Because the ping problem. Please post following info:

Code:

cat /etc/network/interfaces
ip route
ifconfig -a
iptables -L
cat /etc/pve/storage.cfg
mount

This should be enough.

Udo

jcasanova · Mar 18, 2011

1) cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto vmbr0
iface vmbr0 inet static
address 172.17.17.2
netmask 255.255.255.0
gateway 172.17.17.1
bridge_ports eth0
bridge_stp off
bridge_fd 0
kansas:~# ip route
172.17.17.0/24 dev vmbr0 proto kernel scope link src 172.17.17.2
default via 172.17.17.1 dev vmbr0
2) ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:22:19:25:34:29
inet6 addr: fe80::222:19ff:fe25:3429/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3081 errors:0 dropped:0 overruns:0 frame:0
TX packets:3193 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:498065 (486.3 KiB) TX bytes:2676822 (2.5 MiB)
Interrupt:16 Memory:f8000000-f8012800

eth1 Link encap:Ethernet HWaddr 00:22:19:25:34:2b
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Interrupt:16 Memory:f4000000-f4012800

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:806 errors:0 dropped:0 overruns:0 frame:0
TX packets:806 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:247393 (241.5 KiB) TX bytes:247393 (241.5 KiB)

venet0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
UP BROADCAST POINTOPOINT RUNNING NOARP MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

vmbr0 Link encap:Ethernet HWaddr 00:22:19:25:34:29
inet addr:172.17.17.2 Bcast:172.17.17.255 Mask:255.255.255.0
inet6 addr: fe80::222:19ff:fe25:3429/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3128 errors:0 dropped:0 overruns:0 frame:0
TX packets:2406 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:447819 (437.3 KiB) TX bytes:2609503 (2.4 MiB)

vmtab108i0d0 Link encap:Ethernet HWaddr 16:71:56:9c:66:d9
inet6 addr: fe80::1471:56ff:fe9c:66d9/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:46 errors:0 dropped:0 overruns:0 frame:0
TX packets:128 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:6412 (6.2 KiB) TX bytes:7770 (7.5 KiB)

3) iptables -L
Chain INPUT (policy DROP)
target prot opt source destination
ACCEPT all -- anywhere anywhere
LOG all -- loopback/8 anywhere LOG level warning
DROP all -- loopback/8 anywhere
ACCEPT all -- anywhere 255.255.255.255
ACCEPT all -- anywhere server.domain.com (changed the actual values for privacy)
ACCEPT all -- anywhere 172.17.17.255
LOG all -- anywhere anywhere LOG level warning
DROP all -- anywhere anywhere

Chain FORWARD (policy DROP)
target prot opt source destination
LOG all -- anywhere anywhere LOG level warning
DROP all -- anywhere anywhere

Chain OUTPUT (policy DROP)
target prot opt source destination
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere 255.255.255.255
ACCEPT all -- kansas.netnovation.com anywhere
ACCEPT all -- 172.17.17.255 anywhere
LOG all -- anywhere anywhere LOG level warning
DROP all -- anywhere anywhere
kansas:~# cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
content images,iso,vztmpl,rootdir

kansas:~# mount
/dev/mapper/pve-root on / type ext3 (rw,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
procbususb on /proc/bus/usb type usbfs (rw)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
/dev/mapper/pve-data on /var/lib/vz type ext3 (rw)
/dev/sda1 on /boot type ext3 (rw)

Also today the server went down, this is the /var/log/messages of that moment:

Mar 18 11:37:17 kansas kernel: IN=vmbr0 OUT=vmbr0 PHYSIN=eth0 PHYSOUT=vmtab108i0d0 SRC=172.17.17.156 DST=172.17.17.255 LEN=169 TOS=0x00 PREC=0x00 TTL=64 ID=14097 PROTO=UDP SPT=17500 DPT=17500 LEN=149
Mar 18 11:46:12 kansas kernel: vmbr0: port 2(vmtab108i0d0) entering disabled state
Mar 18 11:46:12 kansas kernel: vmbr0: port 1(eth0) entering disabled state
Mar 18 11:46:12 kansas kernel: device eth0 left promiscuous mode
Mar 18 11:46:12 kansas kernel: vmbr0: port 1(eth0) entering disabled state
Mar 18 11:46:12 kansas kernel: vmbr0: port 2(vmtab108i0d0) entering disabled state
Mar 18 11:46:12 kansas kernel: device eth0 entered promiscuous mode
Mar 18 11:46:12 kansas kernel: bnx2: eth0: using MSI
Mar 18 11:46:12 kansas kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
Mar 18 11:46:15 kansas kernel: bnx2: eth0 NIC Copper Link is Up, 1000 Mbps full duplex, receive & transmit flow control ON
Mar 18 11:46:15 kansas kernel: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Mar 18 11:46:15 kansas kernel: vmbr0: port 1(eth0) entering forwarding state
Mar 18 17:06:50 kansas kernel: imklog 3.18.6, log source = /proc/kmsg started.
Mar 18 17:06:50 kansas rsyslogd: [origin software="rsyslogd" swVersion="3.18.6" x-pid="2369" x-info="http://www.rsyslog.com"] restart
Mar 18 17:06:50 kansas kernel: Linux version 2.6.32-4-pve (unknown) (root@oahu) (gcc version 4.3.2 (Debian 4.3.2-1.1) ) #1 SMP Fri Nov 26 06:42:28 CET 2010
... (Boot) ...
Mar 18 17:07:02 kansas kernel: device vmtab108i0d0 entered promiscuous mode
Mar 18 17:07:02 kansas kernel: vmbr0: port 2(vmtab108i0d0) entering forwarding state
Mar 18 17:07:03 kansas kernel: warning: `ntpd' uses 32-bit capabilities (legacy support in use)
Mar 18 17:07:21 kansas kernel: IN=vmbr0 OUT=vmbr0 PHYSIN=eth0 PHYSOUT=vmtab108i0d0 SRC=172.17.17.156 DST=172.17.17.255 LEN=169 TOS=0x00 PREC=0x00 TTL=64 ID=35160 PROTO=UDP SPT=17500 DPT=17500 LEN=149
Mar 18 17:07:43 kansas kernel: IN=vmbr0 OUT=vmbr0 PHYS IN=eth0 PHYSOUT=vmtab108i0d0 SRC=172.17.17.10 DST=239.255.255.250 LEN=331 TOS=0x00 PREC=0x00 TTL=4 ID=0 DF PROTO=UDP SPT=1900 DPT=1900 LEN=311
Mar 18 17:07:43 kansas kernel: IN=vmbr0 OUT=vmbr0 PHYSIN=eth0 PHYSOUT=vmtab108i0d0 SRC=172.17.17.10 DST=239.255.255.250 LEN=322 TOS=0x00 PREC=0x00 TTL=4 ID=0 DF PROTO=UDP SPT=1900 DPT=1900 LEN=302
Mar 18 17:07:43 kansas kernel: IN=vmbr0 OUT=vmbr0 PHYSIN=eth0 PHYSOUT=vmtab108i0d0 SRC=172.17.17.10 DST=239.255.255.250 LEN=374 TOS=0x00 PREC=0x00 TTL=4 ID=0 DF PROTO=UDP SPT=1900 DPT=1900 LEN=354

Thanks for the help!!!

dgamez · Mar 19, 2011

The way I fixed it...

There were some Scripts on /etc/ipmasq/rules that established some iptables rules on Debian server. We never put those in there, I thing it was a hack.

So basically I deleted those rules and replaced them with new iptable rules.

/* List iptables rules */
iptables -L
Chain INPUT (policy DROP) destination
Chain FORWARD (policy DROP) destination
Chain OUTPUT (policy DROP) destination

/* Flush iptables rules */
/* Be careful with this command, you'll loose conection to your server (I managed it by reconnecting via DELL RAC) */
/* If you have physical access to your server, then you'll be fine doing this */
iptables -F

/* Delete any ipmasq rule */
/* Make any backup you need before this step */
rm -rf /etc/ipmasq/rules/*

/* Create new ipmasq rule */
/* The following disables the firewall completely, be aware of this! */

vi /etc/ipmasq/rules/F00chain.rul

#:
#: **********************************************************
#: *** FORWARD CHAIN ***
#: **********************************************************
#:
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

/* Give execution perms to script */
chmod +x /etc/ipmasq/rules/F00chain.rul

/* Start ipmasq */
ipmasq

/* You're Done! */

udo · Mar 21, 2011

dgamez said:
The way I fixed it...

There were some Scripts on /etc/ipmasq/rules that established some iptables rules on Debian server. We never put those in there, I thing it was a hack.

So basically I deleted those rules and replaced them with new iptable rules.

/* List iptables rules */
iptables -L
Chain INPUT (policy DROP) destination
Chain FORWARD (policy DROP) destination
Chain OUTPUT (policy DROP) destination

/* Flush iptables rules */
/* Be careful with this command, you'll loose conection to your server (I managed it by reconnecting via DELL RAC) */
/* If you have physical access to your server, then you'll be fine doing this */
iptables -F

/* Delete any ipmasq rule */
/* Make any backup you need before this step */
rm -rf /etc/ipmasq/rules/*

/* Create new ipmasq rule */
/* The following disables the firewall completely, be aware of this! */

vi /etc/ipmasq/rules/F00chain.rul

#:
#: **********************************************************
#: *** FORWARD CHAIN ***
#: **********************************************************
#:
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

/* Give execution perms to script */
chmod +x /etc/ipmasq/rules/F00chain.rul

/* Start ipmasq */
ipmasq

/* You're Done! */

Hi,
delay from my side because i was on journey (chemnitzer linux-day).

Your firewall-script is responsible for the ping-issue but that the host went down must be another problem (ram, powersupply??).

Udo

dgamez · Mar 22, 2011

Hey udo!
Thanks for your concern...

Actually I did reboot the server that day, is just that jcasanova didn't notice it. So services are doing fine!

Search

Search

Connectivity Problem

jcasanova

New Member

dietmar

Proxmox Staff Member

jcasanova

New Member

dietmar

Proxmox Staff Member

dgamez

New Member

udo

Distinguished Member

jcasanova

New Member

dgamez

New Member

udo

Distinguished Member

dgamez

New Member