Re: iSCSI storage offline
Hi tom,
i have run the command on all the nodes and its not made any difference although one of my multipath active paths was reporting a fault - ive now manually switched the controller that the virtial disk on the SAN is using and the failed multipath is now in the 'ghost' connections:
NEXUS (3690b11c00008da2c0000293351162510) dm-5 DELL,MD32xxi
size=2.0T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=3 status=enabled
| |- 12:0:0:0 sdq 65:0 failed faulty running
| |- 6:0:0:0 sdd 8:48 active ghost running
| |- 8:0:0:0 sdj 8:144 active ghost running
| `- 10:0:0:0 sdn 8:208 active ghost running
`-+- policy='round-robin 0' prio=24 status=active
|- 7:0:0:0 sde 8:64 active ready running
|- 5:0:0:0 sdf 8:80 active ready running
|- 9:0:0:0 sdl 8:176 active ready running
`- 11:0:0:0 sdp 8:240 active ready running
NEXUS-CMS (3690b11c00008daa2000021f951161ae3) dm-6 DELL,MD32xxi
size=2.0T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
|-+- policy='round-robin 0' prio=24 status=active
| |- 5:0:0:1 sdh 8:112 active ready running
| |- 7:0:0:1 sdi 8:128 active ready running
| |- 9:0:0:1 sdm 8:192 active ready running
| `- 11:0:0:1 sdr 65:16 active ready running
`-+- policy='round-robin 0' prio=3 status=enabled
|- 12:0:0:1 sds 65:32 failed faulty running
|- 6:0:0:1 sdg 8:96 active ghost running
|- 8:0:0:1 sdk 8:160 active ghost running
`- 10:0:0:1 sdo 8:224 active ghost running
All my VMs are still running and there is traffic going through all the NICs - is there anything i need to run on the server to recheck the connection in proxmox UI? so that it can stop the error - i suspect a faulty cable on the SAN at the datacentre although ALL SAN IPs are responding to pings and all ports are showing live on the switches.
This below is worrying though - i rebooted this box and it has lost the 'ghost' connections in multipath - but it can ping ALL Ip addresses for the SAN
root@enterprise:~# multipath -ll
NEXUS-CMS (3690b11c00008daa2000021f951161ae3) dm-4 DELL,MD32xxi
size=2.0T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
`-+- policy='round-robin 0' prio=24 status=active
|- 8:0:0:1 sdk 8:160 active ready running
|- 3:0:0:1 sdd 8:48 active ready running
|- 4:0:0:1 sde 8:64 active ready running
`- 7:0:0:1 sdi 8:128 active ready running
IMAGES (360014052e1172000370d002000000000) dm-5 LIO-ORG,FILEIO
size=512G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 6:0:0:0 sdf 8:80 active ready running
NEXUS (3690b11c00008da2c0000293351162510) dm-3 DELL,MD32xxi
size=2.0T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
`-+- policy='round-robin 0' prio=24 status=active
|- 8:0:0:0 sdj 8:144 active ready running
|- 3:0:0:0 sdb 8:16 active ready running
|- 4:0:0:0 sdc 8:32 active ready running
`- 7:0:0:0 sdh 8:112 active ready running
IMAGES-TWO (360014052e1172000370d003000000000) dm-6 LIO-ORG,FILEIO
size=512G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 5:0:0:0 sdg 8:96 active ready running
root@enterprise:~# ping 192.168.130.101
PING 192.168.130.101 (192.168.130.101) 56(84) bytes of data.
64 bytes from 192.168.130.101: icmp_req=1 ttl=64 time=0.226 ms
64 bytes from 192.168.130.101: icmp_req=2 ttl=64 time=0.213 ms
64 bytes from 192.168.130.101: icmp_req=3 ttl=64 time=0.217 ms
64 bytes from 192.168.130.101: icmp_req=4 ttl=64 time=0.210 ms
^C
--- 192.168.130.101 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2999ms
rtt min/avg/max/mdev = 0.210/0.216/0.226/0.015 ms
root@enterprise:~# ping 192.168.130.102
PING 192.168.130.102 (192.168.130.102) 56(84) bytes of data.
64 bytes from 192.168.130.102: icmp_req=1 ttl=64 time=0.226 ms
64 bytes from 192.168.130.102: icmp_req=2 ttl=64 time=0.212 ms
64 bytes from 192.168.130.102: icmp_req=3 ttl=64 time=0.205 ms
64 bytes from 192.168.130.102: icmp_req=4 ttl=64 time=0.209 ms
^C
--- 192.168.130.102 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3000ms
rtt min/avg/max/mdev = 0.205/0.213/0.226/0.007 ms
root@enterprise:~# ping 192.168.131.101
PING 192.168.131.101 (192.168.131.101) 56(84) bytes of data.
64 bytes from 192.168.131.101: icmp_req=1 ttl=64 time=1.13 ms
64 bytes from 192.168.131.101: icmp_req=2 ttl=64 time=0.219 ms
64 bytes from 192.168.131.101: icmp_req=3 ttl=64 time=0.213 ms
64 bytes from 192.168.131.101: icmp_req=4 ttl=64 time=0.204 ms
^C
--- 192.168.131.101 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3000ms
rtt min/avg/max/mdev = 0.204/0.443/1.139/0.402 ms
root@enterprise:~# ping 192.168.131.102
PING 192.168.131.102 (192.168.131.102) 56(84) bytes of data.
64 bytes from 192.168.131.102: icmp_req=1 ttl=64 time=0.497 ms
64 bytes from 192.168.131.102: icmp_req=2 ttl=64 time=0.210 ms
64 bytes from 192.168.131.102: icmp_req=3 ttl=64 time=0.214 ms
^C
--- 192.168.131.102 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.210/0.307/0.497/0.134 ms
root@enterprise:~# ping 192.168.132.101
PING 192.168.132.101 (192.168.132.101) 56(84) bytes of data.
64 bytes from 192.168.132.101: icmp_req=1 ttl=64 time=1.18 ms
64 bytes from 192.168.132.101: icmp_req=2 ttl=64 time=0.220 ms
^C
--- 192.168.132.101 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.220/0.704/1.188/0.484 ms
root@enterprise:~# ping 192.168.132.102
PING 192.168.132.102 (192.168.132.102) 56(84) bytes of data.
64 bytes from 192.168.132.102: icmp_req=1 ttl=64 time=0.505 ms
64 bytes from 192.168.132.102: icmp_req=2 ttl=64 time=0.217 ms
64 bytes from 192.168.132.102: icmp_req=3 ttl=64 time=0.219 ms
64 bytes from 192.168.132.102: icmp_req=4 ttl=64 time=0.214 ms
^C
--- 192.168.132.102 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2999ms
rtt min/avg/max/mdev = 0.214/0.288/0.505/0.126 ms
root@enterprise:~# ping 192.168.133.101
PING 192.168.133.101 (192.168.133.101) 56(84) bytes of data.
64 bytes from 192.168.133.101: icmp_req=1 ttl=64 time=1.39 ms
64 bytes from 192.168.133.101: icmp_req=2 ttl=64 time=0.210 ms
64 bytes from 192.168.133.101: icmp_req=3 ttl=64 time=0.210 ms
^C
--- 192.168.133.101 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.210/0.604/1.394/0.558 ms
root@enterprise:~# ping 192.168.133.102
PING 192.168.133.102 (192.168.133.102) 56(84) bytes of data.
64 bytes from 192.168.133.102: icmp_req=1 ttl=64 time=0.525 ms
64 bytes from 192.168.133.102: icmp_req=2 ttl=64 time=0.215 ms
64 bytes from 192.168.133.102: icmp_req=3 ttl=64 time=0.205 ms
^C
--- 192.168.133.102 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.205/0.315/0.525/0.148 ms
