Proxmox cluster one node hang frequently

emunt6 · Apr 6, 2024

premjith_r said:
>> We are using iSCSI SAN and not configured any zones

Hi!

This is my last idea for the problem:
> Proxmox node: Turn off all Offload cap. on Network Cards - The Offload driver maybe "corrupting" the network packets.
( I always disable Offload on all Network card/Interface. )

FIX:

Code:

 /etc/network/interfaces

auto eth0
iface eth0 inet manual
post-up /sbin/ethtool -K eth0 rx off tx off sg off tso off ufo off gso off gro off lro off

You can run the command manually, doesn't need to restart the Proxmox server it will apply immediately.

More info:

Code:

https://michael.mulqueen.me.uk/2018/08/disable-offloading-netplan-ubuntu/

bbgeek17 · Apr 8, 2024

@premjith_r what do these commands return?
multipath -ll
pvs
vgs
lvs

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

premjith_r · Apr 9, 2024

emunt6 said:
Hi!

This is my last idea for the problem:
> Proxmox node: Turn off all Offload cap. on Network Cards - The Offload driver maybe "corrupting" the network packets.
( I always disable Offload on all Network card/Interface. )

FIX:

Code:

/etc/network/interfaces auto eth0 iface eth0 inet manual post-up /sbin/ethtool -K eth0 rx off tx off sg off tso off ufo off gso off gro off lro off

You can run the command manually, doesn't need to restart the Proxmox server it will apply immediately.

More info:

Code:

https://michael.mulqueen.me.uk/2018/08/disable-offloading-netplan-ubuntu/

Thanks for this suggestion. Shall try this and update.

premjith_r · Apr 9, 2024

bbgeek17 said:
@premjith_r what do these commands return?
multipath -ll
pvs
vgs
lvs

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Please note following out put,

Command : multipath -ll

Output >>

mpatha (3600c0ff000530ac4c81d646201000000) dm-6 HPE,MSA 1060 iSCSI

size=28T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw

|-+- policy='round-robin 0' prio=50 status=active

| `- 3:0:0:1 sdd 8:48 active ready running

|-+- policy='round-robin 0' prio=50 status=enabled

| `- 4:0:0:1 sdh 8:112 active ready running

|-+- policy='round-robin 0' prio=10 status=enabled

| `- 2:0:0:1 sde 8:64 active ready running

`-+- policy='round-robin 0' prio=10 status=enabled

`- 5:0:0:1 sdj 8:144 active ready running

mpathb (3600c0ff0005302c3f51d646201000000) dm-7 HPE,MSA 1060 iSCSI

size=14T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw

|-+- policy='round-robin 0' prio=50 status=active

| `- 2:0:0:2 sdg 8:96 active ready running

|-+- policy='round-robin 0' prio=50 status=enabled

| `- 5:0:0:2 sdk 8:160 active ready running

|-+- policy='round-robin 0' prio=10 status=enabled

| `- 3:0:0:2 sdf 8:80 active ready running

`-+- policy='round-robin 0' prio=10 status=enabled

`- 4:0:0:2 sdi 8:128 active ready running

Command : pvs

PV VG Fmt Attr PSize PFree

/dev/mapper/mpatha iscsi_sana lvm2 a-- 27.90t 2.87t

/dev/mapper/mpathb iscsi_sanb lvm2 a-- 13.94t 11.90t

/dev/sda3 pve lvm2 a-- <446.60g 15.99g

/dev/sdb ceph-bfed1016-bb10-4c15-9898-71790bb510be lvm2 a-- <4.37t 0

Command : vgs

vgs

VG #PV #LV #SN Attr VSize VFree

ceph-bfed1016-bb10-4c15-9898-71790bb510be 1 1 0 wz--n- <4.37t 0

iscsi_sana 1 134 0 wz--n- 27.90t 2.87t

iscsi_sanb 1 9 0 wz--n- 13.94t 11.90t

pve 1 3 0 wz--n- <446.60g 15.99g

Command lvs

lvs

LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert

osd-block-bfd8a4e0-d228-4b25-936b-16d4e31087e4 ceph-bfed1016-bb10-4c15-9898-71790bb510be -wi-ao---- <4.37t

vm-102-disk-0 iscsi_sana -wi-a----- 150.00g

vm-103-disk-0 iscsi_sana -wi-a----- 150.00g

vm-104-disk-0 iscsi_sana -wi-a----- 150.00g

vm-106-disk-0 iscsi_sana -wi-a----- 250.00g

vm-111-disk-0 iscsi_sana -wi-a----- 40.00g

vm-112-disk-0 iscsi_sana -wi-a----- 80.00g

vm-113-disk-0 iscsi_sana -wi-a----- 150.00g

vm-114-disk-0 iscsi_sana -wi-a----- 150.00g

vm-115-disk-0 iscsi_sana -wi-a----- 150.00g

vm-116-disk-0 iscsi_sana -wi-a----- 100.00g

vm-117-disk-0 iscsi_sana -wi-a----- 250.00g

vm-118-disk-0 iscsi_sana -wi-a----- 250.00g

vm-119-disk-0 iscsi_sana -wi-a----- 250.00g

vm-220001-disk-0 iscsi_sana -wi-a----- 150.00g

vm-220002-disk-0 iscsi_sana -wi-a----- 150.00g

vm-220003-disk-0 iscsi_sana -wi-a----- 150.00g

vm-220004-disk-0 iscsi_sana -wi-a----- 150.00g

vm-220005-disk-0 iscsi_sana -wi-a----- 250.00g

vm-220006-disk-0 iscsi_sana -wi-a----- 150.00g

vm-220016-disk-0 iscsi_sana -wi-a----- 150.00g

vm-220017-disk-0 iscsi_sana -wi-a----- 250.00g

vm-220019-disk-0 iscsi_sana -wi-a----- 150.00g

vm-220031-disk-0 iscsi_sana -wi-ao---- 150.00g

vm-220032-disk-0 iscsi_sana -wi-a----- 150.00g

vm-220033-disk-0 iscsi_sana -wi-ao---- 150.00g

vm-220034-disk-0 iscsi_sana -wi-a----- 150.00g

vm-220035-disk-0 iscsi_sana -wi-ao---- 150.00g

vm-220036-disk-0 iscsi_sana -wi-ao---- 150.00g

vm-220037-disk-0 iscsi_sana -wi-a----- 150.00g

vm-220038-disk-0 iscsi_sana -wi-a----- 300.00g

vm-220039-disk-0 iscsi_sana -wi-a----- 150.00g

vm-220040-disk-0 iscsi_sana -wi-a----- 250.00g

vm-220041-disk-0 iscsi_sana -wi-a----- 150.00g

vm-220042-disk-0 iscsi_sana -wi-a----- 270.00g

vm-220043-disk-0 iscsi_sana -wi-a----- 310.00g

vm-220064-disk-0 iscsi_sana -wi-a----- 150.00g

vm-220065-disk-0 iscsi_sana -wi-ao---- 150.00g

vm-220066-disk-0 iscsi_sana -wi-a----- 150.00g

vm-220067-disk-0 iscsi_sana -wi-a----- 150.00g

vm-220068-disk-0 iscsi_sana -wi------- 150.00g

vm-220070-disk-0 iscsi_sana -wi-ao---- 150.00g

vm-220071-disk-0 iscsi_sana -wi-ao---- 150.00g

vm-220072-disk-0 iscsi_sana -wi-ao---- 150.00g

vm-220073-disk-0 iscsi_sana -wi-ao---- 150.00g

vm-220074-disk-0 iscsi_sana -wi-a----- 250.00g

vm-220075-disk-0 iscsi_sana -wi-a----- 300.00g

vm-220076-disk-0 iscsi_sana -wi-a----- 150.00g

vm-220077-disk-0 iscsi_sana -wi-a----- 500.00g

vm-220079-disk-0 iscsi_sana -wi-a----- 150.00g

vm-220080-disk-0 iscsi_sana -wi-a----- 250.00g

vm-220081-disk-0 iscsi_sana -wi-ao---- 150.00g

vm-220082-disk-0 iscsi_sana -wi-ao---- 250.00g

vm-220083-disk-0 iscsi_sana -wi-ao---- 150.00g

vm-220084-disk-0 iscsi_sana -wi-a----- 250.00g

vm-220085-disk-0 iscsi_sana -wi-ao---- 150.00g

vm-220086-disk-0 iscsi_sana -wi-ao---- 150.00g

vm-220087-disk-0 iscsi_sana -wi-ao---- 150.00g

vm-220088-disk-0 iscsi_sana -wi-ao---- 150.00g

vm-220089-disk-0 iscsi_sana -wi-ao---- 250.00g

vm-220091-disk-0 iscsi_sana -wi-a----- 250.00g

vm-220092-disk-0 iscsi_sana -wi-a----- 150.00g

vm-220093-disk-0 iscsi_sana -wi-a----- 150.00g

vm-220094-disk-0 iscsi_sana -wi-ao---- 150.00g

vm-220095-disk-0 iscsi_sana -wi-ao---- 150.00g

vm-220096-disk-0 iscsi_sana -wi-ao---- 150.00g

vm-220097-disk-0 iscsi_sana -wi-ao---- 150.00g

vm-220098-disk-0 iscsi_sana -wi-ao---- 150.00g

vm-220099-disk-0 iscsi_sana -wi-a----- 150.00g

vm-220100-disk-0 iscsi_sana -wi-ao---- 250.00g

vm-220101-disk-0 iscsi_sana -wi-a----- 250.00g

vm-220102-disk-0 iscsi_sana -wi-a----- 150.00g

vm-220103-disk-0 iscsi_sana -wi-a----- 150.00g

vm-220104-disk-0 iscsi_sana -wi-a----- 150.00g

vm-220105-disk-0 iscsi_sana -wi-a----- 150.00g

vm-220106-disk-0 iscsi_sana -wi-a----- 150.00g

vm-220107-disk-0 iscsi_sana -wi-a----- 250.00g

vm-220108-disk-0 iscsi_sana -wi-a----- 250.00g

vm-221006-disk-0 iscsi_sanb -wi-a----- 150.00g

vm-221017-disk-0 iscsi_sanb -wi-a----- 1000.00g

data pve twi-a-tz-- <320.07g 0.00 0.52

root pve -wi-ao---- 96.00g

swap pve -wi-ao---- 8.00g

premjith_r · Apr 9, 2024

Small change done on /etc/multipath.conf, after last hang which happened two days back

preferredip=192.168.2.2

to

preferredip=192.168.2.3

We have two controller on SAN. So gave second controller IP to node2, which making hang issue frequently.

Now monitoring the system.

Two node using controller A IP and one node using controller B IP for SAN communication. Can we use proxmox cluster like that?

bbgeek17 · Apr 9, 2024

premjith_r said:
We have two controller on SAN. So gave second controller IP to node2, which making hang issue frequently.

Now monitoring the system.

Two node using controller A IP and one node using controller B IP for SAN communication. Can we use proxmox cluster like that?

That depends on your SAN really. Do you have to assign a LUN to specific controller so that controller is "active"? Do you have to create second LUN on second controller to truly balance it? Is it using ALUA to move LUN ownership back and forth? Is it a single LUN that is being bounced between controllers for some reason due to access from multiple hosts/paths/controllers and exposes some sort of firmware issue in your SAN?

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

premjith_r · Apr 10, 2024

bbgeek17 said:
That depends on your SAN really. Do you have to assign a LUN to specific controller so that controller is "active"? Do you have to create second LUN on second controller to truly balance it? Is it using ALUA to move LUN ownership back and forth? Is it a single LUN that is being bounced between controllers for some reason due to access from multiple hosts/paths/controllers and exposes some sort of firmware issue in your SAN?

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

We have two controllers and two LUNs on SAN. Both controller will handle both LUNs as active mode. So we can access both LUNs through any of the controller IP.

Dont think we have done any such specific configurations you have mentioned. I can share any other config file needed.

After changing to preferredip=192.168.2.3, the Proxmox Node2 has been running without hang since Apr 6th
Though it's early to comment, but not sure, how changing this above config (preferredip) can stabilize the I/O operations on Node2.

Search

Search

Proxmox cluster one node hang frequently

emunt6

Active Member

bbgeek17

Distinguished Member

premjith_r

New Member

premjith_r

New Member

premjith_r

New Member

bbgeek17

Distinguished Member

premjith_r

New Member