OpenVZ over HA duplicated NAS/SAN based on NFS

dbloquel

New Member
Mar 28, 2014
4
0
1
Hi,

I am currently trying to make proxmox work properly with an HA NAS and openVZ.

My ha nas is using Heartbeat (for failover), DRBD (for replication), NFS and is using a virtual IP adress which will be used to switch between both node for failover in case the primary node is down.
The nas has been added to proxmox using this virtual IP address so that it always reaches the functionnal Nas.

My Openvz containers are stored on the ha nas which is mounted by proxmox, the whole system is working well until a failover occurs which is a real problem given that this is the point of the whole thing.

When a node is down and the failover starts, if one of my container is writting something on the nas the whole system will be locked (if my containers are not writting on the disk the switch is done without any problem).

After such a failover my container are not responding, shutdown is not working (from proxmox gui or even using vzctl stop --fast) and the GUI shows that the nas is not responding anymore.

I think that when a failover occurs proxmox cannot switch from the failed nas to the functionnal one because a container is locking the device. However the container is not responding because it's stored on the nas and since proxmox is trying to reach a nas that is not working anymore the container cannot be unlock which makes a loop.

Here is the error that proxmox syslog is giving me:
Mar 28 11:53:41 myserver pvedaemon[192777]: WARNING: command 'df -P -B 1 /mnt/pve/mynas' failed: got timeout

This one occurs when I am trying to shutdown the container with vzctl stop --fast:
Mar 28 12:02:30 virt3 pvedaemon[202530]: command '/usr/sbin/vzctl exec 10025 /bin/cat /proc/net/dev' failed: exit code 8

dmesg output on proxmox server:

INFO: task tar:55915 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
tar D ffff8810759992f0 0 55915 55912 10025 0x00000000
ffff880f91e6fc78 0000000000000086 ffff880f91e6fbd8 ffffffffa0235fe0
ffff880f91e6fc68 ffffffffa02cdd86 ffff8810759992f0 ffffffffa030b7c8
ffff88104d92a428 ffff88104d92a478 ffff880f91e6ffd8 ffff880f91e6ffd8
Call Trace:
[<ffffffffa0235fe0>] ? rpc_put_task+0x10/0x20 [sunrpc]
[<ffffffffa02cdd86>] ? nfs_initiate_write+0x126/0x1c0 [nfs]
[<ffffffff8112a900>] ? sync_page+0x0/0x50
[<ffffffff815416a3>] io_schedule+0x73/0xc0
[<ffffffff8112a93b>] sync_page+0x3b/0x50
[<ffffffff815425b0>] __wait_on_bit+0x60/0x90
[<ffffffff8112aad0>] wait_on_page_bit+0x80/0x90
[<ffffffff8109ba10>] ? wake_bit_function+0x0/0x50
[<ffffffff8112ae0a>] wait_on_page_writeback_range.part.36+0xea/0x180
[<ffffffff8112b7c5>] wait_on_page_writeback_range+0x15/0x20
[<ffffffff8112c3ef>] filemap_fdatawait+0x2f/0x40
[<ffffffff8112c444>] filemap_write_and_wait+0x44/0x60
[<ffffffffa02c0024>] nfs_getattr+0x104/0x120 [nfs]
[<ffffffff811a4447>] vfs_getattr+0x37/0x70
[<ffffffff811a5955>] vfs_fstat+0x35/0x60
[<ffffffff811a5bfa>] sys_newfstat+0x1a/0x40
[<ffffffff8119ffda>] ? sys_write+0x4a/0x90
[<ffffffff8100b182>] system_call_fastpath+0x16/0x1b

My /etc/pve/storage.cfg

nfs: mynas
path /mnt/pve/blgnas
server 172.x.x.x (my nas virtual Ip adress)
export /drbd/myshare
options vers=3,rsize=1048576,wsize=1048576,nolock,noacl,ac,actimeo=10800,nocto,noatime,nodiratime,timeo=50
content images,iso,vztmpl,rootdir,backup
maxfiles 1

I really need this HA NAS to work with proxmox and my containers but currently I am running out of ideas.

Do you think that it could work with another file sharing system such as iSCSI?

I hope that someone here could gives me some answer or clue about what is going wrong and how I could fix it.

NB: Sorry for my english, I hope it is good enough to be understood.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!