NFS outage, VMs/fs damaged

czechsys

Renowned Member
Nov 18, 2015
422
43
93
Hello,

i have testing Proxmox 4.4 currently connected via NFS3 to our storage NFS HA cluster. If we create situation when storage is out (just stopped NFS for enough time), VMs are going broken, for example one of the situation:

1] freeipa VM - services crash, fs=rw, so reboot VM -> ldap database broken
2] freeipa VM - services crash, fs=rw, restart services -> nothing broken
3] VM - fs=ro, hard reset -> fs damaged

VMs setup:
cache - default(no cache)
disk - virtual scsi, ext4

NFS server:
vm storage - option sync

I saw, that after some time usually VMs are going to fs=ro and those problems probably has some point in VM RAM caching before syncing with fs (on proxmox? on nfs?), but still, too much problems. I just dont understand, how VM hardreset can damage fs for example...It looks as unreliable even with different type network storage if there will be an outage...
Why we see so much damage and what is best way to handle outage in view of the VMs? Or, where is the main problem?
 
the storage settings you choosed (nocache on VM disk), sync on the NFS mount are good settings for a conservative setup for hosting VMs on NFS.

3] can always happen if any storage is being unavailable, and with iscsi you would have here exactly the same problem.

When the VM guest kernel notices it cannot write blocks on the disk anymore after some time, it will issue a panic message and remount the file system read only. On reboot the kernel will notice that the FS was not properly unmounted and will force a fsck.
Now it does not mean that the data of your *application* have to be inconsistent.
For instance, PostgreSQL will return a transaction as completed only when its write ahead log has been persisted to durable storage, so the database state should always be consistent.
 
Thanks for reply Manu.

So, it's as i expected. I hope, PSQL will handle it that way, but FreeIPA with Berkeley DB is going crazy - even with transaction logs or whatever it's using, ldap db broken is crazy. Well, there will be ways to handle it (IPA on local, more regular stop backups etc).

We can't use ceph now - our old virtualization setup is fully HW raided hypervisors, new storage is full SSD - but only 2 nodes. Can i expect same problem with ceph outage on VMs?

Anyway, any chance to make VMs paused, when outage? Anybody knows, how Vmware handle this? I can't use Vmware there due costs on HA requirements.
 
>o, it's as i expected. I hope, PSQL will handle it that way, but FreeIPA with Berkeley DB is going crazy - even with transaction logs or whatever it's using, ldap db broken is crazy. Well, there will be ways to handle it (IPA on local, more regular stop backups etc).

you can try to mount the ext4fs of your VM with data=journal see ext4 man page
this should improve the coherency of your FS in case of an outage, but this will slowdown the writes.
 
For instance, PostgreSQL will return a transaction as completed only when its write ahead log has been persisted to durable storage, so the database state should always be consistent.
Only true if you haven't changed the default setting for synchronous commits.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!