Cloning VM via NFS - VM fs damaged

czechsys

Renowned Member
Nov 18, 2015
506
56
93
Hi,

i found today, that cloning VM damaged fs on the new created VM. The source VM was offline. The storage is mounted via NFS4. The clonning process (12G raw, network 10Gbps):
Code:
May  9 11:36:11 pve-01 pvedaemon[1987]: <root@pam> starting task UPID:pve-01:00000B2B:0000C598:59118D8B:qmclone:200:root@pam:
May  9 11:36:31 pve-01 kernel: [  526.463678] nfs: server NFS_IP not responding, still trying
May  9 11:36:32 pve-01 kernel: [  526.572852] nfs: server NFS_IP OK
May  9 11:36:32 pve-01 pvedaemon[1987]: <root@pam> end task UPID:pve-01:00000B2B:0000C598:59118D8B:qmclone:200:root@pam: OK
The error is almost on end of the clonning.

Tue May 9 11:36:08 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:09 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:10 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:11 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:12 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:13 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:14 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:15 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:16 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:17 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:18 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:19 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:20 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:21 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:22 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:23 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:24 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:25 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:26 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:27 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:28 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:29 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:30 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:31 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:32 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:33 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:34 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:35 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:36 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:37 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:38 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:39 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:40 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:41 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24
Tue May 9 11:36:42 CEST 2017
Export list for NFS_IP: /srv/nfs/export3 C.C.C.C/32,A.A.A.A/24 /srv/nfs/export2 A.A.A.A/24 /srv/nfs B.B.B.B/22,A.A.A.A/24

Server rpc stats:
calls badcalls badfmt badauth badclnt
27138 0 0 0 0

Server nfs v4:
null compound
1 0% 27137 99%

Server nfs v4 operations:
op0-unused op1-unused op2-future access close
0 0% 0 0% 0 0% 197 0% 10 0%
commit create delegpurge delegreturn getattr
3 0% 0 0% 0 0% 5 0% 9383 10%
getfh link lock lockt locku
18 0% 0 0% 0 0% 0 0% 0 0%
lookup lookup_root nverify open openattr
16 0% 0 0% 0 0% 10 0% 0 0%
open_conf open_dgrd putfh putpubfh putrootfh
0 0% 0 0% 27074 30% 0 0% 3 0%
read readdir readlink remove rename
17707 19% 19 0% 0 0% 1 0% 0 0%
renew restorefh savefh secinfo setattr
0 0% 0 0% 0 0% 0 0% 2 0%
setcltid setcltidconf verify write rellockowner
0 0% 0 0% 0 0% 8294 9% 0 0%
bc_ctl bind_conn exchange_id create_ses destroy_ses
0 0% 0 0% 2 0% 3 0% 1 0%
free_stateid getdirdeleg getdevinfo getdevlist layoutcommit
0 0% 0 0% 0 0% 0 0% 0 0%
layoutget layoutreturn secinfononam sequence set_ssv
0 0% 0 0% 0 0% 27132 30% 0 0%
test_stateid want_deleg destroy_clid reclaim_comp allocate
0 0% 0 0% 0 0% 2 0% 0 0%
copy copy_notify deallocate ioadvise layouterror
0 0% 0 0% 0 0% 0 0% 0 0%
layoutstats offloadcancel offloadstatus readplus seek
0 0% 0 0% 0 0% 0 0% 0 0%
write_same
0 0%

proxmox-ve: 4.4-87 (running kernel: 4.4.59-1-pve)
pve-manager: 4.4-13 (running version: 4.4-13/7ea56165)
pve-kernel-4.4.35-1-pve: 4.4.35-77
pve-kernel-4.4.59-1-pve: 4.4.59-87
pve-kernel-4.4.44-1-pve: 4.4.44-84
pve-kernel-4.4.49-1-pve: 4.4.49-86
pve-kernel-4.4.40-1-pve: 4.4.40-82
lvm2: 2.02.116-pve3
corosync-pve: 2.4.2-2~pve4+1
libqb0: 1.0.1-1
pve-cluster: 4.0-49
qemu-server: 4.0-110
pve-firmware: 1.1-11
libpve-common-perl: 4.0-94
libpve-access-control: 4.0-23
libpve-storage-perl: 4.0-76
pve-libspice-server1: 0.12.8-2
vncterm: 1.3-2
pve-docs: 4.4-4
pve-qemu-kvm: 2.7.1-4
pve-container: 1.0-99
pve-firewall: 2.0-33
pve-ha-manager: 1.0-40
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u3
lxc-pve: 2.0.7-4
lxcfs: 2.0.6-pve1
criu: 1.6.0-1
novnc-pve: 0.5-9
smartmontools: 6.5+svn4324-1~pve80
zfsutils: 0.6.5.9-pve15~bpo80
openvswitch-switch: 2.6.0-2

Network is on pve nodes via openvswitch, on storage nodes standard networking (vlan based). Storage nodes are debian9. All nodes are LACPed via switches. Bandwitch used is between 5-10Gbps for NFS network.

Any tip how to check/debug why proxmox is "losing" connection?
 
Well, the proxmox error can be with agressive nfs client setup: timeo=14, retrans=2.
But still, i have too often damaged filesystem in VMs...and still with nocache+sync everywhere...