vm grayed ...

[raqdedicados]

Hi there,. can someone could explain me why the cluster stop syncing the state of the vms and nodes, is like they come on grey and i cannot see nothing. Also i remove a node and is like it has not been remove ... is happens me twice... Thanks!

 

[dietmar]

make sure pvestatd is running on all nodes (any why do you stop pvedaemon?)

 

[raqdedicados]

make sure pvestatd is running on all nodes (any why do you stop pvedaemon?)

HI . it was the solution i found so they can sync again, all this thing start when i try to restore any vps that was backed up correctly on the nfs storage, that has the correct permission coz the backup goes ok. but it can be restored, any idea also this thing '? it happens in all nodes

Dietmar, if i need to pay a suscription to fix this it will be ok. The project is awsome and i belive in this . Can u help me fixing this ?

The nfs was mounted on w2012 with write read root permission and anonymous login... so if they can be backed up ok why they can t be restored.

Any clue ?

 

[wahmed]

I can truly understand the language gap, but could you please try to explain the issue little bit more?

What did you mean by this:

The nfs was mounted on w2012 with write read root permission and anonymous login...

Are you saying when you start Restore, the sync between nodes stops and all goes blank? It is most probably because the machine where NFS is installed. When proxmox loses connection with NFS during restore due to high bandwidth usage, it sometimes goes blank. I used to happen to me when i used under powered machine with FreeNAS. What are your using for NFS Shared Storage?

 

[raqdedicados]

Brief

the backup of the containers locally stored on each proxmox node to the nfs goes ok except in one cluster that prints this message

"mkdir /mnt/pve/backup: File exists at /usr/share/perl5/PVE/Storage/NFSPlugin.pm line 139 (500)"

in the other four it connects perfect, all the proxmox nodes are 3.1 version

We find out that the restores goes at ultra slow speed an that is why it seems to hang...this we check it doing ssh on a node that connects ok going to /mnt/pve/backup and copying the tar archive of the vm to a locally dir ex /home

we re check the routes on the interfaces (coz we think this was the problem) and they are ok, so the backup goes faster why the restores cant ? they are all in same ip block with no restriction at all ? ..we do not understand ...

well we re stuck at this point

w2012 storage server running nfs has read write root permission and firewall off

Any clue ?

 

[wahmed]

Did the restore worked before and it just not working anymore? Or is it a new setup?
Can you simply copy the backup from nfs to local storage to test that transfer works?


Sent from my SGH-T989D using Tapatalk 2

 

[Raymond Burns]

Can you post a screenshot of a backup? I used to have an issue where it would backup really fast, but when I looked at the file size, there was nothing there. Surely Server 2012 is more than 40gb, so it should take more than one minute.

 

[wahmed]

. Surely Server 2012 is more than 40gb, so it should take more than one minute.

I think his nfs shared storage server itself is windows server 2012. Not that he is trying to backup a windows server 2012 vm.


Sent from my SGH-T989D using Tapatalk 2

 

[raqdedicados]

Brief

the backup of the containers locally stored on each proxmox node to the nfs goes ok except in one cluster that prints this message

"mkdir /mnt/pve/backup: File exists at /usr/share/perl5/PVE/Storage/NFSPlugin.pm line 139 (500)"

in the other four it connects perfect, all the proxmox nodes are 3.1 version

We find out that the restores goes at ultra slow speed an that is why it seems to hang...this we check it doing ssh on a node that connects ok going to /mnt/pve/backup and copying the tar archive of the vm to a locally dir ex /home

we re check the routes on the interfaces (coz we think this was the problem) and they are ok, so the backup goes faster why the restores cant ? they are all in same ip block with no restriction at all ? ..we do not understand ...

well we re stuck at this point

w2012 storage server running nfs has read write root permission and firewall off

Any clue ?

 

[Raymond Burns]

Can you perform some network benchmark tests to the NFS and from the NFS? Also, I know that someone posted some NFS tuning tools such as using tcp and nfsv3 for connection protocols. I'll look for the post, and post the link here. As of right now, I'm not sure why it takes so long. To get rid of the theory that Proxmox is the issue, just copy the backup to the local Proxmox storage and do a restore from local storage. If it functions properly, then it has to be the network, NAS, or both.

 

[raqdedicados]

Raymon thanks, i will look up . the other thing, the node that can t connecto to the nfs ? any idea ? why that message ?

 

[wahmed]

Try the following:

1. Run program like CrystalDiskmark from http://crystalmark.info/download/index-e.html#CrystalDiskMark on Windows Server 2012 NFS Storage. See how is your read/write there.

2. If possible, setup a physical machine with FreeNAS, connect it with Proxmox then try backup/restore to eliminate the cause completely that Proxmox is not issue.

3. Delete NFS Share from Proxmox then recreate a new NFS Storage connection with different name and try backup/restore.

4. Setup a physical machine with different OS like Ubuntu. Connect your Windows Server 2012 NFS Share to Ubuntu then try to copy some backup files from Windows 2012 to new Ubuntu machine.

Routes on the interfaces are not the issue at all, if they were you wont even see any NFS share.

 

[Raymond Burns]

Raymon thanks, i will look up . the other thing, the node that can t connecto to the nfs ? any idea ? why that message ?

How are you adding the NFS? When you input the IP, the path should populate with the dropdown. DO NOT type in your own path. Also, it is best practice to have separate backup Datasets apart from your images and containers. In ZFS, you should have at least two datasets, one for backups and one for operations. In Windows NFS, I have no idea what it's called, but try creating a different share and see if it mounts properly.

 

[raqdedicados]

hi . Thanks as u told me i delete the nfs and rename it to another name, now all nodes are syncing with the nfs .
The only thing we cant do is restore coz is very slow...

we do not have a free nass but we hooked it to an esxi server create a vm machine and store it there ... it performs ok

here are the results of the benchmark

 

[mir]

Your random read or writes seems extremely poor. especially for 4k block sizes and using queue depth 32 looks very bad. See attached readings from by storage which was done while service 12 VM's and 2 CT's

 

[dietmar]

What is the output of

# stat /mnt/pve/backup

 

[raqdedicados]

Diet, we change the name of the nfs volumen now works in all the nodes, the only thing is the restore ultra slow that hangs... any clue ? and thanks!

here is the output

root@raqxxxx:~# stat /mnt/pve/storage2
File: `/mnt/pve/storage2'
Size: 64 Blocks: 1 IO Block: 512 directory
Device: 6ch/108d Inode: 562949953421435 Links: 2
Access: (0700/drwx------) Uid: (4294967294/ UNKNOWN) Gid: (4294967294/ UNKNOWN)
Access: 2013-10-03 01:59:10.357957500 -0300
Modify: 2013-10-03 01:59:10.357957500 -0300
Change: 2013-10-03 01:59:10.357957500 -0300
Birth: -
root@raq99:~#