[SOLVED] can't login into website anymore

O

offerlam

Renowned Member
Dec 30, 2012
218
0
81
Denmark
Hi all,

It started by me removing the backup share and adding it again from another ip via NFS..

after doing that all of a sudden my prox website showed all nodes as fenced and all VMs as offline.. also it couldn't show any content from any storage..

but there were still access and VMs was still running..

now closing and opening browsers i can't even login anymore.. I do still have cli access..

any suggestions??

Thanks

Casper
 
Last edited:
offerlam said:
Hi all,

It started by me removing the backup share and adding it again from another ip via NFS..

after doing that all of a sudden my prox website showed all nodes as fenced and all VMs as offline.. also it couldn't show any content from any storage..

but there were still access and VMs was still running..

now closing and opening browsers i can't even login anymore.. I do still have cli access..

any suggestions??

Thanks

Casper
Click to expand...
Hi Casper,
sounds like you have trouble with an defined storage. Look on storage.cfg and the log-files.

Udo
 
Hello again Udo :),

I think you are right..

I went to the storage.cfg file and found the path for the backu01 storage i added before all this.. /mnt/pve/Backup01

but i can't browse it.. I CAN brows the other storage nodes which makes sense since they were working fine before..

Im tempted to edit storage.cfg on all 3 nodes and remove entries for the backup01 assuming that will solve the problem but im puzzled why i have it in the first place..

as you see my storage01 is in a different subnet.. actually different VLAN..

Im using my firewall inter-vlan routing capebilities to have the two VLANs talk to eachother and I can ping and all that stuff.. so im not getting why i have this problem...

could you perhaps give your 10 cents worth?

also do you know where the logs are? and what logs to look in? this part of proxmox is new to me..

here is the content of storage.cfg

Code: 
dir: local
        path /var/lib/vz
        content images,iso,vztmpl,rootdir
        maxfiles 0


nfs: Storage01_Vms
        path /mnt/pve/Storage01_Vms
        server 10.10.99.10
        export /volume1/VMs
        options vers=3
        content images,rootdir
        maxfiles 1


nfs: Storage01_ISO
        path /mnt/pve/Storage01_ISO
        server 10.10.99.10
        export /volume1/ISO
        options vers=3
        content iso
        maxfiles 1


nfs: Storage01_Templates
        path /mnt/pve/Storage01_Templates
        server 10.10.99.10
        export /volume1/Templates
        options vers=3
        content vztmpl
        maxfiles 1


nfs: Backup01
        path /mnt/pve/Backup01
        server 10.10.98.2
        export /volume1/Backup01
        options vers=3
        content backup
        maxfiles 1

Here is the syslog 100 lines..

Code: 
root@proxmox00:/etc/pve# tail -100f /var/log/syslog
Feb  1 13:06:25 proxmox00 pvestatd[3388]: WARNING: storage 'Backup01' is not onl                                      ine
Feb  1 13:06:35 proxmox00 pvestatd[3388]: WARNING: storage 'Backup01' is not onl                                      ine
Feb  1 13:06:45 proxmox00 pvestatd[3388]: WARNING: storage 'Backup01' is not onl                                      ine
Feb  1 13:06:55 proxmox00 pvestatd[3388]: WARNING: storage 'Backup01' is not onl                                      ine
Feb  1 13:07:05 proxmox00 pvestatd[3388]: WARNING: storage 'Backup01' is not onl                                      ine
Feb  1 13:07:07 proxmox00 pvedaemon[569148]: <root@pam> successful auth for user                                       'root@pam'
Feb  1 13:07:15 proxmox00 pvestatd[3388]: WARNING: storage 'Backup01' is not onl                                      ine
Feb  1 13:07:25 proxmox00 pmxcfs[2508]: [dcdb] notice: data verification success                                      ful
Feb  1 13:07:25 proxmox00 rrdcached[2475]: flushing old values
Feb  1 13:07:25 proxmox00 rrdcached[2475]: rotating journals
Feb  1 13:07:25 proxmox00 rrdcached[2475]: started new journal /var/lib/rrdcache                                      d/journal/rrd.journal.1391256445.269701
Feb  1 13:07:25 proxmox00 rrdcached[2475]: removing old journal /var/lib/rrdcach                                      ed/journal/rrd.journal.1391249245.269682
Feb  1 13:07:25 proxmox00 pvestatd[3388]: WARNING: storage 'Backup01' is not onl                                      ine
Feb  1 13:07:26 proxmox00 pvedaemon[569148]: WARNING: storage 'Backup01' is not                                       online
Feb  1 13:07:34 proxmox00 pvestatd[3388]: WARNING: storage 'Backup01' is not onl                                      ine
Feb  1 13:07:37 proxmox00 pvedaemon[482514]: WARNING: storage 'Backup01' is not                                       online
Feb  1 13:07:45 proxmox00 pvestatd[3388]: WARNING: storage 'Backup01' is not onl                                      ine
Feb  1 13:07:55 proxmox00 pvestatd[3388]: WARNING: storage 'Backup01' is not onl                                      ine
Feb  1 13:08:05 proxmox00 pvestatd[3388]: WARNING: storage 'Backup01' is not onl                                      ine
Feb  1 13:08:15 proxmox00 pvestatd[3388]: WARNING: storage 'Backup01' is not onl                                      ine
Feb  1 13:09:05 proxmox00 pveproxy[611058]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 13:16:49 proxmox00 pveproxy[611045]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 13:17:01 proxmox00 /USR/SBIN/CRON[921120]: (root) CMD (   cd / && run-par                                      ts --report /etc/cron.hourly)
Feb  1 13:22:02 proxmox00 pvedaemon[569090]: <root@pam> successful auth for user                                       'root@pam'
Feb  1 13:37:02 proxmox00 pvedaemon[482514]: <root@pam> successful auth for user                                       'root@pam'
Feb  1 13:37:53 proxmox00 pveproxy[611058]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 13:37:54 proxmox00 pveproxy[611045]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 13:38:53 proxmox00 pveproxy[611058]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 13:42:34 proxmox00 pveproxy[611054]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 13:46:11 proxmox00 pmxcfs[2508]: [status] notice: received log
Feb  1 13:47:24 proxmox00 pveproxy[611054]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 13:47:24 proxmox00 pveproxy[611045]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 13:47:27 proxmox00 pveproxy[611054]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 13:47:54 proxmox00 pveproxy[611045]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 13:48:25 proxmox00 pveproxy[611054]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 13:48:55 proxmox00 pveproxy[611058]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 13:49:26 proxmox00 pveproxy[611058]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 13:57:14 proxmox00 pveproxy[611054]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 13:57:14 proxmox00 pveproxy[611058]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 13:58:01 proxmox00 pveproxy[611045]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:05:11 proxmox00 pveproxy[611054]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:05:11 proxmox00 pveproxy[611058]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:05:11 proxmox00 pveproxy[611045]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:05:41 proxmox00 pveproxy[611054]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:06:12 proxmox00 pveproxy[611054]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:06:42 proxmox00 pveproxy[611045]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:07:12 proxmox00 pveproxy[611058]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:07:17 proxmox00 pveproxy[611058]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:07:18 proxmox00 pveproxy[611045]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:07:18 proxmox00 pveproxy[611054]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:07:25 proxmox00 rrdcached[2475]: flushing old values
Feb  1 14:07:25 proxmox00 rrdcached[2475]: rotating journals
Feb  1 14:07:25 proxmox00 rrdcached[2475]: started new journal /var/lib/rrdcache                                      d/journal/rrd.journal.1391260045.269842
Feb  1 14:07:25 proxmox00 rrdcached[2475]: removing old journal /var/lib/rrdcach                                      ed/journal/rrd.journal.1391252845.269611
Feb  1 14:07:25 proxmox00 pmxcfs[2508]: [dcdb] notice: data verification success                                      ful
Feb  1 14:07:48 proxmox00 pveproxy[611058]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:08:18 proxmox00 pveproxy[611058]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:08:20 proxmox00 pveproxy[611045]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:08:20 proxmox00 pveproxy[611054]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:08:21 proxmox00 pveproxy[611058]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:08:42 proxmox00 pveproxy[611058]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:08:42 proxmox00 pveproxy[611045]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:08:43 proxmox00 pveproxy[611054]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:09:13 proxmox00 pveproxy[611058]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:09:43 proxmox00 pveproxy[611058]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:10:14 proxmox00 pveproxy[611054]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:10:44 proxmox00 pveproxy[611045]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:17:01 proxmox00 /USR/SBIN/CRON[922710]: (root) CMD (   cd / && run-par                                      ts --report /etc/cron.hourly)
Feb  1 14:28:41 proxmox00 pveproxy[611058]: problem with client 10.81.234.9; Con                                      nection reset by peer
Feb  1 14:28:41 proxmox00 pveproxy[611058]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:29:33 proxmox00 pveproxy[611058]: problem with client 10.81.234.9; Con                                      nection reset by peer
Feb  1 14:29:33 proxmox00 pveproxy[611058]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:30:43 proxmox00 pveproxy[611054]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 14:31:10 proxmox00 pveproxy[611045]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 15:07:25 proxmox00 rrdcached[2475]: flushing old values
Feb  1 15:07:25 proxmox00 rrdcached[2475]: rotating journals
Feb  1 15:07:25 proxmox00 rrdcached[2475]: started new journal /var/lib/rrdcache                                      d/journal/rrd.journal.1391263645.269773
Feb  1 15:07:25 proxmox00 rrdcached[2475]: removing old journal /var/lib/rrdcach                                      ed/journal/rrd.journal.1391256445.269701
Feb  1 15:07:25 proxmox00 pmxcfs[2508]: [dcdb] notice: data verification success                                      ful
Feb  1 15:17:01 proxmox00 /USR/SBIN/CRON[923478]: (root) CMD (   cd / && run-par                                      ts --report /etc/cron.hourly)
Feb  1 15:35:08 proxmox00 pveproxy[611045]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 15:54:09 proxmox00 pveproxy[611045]: WARNING: proxy detected vanished cli                                      ent connection
Feb  1 16:07:25 proxmox00 rrdcached[2475]: flushing old values
Feb  1 16:07:25 proxmox00 rrdcached[2475]: rotating journals
Feb  1 16:07:25 proxmox00 rrdcached[2475]: started new journal /var/lib/rrdcache                                      d/journal/rrd.journal.1391267245.269681
Feb  1 16:07:25 proxmox00 rrdcached[2475]: removing old journal /var/lib/rrdcach                                      ed/journal/rrd.journal.1391260045.269842
Feb  1 16:07:25 proxmox00 pmxcfs[2508]: [dcdb] notice: data verification success                                      ful
Feb  1 16:17:01 proxmox00 /USR/SBIN/CRON[924200]: (root) CMD (   cd / && run-par                                      ts --report /etc/cron.hourly)
Feb  1 17:07:25 proxmox00 pmxcfs[2508]: [dcdb] notice: data verification success                                      ful
Feb  1 17:07:25 proxmox00 rrdcached[2475]: flushing old values
Feb  1 17:07:25 proxmox00 rrdcached[2475]: rotating journals
Feb  1 17:07:25 proxmox00 rrdcached[2475]: started new journal /var/lib/rrdcache                                      d/journal/rrd.journal.1391270845.269688
Feb  1 17:07:25 proxmox00 rrdcached[2475]: removing old journal /var/lib/rrdcach                                      ed/journal/rrd.journal.1391263645.269773
Feb  1 17:17:01 proxmox00 /USR/SBIN/CRON[925006]: (root) CMD (   cd / && run-par                                      ts --report /etc/cron.hourly)
Feb  1 18:07:25 proxmox00 rrdcached[2475]: flushing old values
Feb  1 18:07:25 proxmox00 rrdcached[2475]: rotating journals
Feb  1 18:07:25 proxmox00 rrdcached[2475]: started new journal /var/lib/rrdcache                                      d/journal/rrd.journal.1391274445.269694
Feb  1 18:07:25 proxmox00 rrdcached[2475]: removing old journal /var/lib/rrdcach                                      ed/journal/rrd.journal.1391267245.269681
Feb  1 18:07:25 proxmox00 pmxcfs[2508]: [dcdb] notice: data verification success                                      ful
Feb  1 18:17:01 proxmox00 /USR/SBIN/CRON[925729]: (root) CMD (   cd / && run-par                                      ts --report /etc/cron.hourly)

But i don't see what could be wrong

Thanks

casper
 
Last edited:
Hi,
Backup01 isn't online?!
What is the output of
Code: 
mount
df -h
Im tempted to edit storage.cfg on all 3 nodes and remove entries for the backup01 assuming that will solve the problem but im puzzled why i have it in the first place..
Click to expand...
Wait!! Edit on all 3 nodes?? /etc/pve must be on all (active) cluster member the same! If you cange storage.cfg on one host, the content must be the same on all host.
On hosts with different content you shouldn't be able to change the file, because of missing qorum!

What is the output of following commands
Code: 
pvecm status
pvecm nodes
# looks "pvecm nodes" from other nodes different?
Udo
 
Hi udo,

Storage.cfg is the same on all nodes... so what i ment was to remove the backup01 entry on all nodes... these problems started when i added backup01 back into the cluster as storage for backup after moving it to a seperate vlan... so i just assumed the quick fix was to remove the backup01 entries in all storage.cfg files to correct the problem..

that being said i would really like to know what is wrong...

i know it says backup01 is offline but its not

Code: 
root@proxmox01:~# ping 10.10.98.2PING 10.10.98.2 (10.10.98.2) 56(84) bytes of data.
64 bytes from 10.10.98.2: icmp_req=1 ttl=63 time=2.79 ms
64 bytes from 10.10.98.2: icmp_req=2 ttl=63 time=0.212 ms

as for your output request here they come:

mount

Code: 
root@proxmox00:~# mountsysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=4114924,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=3293860k,mode=755)
/dev/mapper/pve-root on / type ext3 (rw,relatime,errors=remount-ro,user_xattr,acl,barrier=0,data=ordered)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=6587700k)
/dev/mapper/pve-data on /var/lib/vz type ext3 (rw,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered)
/dev/sda1 on /boot type ext3 (rw,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
none on /sys/kernel/config type configfs (rw,relatime)
beancounter on /proc/vz/beancounter type cgroup (rw,relatime,blkio,name=beancounter)
container on /proc/vz/container type cgroup (rw,relatime,freezer,devices,name=container)
fairsched on /proc/vz/fairsched type cgroup (rw,relatime,cpuacct,cpu,cpuset,name=fairsched)
10.10.99.10:/volume1/Templates on /mnt/pve/Storage01_Templates type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.99.10,mountvers=3,mountport=892,mountproto=udp,local_lock=none,addr=10.10.99.10)
10.10.99.11:/volume1/Backup01 on /mnt/pve/Backup01 type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.99.11,mountvers=3,mountport=892,mountproto=udp,local_lock=none,addr=10.10.99.11)
10.10.99.10:/volume1/ISO on /mnt/pve/Storage01_ISO type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.99.10,mountvers=3,mountport=892,mountproto=udp,local_lock=none,addr=10.10.99.10)
10.10.99.10:/volume1/VMs on /mnt/pve/Storage01_Vms type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.99.10,mountvers=3,mountport=892,mountproto=udp,local_lock=none,addr=10.10.99.10)

df -h

well it doesn't give any output back. its not getting to waiting for new input.. it just steps down one line and nothing happens until i breka it with ctrl+c

Code: 
root@proxmox00:~# df -h^C
root@proxmox00:~#

pvecm status

Code: 
root@proxmox00:~# pvecm statusVersion: 6.2.0
Config Version: 34
Cluster Name: DingITCluster
Cluster Id: 44340
Cluster Member: Yes
Cluster Generation: 1168
Membership state: Cluster-Member
Nodes: 3
Expected votes: 3
Total votes: 3
Node votes: 1
Quorum: 2
Active subsystems: 6
Flags:
Ports Bound: 0 177
Node name: proxmox00
Node ID: 1
Multicast addresses: 239.192.173.225
Node addresses: 10.10.99.20

pvecm nodes

Code: 
PROXMOX00
root@proxmox00:~# pvecm nodesNode  Sts   Inc   Joined               Name
   1   M   1160   2014-01-25 13:07:27  proxmox00
   2   M   1168   2014-01-25 13:07:57  proxmox01
   3   M   1168   2014-01-25 13:07:57  proxmox02
root@proxmox00:~#

PROXMOX01
root@proxmox01:~# pvecm nodes
Node  Sts   Inc   Joined               Name
   1   M   1168   2014-01-25 13:07:57  proxmox00
   2   M   1104   2014-01-25 12:20:22  proxmox01
   3   M   1120   2014-01-25 12:22:32  proxmox02
root@proxmox01:~#


PROXMOX02
root@proxmox02:~# pvecm nodes
Node  Sts   Inc   Joined               Name
   1   M   1168   2014-01-25 13:07:57  proxmox00
   2   M   1120   2014-01-25 12:22:32  proxmox01
   3   M   1108   2014-01-25 12:21:48  proxmox02
root@proxmox02:~#

Thanks

Casper
 
Are you sure the nfs server is running on the storage?
What output do you get from:
rpcinfo -p 10.10.98.2
showmount -e 10.10.98.2
 
Uhhh the mighty Mir :) welcome to the thread and thanks for helping out!

here is a combined output of what you asked

Code: 
root@proxmox00:/etc/pve# rpcinfo -p 10.10.98.2   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100005    1   udp    892  mountd
    100005    1   tcp    892  mountd
    100005    2   udp    892  mountd
    100005    2   tcp    892  mountd
    100005    3   udp    892  mountd
    100005    3   tcp    892  mountd
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100021    1   udp  41574  nlockmgr
    100021    3   udp  41574  nlockmgr
    100021    4   udp  41574  nlockmgr
    100021    1   tcp  39041  nlockmgr
    100021    3   tcp  39041  nlockmgr
    100021    4   tcp  39041  nlockmgr
    100024    1   udp  59015  status
    100024    1   tcp  51378  status
root@proxmox00:/etc/pve# showmount -e 10.10.98.2
Export list for 10.10.98.2:
/volume1/mail01_backup 192.168.253.3
/volume1/Backup01      10.10.99.22,10.10.99.21,10.10.99.20
root@proxmox00:/etc/pve#

I must say though the showmount took a long time to produce a output.. also the /volume1/mail01_backup 192.168.253.3 shouldn't be there anymore.. its correct that i was trying to connect the server to that NFS share but from that server perspektive i never got it to work.. hmm maybe it did work.. anyway I removed all NFS related install and setup on the server and rebooted so i would assume it shouldn't be there?

Thanks

Casper
 
offerlam said:
Hi udo,

Storage.cfg is the same on all nodes... so what i ment was to remove the backup01 entry on all nodes... these problems started when i added backup01 back into the cluster as storage for backup after moving it to a seperate vlan... so i just assumed the quick fix was to remove the backup01 entries in all storage.cfg files to correct the problem..

that being said i would really like to know what is wrong...

i know it says backup01 is offline but its not

Code: 
root@proxmox01:~# ping 10.10.98.2PING 10.10.98.2 (10.10.98.2) 56(84) bytes of data.
64 bytes from 10.10.98.2: icmp_req=1 ttl=63 time=2.79 ms
64 bytes from 10.10.98.2: icmp_req=2 ttl=63 time=0.212 ms

as for your output request here they come:

mount

Code: 
root@proxmox00:~# mountsysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=4114924,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=3293860k,mode=755)
/dev/mapper/pve-root on / type ext3 (rw,relatime,errors=remount-ro,user_xattr,acl,barrier=0,data=ordered)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=6587700k)
/dev/mapper/pve-data on /var/lib/vz type ext3 (rw,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered)
/dev/sda1 on /boot type ext3 (rw,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
none on /sys/kernel/config type configfs (rw,relatime)
beancounter on /proc/vz/beancounter type cgroup (rw,relatime,blkio,name=beancounter)
container on /proc/vz/container type cgroup (rw,relatime,freezer,devices,name=container)
fairsched on /proc/vz/fairsched type cgroup (rw,relatime,cpuacct,cpu,cpuset,name=fairsched)
10.10.99.10:/volume1/Templates on /mnt/pve/Storage01_Templates type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.99.10,mountvers=3,mountport=892,mountproto=udp,local_lock=none,addr=10.10.99.10)
10.10.99.11:/volume1/Backup01 on /mnt/pve/Backup01 type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.99.11,mountvers=3,mountport=892,mountproto=udp,local_lock=none,addr=10.10.99.11)
10.10.99.10:/volume1/ISO on /mnt/pve/Storage01_ISO type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.99.10,mountvers=3,mountport=892,mountproto=udp,local_lock=none,addr=10.10.99.10)
10.10.99.10:/volume1/VMs on /mnt/pve/Storage01_Vms type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.99.10,mountvers=3,mountport=892,mountproto=udp,local_lock=none,addr=10.10.99.10)

df -h

well it doesn't give any output back. its not getting to waiting for new input.. it just steps down one line and nothing happens until i breka it with ctrl+c

Code: 
root@proxmox00:~# df -h^C
root@proxmox00:~#

pvecm status

Code: 
root@proxmox00:~# pvecm statusVersion: 6.2.0
Config Version: 34
Cluster Name: DingITCluster
Cluster Id: 44340
Cluster Member: Yes
Cluster Generation: 1168
Membership state: Cluster-Member
Nodes: 3
Expected votes: 3
Total votes: 3
Node votes: 1
Quorum: 2
Active subsystems: 6
Flags:
Ports Bound: 0 177
Node name: proxmox00
Node ID: 1
Multicast addresses: 239.192.173.225
Node addresses: 10.10.99.20

pvecm nodes

Code: 
PROXMOX00
root@proxmox00:~# pvecm nodesNode  Sts   Inc   Joined               Name
   1   M   1160   2014-01-25 13:07:27  proxmox00
   2   M   1168   2014-01-25 13:07:57  proxmox01
   3   M   1168   2014-01-25 13:07:57  proxmox02
root@proxmox00:~#

PROXMOX01
root@proxmox01:~# pvecm nodes
Node  Sts   Inc   Joined               Name
   1   M   1168   2014-01-25 13:07:57  proxmox00
   2   M   1104   2014-01-25 12:20:22  proxmox01
   3   M   1120   2014-01-25 12:22:32  proxmox02
root@proxmox01:~#


PROXMOX02
root@proxmox02:~# pvecm nodes
Node  Sts   Inc   Joined               Name
   1   M   1168   2014-01-25 13:07:57  proxmox00
   2   M   1120   2014-01-25 12:22:32  proxmox01
   3   M   1108   2014-01-25 12:21:48  proxmox02
root@proxmox02:~#

Thanks

Casper
Click to expand...
Hi Caspar,
you have still mounted Backup01 from 10.10.99.11
Code: 
10.10.99.11:/volume1/Backup01 on /mnt/pve/Backup01 type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.10.99.11,mountvers=3,mountport=892,mountproto=udp,local_lock=none,addr=10.10.99.11)
But in your storage.cfg the nfs-server is on 10.10.98.2!

Do an "unmount /mnt/pve/Backup01" and see if an "df -h" is running without problem.
If you than can mount the nfs-server with 10.10.98.2 all schould be fine!

Udo
 
Hi Udo..

you are right..??? why has this happend? I removed the backup01 storage before adding it again with the different ip??? do i need to apply it somewhere when i remove storage from proxmox?

the unmount command is not there??

Thanks

Casper
 
udo said:
Sorry,
umount is the right command.

Udo
Click to expand...

lol why didn't i try that before posting.. oh well..

I get this

Code: 
root@proxmox00:/etc/pve# umount /mnt/pve/Backup01umount.nfs: /mnt/pve/Backup01: device is busy
umount.nfs: /mnt/pve/Backup01: device is busy
 
offerlam said:
lol why didn't i try that before posting.. oh well..

I get this

Code: 
root@proxmox00:/etc/pve# umount /mnt/pve/Backup01umount.nfs: /mnt/pve/Backup01: device is busy
umount.nfs: /mnt/pve/Backup01: device is busy
Click to expand...
-hi,
look which process are bind to this directory:
Code: 
fuser -cu /mnt/pve/Backup01
then decide, if you can kill the process.

but you can also use the force-switch by umount ( umount -f ).

Udo
 
This has happened to me numerous times. Mostly when i changed NFS share from one ip to another, or deleted the share recreated a new one or interrupted a backup to NFS share. In almost all case after deleting the NFS share completely and rebooting the node fixed all issue. Also try to run this:

If all your VMs are running but the GUI shows all offline or GUI having issues, try to run the following and see if GUI starts behaving.
#service pvedaemon restart
#service pvestatd restart
#service pveproxy restart
 
Hi guys..

first of a huge thank you for the interest this post has taken! .. I appreciate every input..

Doing umount -f did not help but m.ardito suggestion with umount -f -l did the trick..

now first a little background cause maybe ...just maybe we have a bug here..

when i removed backup01 and added it again with the new ip on the different vlan I did it from proxmox00 10.10.99.20..

so i went and did umount -f -l /mnt/pve/Backup01 and now can log back into the web interface and even brows backup01 storage from web.. but i can't from the 2 other nodes.. proxmox01 and 02 IP 10.10.99.21 and .22

so i tried the umount -f -l /mnt/pve/Backup01 on both proxmox01 and 02 but that didn't change anything. I tried to log into web from 01 and 02 with no luck.

So i went with symmcom suggestion and tried to restart the service.. this fails on all nodes with the following:

Code: 
root@proxmox00:/etc/pve# pvedaemon restartunable to create socket - IO::Socket::INET: Address already in use
root@proxmox00:/etc/pve# pvestatd restart
ERROR: can't aquire lock '/var/run/pvestatd.pid.lock' - Resource temporarily unavailable
root@proxmox00:/etc/pve# pveproxy restart
unable to create socket - IO::Socket::INET: Address already in use

Now im pretty confident that if i just rebooted all the nodes i would perhaps fix my problemes.. but this i ment to be used as hosting real soon and when that happens i can't just go around rebooting nodes left and right when ever i have a problem. This is why i'm really intrested in a solution that does NOT require reboot..

also i'm pretty proud (on behalf of the proxmox team) of proxmox compared to other virtualization products and try to promote it where ever i can.. so stuff like this is a thorn in my side cause i know you wouldn't have such issues on vmware or hyper-v

Ok so i went back to the web interface of proxmox00 which was working after i did the umount and removed the storage backup01 from the web interfaces.. this does not remove the mount point in /mnt/pve/

So im thinking symmcon has a point here? and shouldn't proxmox remove the mountpoint? and is there a way i can achive symmcom solution WITHOUT having to boot?

Thanks!

Casper
 
ok just as a test through 10.10.99.20 proxmox00 i created a new storage pointing to the same NAS with NFS but instead of using backup01 I used backup01_v2

after doing that i went to the cli and did ls /mnt/pve/ expecting to see backup01_v2 appear.. but it didn't.. first it was not on proxmox00 but it was on proxmox02.. than i went to proxmox01 and it didn't have the mount point.. went back to proxmox00 and now it had the mount point...

shouldn't the mount point be created instantaneously? I thinking maybe im moving to quick for proxmox? from creating the storage to try and use it before it has actually done all it has to do in the background?
 
ok here is an update...

So /mnt/pve/backup01_v2 is still only located on proxmox00 and proxmox02 NOT proxmox01..

If i go to the web interface and ask for content of backup01_v2 I get the content ONLY from proxmox00 - this is done connecting to the web interface of proxmox00..

If i go to cli and do ls /mnt/pve/Backup01_v2/dump i get the content when done on proxmox00 which was expected since it works with the web interface.. but i do ALSO get it on proxmox02 even though i can't get it from the web interface logged in from proxmox00.. I can't log in to the web interface from proxmox01 and 02 still..
 
Udu if i do the fuser command you ask i get this output

Code: 
/mnt/pve/Backup01:       1rce(root)     2rc(root)     3rc(root)     4rc(root)     5rc(root)     6rc(root)     7rc(root)     8rc(root)     9rc(root)    10rc(root)    11rc(root)    12rc(root)    13rc(root)    14rc(root)    15rc(root)    16rc(root)    17rc(root)    18rc(root)    19rc(root)    20rc(root)    21rc(root)    22rc(root)    23rc(root)    24rc(root)    25rc(root)    26rc(root)    27rc(root)    28rc(root)    29rc(root)    30rc(root)    31rc(root)    32rc(root)    33rc(root)    34rc(root)    35rc(root)    36rc(root)    37rc(root)    38rc(root)    39rc(root)    40rc(root)    41rc(root)    42rc(root)    43rc(root)    44rc(root)    45rc(root)    46rc(root)    47rc(root)    48rc(root)    49rc(root)    50rc(root)    51rc(root)    52rc(root)    53rc(root)    54rc(root)    55rc(root)    56rc(root)    57rc(root)    58rc(root)    59rc(root)    60rc(root)    61rc(root)    62rc(root)    63rc(root)    64rc(root)    65rc(root)    66rc(root)    67rc(root)    68rc(root)    69rc(root)    70rc(root)    71rc(root)    72rc(root)    73rc(root)    74rc(root)    75rc(root)    76rc(root)    77rc(root)    78rc(root)    79rc(root)    80rc(root)    81rc(root)    82rc(root)    83rc(root)    84rc(root)    85rc(root)    86rc(root)    87rc(root)    88rc(root)    89rc(root)    90rc(root)    91rc(root)    92rc(root)    93rc(root)    94rc(root)    95rc(root)    96rc(root)    97rc(root)    98rc(root)    99rc(root)   100rc(root)   101rc(root)   102rc(root)   103rc(root)   104rc(root)   105rc(root)   106rc(root)   107rc(root)   108rc(root)   109rc(root)   110rc(root)   111rc(root)   112rc(root)   113rc(root)   114rc(root)   115rc(root)   116rc(root)   117rc(root)   118rc(root)   119rc(root)   120rc(root)   121rc(root)   122rc(root)   123rc(root)   124rc(root)   125rc(root)   126rc(root)   127rc(root)   128rc(root)   129rc(root)   130rc(root)   131rc(root)   132rc(root)   133rc(root)   134rc(root)   135rc(root)   136rc(root)   137rc(root)   138rc(root)   139rc(root)   140rc(root)   141rc(root)   142rc(root)   143rc(root)   144rc(root)   145rc(root)   146rc(root)   147rc(root)   148rc(root)   149rc(root)   150rc(root)   151rc(root)   152rc(root)   153rc(root)   154rc(root)   155rc(root)   156rc(root)   157rc(root)   158rc(root)   159rc(root)   160rc(root)   161rc(root)   162rc(root)   163rc(root)   164rc(root)   165rc(root)   166rc(root)   167rc(root)   168rc(root)   169rc(root)   170rc(root)   171rc(root)   172rc(root)   173rc(root)   174rc(root)   175rc(root)   176rc(root)   178rc(root)   179rc(root)   180rc(root)   181rc(root)   182rc(root)   183rc(root)   184rc(root)   185rc(root)   186rc(root)   187rc(root)   188rc(root)   189rc(root)   190rc(root)   191rc(root)   192rc(root)   193rc(root)   194rc(root)   195rc(root)   196rc(root)   197rc(root)   198rc(root)   199rc(root)   200rc(root)   201rc(root)   202rc(root)   203rc(root)   204rc(root)   205rc(root)   206rc(root)   207rc(root)   208rc(root)   209rc(root)   210rc(root)   211rc(root)   212rc(root)   213rc(root)   214rc(root)   215rc(root)   220rc(root)   221rc(root)   222rc(root)   223rc(root)   224rc(root)   225rc(root)   226rc(root)   227rc(root)   228rc(root)   229rc(root)   230rc(root)   231rc(root)   232rc(root)   233rc(root)   234rc(root)   235rc(root)   236rc(root)   238rc(root)   239rc(root)   240rc(root)   241rc(root)   242rc(root)   243rc(root)   366rc(root)   369rc(root)   370rc(root)   371rc(root)   427rc(root)   428rc(root)   431rc(root)   432rc(root)   452rc(root)   465rc(root)   599rce(root)   789rc(root)  1347rc(root)  1350rc(root)  1574rc(root)  1628rc(root)  1641rc(root)  1837rc(root)  2046rce(root)  2063rce(statd)  2068rc(root)  2069rc(root)  2070rc(root)  2071rc(root)  2072rc(root)  2073rc(root)  2074rc(root)  2075rc(root)  2076rc(root)  2077rc(root)  2078rc(root)  2079rc(root)  2080rc(root)  2081rc(root)  2082rc(root)  2083rc(root)  2085rc(root)  2086rc(root)  2087rc(root)  2094rce(root)  2177rc(root)  2179rc(root)  2180rc(root)  2181rc(root)  2182rc(root)  2183rc(root)  2184rc(root)  2185rc(root)  2186rc(root)  2187rc(root)  2188rc(root)  2189rc(root)  2190rc(root)  2191rc(root)  2192rc(root)  2193rc(root)  2194rc(root)  2195rc(root)  2196rc(root)  2197rc(root)  2198rc(root)  2199rc(root)  2200rc(root)  2201rc(root)  2202rc(root)  2203rc(root)  2204rc(root)  2205rc(root)  2206rc(root)  2207rc(root)  2208rc(root)  2209rc(root)  2210rc(root)  2211rc(root)  2212rc(root)  2213rc(root)  2214rc(root)  2216rce(root)  2217rce(root)  2329rce(root)  2338rce(root)  2344rce(root)  2351rce(root)  2393rce(daemon)  2428rce(ntp)  2463rce(root)  2473rce(root)  2486rce(root)  2522rce(root)  2528rce(postfix)  2560rce(root)  2638rce(root)  2705rc(root)  2759rc(root)  2789rc(root)  2849rce(root)  2852rce(root)  2855rce(root)  2859rce(root)  2873rce(root)  2878rce(www-data)  2901rce(www-data)  2902rce(www-data)  2917rc(root)  2973rce(root)  2974rce(root)  2975rce(root)  2976rce(root)  2977rce(root)  2978rce(root)  2980rce(root)  2981rce(root)  3727rce(root)  3743rce(root)  3858rce(root)  3859rce(root)  3866rc(root)  3867rc(root)  3868rc(root)  3869rc(root)  3870rc(root) 229321rce(root) 229322rce(root) 229323rce(root) 229422rce(root) 263295rce(root) 263296rce(root) 263297rce(root) 263451rce(root) 419389rce(root) 419390rce(root) 419391rce(root) 419477rce(root) 419481rce(root) 457521rce(root) 457522rce(root) 457523rce(root) 457661rce(root) 475192rce(root) 475193rce(root) 475194rce(root) 475276rce(www-data) 475285rce(www-data) 475287rce(www-data) 475307rce(root) 479139rce(postfix) 480025rce(root) 480027rce(root) 480128rce(root) 605286rce(root) 605287rce(root) 605288rce(root) 605388rce(root) 951557rce(root) 951558rce(root) 951559rce(root) 951673rce(root)root@proxmox01:~#
 
Hi guys..

thanks for all your help..

in the ned i ended up with rebooting all nodes... I had some downtime on VMs because i lost qurom.. I suppose when you have a three node cluster you should one boot one server at a time to avoide this?

I can confirm after reboot that backup01_v2 seems to be working fine.. currently im backing up a vm to see how it goes..

I can also confirm that the old mount point backup01 still exist in my /mnt/pve which to me must be a bug?

Thanks

Casper
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back