open-iscsi automount, with a NFS on top

[zccopwrx]

Having some issues regarding setting up an ISCSI target for /var/lib/vz.

The problem is, open-iscsi tries to start before vmbr0 is up and running, therefor fails, and leaves the node stuck at a Maintenance prompt.

Unfortunately, im not well enough versed in Debian to figure out how to get vmbr0 up first, then start open-iscsi, THEN mount nfs..

From what I can tell, in /etc/network/if-up.d/mountnfs, that means when vmbr0 starts, it immediately mounts nfs, which wont work, considering the ISCSI target needs to be there before NFS can mount on top of it. (for images and templates dirs).

I really didn't want to have to mount these somewhere like rc.local, does anyone have a better idea?

 

[zccopwrx]

For now I did it the rc.local (dirty) way.

mount /dev/sdc1 /var/lib/vz -t ext3 -o _netdev
mount XXXXXXX:/mnt/ofsan1/store1/proxmox-images/ /var/lib/vz/images -t nfs -o rsize=8k,wsize=8k,noatime,hard,_netdev
mount XXXXXXX:/mnt/ofsan1/store1/proxmox-templates/ /var/lib/vz/template -t nfs -o rsize=8k,wsize=8k,noatime,hard,_netdev
/etc/init.d/vz restart
/etc/init.d/qemu-server restart

It works.. but I dont think its the most efficient way of doing things.

 

[thefool808]

I've gotten iSCSI to automount... testing NFS on top of that now.

mini HowTo:

NOTE: this will only work with one iSCSI target until Debian adds some udev rules that allow for consistent iSCSI mapping to /dev targets (i.e. can't use /dev/sdc1 in fstab because with two iSCSI targets you can't be sure which one will get /dev/sdc...)

I followed this to attach to my iSCSI target ->

http://www.cyberciti.biz/faq/howto-setup-debian-ubuntu-linux-iscsi-initiator/

After that I added this line to /etc/fstab (note the _netdev option):

/dev/sdc1       /var/lib/vz     ext3    _netdev 0       0

Debian will not mount filesystems with the _netdev option by default. This can be seen in /etc/init.d/mountall.sh:

mount -a -t nonfs,nfs4,smbfs,cifs,ncp,ncpfs,coda,ocfs2,gfs \
      -O no_netdev

This seems to be the way RH and maybe Suse are doing their iSCSI automounting so it should hold up.

Then just add this file at /etc/init.d/mount_netdev.sh

#! /bin/sh
### BEGIN INIT INFO
# Provides:          mount_netdev
# Required-Start:    network, open-iscsi
# Required-Stop: 
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Mount/Unmount filesystems with _netdev option (iSCSI)
# Description:
### END INIT INFO

PATH=/sbin:/bin
. /lib/lsb/init-functions

do_start() {
        #
        # Mount local file systems in /etc/fstab.
        #
        mount_all_netdev() {
            mount -a -O _netdev
        }
        if [ "$VERBOSE" = no ]
        then
                log_action_begin_msg "Mounting _netdev filesystems"
                mount_all_netdev
                log_action_end_msg $?
        else
                log_daemon_msg "Will now mount _netdev filesystems"
                mount_all_netdev
                log_end_msg $?
        fi
}

do_stop() {
        #
        # Mount local file systems in /etc/fstab.
        #
        umount_all_netdev() {
            umount -a -O _netdev
        }
        if [ "$VERBOSE" = no ]
        then
                log_action_begin_msg "Unmounting _netdev filesystems"
                umount_all_netdev
                log_action_end_msg $?
        else
                log_daemon_msg "Will now umount _netdev filesystems"
                umount_all_netdev
                log_end_msg $?
        fi
}

case "$1" in
  start|"")
    do_start
    ;;
  restart|reload|force-reload)
    echo "Error: argument '$1' not supported" >&2
    exit 3
    ;;
  stop)
    do_stop
    ;;
  *)
    echo "Usage: mountall.sh [start|stop]" >&2
    exit 3
    ;;
esac

:

and make this script executable:

chmod 755 /etc/init.d/mount_netdev.sh

Finally enable this script with the command:

update-rc.d mount_netdev.sh start 42 2 3 4 5 . stop 33 0 1 6 .

This makes sure it mounts after networking has started and unmounts before networking has been stopped.

Finally, I changed the sequence numbers of the open-iscsi startup script in /etc/rc2.d to start up after networking, but then I changed it back to 20 and it still seems to work... It seems that iSCSI is smart enough to do it's thing once networking comes up. Which is nice.

Also, I'm not sure it's always necessary, but, make sure you have

node.startup = automatic

in /etc/iscsd.conf and possibly everywhere else node.startup (or node[0].startup) is mentioned which includes /etc/iscsi/nodes/node_name/ip_address,port

 

[zccopwrx]

Interestingly enough, the only place a networking script is linked in any /etc/rcX.d's is in rc6.. no where else... how does THAT happen?

Ill try to readd it.. but all 3 of my nodes have the same thing.

 

[zccopwrx]

Ok I got it semi-working.. No more rc.local script but the order of boot still isnt right.. I added S19networking to rc2.d right before iscsi, problem is that in my /etc/network/if-up.d/ theres a mountnfs script that runs as soon as the networking is online... which means nfs tries to mount before iscsi is active and it causes some issue (like openvz starts but without any vz's running until I restart it)..

Im wondering why you have a networking script in all your runlevels and I only have runlevel 6 ?

 

[thefool808]

I think debian runs rcS.d scripts before running any of the specific run-level scripts. I actually moved my script to that folder instead of rc2.d, but I was still having problems (mainly because of what you pointed out about the networking script actually doing the NFS mounting through if-up.d).

My new solution which I've been thinking of implementing is making folders in the /mnt folder, such as /mnt/iscsi and /mnt/nfs. Then, mounting the respective network shares to those locations. Finally, I'll just have /var/lib/vz be a link to /mnt/iscsi/vz and within /mnt/iscsi/vz i'll have links to /mnt/nfs such as /mnt/iscsi/vz/template -> /mnt/nfs/vz/template. Then it shouldn't matter in what order the mounts come up.

The only problem I can forsee is making sure no init scripts try to access files on those links. Therefore, I still beleive I'll put the mount_netdev.sh in rcS.d instead of rc2.d.

 

[zccopwrx]

Hehe, my rc.local script is starting to look more appealing now.. It actually seems like much less work.

Perhaps as ProxMox starts going down the shared storage roadmap, this functionality will be automatic, so I suppose I won't spend too much more time on it.

 

[thefool808]

Looking through the Debian Lenny open-iscsi package it appears that they have tried to solve these problems.

http://packages.debian.org/lenny/open-iscsi

It adds the "mount -a -O _netdev" line to the /etc/init.d/open-iscsi startup script. Also, it puts that script in /etc/rcS.d at S45 (same as the mountnfs script). This is the route I've been trying to follow.

They also solve the shutdown script problems by adding a script unmountiscsi.sh that runs right before the open-iscsi script is sent a 'stop'.

It still will not solve the problem of controlling which shares are mounted when, and it seems the symbolic links are the easiest solution I can come up with.