1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

sshfs shared storage

Discussion in 'Proxmox VE: Installation and configuration' started by bread-baker, May 30, 2012.

  1. bread-baker

    bread-baker Member

    Joined:
    Mar 6, 2010
    Messages:
    432
    Likes Received:
    0
    I think it would be good to allow sshfs for shared storage.

    For us sshfs is more reliable then nfs . We use sshfs for user documents - with $HOME/Documents targeted to a kvm sshfs . We've done that for 3 years and have had very few issues.
     
  2. tom

    tom Proxmox Staff Member
    Staff Member

    Joined:
    Aug 29, 2006
    Messages:
    11,053
    Likes Received:
    49
    never tried but just mount via /etc/fstab and then configure a shared directory storage, does that work for you?
     
  3. bread-baker

    bread-baker Member

    Joined:
    Mar 6, 2010
    Messages:
    432
    Likes Received:
    0
    the mount and share work, thanks for the idea.

    however restoring an openvz template does not work. If I remember correctly, sshfs and links do wot get along. here is some of the error message at pve :
    Code:
    Creating container private area (/mnt/pve/fbc1-nfs/template/cache/debian-7.0-FBC_7.0_amd64.tar.gz)
    tar: ./lib/libnl-3.so.200: Cannot change ownership to uid 0, gid 0: No such file or directory
    tar: ./lib/terminfo/r/rxvt-m: Cannot change ownership to uid 0, gid 0: No such file or directory
    tar: ./lib/libxtables.so.7: Cannot change ownership to uid 0, gid 0: No such file or directory
    tar: ./lib/x86_64-linux-gnu/libnsl.so.1: Cannot change ownership to uid 0, gid 0: No such file or directory
    tar: ./lib/x86_64-linux-gnu/librt.so.1: Cannot change ownership to uid 0, gid 0: No such file or directory
    tar: ./lib/x86_64-linux-gnu/libuuid.so.1: Cannot change ownership to uid 0, gid 0: No such file or directory
    tar: ./lib/x86_64-linux-gnu/libudev.so.0: Cannot change ownership to uid 0, gid 0: No such file or directory
    tar: ./lib/x86_64-linux-gnu/libpng12.so.0: Cannot change ownership to uid 0, gid 0: No such file or directory
    tar: ./lib/x86_64-linux-gnu/libncurses.so.5: Cannot change ownership to uid 0, gid 0: No such file or directory
    tar: ./lib/x86_64-linux-gnu/libpamc.so.0: Cannot change ownership to uid 0, gid 0: No such file or directory
    tar: ./lib/x86_64-linux-gnu/libbz2.so.1.0: Cannot change ownership to uid 0, gid 0: No such file or directory
    tar: ./lib/x86_64-linux-gnu/libbz2.so.1: Cannot change ownership to uid 0, gid 0: No such file or directory
    ...
    [COLOR=#000000][FONT=tahoma]tar: ./usr/lib/pymodules/python2.7/SOAPpy-0.12.0.egg-info: Cannot change ownership to uid 0, gid 0: No such file or directory[/FONT][/COLOR]
    [COLOR=#000000][FONT=tahoma]tar: ./usr/lib/pymodules/python2.7/python_debianbts-1.10.egg-info: Cannot change ownership to uid 0, gid 0: No such file or directory[/FONT][/COLOR]
    [COLOR=#000000][FONT=tahoma]tar: ./usr/lib/pymodules/python2.7/reportbug-6.3.1.egg-info/SOURCES.txt: Cannot change ownership to uid 0, gid 0: No such file or directory[/FONT][/COLOR]
    [COLOR=#000000][FONT=tahoma]tar: ./usr/lib/pymodules/python2.7/reportbug-6.3.1.egg-info/top_level.txt: Cannot change ownership to uid 0, gid 0: No such file or directory[/FONT][/COLOR]
    [COLOR=#000000][FONT=tahoma]tar: ./usr/lib/pymodules/python2.7/reportbug-6.3.1.egg-info/PKG-INFO: Cannot change ownership to uid 0, gid 0: No such file or directory[/FONT][/COLOR]
    [COLOR=#000000][FONT=tahoma]tar: ./usr/lib/pymodules/python2.7/reportbug-6.3.1.egg-info/dependency_links.txt: Cannot change ownership to uid 0, gid 0: No such file or directory[/FONT][/COLOR]
    [COLOR=#000000][FONT=tahoma]tar: ./usr/lib/pymodules/python2.7/debianbts.py: Cannot change ownership to uid 0, gid 0: No such file or directory[/FONT][/COLOR]
    [COLOR=#000000][FONT=tahoma]tar: ./usr/lib/python2.7/sitecustomize.py: Cannot change ownership to uid 0, gid 0: No such file or directory[/FONT][/COLOR]
    [COLOR=#000000][FONT=tahoma]tar: ./lib64/ld-linux-x86-64.so.2: Cannot change ownership to uid 0, gid 0: No such file or directory[/FONT][/COLOR]
    [COLOR=#000000][FONT=tahoma]tar: Exiting with failure status due to previous errors[/FONT][/COLOR]
    [COLOR=#000000][FONT=tahoma]vps-create ERROR: Unpack /mnt/pve/fbc1-nfs/template/cache/debian-7.0-FBC_7.0_amd64.tar.gz failed[/FONT][/COLOR]
    [COLOR=#000000][FONT=tahoma]Creation of container private area failed[/FONT][/COLOR] 
    
    TASK ERROR: command 'vzctl --skiplock create 8010 --ostemplate /mnt/pve/fbc1-nfs/template/cache/debian-7.0-FBC_7.0_amd64.tar.gz --private /fbc246-data/private/8010' failed: exit code 48
    
    I think the error lines are all due to symlinks ...

    Next I'll try a kvm on sshfs.
     
    #3 bread-baker, May 30, 2012
    Last edited: May 30, 2012
  4. bread-baker

    bread-baker Member

    Joined:
    Mar 6, 2010
    Messages:
    432
    Likes Received:
    0
    KVM restore works.


    I'll try migration next.
     
  5. bread-baker

    bread-baker Member

    Joined:
    Mar 6, 2010
    Messages:
    432
    Likes Received:
    0
    Code:
    [COLOR=#000000][FONT=tahoma]May 30 12:59:58 starting migration of VM 8020 to node 'fbc240' (10.100.100.240)[/FONT][/COLOR]
    [COLOR=#000000][FONT=tahoma]May 30 12:59:58 copying disk images[/FONT][/COLOR]
    [COLOR=#000000][FONT=tahoma]May 30 12:59:58 starting VM 8020 on remote node 'fbc240'[/FONT][/COLOR]
    [COLOR=#000000][FONT=tahoma]May 30 12:59:59 starting migration tunnel[/FONT][/COLOR]
    [COLOR=#000000][FONT=tahoma]May 30 13:00:00 starting online/live migration on port 60000[/FONT][/COLOR]
    [COLOR=#000000][FONT=tahoma]May 30 13:00:02 migration status: completed[/FONT][/COLOR]
    [COLOR=#000000][FONT=tahoma]May 30 13:00:02 migration speed: 256.00 MB/s[/FONT][/COLOR]
    [COLOR=#000000][FONT=tahoma]May 30 13:00:04 migration finished successfuly (duration 00:00:06)[/FONT][/COLOR]
    [COLOR=#000000][FONT=tahoma]TASK OK
    
    so on line migration works . but I'll use the kvm a while and check for problems.[/FONT][/COLOR]
     
  6. bread-baker

    bread-baker Member

    Joined:
    Mar 6, 2010
    Messages:
    432
    Likes Received:
    0
    I had too many errors with rsync and the kvm on sshfs .

    the issues may have had nothing to do with sshfs, b ut the system i restored works without issues on drbd and local storage.

    So I give up on sshfs for now. Will use nfs instead.

    here were some of the issues:
    Code:
    [mntent]: warning: no final newline at the end of /etc/mtab                                                                                    
    [mntent]: warning: no final newline at the end of /etc/mtab                                                                                    
    [mntent]: warning: no final newline at the end of /etc/mtab
    
    
    inflate returned -3 (0 bytes)                                                                                                                  
    rsync error: error in rsync protocol data stream (code 12) at token.c(547) [receiver=3.0.3]                                                    
    rsync: connection unexpectedly closed (10290 bytes received so far) [generator]                                                                
    rsync error: error in rsync protocol data stream (code 12) at io.c(635) [generator=3.0.3]     
    
    
    rsync: readlink "/bkup/u/hadjx1.pro" failed: Input/output error (5)                                                                            
    rsync: recv_generator: failed to stat "/bkup/u/hadjx1.pro": Input/output error (5)                                                             
    rsync error: some files could not be transferred (code 23) at main.c(1524) [generator=3.0.3]         
    
    
     
    #6 bread-baker, May 30, 2012
    Last edited: May 30, 2012
  7. e100

    e100 Active Member
    Proxmox VE Subscriber

    Joined:
    Nov 6, 2010
    Messages:
    1,159
    Likes Received:
    8
    Using sshfs for VM storage is certainly an intriguing idea.

    The main issue I see is that SSH uses a lot of CPU unless you have hardware accelerated crypto.
    Many CPU's have AES-NI acceleration but squeeze does not have AES-NI enabled for SSH.

    The fastest cipher, without hardware acceleration, is arcfour.
    If you are using sshfs on a trusted network using arcfour cipher for the sshfs connection should help with performance and reduce your CPU usage.

    Just pass the option when mounting sshfs:
    Code:
    -o Ciphers=arcfour
     
  8. RobFantini

    RobFantini Member
    Proxmox VE Subscriber

    Joined:
    May 24, 2012
    Messages:
    956
    Likes Received:
    5
    hi there e100

    i have this in our global ssh_config. do you know if arcfour weaknesses are fixed? In any case It would be great id sshfs issue could get fixed. I've an extra server that I can use for testing if any one has suggestions for retesting sshfs.
    Code:
    # Changing the ciphers used by SSH to less cpu-demanding ones can improve speed. In this aspect, the best choices are arcfour and blowfish-cbc. Please do not do this unless you know what you are doing; arcfour has a number of known weaknesses. To use them, run SSH with the "c" flag, like this: 
    # 2012-04-21 put blowfish-cbc 1-st as 'arcfour has a number of known weaknesses.'
    Ciphers blowfish-cbc,arcfour
    
     
    #8 RobFantini, May 31, 2012
    Last edited: May 31, 2012
  9. e100

    e100 Active Member
    Proxmox VE Subscriber

    Joined:
    Nov 6, 2010
    Messages:
    1,159
    Likes Received:
    8
    arcfour, AFAIK, is the weakest cipher that SSH supports and it can not be fixed since the weaknesses are inherent to the algorithm.
     

Share This Page