[SOLVED] Basic setup for simple Home-Server (where to store the data)

[EinsGehtNoch]

Hello Friends,

I have assembled a home-server. Mainly to replace my deprecated 2-bay NAS, give me a real web-server (e.g. NextCloud) and a place to store a remote desktop. Further applications will follow as needed. I like to keep my functions strongly seperated, since I tend to mess around sometimes. This messing around should now not affect my file-server and nextcloud, since other people need it too

I have installed without bigger Problems Proxmox and for testing Ubuntu Server and Ubuntu Desktop (both as VMs) within Proxmox. No big problems here either.

I am struggling now with the basic set-up for the main functions for file-server and data-server for nextcloud. I have been reading for the last weeks, but cannot figure out the right way

What does make more sense to you?

1. Let Proxmox handle the file-server storage system (ZFS mirror / Software RAID 1) and somehow let the file server VM (Ubuntu Server) access it.
   
2. Pass the hard-disks directly to the file-server, and create a RAID there.

The first option seems more straightforward to me. Also the host (Proxmox) claim unused system ressources as needed. But what would be the best way, to access the ZFS storage from within the VM?



==========================================
Hardware (for now):
Supermicro X11SSM-F
G4400
16 GB ECC RAM
32 GB SSD (System HDD: Proxmox, VMs, ISOs, ...)
2x3 TB HDD from old NAS for data storage
(2x 4 TB HDD for various use and for replacing one of the older 3 TB HDDs if necessary)

 

[sigxcpu]

For home storage I'm using some ZFS pools (mirror for important data, striped for other) and export them through FTP, AFP, CIFS, WebDAV using containers (you can mount host folders inside a container).

 

[EinsGehtNoch]

(you can mount host folders inside a container).

But you cannot mount them inside of a VM, right?

 

[sodden]

I'm currently switching from your option 2 to 1 because it makes things a lot simpler and more manageable.
I'm using a combination of NFS and autofs for filesystem access between host and VMs. If you're using ZFS you can simply set the sharenfs option on a dataset and mount it wherever you need.

 

[EinsGehtNoch]

Thanks. I put myself a little bit in a dead end now.

For option 1, I have problems sharing the folders between the host and the guest. Doing this with my limited knowledge is harder than I thought. With the tutorials I can find, I don't get a write access to the folders on the host. Even though I do this by using an internal network, that only talks between the host and one guest, I am not sure if that is safe from a security point of view.

At the end, the file server VM should be running Samba to distribute the files within the network. So basically create a ZFS on the host, share it with NFS to a guest, then distribute it from there with samba to the world . Doing my research, I found then a lot of references, that you shoud avoid exactly that (use samba to share a nfs-mount). If this is still a valid contstraint, is above my knowledge.

At the moment I am reconsidering my whole idea of running proxmox as a hypervisor and some VMs from within and maybe setup a simple file server.

I would be curious, if other users have the same struggle, or if my approach of running a small, dedicated server at home under proxmox is just complete overkill.

 

[sodden]

Overkill...maybe, depends on what you are trying to achieve.

Permissions and NFS can be a pain. The most common gotcha is a UID/GID mismatch. I don't know how you've set it up but if you try to access an NFS share with user X that exists on both systems, that user must also have the same UID/GID on the system you are accessing from to be able to read or write. (Hope that makes sense)

By default NFS maps IDs 1:1, to circumvent this you can set some options that map incommig IDs to a specific ID.
I can really recommend reading the archlinux wiki on NFS. (https://wiki.archlinux.org/index.php/NFS and https://wiki.archlinux.org/index.php/NFS/Troubleshooting)

As for sharing via Samba. I think you're over complicating things. Just install Samba on your proxmox host and publish your folders from there. No need for a dedicated Samba machine.

Edit:
Made a little diagram.

Host           Storage                  Service
proxmox ->     tank/Videos              NFS/SMB
proxmox ->     tank/Music               NFS/SMB
vm100   ->     (NFS)/Videos               MediaServer
vm101   ->     tank/vmdatastore         MySQL Server

 

[udo]

Thanks. I put myself a little bit in a dead end now.

For option 1, I have problems sharing the folders between the host and the guest. Doing this with my limited knowledge is harder than I thought. With the tutorials I can find, I don't get a write access to the folders on the host. Even though I do this by using an internal network, that only talks between the host and one guest, I am not sure if that is safe from a security point of view.

At the end, the file server VM should be running Samba to distribute the files within the network. So basically create a ZFS on the host, share it with NFS to a guest, then distribute it from there with samba to the world . Doing my research, I found then a lot of references, that you shoud avoid exactly that (use samba to share a nfs-mount). If this is still a valid contstraint, is above my knowledge.

At the moment I am reconsidering my whole idea of running proxmox as a hypervisor and some VMs from within and maybe setup a simple file server.

I would be curious, if other users have the same struggle, or if my approach of running a small, dedicated server at home under proxmox is just complete overkill.

Hi,
why not the default way? OK, it's depends, where you need the data too...
With default way I mean, your host provide (zfs)storage, which you use for an fileserver-VM. And inside the VM you have all data.
You can use qcow2 or install the fileserver as an container (depends which services you need).

Udo

 

[sodden]

Hi,
why not the default way? OK, it's depends, where you need the data too...
With default way I mean, your host provide (zfs)storage, which you use for an fileserver-VM. And inside the VM you have all data.
You can use qcow2 or install the fileserver as an container (depends which services you need).

Udo

This is also possible. But adds another layer to your data that is not really necessary. If your Fileserver-VM breaks that data is gone. However if you just use your native filesystem (ZFS etc.) to store your data you can only really lose it due to a Hardware failure. On top of that you can just plug and play the Harddrives into another system and access your files like normal should your server break.

 

[EinsGehtNoch]

Hi,
why not the default way? OK, it's depends, where you need the data too...
With default way I mean, your host provide (zfs)storage, which you use for an fileserver-VM. And inside the VM you have all data.
Udo

I guess my application is not the default way. Creating container images inside the zfs does simply not make sense to me.

Overkill...maybe, depends on what you are trying to achieve.

Permissions and NFS can be a pain. The most common gotcha is a UID/GID mismatch. I don't know how you've set it up but if you try to access an NFS share with user X that exists on both systems, that user must also have the same UID/GID on the system you are accessing from to be able to read or write. (Hope that makes sense)ySQL Server
[/CODE]

Thanks again for the expertise! As you expect, I am already struggling with the different owner/groups and permission settings between the host and the guest . Running the samba-share on the host sound fine. But I also want to be able to run than a web server in the guest, which will result in the same questions, and then a media server, and then ...
At the end it turns out, I run everything on the host, which was not the idea of separating the appliances

 

[sodden]

I think you misunderstood what I was trying to say.
My point was that it is not necessary to put Virtual Fileserver on top of proxmox. Lets not forget, it's just Debian with a pve enabled kernel. And since it has already most of the packages installed you would need for a basic fileserver anyway, why not use it as such.
For everything else you would use a VM of course.

 

[udo]

I think you misunderstood what I was trying to say.
My point was that it is not necessary to put Virtual Fileserver on top of proxmox. Lets not forget, it's just Debian with a pve enabled kernel. And since it has already most of the packages installed you would need for a basic fileserver anyway, why not use it as such.
For everything else you would use a VM of course.

Hi,
of course you can do that - esp. as homeserver - but it's has some benfits to run on the host hypervisor related things only (security, separation, ...).
Like to expand the Storage to shared storage and use live migration to use the fileserver during pve-updates too.
On my homeserver are some TB for an fileserver-container (not optimal for nfs - for this was kvm better).

Udo

 

[sodden]

Hi,
of course you can do that - esp. as homeserver - but it's has some benfits to run on the host hypervisor related things only (security, separation, ...).
Like to expand the Storage to shared storage and use live migration to use the fileserver during pve-updates too.
On my homeserver are some TB for an fileserver-container (not optimal for nfs - for this was kvm better).

Udo

That's true. For availability you would definitely run a virtual fileserver. But I doubt that many of us home users have a proxmox cluster running

Edit:

@EinsGehtNoch:
As for your NFS troubles, try this.

on proxmox:
create a group that should be able to access files from an nfs share.
groupadd --gid 1200 <groupname>

set ownership/permissions on the nfs share
chgrp -R <groupname> /path/to/nfs/share
chmod g+rw /path/to/nfs/share

on guest:
create the same group on the guest system and add your user
groupadd --gid 1200 <groupname>
usermod -a -G <groupname> <youruser> (you have to logout and back in for this to take effect)

try accessing the nfs share from your guest vm. If you did everything correctly ls -l should also display the same group name on both systems.

 

[EinsGehtNoch]

Thank's for the help, but I just don't get it to work.

On the host the permission of tank/test permission and owner is
775 root:nfs-share (nfs-share beeing gid=1200)

On the guest it is mounted right and i see as permissions
775 root:nfs-share (nfs-share again being gid=1200)

id shows me, that I am also in the group nfs-share (1200), but still no write access).

I can just not figure out, what is wrong. Also at the moment I don't know, where to look for the fault. Is it a problem of Proxmox/Debian or of the guest/Ubuntu. Or is it something wrong with the share of the zfs file system at all.

 

[sodden]

Thank's for the help, but I just don't get it to work.

On the host the permission of tank/test permission and owner is
775 root:nfs-share (nfs-share beeing gid=1200)

On the guest it is mounted right and i see as permissions
775 root:nfs-share (nfs-share again being gid=1200)

id shows me, that I am also in the group nfs-share (1200), but still no write access).

I can just not figure out, what is wrong. Also at the moment I don't know, where to look for the fault. Is it a problem of Proxmox/Debian or of the guest/Ubuntu. Or is it something wrong with the share of the zfs file system at all.

How does your "sharenfs" option for tank/test look like ?
It's definitely a permission problem.

 

[EinsGehtNoch]

root@pve:~# zfs get sharenfs
NAME            PROPERTY  VALUE                             SOURCE
tank       sharenfs  off                               local
tank/be    sharenfs  off                               inherited from tank
tank/fi    sharenfs  off                               inherited from tank
tank/ho    sharenfs  off                               inherited from tank
tank/mu    sharenfs  off                               inherited from tank
tank/test  sharenfs  rw=@192.168.200.2  local

192.168.200.1 being proxmox-host
192.168.200.2 being the guest

root@pve:~# ls -l /tank/
total 3
drwxr-xr-x 2 root root          2 Jan  7 15:37 be
drwxr-xr-x 2 root root          2 Jan  7 15:37 fi
drwxr-xr-x 2 root root          2 Jan  7 15:37 ho
drwxr-xr-x 2 root root          2 Jan  7 15:37 mu
drwxrwxr-x 2 root nfs-exchange 3 Jan  7 14:34 test

 

[sodden]

Try sharing to your local subnet rw=@192.168.200.0/24 or just set sharenfs=on. If I remember correctly you don't need the @ if you're sharing to a single host.

Edit:

From: https://linux.die.net/man/5/exports

netgroups
NIS netgroups may be given as @group. Only the host part of each netgroup members is consider in checking for membership. Empty host parts or those containing a single dash (-) are ignored.

 

[mir]

Add noacl to your proxmox nfs mount options. ZFS' acl implementation is not compatible with the POSIX acl implementation which is used in Linux.

 

[sodden]

Add noacl to your proxmox nfs mount options. ZFS' acl implementation is not compatible with the POSIX acl implementation which is used in Linux.

I'm not using acls myself so I can neither confirm nor deny that. But from reading the man page I think this is not necessary or the issue.

acltype=noacl | posixacl

Controls whether ACLs are enabled and if so what type of ACL to
use. When a file system has the acltype property set to noacl (the
default) then ACLs are disabled. Setting the acltype property to
posixacl indicates Posix ACLs should be used. Posix ACLs are spe-
cific to Linux and are not functional on other platforms. Posix
ACLs are stored as an xattr and therefore will not overwrite any
existing ZFS/NFSv4 ACLs which may be set. Currently only posixacls
are supported on Linux.

To obtain the best performance when setting posixacl users are
strongly encouraged to set the xattr=sa property. This will result
in the Posix ACL being stored more efficiently on disk. But as a
consequence of this all new xattrs will only be accessable from ZFS
implementations which support the xattr=sa property. See the xattr
property for more details.

 

[EinsGehtNoch]

Still no luck. At this moment I am even too annoyed to just give up. Let me walk you through the process, hopefully it helps me and others.

Setup is a server with two hard-disk (/dev/sda /dev/sdb) and a network interface eth0 connected to a router. Commands are kept short. On a real system further tuning might be necessary (e.g. set right -ashift for ZFS)

First steps on the host (Proxmox):

1. Install Proxmox (V4.4) on a Server (/dev/sda is used here for the host system). Apply all updates.
2. Install packages for nfs
   

   apt-get install nfs-kernel-server

3. Add dummy entry to /etc/export to get the nfs-system running at start-up
   

   /mnt localhost(ro)

4. reboot
5. Create ZFS system:
   

   zpool create -f tank /dev/sdb
   zfs create tank/one
   zfs create tank/two
   zfs create tank/test

6. Enable nfs-sharing to the 192.168.200.0 net for one of the file-systems
   

   zfs set sharenfs="rw=@192.168.200.0/24" tank/test

7. Add a group for nfs and assign a zfs-filesystem to that group with rw-Rights
   

   groupadd -gid 1300 nfs-share
   chgrp -R nfs-share /tank/test
   chmod g+rw /tank/test

Guest creation and set-up (Ubuntu-Desktop 16.04):
Network device eth0 bridged to vmbr0 to connect to the internet

1. Still on the proxmox node add a "Linux Bridge" (vmbr100) and assign 192.168.200.1 to it
   
2. Create a VM and add a second "network device" with bridge=vmbr100
3. Install Ubuntu and update
   

   sudo apt update && sudo apt dist-upgrade

4. Install packages for nfs
   

   sudo apt-get install nfs-common

5. Apply IP 192.168.200.2 to the network card bridget to vmbr100. The first one will get an IP from the router from a different sub-net.
   
6. Add the same group with same gid and assign group to user
   

   sudo groupadd -gid 1300 nfs-share
   sudo usermod -aG nfs-share <current-user>

7. reboot
8. create mount points for the nfs-share
   

   sudo mkdir /srv/tank
   sudo mkdir /srv/tank/test

9. Add entry to /etc/fstab
   

   192.168.200.1:/tank/test /srv/tank/test nfs rw 0 0

10. reboot

After this I am still not able to create and files or folder is the /srv/tank/test folder. If I create a test file from within proxmox, I can see and open it, but cannot change anything. What did I miss?

On the host:

root@pve:~# ls -n /tank/
total 2
drwxr-xr-x 2 0    0 2 Jan  7 17:31 one
drwxrwxr-x 2 0 1300 2 Jan  7 17:31 test
drwxr-xr-x 2 0    0 2 Jan  7 17:31 two

On the guest:

ich@heimserver:~$ id
[...] 1300(nfs-share)

ich@heimserver:~$ ls -n /srv/tank/
insgesamt 1
drwxrwxr-x 2 0 1300 3 Jan  7 17:47 test

Trying to create a file does result in an error:

ich@heimserver:~$ touch /srv/tank/test/testfile.txt
touch: '/srv/tank/test/testfile.txt' kann nicht berührt werden: Keine Berechtigung

 

[mir]

I'm not using acls myself so I can neither confirm nor deny that. But from reading the man page I think this is not necessary or the issue.

It is because ZFS NFS uses nfsv4 acl so not explicitly adding noacl means acl will be active since Linux nfs clients detects active acl and will use this. The problem with this is that the ZFS nfsv4 acl is incompatible with the Linux acl which is posix acl.