Proxmox File Server - Attach existing drive, reallocate storage or ???

[kamiller42]

I'm looking for advice from Proxmox pros.

I currently have a 2 ext4 2TB drives for storage. They are part of home server which has been running for 6 years in a configuration I call a poor man's RAID 1. I would mirror 1 drive to the other via a nightly rsync cron job.

I've rebuilt my server with new hardware and am setting up Proxmox containers and VMs to accomplish certain tasks (pfSense, Win IIS server, media serving, etc.). Containers sounded great for a file serving service, but NFS4 is not possible in a container. So I setup a KVM with the intent to share the root of 1 2TB drive via a shared folder like in VMware or maybe bind mount points in a container. Bind mount points are not an option for a KVM, and the closest I found to shared folders is setting up a NFS share, which seems far from optional.

I read the threads saying running a file server in a KVM is not advised, which is why I wanted a container. However, I don't want to turn the host into a file server. In my mind, the host should be for virtualization and management thereof and that's it. Am I wrong?

The question:
Should I abandon efforts to share or attach my storage to a KVM and allocate 1 drive to KVM's storage using RAW storage, or proceed to attempt sharing the drives via an NFS share created on the host (or via some other method)? Or, do something else entirely? I'm open to ideas.

 

[kamiller42]

No one uses Proxmox with a file server?

This is as close to advice I could find on setting one up and even he is unsure its optimal setup.

http://people.binf.ku.dk/~hanne/b2evolution/blogs/index.php/2011/08/18/virtualization-with-proxmox

 

[1nerdyguy]

No one uses Proxmox with a file server?

This is as close to advice I could find on setting one up and even he is unsure its optimal setup.

http://people.binf.ku.dk/~hanne/b2evolution/blogs/index.php/2011/08/18/virtualization-with-proxmox

I've setup Fileservers in Proxmox just like I would on hardware. KVM images, either Win2k8/Win2k12 or Ubuntu 12.x, both currently running. With VirtIO drivers and bonded nics, I'm able to get decent bandwidth, and haven't had a problem. The Win2k8 machine has been up serving files for 2 years now, so I call it good.

 

[kamiller42]

I've setup Fileservers in Proxmox just like I would on hardware. KVM images, either Win2k8/Win2k12 or Ubuntu 12.x, both currently running. With VirtIO drivers and bonded nics, I'm able to get decent bandwidth, and haven't had a problem. The Win2k8 machine has been up serving files for 2 years now, so I call it good.

Thanks for the reply. Is the Win2k8 server serving files from directories stored on its virtual disks, i.e. RAW or qcow2 or VMDK? Or, is the storage source independent and linked into the KVM?

 

[1nerdyguy]

Thanks for the reply. Is the Win2k8 server serving files from directories stored on its virtual disks, i.e. RAW or qcow2 or VMDK? Or, is the storage source independent and linked into the KVM?

It's serving from it's own virtual disks, yes. Setup in qcow2. They are hosted on a OmniOS NAS, which has a 30 disk ZFS pool as storage with 5 256GB SSD's for cache drives in front of them. It's connected to the proxmox cluster via 4 bonded 1gig nics into a gig switch, each node connected in. Mounted to proxmox as NFS storage.

Works great!

 

[WaltervdSchee]

I'm running a fileserver in a KVM , using a RAID-5 device which is configured as a SCSI device to the KVM.
scsi0: /dev/sdc1 (partitioned RAID5 device)
It works fantastically, I get 105 to 110 MBytes/s with it, in both up and down link to the KVM based fileserver (CentOS 6.5).
Hardware has recently changed to a 8-core Avoton CPU (C2750)

 

[kamiller42]

So, if you were to examine the contents of the RAID5 in a host OS SSH session, it would be holding a giant qcow file which holds the files being served. Correct?

If you have another KVM or a container needs access to file server storage, it's done via a NFS share?

 

[WaltervdSchee]

So, if you were to examine the contents of the RAID5 in a host OS SSH session, it would be holding a giant qcow file which holds the files being served. Correct?

If you have another KVM or a container needs access to file server storage, it's done via a NFS share?

No, the partitioned RAID-5 device is handed over as a 'raw' device to the KVM (SCSI disk).
Within the KVM it is handled as "/dev/sda"

The device is then formatted within the KVM as a large single disk, (16.5 TB) and formatted XFS.

The RAID-5 disk as seen on the ProxMox host :

sd 16:0:0:0: [sdd] 39069286400 512-byte logical blocks: (20.0 TB/18.1 TiB)

gfdisk output:
   1            2048      3221227519   1.5 TiB     0700
   2      3221227520      4164945919   450.0 GiB   0700
   3      4164945920     39069284351   16.3 TiB    0700

Device /dev/sdd3 is then placed as scsi0: device in the config for the KVM

(Config file)
bootdisk: virtio0
cores: 4
cpuunits: 35000
memory: 4096
name: <kvm server>
net0: virtio=<mac>,bridge=vmbr0
onboot: 1
ostype: l26
scsi0: /dev/sdd3
sockets: 2
startup: order=2,up=20
virtio0: VMStore:vm-1001-disk-1,size=50G
virtio1: local:1001/vm-1001-disk-1.qcow2,format=qcow2,size=75G

Device as seen by KVM based guest

sd 2:0:0:0: [sda] 34904338432 512-byte logical blocks: (17.8 TB/16.2 TiB)
sd 2:0:0:0: [sda] Attached SCSI disk
sd 2:0:0:0: Attached scsi generic sg0 type 0
XFS (sda): Mounting Filesystem
XFS (sda): Starting recovery (logdev: internal)
XFS (sda): Ending recovery (logdev: internal)

mount output:

/dev/sda                        17T  7.1T  9.3T  44% /data

In the KVM guest, you can then export the filesystem as NFSv3/4 and/or SMB as you see fit to configure, since its a local disk as far as the KVM is concerned.
This way of working is definitely NOT cluster/HA compatible but on a single server at home, it works like a champ !
Hope this helps clarify things.

 

[kamiller42]

Can you give me a reference to how to attach raw drives to a KVM? What I know of storage in Proxmox is bind mount points for containers and this... http://pve.proxmox.com/wiki/Storage_Model

 

[kamiller42]

Never mind question. I found this... http://forum.proxmox.com/threads/9451-Adding-Multiple-Physical-Disks-to-VM

Looks like 2 strategies to setting up file server:
1. The way 1nerdyguy did it with drives hosting a qcow2 file holding all of the files being served.
2. Attaching a physical device to file serving VM and setup the devices in the VM.

#2 seems the most intuitive to me, but does have the cons of not being an emulated drive.

Any think of a simple pro/con list of each approach?