additional hdd as storage

[napalmsilly]

hi,

proxmox VE 4.1 is running on an ssd on my home server. i would like to add three attached hdd to the filesystem; preferably as one logical volume (raid 5?).

i have never configured a raid, or even attached additional storage, on linux before. usually i worked with one drive which had been auto configured during install. did some google searching...

lsblk shows my drives like this
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 232.9G 0 disk
├─sda1 8:1 0 1007K 0 part
├─sda2 8:2 0 127M 0 part
└─sda3 8:3 0 232.8G 0 part
├─pve-root 252:0 0 58G 0 lvm /
├─pve-swap 252:1 0 19G 0 lvm [SWAP]
└─pve-data 252:2 0 139.8G 0 lvm /var/lib/vz
sdb 8:16 0 3.7T 0 disk
└─sdb1 8:17 0 3.7T 0 part
sdc 8:32 0 3.7T 0 disk
└─sdc1 8:33 0 3.7T 0 part
sdd 8:48 0 3.7T 0 disk
└─sdd1 8:49 0 3.7T 0 part

sdb, sdc and sdd need to be configured. i have created partitions with gdisk for each physical drive. i might have already made a mistake there because those created "subdrives" (sorry don't know the correct term) have different file systems than the main ones.

i have tried to mount those files by entering them in /etc/fstab but something must have gone wrong here as well because proxmox would not boot correctly anymore. so i re-installed the system.

am i approaching this the right way? i had already united those drives with hp smart storage administrator tool. this only offered raid0 though. however, no raid seems to be existing under the proxmox environment (or i don't know the correct commands to find it).

i would like to provide this raid via proxmox to a virtual machine, which will encrypt it and then provide this encrypted drive to other vm as a file server. so each user can have their own directories and permissions. or should i do this directly through proxmox and then provide it to all the machines? i was planning on leaving the proxmox installation as plain as possible and have different vm handle administrative work.

sorry if this is a noob question. i would just like to make sure to configure proxmox the best way possible as it will be the foundation for all the vms. don't mind trial and error when working on a specific vm which i can just reset to a working snapshot. it is, however, a bit of a pain to keep reinstalling proxmox itself...

thanks in advance for your help.

 

[LnxBil]

hi,

proxmox VE 4.1 is running on an ssd on my home server. i would like to add three attached hdd to the filesystem; preferably as one logical volume (raid 5?).

i have never configured a raid, or even attached additional storage, on linux before. usually i worked with one drive which had been auto configured during install. did some google searching...

lsblk shows my drives like this
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 232.9G 0 disk
├─sda1 8:1 0 1007K 0 part
├─sda2 8:2 0 127M 0 part
└─sda3 8:3 0 232.8G 0 part
├─pve-root 252:0 0 58G 0 lvm /
├─pve-swap 252:1 0 19G 0 lvm [SWAP]
└─pve-data 252:2 0 139.8G 0 lvm /var/lib/vz
sdb 8:16 0 3.7T 0 disk
└─sdb1 8:17 0 3.7T 0 part
sdc 8:32 0 3.7T 0 disk
└─sdc1 8:33 0 3.7T 0 part
sdd 8:48 0 3.7T 0 disk
└─sdd1 8:49 0 3.7T 0 part

sdb, sdc and sdd need to be configured. i have created partitions with gdisk for each physical drive. i might have already made a mistake there because those created "subdrives" (sorry don't know the correct term) have different file systems than the main ones.

First, a partition (your subdrive) does not have a filesystem. This are different things.

i have tried to mount those files by entering them in /etc/fstab but something must have gone wrong here as well because proxmox would not boot correctly anymore. so i re-installed the system.

Yes, this happens if the wrong fstab-entries are used. Normally, this can be fixed very easily, but a new install is also possible (as the most extreme fix).

am i approaching this the right way? i had already united those drives with hp smart storage administrator tool. this only offered raid0 though. however, no raid seems to be existing under the proxmox environment (or i don't know the correct commands to find it).

What raid controller is that? Normally, you have devices like /dev/cciss or simply /dev/sda as the RAID volumes. Not every RAID-controller is a hardware RAID controller, which are almost all supported in Linux, but they cost (new) at least something around 100 EUR/$ and are not common in consumer hardware.
Proxmox only supports hardware RAID or RAID-z from ZFS, no software raid, which can also be done. If you don't mind, you can also set up software raid, but you will not get support if you'd have payed for it.

i would like to provide this raid via proxmox to a virtual machine, which will encrypt it and then provide this encrypted drive to other vm as a file server. so each user can have their own directories and permissions. or should i do this directly through proxmox and then provide it to all the machines? i was planning on leaving the proxmox installation as plain as possible and have different vm handle administrative work.

Performance-technically speaking, you want to have it on your host, not in a VM. I do not know if this supported in Proxmox, but it is in plain Debian. Please google for LUKS Debian in you language to get some easy instructions.

sorry if this is a noob question. i would just like to make sure to configure proxmox the best way possible as it will be the foundation for all the vms. don't mind trial and error when working on a specific vm which i can just reset to a working snapshot. it is, however, a bit of a pain to keep reinstalling proxmox itself...

Normally, you don't have to do this. A good working knowledge of Linux is surely a plus :-D

In general, I would say that Proxmox is easier than VMware, but Proxmox does support more hardware setups (at least in contrast to VMware 5.5) and more setup options are therefore available. But all hardware that is supported in VMware should work out-of-the-box in Proxmox.

Proxmox is very easy to work with if you're already familiar with Debian.

 

[napalmsilly]

Thank you. I am installing proxmox once more now.

hp smart storage administrator is a tool to create raids to use with hp proliant servers. i did not use it this time. its only option was to create a raid0 from my hdds anyway, which does not suit my needs.

my last attempt was to create a software raid with parted and mdadm; this kinda worked, however, i did not have the correct amount of data available for my raid (only ~7.5TB when it should have been ~12TB). then again, this was just a software raid. i will try raid-z from zfs this time.

once a raid is created and mounted, how do i make it available to proxmox VM? i tried adding it as storage on the web gui. this works, but several folders are automatically created (images, backup, lost+found, depending on settings). i do not need any of this on my raid.

sda is my SSD, containing root, data and swap partitions for proxmox.
sdb, sdc and sdd are HDD, supposed to be a raid containing all kinds of data (and also for snapshots of pve vm).

what is the best solution to share this raid between several vm? each vm is supposed to have different access to this raid. for example
vm1 - fileserver - /files/ and subfolders
vm2 - media server - only /files/media and subfolders
vm3 - syncthing - only /files/sync
etc.

 

[LnxBil]

Thank you. I am installing proxmox once more now.

hp smart storage administrator is a tool to create raids to use with hp proliant servers. i did not use it this time. its only option was to create a raid0 from my hdds anyway, which does not suit my needs.

I seldomly non-linux based raid configuration tools. What RAID controller in Proliant servers allows JBOD? None of my Pxxx (standard HP DL-based hardware raid controller) does that. What server is that your working on?

my last attempt was to create a software raid with parted and mdadm; this kinda worked, however, i did not have the correct amount of data available for my raid (only ~7.5TB when it should have been ~12TB). then again, this was just a software raid. i will try raid-z from zfs this time.

Depends on what kind of mdadm raid you used to create the raid. Why didn't you use hardware raid for your data?

once a raid is created and mounted, how do i make it available to proxmox VM? i tried adding it as storage on the web gui. this works, but several folders are automatically created (images, backup, lost+found, depending on settings). i do not need any of this on my raid.

If you use ZFS, then simply add ZFS as storage device and Proxmox internally uses zvol for storing qemu-based vms.

sda is my SSD, containing root, data and swap partitions for proxmox.
sdb, sdc and sdd are HDD, supposed to be a raid containing all kinds of data (and also for snapshots of pve vm).

So, there is no redundancy for your OS? The SSD will probably fail first, depending on the SSD and system usage.

what is the best solution to share this raid between several vm? each vm is supposed to have different access to this raid. for example
vm1 - fileserver - /files/ and subfolders
vm2 - media server - only /files/media and subfolders
vm3 - syncthing - only /files/sync
etc.

You want to access the same files from different VMs? Why do you want this complex setup? Normally you can achieve this by using samba and ACLs only. Your fileserver can also share media and sync-stuff, so it's redundant.

 

[napalmsilly]

I am working on a HP Proliant ML10 v2 server. I could not find much information on how to set up raid on this machine. The only tool by HP allowing me to configure raid is the smart storage administrator. it is an image i can boot from and set up raid. however, it only allows raid0. i did not find any possibility to set up hardware raid at all.

I thought I would let different VM use the some of the same files in order to protect others; for example, I do not need to grant access to my media server to directories only used for syncing. If I do this via samba, will proxmox determine that those samba shares are on its local storage or would everything still be sent through the network even though all storage is on the same machine?

 

[LnxBil]

I am working on a HP Proliant ML10 v2 server. I could not find much information on how to set up raid on this machine. The only tool by HP allowing me to configure raid is the smart storage administrator. it is an image i can boot from and set up raid. however, it only allows raid0. i did not find any possibility to set up hardware raid at all.

Ah okay, entry level. I only worked on the DL- and BL-line (rack-mounted, almost everything redundant or blade).

The Smart Storage Admin is IMHO a Linux-based disk, which boots firefox, so it is (technically) working in Linux and you should get a tool to configure your raid. Sometimes you have to look up the "real" hardware behind the rebranded HP stuff, e.g. with lspci to get the chipset of the raid card and then search the internet for the corresponding management tools, e.g. MegaCli and others.

I thought I would let different VM use the some of the same files in order to protect others; for example, I do not need to grant access to my media server to directories only used for syncing. If I do this via samba, will proxmox determine that those samba shares are on its local storage or would everything still be sent through the network even though all storage is on the same machine?

Yes and no. network in Proxmox does not mean using your hardware network card, but it is still valid network traffic (OSI-speaking), but instead of real transmitting it is done via memory copy from one VM to the other - virtual network so to speak. It is of course slower than using one disk and one vm.

You have the problem that you have to figure out how to share the data between the machines. Technically this is more complex (and therefore slower) than storing all files on one server and using ACLs to distinguish. You can even have samba simulate more server for you that mimic 3 machines, but only allow to access to some directories/files.

 

[napalmsilly]

The Smart Storage Admin is IMHO a Linux-based disk, which boots firefox, so it is (technically) working in Linux and you should get a tool to configure your raid. Sometimes you have to look up the "real" hardware behind the rebranded HP stuff, e.g. with lspci to get the chipset of the raid card and then search the internet for the corresponding management tools, e.g. MegaCli and others.

That is correct. While SSA loads, it even displays Firefox in the window title for a second before maximizing it. There is a tool to configure raid. I have Dynamic HP Smart Array B120i RAID Support. This is enabled (ROM based Setup Utility - System Options - SATA Controller Options).
How do I look this up? Will those management tools be another ISO I need to boot from? The HP one does not work very well. I do not understand why its only option is RAID0, either.

I am almost at the point where I return the HP Proliant and look for a different server. This is all very confusing.

 

[LnxBil]

That is correct. While SSA loads, it even displays Firefox in the window title for a second before maximizing it. There is a tool to configure raid. I have Dynamic HP Smart Array B120i RAID Support. This is enabled (ROM based Setup Utility - System Options - SATA Controller Options).
How do I look this up? Will those management tools be another ISO I need to boot from?

I'm unfamiliar with these "cheap" raid controllers.

The HP one does not work very well. I do not understand why its only option is RAID0, either.

That's easy: The controller is not able to do it or almost certainly because you haven't bought it. A lot of useful stuff is not included to push the price down. I would never buy a server without redundant power supply and disk (again - I used a decade ago, but times and available money changes).

I am almost at the point where I return the HP Proliant and look for a different server. This is all very confusing.

I don't think that it'll change much: You get what you pay for and in this price segment, there is not much variation in features.

It is very hard to say what's best for you, because I don't know your environment, but normally, this "server" should be OK for a small office, yet you need a backup plan if something breaks.

 

[napalmsilly]

Basically, I just need a VM and fileserver for my private network.

Main focus will be keeping files in sync between desktop and two laptops. Besides that, I need to provide my media files to several (well, 2) Kodi clients. In order to keep those in sync (played/unplayed status etc.) I need an sql server. CUPS server in order to print and scan to / from all clients.

Then there is a usenet client connected to CouchPotato and Sonarr.

That is pretty much all I need for regular use. I would like some VM to play around with ubuntu server as well as having some "trial and error machines" to just practice my linux skills on. Or just to test services I don't really need, but would like to understand (local git server,

At the moment, I am still using a Synology DS413 as a fileserver and to sync my data. Everything else is handled by raspberry pis (not very well, I might add). I thought having VMs and file server in one machine would be the right way to handle this. The synology was supposed to become a mere backup server for everything stored on the "new" server in case of data loss.

afaik, all this could be done with an intel NUC as well, except for storing all data as well. If I kept the Synology the way it is now and had a NUC handle the virtual machines, it'd probably be sufficient. Maybe the HP solution was just overkill for my needs.

 

[LnxBil]

For a home setup, this server is good enough. I'd suggest you have your files on your proxmox host and export it via NFS or SAMBA.

I have a similar setup at home and store my files local (only samba and afp is local) and only my "toys" runs in a VM. The additional overhead for using VM for fileserver is IMHO too much.

If you want to use ZFS, which is very cool FS, you can incrementally backup your filesystems. That's very neat.

 

[napalmsilly]

I am just not quite sure how to figure out this raid problem. Will I still get efficient performance if i create a software raid, create a samba share (or ZFS) to provide the entire raid to a VM, then let this VM handle everything else (permissions, encryption)?

 

[LnxBil]

ZFS works better without RAID with just the drives itself - it will be slower on RAID (hardware or software).

Performance will always suffer if you want to do encryption in a VM. Why is there virtualization involved at all?

 

[napalmsilly]

To be honest, the main reason for VM is that I have a usenet subscription which I would like to exclusively have connected through a VPN (for paranoia / tinfoil hat reasons).

Also, I am not that familiar with linux. I know the basic stuff, but (obviously) stuff like setting up a raid, for example, gives me real trouble. I used to just have a couple of raspberry pis as linux (debian/raspbian) machines to learn with. However, if I make a mistake there, I can end up setting up my pi from scratch; in a VM, I'd just go back to my last working snapshot.
(sure, I can just have a VM or two on my laptop or desktop, but those don't run 24/7)

My current setup (3 raspberry pis, Synology DiskStation as NAS) works okay-ish. I was hoping to optimize by having one dedicated VM host that provides hardware and has decent power (2x3.1GHz, 20GB RAM) as well as disk storage (3x4TB) instead of using several ARM devices that need to read and write over the network.
Also, I do not have an extra backup for "unimportant" data. The important files are synced on three computers, but everything else is only stored on the LAN and would be gone in case of hardware failure. So I was planning on using the Synology as an offsite-backup that syncs with my VM server via VPN and can always restore its data (or vice versa - I am not expecting both failing at the same time).

Has somebody here worked with plain Proxmox* on an HP Proliant ml10 v2 and/or knows a fool-proof way to set up ZFS as raid, encrypt it, then provide it to a VM? (*not installing Debian first, then installating Proxmox on top of it)

I am not sure whether or not I had just been to enthusiastic when buying this hardware (wasn't exactly cheap for "just for fun" use) or if it is actually worth it. While I haven't figure out the raid situation yet, I did create a few VM already and it is pretty nice working with those.

 

[LnxBil]

If you only want to use ZFS on the 4 SATA disks, please have a look at
https://pve.proxmox.com/wiki/ZFS#Create_a_new_pool_with_RAIDZ-1

This creates something similar than RAID-5. ZFS on non-Solaris-based machines does not offer encryption directly, so you have to use Luks for that.

I encrypt my external backup disks and put an ZFS on it afterwards. But keep in mind: this is VERY performance consuming! I have a Dual Xeon E5640 (8 cores + 8 HT cores) and I "only" get 80-100 MB/sec on Luks-encrypted-then-on-top-ZFS. For best encryption, data is distributed across the whole disks, so sequential data will not be sequential anymore.

You have to test how fast it is for you.

 

[napalmsilly]

Thanks for all the input.

I will return this ml10 and research some better hardware (including hardware raid) to use. There is no point using this system the way things work. Might as well stick with the raspberry pi solution until I can purchase a better hardware setup.

 

[LnxBil]

Faster for disk encryption? That's going to be very expensive. For home use, your machine is totally OK, but don't expect much speed with full disk encryption. Is partial-disk-encryption maybe going to work?

My home system (PVE 4.1) has also 4 SATA disks with actually software raid 5 and separate ZFS, and it saturates GBit from my client. That's perfect, but it's not encrypted. Encryption is much, much slower. At about 12 MB/sec, enough for most things, but still.