Getting started with PVE storage model - software RAID5 array and storage concepts

[lpallard]

Hello,

So following up my "debuts" with proxmox, I am now trying to grasp the concepts and storage model of PVE. First of all, I'd like to add a bunch of SATA drives (6 of them) that were previously assembled as a software RAID 5 array under Slackware Linux. I would like to reassemble them and reuse the array under a VM running CentOS 6.5

Some questions:

1. Software arrays do not seems to be supported in proxmox at all. How should I proceed? Should I assemble them with mdadm at proxmox's level then add the md device in the vmnumber.conf file with something like: ide0: /dev/md0 so the VM sees it as a single large 10TB drive, or should I pass-through all 6 drives unassembled to the VM then assemble the array at the VM level?

2. I have tried the second option (pass-through all 6 drives unassembled to the VM then assemble the array at the VM level) by adding these lines in the VM config file

ide0: /dev/sdb
ide1: /dev/sdc
ide2: /dev/sdd
ide3: /dev/sde
ide4: /dev/sdf
ide5: /dev/sdg

but unfortunately, in proxmox, I only see 4 of them. I booted up the VM and confirm that only 4 of the 6 drives are passed to the VM. Obviously, the array is not started because there are too many missing drives. Why are only 4 of the 6 drives passed to the VM by PVE? Is it a misconfig of some sort or are VM limited to 4 physical drives?

EDIT: User tom at thread http://forum.proxmox.com/threads/15942-Local-folder-or-IDE-drive-pass-through clearly stated that

you cannot use ide4. there are only 4 ide devices allowed - ide0, ide1, ide2 and ide3.

So how do I add more than 4 !??!

Thanks guys!

 

[lpallard]

Re: Getting started with PVE storage model - software RAID5 array and storage concept

I tried with 'scsi' (from the vm.conf man page) instead of 'ide' but I get a kernel panic with CentOS VM...

looks like sata only goes up to 5 (including 0) so provides 6 possible SATA devices... A bit short but it will work for me until I add more drives.

I may not be using the storage model the way proxmox is intended to be.....

Any feedback appreciated

Thanks and once again congratulations for proxmox!

 

[m.ardito]

Re: Getting started with PVE storage model - software RAID5 array and storage concept

You could do any kind of things with those disk.
IDE is limited to 4 (2channels, each master/slave) and so is obviously also in virtual environments.
If you have to raid them, you should probably do that at the lowest possible level so "hw > sw", "pve host > inside vm"

Marco

 

[lpallard]

Re: Getting started with PVE storage model - software RAID5 array and storage concept

How would you RAID them at the pve host level? Standard mdadm config like I am used to in Slackware?

 

[m.ardito]

Re: Getting started with PVE storage model - software RAID5 array and storage concept

I you can use a good hw raid (better with bbu or even better with flash cache).
otherwise you could also use mdadm, not supported, though.

(other options could be use them for at least three ceph nodes, even within pve 3.2, but I am newbie in this field....)

Marco

 

[lpallard]

Re: Getting started with PVE storage model - software RAID5 array and storage concept

I you can use a good hw raid (better with bbu or even better with flash cache).
otherwise you could also use mdadm, not supported, though.

(other options could be use them for at least three ceph nodes, even within pve 3.2, but I am newbie in this field....)

Marco

Im already using a M5016 for my system drives (two SAS 15k rpm drives in RAID1) and I have another M1015 in HBA mode where my drives are connected to. Right now, passed through to the VM 'as-is' (no tweaking or modifications whatsoever) are being assembled in CentOS with mdadm and it seems to work fine.

Maybe if I can put my hand on a 3TB drive I will copy (backup my stuff), thenplug the drives to the M5016 and create a RAID5 array from that... I just dont like the single point of failure (the M5016) and the fact that I initially create the RAID1 array on the M5016 with the two drives plugged to their respective SAS ports on the controller... If I remove a cable, connect the drive to the same SAS port on the controller with the fanout cable, will the M5016 firmware interpret that as a drive failure??

 

[m.ardito]

Re: Getting started with PVE storage model - software RAID5 array and storage concept

I just dont like the single point of failure (the M5016)

well you have at least another one: the motherboard... for that, only rescue is to have tested spare parts available and/or fast replacement service. and in the meanwhile, move/restore vms to another cluster node...

and the fact that I initially create the RAID1 array on the M5016 with the two drives plugged to their respective SAS ports on the controller... If I remove a cable, connect the drive to the same SAS port on the controller with the fanout cable, will the M5016 firmware interpret that as a drive failure??

sorry I don't understand. About raid contoller firmware, perhaps only manufacturer support could answer you, and it obviously depends also on fw version...

Marco

 

[lpallard]

Re: Getting started with PVE storage model - software RAID5 array and storage concept

well you have at least another one: the motherboard... for that, only rescue is to have tested spare parts available and/or fast replacement service. and in the meanwhile, move/restore vms to another cluster node...



sorry I don't understand. About raid contoller firmware, perhaps only manufacturer support could answer you, and it obviously depends also on fw version...

Marco

I meant that I now have two SAS drives connected to the two of the SAS ports on the card, one drive per port. So port-wise, the M5016 is maxed out. If I get a fanout cable, I can connect both drives on the sanme SAS port, but I am afraid the firnware of the controller interprets the disk address reassignment as a failure leading to a degraded array..

If there;s no problem, then I will get a fanout cable, and plug both drives on the same SAS port leaving a SAS port free to host 4 additional drives. Thats still short of 2 for the 6 drives that I am using now..

I think the simplest until I get another RAID5 capable controller, will be to assemble the 6 drives as software raid in proxmox, then offer the resulting storage to the VM's. That way, the VM's will se a single 10TB drive, but proxmox will see a 6x2TB in RAID5....

How can I do that?

Other topic of interest: How can I have proxmox "talk" to the M5016 to get its temperature, array status, etc? I noticed in the controller's firmware there are no notification features (email, etc) like Supermicro's IPMI... If one of the drives fails, or if the controller overheats, how will I know?

Sensors doesnt even work in proxmox. Usually, on Slackware, i would have a script that would probe the hardware sensors every so often and if values were above threshold, I would get an email..

 

[m.ardito]

Re: Getting started with PVE storage model - software RAID5 array and storage concept

Other topic of interest: How can I have proxmox "talk" to the M5016 to get its temperature, array status, etc? I noticed in the controller's firmware there are no notification features (email, etc) like Supermicro's IPMI... If one of the drives fails, or if the controller overheats, how will I know?

I have M5015 (IBM rebranded version of LSI controller) and use utils from http://hwraid.le-vert.net/ see http://pve.proxmox.com/wiki/Raid_controller#LSI

Marco

 

[lpallard]

Re: Getting started with PVE storage model - software RAID5 array and storage concept

Ok! I am trying to install megacli, megactl and megaraid-status but I am getting

root@proxmox:~# aptitude install megacli
Couldn't find any package whose name or description matched "megacli"
Couldn't find any package whose name or description matched "megacli"
No packages will be installed, upgraded, or removed.
0 packages upgraded, 0 newly installed, 0 to remove and 7 not upgraded.
Need to get 0 B of archives. After unpacking 0 B will be used.

What am I doing wrong?

I followed the instructions at http://hwraid.le-vert.net/wiki/DebianPackages without errors.... Sorry for asking, I am VERY new to debian....

 

[m.ardito]

Re: Getting started with PVE storage model - software RAID5 array and storage concept

I followed the instructions at http://hwraid.le-vert.net/wiki/DebianPackages without errors.... Sorry for asking, I am VERY new to debian....

after you set a repository in /etc/apt/sources.list (or in any file under /etc/apt/sources.list.d/*)
(I have added for megacli "deb http://hwraid.le-vert.net/debian wheezy main")
you have to "update" local db of that repo (it's empty at first):

#apt-get update

then you will find all megacli utils from that repo

#aptitude search megacli

then install with

apt-get install <package>

Marco

 

[lpallard]

Re: Getting started with PVE storage model - software RAID5 array and storage concept

after you set a repository in /etc/apt/sources.list (or in any file under /etc/apt/sources.list.d/*)
(I have added for megacli "deb http://hwraid.le-vert.net/debian wheezy main")
you have to "update" local db of that repo (it's empty at first):

#apt-get update

then you will find all megacli utils from that repo

#aptitude search megacli

then install with

apt-get install <package>

Marco

Before you replied, I had them installed with

aptitude install package

Not sure why with apt-get they were not found...

Now I've got the megacli, megactl & megaraid-status packages installed. The website http://hwraid.le-vert.net/debian mentions that megaraid-status is a wrapper script to monitor the controller and its raid arrays, but doesnt say how to use the script. Do you have a cron job running to initiate the script every so often and interpret the output?

 

[lpallard]

Re: Getting started with PVE storage model - software RAID5 array and storage concept

Now with most questions out of the way, I am back asking more questions regarding proxmox storage and how to use it effectively...

My main storage is using two 300GB SAS drives in RAID1 config.

Disk is partitioned as this:

/dev/mapper/pve-root 32.2GB (4% used)
/dev/mapper/swap 37.3GB
/dev/mapper/pve-data 211.8GB (59% used)

In pve-data I have my VM's, one of them is a CentOS server partitioned as this:

vg_centos-lvm_root 15.7GB (72% used)
vg_centos-lv_swap: 15.7 GB
vg_centos-lv_home: 1.5GB (5% used)
vg_centos-lv_serverdata: 95.5 GB (74% used)

vg_centos-lv_serverdata is used to store VM data (SQL databases, document storage, misc datastores)


Questions at the hypervisor's level:

-With only 4% in use for 'pve-root', would it be advisable to shrink the FS ? Any chance usage goes up all of a sudden? Considering the storage premium on these SAS drives, I'd like to benefit from most of the drive's capacity. Right now, almost 31GB is unused which is a tremendous waste of space.

-I cannot remember if I sized the swap to 37GB manually or if Proxmox sized it automatically, but swap usage is now a little over 50% of installed RAM (64GB). Is it OK for proxmox? Should I reduce it? Should I increase it?

Questions for the CentOS VM:

-Right now 'vg_centos-lv_serverdata' is used for specific VM data.. For reliability and flexibility, should I create another mount point on '/dev/mapper/pve-data' and move the data from inside the VM to the mount point? What are you guys doing with VM data storage?? 'vg_centos-lv_serverdata' is also getting pretty full (74%) but Im not expecting it to grow beyong 100GB... By moving the data outside of the VM I could use some space off of '/dev/mapper/pve-data' which is curently only 59% used...


Any pointers, experience, feedback or advice seeked!

Greetings

 

[lpallard]

Re: Getting started with PVE storage model - software RAID5 array and storage concept

Now with most questions out of the way, I am back asking more questions regarding proxmox storage and how to use it effectively...

My main storage is using two 300GB SAS drives in RAID1 config.

Disk is partitioned as this:

/dev/mapper/pve-root 32.2GB (4% used)
/dev/mapper/swap 37.3GB
/dev/mapper/pve-data 211.8GB (59% used)

In pve-data I have my VM's, one of them is a CentOS server partitioned as this:

vg_centos-lvm_root 15.7GB (72% used)
vg_centos-lv_swap: 15.7 GB
vg_centos-lv_home: 1.5GB (5% used)
vg_centos-lv_serverdata: 95.5 GB (74% used)

vg_centos-lv_serverdata is used to store VM data (SQL databases, document storage, misc datastores)


Questions at the hypervisor's level:

-With only 4% in use for 'pve-root', would it be advisable to shrink the FS ? Any chance usage goes up all of a sudden? Considering the storage premium on these SAS drives, I'd like to benefit from most of the drive's capacity. Right now, almost 31GB is unused which is a tremendous waste of space.

-I cannot remember if I sized the swap to 37GB manually or if Proxmox sized it automatically, but swap usage is now a little over 50% of installed RAM (64GB). Is it OK for proxmox? Should I reduce it? Should I increase it?

Questions for the CentOS VM:

-Right now 'vg_centos-lv_serverdata' is used for specific VM data.. For reliability and flexibility, should I create another mount point on '/dev/mapper/pve-data' and move the data from inside the VM to the mount point? What are you guys doing with VM data storage?? 'vg_centos-lv_serverdata' is also getting pretty full (74%) but Im not expecting it to grow beyong 100GB... By moving the data outside of the VM I could use some space off of '/dev/mapper/pve-data' which is curently only 59% used...


Any pointers, experience, feedback or advice seeked!

Greetings

The documentation is not clear, or I cant find the right information, but storing data IN a VM is sure thing not the best way to do it isn't it?? If the VM gets corrupted, so you lose the machine AND the data. How do you create a storage point on the same physical HDD as proxmox, but you can mount in your VM for storage?

Add storage has only Containers, Images, ISO and Backups, Templates,,.. No "files" or generic data..

 

[dominix]

Re: Getting started with PVE storage model - software RAID5 array and storage concept

-With only 4% in use for 'pve-root', would it be advisable to shrink the FS ?

here is an example for the resize. I do it online but with all service stopped.
[at your own risk, make backup ... so on]
NB : remove comment #* comment *#

init 1
mount / -o remount,ro
umount /dev/pve/data


fsck -f /dev/pve/root
fsck -f /dev/pve/data
resize2fs /dev/pve/root [minimal size]G #* reduce size of the FS before reduce partition or lv*#
lvreduce -L [desired size] /dev/pve/root #* reduce partition/lv now, MUST BE greater than the line above *#
resize2fs /dev/pve/root #* set the size of the FS fit the size off the partition-lv *#
lvextend -l+100%FREE /dev/pve/data #* move the free space of volume group to the partition-lv for DATA fs *#
resize2fs /dev/pve/data #* set the size of the FS fit the size off the partition-lv *#
fsck -f /dev/pve/data

then reboot

any question ?

 

[lpallard]

Re: Getting started with PVE storage model - software RAID5 array and storage concept

any question ?

Thanks!

Yes I do have questions:

1. Would it be better using a bootable CD or is it just fine to do it under init1?
2. Do you advise against shrinking the root FS/lv? Any chance proxmox actually requires the 32.2GB available?
3. I do not understand

]#* reduce partition/lv now, MUST BE greater than the line above *#

lets say I want to shrink root to 10GB, using

resize2fs /dev/pve/root 10G

and then use

lvreduce -L 10G /dev/pve/root

wouldn't do it?

As for SWAP, should I leave it alone? Do you advice increasing/reducing it? Is there actual guidelines for SWAP size? Half of RAM? more / less?

Finally, to backup my VM's, only copying the .raw or .qcow2 files and their VM.conf would be enough? Installing and configuring proxmox was a breeze....

Thanks!

EDIT: Forgot to ask: if I have to play with my current setup that much, wouldn't I be better off reinstalling proxmox fresh, and restore my VM's? Easy to do? Should I setup a backup storage before?

What about filesystems? I am using Ext4 right now but default is Ext3. Am I better with default or something else?


Lots of questions, if someone can shed light on those I will be good to go for a while... I just wished proxmox's documentation had better background info..

 

[dominix]

Re: Getting started with PVE storage model - software RAID5 array and storage concept

lets say I want to shrink root to 10GB, using

Code: resize2fs /dev/pve/root 10G

and then use

Code: lvreduce -L 10G /dev/pve/root

it COULD do it, but sometimes there is light differences in calculation between FS and raw block, so if you want to be sure, shrink the FS more than necessary:

resize2fs /dev/pve/root 9G
lvreduce -L 10G /dev/pve/root
resize2fs /dev/pve/root #* without size it take whatever possible *#

 

[lpallard]

Re: Getting started with PVE storage model - software RAID5 array and storage concept

OK before I resize (or reinstall proxmox) I need to find out how to properly use a new RAID5 array added to the node..

I created a RAID5 array from the M5016 controller and created a PV on the entire resulting 3.8TB volume, then created a single VG using all space called "storage" then two LV's (one of 3.5TB called datastore, the other called backups of 300GB).

I intended to use the 3.5TB LV as a standard "drive" and pass it to my VM's so they can all use it as a shared storage (lets say one VM creates files on it, another VM can use it as well at the same time). The other LV (backups) will strictly be used by proxmox for backups.

I have successfully created a new local storage in proxmox GUI with backups and proxmox is using it to backup my VM's.. Works perfectly.

How do I use the 3.5TB LV (datastore) for my VM's? Proxmox GUI has only "Backups, ISO, images, containers" but no "Files"....

Thanks!

EDIT: I could always pass the logical volume /dev/storage/datastore to my main VM (in the .conf file) but /dev/storage/datastore would not be available to other VM's...

 

[aenima99x]

Re: Getting started with PVE storage model - software RAID5 array and storage concept

OK before I resize (or reinstall proxmox) I need to find out how to properly use a new RAID5 array added to the node..

I created a RAID5 array from the M5016 controller and created a PV on the entire resulting 3.8TB volume, then created a single VG using all space called "storage" then two LV's (one of 3.5TB called datastore, the other called backups of 300GB).

I intended to use the 3.5TB LV as a standard "drive" and pass it to my VM's so they can all use it as a shared storage (lets say one VM creates files on it, another VM can use it as well at the same time). The other LV (backups) will strictly be used by proxmox for backups.

I have successfully created a new local storage in proxmox GUI with backups and proxmox is using it to backup my VM's.. Works perfectly.

How do I use the 3.5TB LV (datastore) for my VM's? Proxmox GUI has only "Backups, ISO, images, containers" but no "Files"....

Thanks!

EDIT: I could always pass the logical volume /dev/storage/datastore to my main VM (in the .conf file) but /dev/storage/datastore would not be available to other VM's...

I'm struggling with this exact same thing right now. Did you ever figure this out?

I'm trying to create an all-in-one setup. Proxmox hosting several VM's (pfSense, Ubuntu, XBMC, etc). I currently have 2 mdadm RAID5's with LVM running and would like to use a similar setup with ProxMox and have this storage available to all of my VM's. Is it possible to setup the mdadm/lvm on the proxmox level and then present it to multiple vm's?

 

[dominix]

Re: Getting started with PVE storage model - software RAID5 array and storage concept

what is exactly your problem ?