Project: proxmox-host and freenas-guest with 4x3tb Riad5

[hoppel118]

Hey guys,

I want to realize a project for my home-entertainment. Here the planned structure:

hardware:

• mainboard: (VT-x/VT-d capable)
• cpu: Intel Core i5 650
• ram: 4x 4GB
• os-hdd: 250GB
• chassis: Fractal Define R3 USB3.0 Titanium Grey

guest1 (storage-vm):

• freenas 8
• 8GB RAM
• Raid-Controller configured as Raid5 (maybe Raid-Z1) with 4x3TB passed through to this guest

guest2 (dvbs2-tv-vm):

• debian with vdr
• 4GB RAM
• dvb-s2-tv-card (Digital Devices Cine S2)passed though to this guest

The Server is already built and freenas is running fine. Now I need to buy the raid-controller and the hdds. I am choosing to buy one of the following controller:

• Areca ARC1231ML
• 3ware 9650-8LMPL

I read that both controllers support freebsd (freenas). Which one would you prefer?

How is it possible to pass through the tv-card and the raid-controller of the choice into guests? With esxi this should be possible, but I like the webui of proxmox. It's great.

With kind regards from germany

Hoppel

 

[macday]

Hi,

I´m from germany, too.

I´m using Proxmox since the 1.1 and I have to tell you FreeBSD of which FreeNAS ist build up is not good as KVM Guest. You do not have paravirtualized storage or network interfaces.

My opinion is build a samba fileserver based on squeeze and ZFSonlinux and you´ll be happy.

Cheers

 

[hoppel118]

What about openmediavault? It has a nice webinterface, too. I want the things to get easier. The terminal is not the problem, tried a great variety of distributions over the years. But now in this time (2012) I like webinterfaces for administration.

It's based on debian. Did you ever try it?

Seemingly no ZFS. I will give both (freenas, openmediavault) a chance, going to install it as a guest.

regards

 

[macday]

openattic is one of the best...with zfs-fuse integrated ...

 

[udo]

Hi Hoppel,
i use some Areca-controller and have good experiences with them (speed). Ok, two of them must be repaired, but this happens during guarantee.

But why you wan't to pass the controller to the VM? I use the controller on the host (so also the pve-os is saved again disk-failure), and use lvm-storage to pass the storage to the guest.
Work fast ans is much more flexible (other VMs can use space on the fast raid-volumes).

Like macday wrote, you should take a look at openattic.org - debian-based, work well in a VM (virtio-disk for lvm usable with command-line instead of gui - but perhaps also yet with the gui, because they develop much on this system) and also good performance.

Udo

 

[hoppel118]

openattic is one of the best...with zfs-fuse integrated ...

Yes, why not, talked to the people from it-novum on the cebit. Will have a look on it!

i use some Areca-controller and have good experiences with them (speed). Ok, two of them must be repaired, but this happens during guarantee.

Oh, it's not a real good result. How many controllers do you use?

But why you wan't to pass the controller to the VM? I use the controller on the host (so also the pve-os is saved again disk-failure), and use lvm-storage to pass the storage to the guest. Work fast ans is much more flexible (other VMs can use space on the fast raid-volumes).

I am new to this and it was the best logic for me. Tell me something more about a real good solution.

What about Raid-Z1/Raid-Z2? Is it a good alternative against a hardware Raid 5 or Raid 6?

 

[udo]

Yes, why not, talked to the people from it-novum on the cebit. Will have a look on it!

Oh, it's not a real good result. How many controllers do you use?

Seven (four types) plus some areca-controller in hardware-raids. Of course, one dead controller is one to much, but if i had the choice I think i got again an areca-controller.

I am new to this and it was the best logic for me. Tell me something more about a real good solution.

like i wrote before - i think lvm-storage for the guest is very flexible. Just play around with an test-installation.

What about Raid-Z1/Raid-Z2? Is it a good alternative against a hardware Raid 5 or Raid 6?

I don't know. I have no experiences with zfs on linux. Because of the "wrong" license, you can only use ZFS in userspace, which can't be very fast... and for virtualisation you are happy about fast IO...
Some people in this forum use normal linux softwareraid (mdraid), but it's not supported from the proxmox-staff (I know, there are a lot of pro and cons).

Udo

 

[hoppel118]

Seven (four types) plus some areca-controller in hardware-raids. Of course, one dead controller is one to much, but if i had the choice I think i got again an areca-controller.

So the performance must be realy great. Do you have some experience with areca and 3TB-HDDs? I made the follwoing comparison:

Variant 1:
modell: 4x Hitachi Deskstar (HDS5C3030ALA630) 3TB SATA III 5300
cost per hdd: 162€
cost per 4hdds: 648€
TB without raid: 12
TB with raid5: 9

Variant2 2:
modell: 4x Hitachi Deskstar (HDS5C3020ALA632) 2TB SATA III 5300
cost per hdd: 128€
cost per 6hdds: 768€
TB without raid: 12
TB with raid5: 10
TB with raid6: 8

What do you think? What variant would be better? That's round about my cost-limit for the hdds.

like i wrote before - i think lvm-storage for the guest is very flexible. Just play around with an test-installation.

I will do so, if I have all the hardware.

 

[udo]

So the performance must be realy great. Do you have some experience with areca and 3TB-HDDs? I made the follwoing comparison:

Variant 1:
modell: 4x Hitachi Deskstar (HDS5C3030ALA630) 3TB SATA III 5300
cost per hdd: 162€
cost per 4hdds: 648€
TB without raid: 12
TB with raid5: 9

Variant2 2:
modell: 4x Hitachi Deskstar (HDS5C3020ALA632) 2TB SATA III 5300
cost per hdd: 128€
cost per 6hdds: 768€
TB without raid: 12
TB with raid5: 10
TB with raid6: 8

What do you think? What variant would be better? That's round about my cost-limit for the hdds.



I will do so, if I have all the hardware.

Hi,
I don't have experiences with 3TB-disks yet. And HDS change the disk discription with each disk... I made good experiences with the Ultrastar HUA722020ALA330 (20 pieces in production), and for my private computer i buy two Ultrastar A7K3000 (different HDS-Number) - which both d.o.a. (dead on arival), but after changing both disks run well ;-)
Deskstar are normaly not for raid-usage, or?

In variant2 you mean six 2TB-disk?! For speed reasons more disks are better, but you need also more power and if your VMs need the speed, you should think about raid-10 (or much better sas-disk, but they have only 600GB).
I think for an normal use fileserver is an 4-disk-setup good enough (the cache from the raidcontroller must bring the speed). But with such big disks i normaly use raid-6 instead of raid-5, because at disk-fail the rebuild takes a whole day (depends on the rebuild-prio).

Udo

 

[Equand]

you can install freenas-9 it should support freebsd's virtio drivers from the box.

 

[e100]

This might be a little off topic, but I own lots of RAID cards and wanted to share my experiences so you can have a better picture.

I own many Areca controllers for many years.
The only ones I had issue with were the 1880 series SATA/SAS controllers.
I purchased many of them as soon as they were released, had some firmware bugs and the heat sinks were too small.
Being an early adopter comes with such risks.
Areca fixed the firmware and replaced every card with one that was redesigned with a larger heat sink.
They even cross shipped a few at a time so I could replace them quicker.

The disks I was using with these also had a firmware issue, that was also contributing to my issues.
Talk about Areca service/support, I sent them the WD firmware and they sent me back a special firmware file for my cards.
Uploaded that file into the RAID card(while the server is running and online) and it updated the firmware on my disks without disruption!
I do not know if other manufacturers can do that or not, but that was pretty awesome.
Also, not sure if every model of Areca card can do that or not.

We have at least 22 Areca cards, 15 are the 1880ix-12 we initially had issues with but have been working flawless since Areca implemented fixes.
We even have a ARC1231ML, been a good card for many years.

Areca cards are good and they stand behind their products.


To contrast I had some issues with Adaptec Cards.
Bought their very first SAS card, never did get that thing working perfect.
Support was no help, like pulling teeth just to get generic replies like "check your cables"
Using some Adaptec "host RAID" integrated into some supermicro servers for mirroring, buggy drivers and BIOS there too.
Ended up purchasing a Highpoint card to fix that problem since Adaptec's solution required reinstalling windows which I did not have time to do on so many machines. Update driver to fix issue and Windows no longer boots and only fix is to reinstall. Oh and it took months to convince them there was a problem. That is the sort of crappy service I got from Adaptec.

The few LSI SATA cards I owned worked well, just were slow. (early adopter again I suspect)
I suspect they are better now, especially considering that the Areca 1880 cards use LSI processors.
I also see lots of forum members using LSI, they too had some recent drivers issues but LSI seemed to help them get the issue fixed.

Never used 3ware so no opinion there but I do not recall reading anything bad about them.

Bottom line:
I would not hesitate to purchase any Areca product.
If I needed a particular feature/performance not available from Areca that an LSI card could provide I would purchase the LSI card.
If a 3ware card offered some compelling reason(price/performance/features) to purchase it, I would consider giving them a try.
I have not purchased an Adaptec product for 7 years and likely never will again, tho I do admit maybe they are have improved in that time.

 

[hoppel118]

I made good experiences with the Ultrastar HUA722020ALA330 (20 pieces in production), and for my private computer i buy two Ultrastar A7K3000 (different HDS-Number) - which both d.o.a. (dead on arival), but after changing both disks run well ;-)
Deskstar are normaly not for raid-usage, or?

No, you're right, the Deskstar is not really build for raid-usage. But in other forums I read about the good compatibility to Raid-Systems. The Ultrastar is to expensive for my home-equipment.

In variant2 you mean six 2TB-disk?! For speed reasons more disks are better, but you need also more power and if your VMs need the speed, you should think about raid-10 (or much better sas-disk, but they have only 600GB). I think for an normal use fileserver is an 4-disk-setup good enough (the cache from the raidcontroller must bring the speed). But with such big disks i normaly use raid-6 instead of raid-5, because at disk-fail the rebuild takes a whole day (depends on the rebuild-prio).

Yes, I meant the 2TB-HDDs, because of possible incompatibility-issues with 3tb-disks. This was an alternative. But now I dicided to buy 4x Hitachi Deskstar 3TB 5k3000 (HDS5C3030ALA630). It should be enough performance for my 1 or 2 streams per time and this modell has a "good" cost-to-benefit-rate. Later I am going to build a raid6, but for the moment the project is expensive enough and I only need capacity for round about 5TB.

you can install freenas-9 it should support freebsd's virtio drivers from the box.

At the moment I am trying freenas 8.2 openmediavault and openattic. Freenas 9, Alpha-Release? Where can I download it?

This might be a little off topic, but I own lots of RAID cards and wanted to share my experiences so you can have a better picture.

Thank you for this statement it underlines my decision in the morning. I bought an areca arc-1231ml. I think it's much more than I need for my homeequipment.

regards

 

[hoppel118]

Hey guys,

after I recognized that my available motherboard is vendor-proprietary (PCIe-Slots are above the I/O-Panel), I have to buy another one. I want to use the existing CPU "Core i5-650", but I don't really know if it fits onto this board:

Intel Server Board S3420GPLX

Here the board/cpu-combination is tested positively, but on the vendor-site (intel) the core i5 is not mentioned. What do you think? Is it compatible?


Otherwise I am interested in the new intel desktop boards with z77 chipset. They are also VT-d capable, written here:

http://www.intel.com/support/motherboards/desktop/sb/CS-030922.htm

Which board is the better choice?

Regards from Germany

 

[jekader]

Why is the PCIe slot position critical for a RAID controller? Just buy a riser card and fix it in some position inside the case, then connect the HDDs!

 

[hoppel118]

Why is the PCIe slot position critical for a RAID controller? Just buy a riser card and fix it in some position inside the case, then connect the HDDs!

Riser cards..., I hate riser cards, because I testet round about 6 risers in my htpc and nothing worked correctly. But in no way a riser could be a solution, the mainboard doesn't fit into any chassis, because of the wrong I/O-Panel position. The motherboard construction is proprietary from the vendor, have never seen this before.

 

[jekader]

oh, how I get it. Thanks for the explanation!

 

[hoppel118]

Hey guys,

my hardware for the homeentertainment-server is complete and works perfect. Thanks for the helpfull advices.

My Hardware:

- chassis: Fractal Design Define R3
- mainboard: Intel S3420 GPLX (incl. RMM3)
- cpu: Intel X3440
- ram: 4x 4GB
- raid-controller: Areca ARC1231ML (incl. BBU)
- oss-hdds: 2x 250GB Seagate 7200rpm
- data-hdds: 4x 3TB Hitachi Deskstar 5K3000 (Raid5)

The array initialization and volume check took round about 3 and a half day together. On the raid5 I built one 9TB primary gpt partition and activated lvm. Now I use this partition partwise (350GB) for my virtual machines and the rest for the nas-vm (openmediavault) as data-space. Thanks for the advice into this direction, udo. I never used lvm before, its great!

Streaming 1080p movies over my network works pretty good. That was the first goal I had to realize.

Now I want to learn more about the functions of proxmox and openmediavault, like backup, snapshot and storage.

When I am safe with this new world of kvm, I want to built the next vm to share tv-streams in my network. So I have to passthrough a still to buy "digital devices cineS2 octopus" pcie tuner-card. The distributions in the near choice for this are debian with manually compiled vdr, yavdr as a headless server or tvheadend. Maybe it's good to try passing the pcie-card to a windows-vm and check the function with Media-Portal as a tv-backend. Driver installation under windows for testing is really simple, but here the goal is also linux.

Later I will have other vms to compile and test without touching the productive vms. My girlfriend will be happy about this and never again compiling (black console) at the htpc! ;-)

And there is another idea for virtualization in my home-environment. At the moment I have two routers. One which I use because of the very good firewall-possibilities and the other because of the wlan-function. So it's a good starting point to virtualize the router with endian, pfsense or ipcop and buy 802.11n-AccessPoints getting powered by PoE from a new managable Switch. ;-)

In the past I used ipcop. What is the best choice distribution to virtualize the router/firewall?

Regards









Sent from my mobile phone using Tapatalk

 

[JimBeam]

For router distro: have a look at Vyatta. Debian-based, thus works with virtio net drivers (don't know if the BSD ones do already - had used pfsense before but it was a bit slow).
Vyatta can do routing, NAT, firewall etc, but doesn't have intrusion detection/prevention like others do.

 

[wetwilly]

Another option, if you dont need the advanced things that Vyatta can do, is to use ipfire. It also supports virtio drivers.

 

[udo]

...
When I am safe with this new world of kvm, I want to built the next vm to share tv-streams in my network. So I have to passthrough a still to buy "digital devices cineS2 octopus" pcie tuner-card. The distributions in the near choice for this are debian with manually compiled vdr, yavdr as a headless server or tvheadend. Maybe it's good to try passing the pcie-card to a windows-vm and check the function with Media-Portal as a tv-backend. Driver installation under windows for testing is really simple, but here the goal is also linux.

I don't have experiences with pcie-passtrough but some posts shows that's depends on Motherboard/bios if passthrough work or not.

...
And there is another idea for virtualization in my home-environment. At the moment I have two routers. One which I use because of the very good firewall-possibilities and the other because of the wlan-function. So it's a good starting point to virtualize the router with endian, pfsense or ipcop and buy 802.11n-AccessPoints getting powered by PoE from a new managable Switch. ;-)

In the past I used ipcop. What is the best choice distribution to virtualize the router/firewall?

As firewall i like devil-linux (hardened cd-distro) with a small disk to save the config (or a little bit bigger for log/services...). Firewall rules can be easily defined on the desktop-linux with fwbuilder.
pfsense is nice for an captive portal (wifi guests).

Udo