Need your help about proxmox strategy

[Sebastian2000]

I have actually virtual server (KVM) on an very low profiles servers (Atom C2750, with 8G RAM and one no entreprise SSD disk).

As I don't have redundancy (only 1 disk), I firstly use DRBD between 2 server to have some RAID1 security, but DRBD was often broken, so I buy an QNAP (ES1640DC-V2) in RAID10 with SAS NL disk to use an share storage and not have problem of security data, connection with iscsi multipath.. But also have an heavy problem of stability, Atom seem have problem of stability with iscsi connection... So finally have try the replication feature with proxmox5, but have problem that replication break (there is some thread about this).

I have decide to change really of hardware, and now have bought an DELL C6220 with 4 nodes with 64G RAM, 2xES2650 and 6 disks SSD each node. I think using RAID10 hardware for now, but have some doubt about use ZFS or LVM in proxmox server... Virtual kvm are in ext4 and this will not change, just move our disk on new proxmox server. I will not use the replication function (that only work on zfs)...

What do you think? better ZFS or better LVM in my case?

 

[samontetro]

Hi Sebastian2000

and what about unsing ceph with several raid0 devices (I think raid10 is also for high bandwith) ? Replication of your data would allow you to move your VMs from one node to the other even if with a C6220 you have some spof with the fans, the bmc...
Howerver C6220 has only gigabit ethernet interface and replication off ssds storage could require ethernet 10Gb/s for efficiency...
Note: I'm not an expert but a beginer with proxmox, using a HA cluster with 3 DELL R530 and replicated storage. Using 3 nodes allows me to move one server in a separate building to ensure also data security.

Patrick

 

[Sebastian2000]

Hi Sebastian2000

and what about unsing ceph with several raid0 devices (I think raid10 is also for high bandwith) ? Replication of your data would allow you to move your VMs from one node to the other even if with a C6220 you have some spof with the fans, the bmc...
Howerver C6220 has only gigabit ethernet interface and replication off ssds storage could require ethernet 10Gb/s for efficiency...
Note: I'm not an expert but a beginer with proxmox, using a HA cluster with 3 DELL R530 and replicated storage. Using 3 nodes allows me to move one server in a separate building to ensure also data security.

Patrick

Well, I have dual 10Gbs on all node, I need one of the 10Gbs will be for QNAP connection and backup but other will be available. I have read a lot of problem perf with ceph and for this I have not review this option for now... I need really an good stability now and for this, think to RAID10 hardware solution as I will have 6 disks for nodes and RAID card to do it...

What disk do you use (Capacity and suppose that they are SSD)? How many VMs on each node?

Thanks a lot for your help, I also not an expert and for this, appreciate the hekp of this forum!

 

[samontetro]

My use is quite different than yours I think: I need high available storage and services for my servers, cluster, users desktop, data integrity and resilience of the config. But I have few users (10 to 20) and large data. SSDs are too much expensive for our organisation so I use slower but capacitive devices (2 disks/node, 8Tb each). Hardware raid1 is only set for the OS (2 x 300GB at 15Krpm disks), all data disks are managed with ceph (1 osd = 1 disk).
Yes, I have not high I/O bandwith with this setup but PVE HA + CEPH is very resilient: I test it with unpluging network cables, crashing VM or nodes...etc in the pre-prod step. I've just added a local script that check the separate ceph network availability (don't know how to set PVE-HA to do this).
In prod, from 6 months, I've got some electrical power failure (on one node), one network switch crash, a "network jam" (misconfiguration by the network administrator)... but always PVE-HA + CEPH continue to provide services and data with no corruption.

But as shown on

("Install Ceph Server on Proxmox VE"), high bandwith can be reached with SSDs.

 

[cbx]

My use is quite different than yours I think: I need high available storage and services for my servers, cluster, users desktop, data integrity and resilience of the config. But I have few users (10 to 20) and large data. SSDs are too much expensive for our organisation so I use slower but capacitive devices (2 disks/node, 8Tb each). Hardware raid1 is only set for the OS (2 x 300GB at 15Krpm disks), all data disks are managed with ceph (1 osd = 1 disk).
Yes, I have not high I/O bandwith with this setup but PVE HA + CEPH is very resilient: I test it with unpluging network cables, crashing VM or nodes...etc in the pre-prod step. I've just added a local script that check the separate ceph network availability (don't know how to set PVE-HA to do this).
In prod, from 6 months, I've got some electrical power failure (on one node), one network switch crash, a "network jam" (misconfiguration by the network administrator)... but always PVE-HA + CEPH continue to provide services and data with no corruption.

But as shown on

("Install Ceph Server on Proxmox VE"), high bandwith can be reached with SSDs.

Thanks for your help.. In our case, it's for hosting service, so I don't think that ceph can be the better way for us, because I need an good (or almost, acceptable) I/O on each node, I suppose that ceph always will have less performance that RAID10 hardware. I review to do some RAID 10 with ZFS but not sure is the best as I will have an RAID card to do it with hardware. Then, I not sure if better install proxmox with ext4/LVM or ZFS (0 as it will be on an RAID10 hardware), our VMs are in ext4 and we can't change this...

 

[guletz]

ZFS work better without any HW Raid Controller. It is very possible to have lower performance in term of IOPs with zfs compared with non-zfs(LVM). But you can get others advantages with zfs, and maybe this will be more important, like:
- compression(so if you have many compressible files, you can save even 50% of your storage capacity, and your IOPs will be better)
- snapshots in less then a second
- you can clone very easy any of your VM in seconds
- you can replicate VMs(asyncron) from one node to other node(is more like a differential backup) - so if one Proxmox node is broken, in a couple of minutes you can start all your replicated VM on online node(where you have the replicated VMs)!
- sometime in the near-future, zfs will have the encryption suport built in

Good luck!

 

[Sebastian2000]

Hello

I will finaly choose to do an RAID10 Hardware, and Install Proxmox on it in ZFS because I think that HW raid had to use less CPU resources than RAID software...

Then, I have another important doubt, I think I will use Consumer SSD disk, I know that a lot of person notify about this and recommand Entreprise SSD, but An 500G SSD consumer is about 150€, an entreprise SSD, the better price I have seen is about 500€ for an 960G (see only on some german reseller on ebay at this price)... As I need almost 1,2T for each node, 6x500G in RAID10= 900€, 4x960G in RAID10=2000€...
I know that :
- The lifetime of SSD consumer is lower than SSD entreprise
- They are a more sensible as power down
- Performance are not the same

but in our case, there is more than 4000€ of difference on the entire node in the better case... The other option is to put SAS disk, but seem much worst than SSD, included consumer SSD...

As we have RAID10 monitoring and SSD spare to change it ASAP in case of disk failure, is really so bad to do this? Please, give me your opinion...

 

[guletz]

I will finaly choose to do an RAID10 Hardware, and Install Proxmox on it in ZFS because I think that HW raid had to use less CPU resources than RAID softwar

Wrong. zfs on top of hw raid cand reduce the zfs performance. Also it could corruption your zfs. zfs will use the same cpu resurces (almost) with or without hw raid.

but in our case, there is more than 4000€ of difference on the entire node in the better case... The other option is to put SAS disk, but seem much worst than SSD, included consumer SSD...

You must consider your costs/money for let say 3 years. I has to choosing like you some time ago. My SSD was die after 3 year. And I need to get another one. So in the end I waste my money and my time and performance. An Datacenter SSD was better and it was working evean now.

But if your budget is very low, maybe consider to buy only some SSD data center(not all)

 

[BEHIND-IT]

Hey there,

it's my first post, therefore - i'm the new guy around in this board.
My name is Oliver - nice to read you.

I'm nearly exactly at the same point, i've got my Dell C6220 last Monday, today i'm running Proxmox 5.1 on all 4 x Nodes (32 GB RAM each, 2 x 10 Gbit/s with 2 x & 146 GB 10k SAS HDDs on LSI 1GB Cache HW Raid 1 ). Therefore thumbs up for your choice Sebastian2000.

Hello
I know that :
- The lifetime of SSD consumer is lower than SSD entreprise
- They are a more sensible as power down
- Performance are not the same

As we have RAID10 monitoring and SSD spare to change it ASAP in case of disk failure, is really so bad to do this? Please, give me your opinion...

My opinion:

I think you have the right knowledge - keep MTBF / DWPD figures in mind, by choosing your consumer SSDs and monitor them properly, than this should work for a normal read intensive workload.

I've planned to manage it this / the same way:

- All 4 Nodes will run Proxmox from the SAS RAID 1

Performance Node

- Local Storage will be cheap consumer SSDs like EVO 850 250 GB with ZFS RAID-Z2 if i manage to get them passed via the LSI Controller, otherwise LSI HW RAID 10
- Scale Out Storage will be 16 / later 32 ARM based microservers with CEPH

Normal Node

- No local Storage
- Scale Out Storage like above

Now i'm reading the manuals to get the cluster between the 4 nodes up and running.

Best regards and have a good time

Oliver

 

[Sebastian2000]

Thanks Oliver and thanks guletz
@guletz, I will revieve the server the next week, I will try to do some I/O test with and without HW RAID to review performance before choosing, I admit that... not use HW RAID, having an RAID card, it's a few dificil to take this decision (I have seached an server with this, as there was my problem on other microservers that have only 1 disk and so, no RAID cards) but had to do some test for this.

I understand also your opinion about SSD and good see that you have use this option in the past, I assume they will have a less lifetime, but, it's possible that not 2 less time than an SSD entreprise. I have also the buget problem, today to have 16 SSD entreprise of 960G and to RAID10 or ZFS RAID 10, the cost is more than 6500/7000€, RAID5 or ZFS RAID-Z1,, more than 4500/5000€ (search it in ebay reseller german)... RAID 10 or ZFS RAID 10 with 24*525G of SSD consumer : 3500€...

What SSD type of disk and space do you use?
@oliver also nice to see that you have do some similar choice

 

[guletz]

assume they will have a less lifetime, but, it's possible that not 2 less time than an SSD entreprise.

Maybe more... I use a mid range SSD(non enterprise ), for a low usage pool(home server). But on many enterprise SSD with a high load zfs, after 4 years all of them are alive. I think it depends a lot how is the load, and what models do you use.
Maybe you can use a mixed solution (for risk perspective ) with some nodes with enterprise SSD and the rest with cheap SSDs. Another ideea is to use only cheap SSDs, with let say +1/2 extra SSDs. You will have let say 1 or 2 SSDs as reserved. Then let say in each month you will replace (in round robin way) your spare SSD with a SSD that is on a node. So in time any SSD will have some work time on a server and a not-used period. Another advantage will be that if in a day one SSD will be broken than you can replace in a very short time.

Good luck!

 

[Sebastian2000]

Maybe more... I use a mid range SSD(non enterprise ), for a low usage pool(home server). But on many enterprise SSD with a high load zfs, after 4 years all of them are alive. I think it depends a lot how is the load, and what models do you use.
Maybe you can use a mixed solution (for risk perspective ) with some nodes with enterprise SSD and the rest with cheap SSDs. Another ideea is to use only cheap SSDs, with let say +1/2 extra SSDs. You will have let say 1 or 2 SSDs as reserved. Then let say in each month you will replace (in round robin way) your spare SSD with a SSD that is on a node. So in time any SSD will have some work time on a server and a not-used period. Another advantage will be that if in a day one SSD will be broken than you can replace in a very short time.

Good luck!

Well, finaly I will buy kingstom DC400 (SEDC400S37), as they are for entreprise, but no so expensive as intel. The lifetime is no so good of an intel, but better than the crucial MX300 that I would use at first time, and are MLC and not TLC... Is this an good choice for limited budget?

 

[guletz]

and not TLC... Is this an good choice for limited budget?

I do not know if it is a good choice. You will see after several month. But for sure is a better option compared with your initial ideea about your storage. Your choice is at the middle (not the best like Intel DC but not so bad like Crucial). Only the time will show if your choice it was wrong or not!
The same apply to my observations. And I will be very glad if you find some time in the future and say what it my observation / your decision. In any case this will be also very useful for me, because I can understand more things about zfs and I can help others guys (with your feedback from you) at the next post related with this topic.
And after some time if I will be alive I will be able to respond better on of your next question about ... what is the best ....
Many guys have fail in some circumstances using proxmox. Many of them are post some questions. But only few of them are trying to share What was not working for them (with details).
In my case I can say that I learn many things reading feedback on this Forum about things that are not working like we estimate. But I try to help if I guess if I can/know because I can not pay for the great job that Proxmox is doing.
I can not pay a paid support but I can sometime help guys like you so, Proxmox employees can do something better(and they can do)



Good luck to everyone who use Proxmox!

 

[Sebastian2000]

I do not know if it is a good choice. You will see after several month. But for sure is a better option compared with your initial ideea about your storage. Your choice is at the middle (not the best like Intel DC but not so bad like Crucial). Only the time will show if your choice it was wrong or not!
The same apply to my observations. And I will be very glad if you find some time in the future and say what it my observation / your decision. In any case this will be also very useful for me, because I can understand more things about zfs and I can help others guys (with your feedback from you) at the next post related with this topic.
And after some time if I will be alive I will be able to respond better on of your next question about ... what is the best ....
Many guys have fail in some circumstances using proxmox. Many of them are post some questions. But only few of them are trying to share What was not working for them (with details).
In my case I can say that I learn many things reading feedback on this Forum about things that are not working like we estimate. But I try to help if I guess if I can/know because I can not pay for the great job that Proxmox is doing.
I can not pay a paid support but I can sometime help guys like you so, Proxmox employees can do something better(and they can do)



Good luck to everyone who use Proxmox!

Firstly, thanks a lot for your help!

Be sure I will try remenber in some month to tell us if I will in the good way with this SSD type or not (Our hosting user are not heavy user and we have an cheaper service, for this it's some dificult to have an very expensive SSD...).

For now we have do various mistake about hardware (microserver with only 1 disk... then drbd on this... then an QNAP who microserver not able to use with the require stability, replication function with these microserver, and finally, change to some much more better server (C6220 with 2*ES2650 and 64G RAM)

I still not have take decision about RAID10 HW or ZFS RAID for now... Firstly I would use replication, so ZFS or ZFS, but, finally, it not seem so stable for production environement, so this feature of ZFS I will probably not use it and still have the choose of ZFS RAID or RAID HW with... ZFS or LVM.

 

[guletz]

Firstly I would use replication, so ZFS or ZFS, but, finally, it not seem so stable for production

ZFS it is and it was stable in production for at least 5 years. I use/had have use zfs for many years, on many different servers in very different zfs configs. I have never see any serious / critical problems. And I use zfs on various linux platforms : debian, centos, fedora, linux-mint. So on desktop and server, and belive me, I am not so stupid to run .... not-so-production-ready stuff like zfs. As a funny thing on a linux mail list I was read a respond (5 years ago) that running zfs is a real stupid ...

But you must start to learn... zfs/disk storage.

 

[Sebastian2000]

ZFS it is and it was stable in production for at least 5 years. I use/had have use zfs for many years, on many different servers in very different zfs configs. I have never see any serious / critical problems. And I use zfs on various linux platforms : debian, centos, fedora, linux-mint. So on desktop and server, and belive me, I am not so stupid to run .... not-so-production-ready stuff like zfs. As a funny thing on a linux mail list I was read a respond (5 years ago) that running zfs is a real stupid ...

But you must start to learn... zfs/disk storage.

I talk about proxmox replication funcionality, not ZFS system, sorry for the confusion of my last message...

 

[Sebastian2000]

ZFS it is and it was stable in production for at least 5 years. I use/had have use zfs for many years, on many different servers in very different zfs configs. I have never see any serious / critical problems. And I use zfs on various linux platforms : debian, centos, fedora, linux-mint. So on desktop and server, and belive me, I am not so stupid to run .... not-so-production-ready stuff like zfs. As a funny thing on a linux mail list I was read a respond (5 years ago) that running zfs is a real stupid ...

But you must start to learn... zfs/disk storage.

Also understand that always use software RAID (is with various disk, deskop sometime are with an single disk)? Do you use trim funcion in this case with SSD?

 

[guletz]

Hi,

Sorry for my late replay. Yes I use the trim for any SSD that I used in zfs (even for non-mirror setup). I trim any SSD at least one time in 6 month. For SSD mirror is simple to shut down the server, remove one SSD, then I put this SSD on my linux desktop and I make a ext4 on the all ssd then trim . Then I put it back in the server and I replace my pool with this trimmed ssd.