Some Newbie questions....!

[Michael S Ortega]

Hello guys, my name is MIchael and i have been playing around with proxmox the last few weeks, i am currently planning of using it in production for my business and maybe some of my customers but before i proceed with such a move, i want to be clear about some things:

1- Raid level: most of the computers will have max 4 hard drives, so ill be using a zfs raid 10 level (mirror when only 2 drives are present), this zfs will be for the Whole DATA, ill be adding 2 enterprise level ssd 120gb (raid1) for proxmox system itself. i would like any feedback on this?

2-for windows share ill be using the built in template turnkey file server which in where i will mount the DATA folder from the zfs Data pool, i was thinking of using OMV, but havent play around with that so not sure how reliable is on proxmox. Any suggestion about this? asking because last week i have to reinstall the whole proxmox cuz the containers stop working (wont start) after proxmox update.

3- talking about the proxmox update, my containers stop working after APT UPDATE command, i was reading and people say you need to comment out the enterprise repository for the source list and add the free one... should i do that in order to prevent this kinda of behavior? if so whats the process? I am not clear at this yet....i had that proxmox server for quite sometime, but never did the update before till last week, then everything stop working.

4- what raid controller do you recommend, i have this one https://www.ebay.com/itm/Dell-47MCV...-Flashed-IT-Mode-withcable-U039M/333108741298 on IT Mode, maybe you guys have a better suggestion since i will another one!

5-what is the best practice for the host backup, i have no problem with the DATA zfs since i can always import it into another machine, but what about the host boot drives goes bad, any suggestion to do a disaster recovery for such a scenario.

6- Is there any script or software to do automatically/schedule snapshots for vms and so? would like that since is faster and takes less space, so would be great if you guys guide me on this matter.

7-What Unprivileged container mean, i notice sometimes when creating if check marked it the container wont start... i get error or something, whats the deal?

thats for now. thank you in advanced!

 

[ertanerbek]

Hello guys, my name is MIchael and i have been playing around with proxmox the last few weeks, i am currently planning of using it in production for my business and maybe some of my customers but before i proceed with such a move, i want to be clear about some things:

1- Raid level: most of the computers will have max 4 hard drives, so ill be using a zfs raid 10 level (mirror when only 2 drives are present), this zfs will be for the Whole DATA, ill be adding 2 enterprise level ssd 120gb (raid1) for proxmox system itself. i would like any feedback on this?

Raidz ok for you. For traditional RAID system ZFS not make all job on DISK and generaly send sequential READ and Write to disk so ZFS will change all RANDOM request to sequential for your disk.

2-for windows share ill be using the built in template turnkey file server which in where i will mount the DATA folder from the zfs Data pool, i was thinking of using OMV, but havent play around with that so not sure how reliable is on proxmox. Any suggestion about this? asking because last week i have to reinstall the whole proxmox cuz the containers stop working (wont start) after proxmox update.

If you will use that system on your home lab, use Proxmox host for file sharing, if you will use on production area you can use FreeNAS. I was experiance some problem with container, Proxmox can not mount loop disk anyway actually on Container side you can mount one folder and uyou can use for file share if container was experiance problem just build new one and mount same folder to that.

3- talking about the proxmox update, my containers stop working after APT UPDATE command, i was reading and people say you need to comment out the enterprise repository for the source list and add the free one... should i do that in order to prevent this kinda of behavior? if so whats the process? I am not clear at this yet....i had that proxmox server for quite sometime, but never did the update before till last week, then everything stop working.

4- what raid controller do you recommend, i have this one https://www.ebay.com/itm/Dell-47MCV...-Flashed-IT-Mode-withcable-U039M/333108741298 on IT Mode, maybe you guys have a better suggestion since i will another one!

I was use my RAID card with IT and IR mode also Jbod mode all have same performance, if you use company based SAS drive because lowly company ( IBM, FP, Fujitsu, DELL etc .. ) use unic firmware file for disk so my disk never tell to cache capacity to ZFS . With normal SAS/SATA/NL-SAS you can use IT mode with ZFS

5-what is the best practice for the host backup, i have no problem with the DATA zfs since i can always import it into another machine, but what about the host boot drives goes bad, any suggestion to do a disaster recovery for such a scenario.

Proxmox already have backup solution also that backup system was work with snapshot, why you are searching new one ? Add one external disk and get backup from proxmox GUI..

6- Is there any script or software to do automatically/schedule snapshots for vms and so? would like that since is faster and takes less space, so would be great if you guys guide me on this matter.

Snapshoot is not backup system, use automatic backup system.

7-What Unprivileged container mean, i notice sometimes when creating if check marked it the container wont start... i get error or something, whats the deal?

https://pve.proxmox.com/wiki/Unprivileged_LXC_containers

thats for now. thank you in advanced!

 

[udo]

Hello guys, my name is MIchael and i have been playing around with proxmox the last few weeks, i am currently planning of using it in production for my business and maybe some of my customers but before i proceed with such a move, i want to be clear about some things:

1- Raid level: most of the computers will have max 4 hard drives, so ill be using a zfs raid 10 level (mirror when only 2 drives are present), this zfs will be for the Whole DATA, ill be adding 2 enterprise level ssd 120gb (raid1) for proxmox system itself. i would like any feedback on this?

Hi Michael,
depends on the SSDs. ZFS on Linux is very flexible, but not really fast... with SSD-only raids it's ok, but with HDDs it's depends on your IO-workload. An SSD or NVMe for journaling and cache can be very helpfull. But use the right SSD (enterprise - look, which ssd is usable for ceph-journaling - then they is for zfs fine too).

2-for windows share ill be using the built in template turnkey file server which in where i will mount the DATA folder from the zfs Data pool, i was thinking of using OMV, but havent play around with that so not sure how reliable is on proxmox. Any suggestion about this? asking because last week i have to reinstall the whole proxmox cuz the containers stop working (wont start) after proxmox update.

3- talking about the proxmox update, my containers stop working after APT UPDATE command, i was reading and people say you need to comment out the enterprise repository for the source list and add the free one... should i do that in order to prevent this kinda of behavior? if so whats the process? I am not clear at this yet....i had that proxmox server for quite sometime, but never did the update before till last week, then everything stop working.

without an subscription it's important to disable the enterprise-repo, enable the pve-no-subscription-repo and do allways "apt dist-upgrade".

4- what raid controller do you recommend, i have this one https://www.ebay.com/itm/Dell-47MCV...-Flashed-IT-Mode-withcable-U039M/333108741298 on IT Mode, maybe you guys have a better suggestion since i will another one!

This isn't an raid-controller anymore - it mode mean hba. But you are lucky - this is the right thing you need for zfs (zfs "need to"/"should" see the whole, real disk)

5-what is the best practice for the host backup, i have no problem with the DATA zfs since i can always import it into another machine, but what about the host boot drives goes bad, any suggestion to do a disaster recovery for such a scenario.

you need only some files to backup (meanly /etc (incl. /etc/pve)). A new instsllation of pve is very fast - than copy back your config, which you are need.

6- Is there any script or software to do automatically/schedule snapshots for vms and so? would like that since is faster and takes less space, so would be great if you guys guide me on this matter.

something like zfs-auto-sanpshot? https://github.com/zfsonlinux/zfs-auto-snapshot/

7-What Unprivileged container mean, i notice sometimes when creating if check marked it the container wont start... i get error or something, whats the deal?

If you create an unpriviledged container, they start... but if you change an container, they won't.

Udo

 

[Michael S Ortega]

thank you for the replies, one last question... you suggest to used RAIDZ but i read that its not recommend for production use, instead use raidz2, but since i dont have the amount of disk nor the HD Bays on the computer, is not an option, so whats wrong if i use zfs rad10 mirror? thanks

 

[ertanerbek]

thank you for the replies, one last question... you suggest to used RAIDZ but i read that its not recommend for production use, instead use raidz2, but since i dont have the amount of disk nor the HD Bays on the computer, is not an option, so whats wrong if i use zfs rad10 mirror? thanks

With RAID10 ( miror + mirror ) firstly loss of space %50 percent with 4 Disk. Raid 10 also use two disk at same time for WRITE request, but read performance will be grow up. With Raidz ( one disk for parite, 3 disk for data. Same with RADI5 ) you will get 3 disk space so write random write request should be grow up, squential will grow up but not more due sync issue, read speed should be grow up. With four disk write speed never be have huge difference so why you want lost one disk area ?

You can test your system with Raıd10 and Raıdz and then you will see result.


RAID 10

zpool create tank mirror disk1 disk2
zpool add tank mirror disk3 disk4


RAID 5

zpool create -o ashift=12 tank raidz disk1 disk2 disk3 disk4


BW test with 1 MB file 32 Q and 8 job via 5Gb File..

fio --randrepeat=1 --ioengine=libaio --direct=0 --gtod_reduce=1 --name=test --filename=test --bs=1M --iodepth=32 --size=5G --readwrite=randrw --rwmixread=50 --numjobs=8 --time_based --runtime=300

KVM IO test with 64K file 32 Q and 50 job

fio --randrepeat=1 --ioengine=libaio --direct=0 --gtod_reduce=1 --name=test --filename=test --bs=64K --iodepth=32 --size=5G --readwrite=randrw --rwmixread=50 --numjobs=50 --time_based --runtime=300

you can watch your system during test time with;

dd -d -D total,disk1,disk2,disk3,disk4 ( This command will show you BW )
zpool iostat -v tank 1 ( this command will show you IO )

 

[LnxBil]

5-what is the best practice for the host backup, i have no problem with the DATA zfs since i can always import it into another machine, but what about the host boot drives goes bad, any suggestion to do a disaster recovery for such a scenario.

If you install PVE on your ZFS (spinners), you can just send/receive your whole installation to backup place and restore if by reversing the sync-direction to another host. This relates to your question 6.

Another solution is to use swap disks in a mirror 1: You start with 3 disks, 2 installed and one stored somewhere. Every other week you swap out the disks so that the RAID will resynchronise and you will always have one system backup laying around.

6- Is there any script or software to do automatically/schedule snapshots for vms and so? would like that since is faster and takes less space, so would be great if you guys guide me on this matter.

Yes, you can do automatic snapshots of your whole pool with e.g. cron. I'd also send/receive backup your snapshots to another machine.

2-for windows share ill be using the built in template turnkey file server which in where i will mount the DATA folder from the zfs Data pool, i was thinking of using OMV, but havent play around with that so not sure how reliable is on proxmox. Any suggestion about this? asking because last week i have to reinstall the whole proxmox cuz the containers stop working (wont start) after proxmox update.

Yes, good idea. It's always the best to separate your environments: PVE does only virtualisation and the "real" work is in your containers. I'd bind mount (via GUI) some datasets to your fileserver and install samba with shadow copy feature that it'll export the ZFS snapshots directly to your windows clients.

 

[Michael S Ortega]

thank you guys for all the replies, in conclusion ill be using a proxmox server with 4 disk (Dell enterprise) and 256 SDD drive (enterprise) for my ZFS POOL, So is separate from the boot system and can easly import it into a new installation, the host itself will be installed on the 2 mirrored 60gb ssd (no data or vm will be saved on this), for backing up the host i just going to save the actual etc/pve folder since it contains pretty much what is important for the PVE restoration. this proxmox will be serving 2 windows server vm and an LXC file server container. OH almost forgot ill comment out the enterprise repository so i dont break the PVE using the "apt update" command, ill be using the apt dist-upgrade! so everything is stable for my first production enviroment. thank you all for the help. Greetings

 

[LnxBil]

OH almost forgot ill comment out the enterprise repository so i dont break the PVE using the "apt update" command

Just to be clear: That will not break anything. You will however get a permission denied on every apt update, so it's ok to comment it out. Don't forget to add the no-subscription repository if you do not plan to support PVE with a subscription.

 

[Michael S Ortega]

got it!

Just to be clear: That will not break anything. You will however get a permission denied on every apt update, so it's ok to comment it out. Don't forget to add the no-subscription repository if you do not plan to support PVE with a subscription.

got it