Help me understand the basics

[Dobbler]

Hi guys,

I just started setting up a new server (for dev & storage/cloud stuff) and I want to use virtualization.
Up to now I have tried unRaid and Proxmox for a few days. Proxmox offers much better performance but I haven't figured out how to set it up properly.

This is my hardware:
HP Microserver Gen 8
-G1610T (desperately looking for a 1265L V2 xeon!)
-16GB ECC Ram
-2x 3TB WD Red (for RAID Storage)
-1x 120GB Samsung Pro SSD (for Proxmox OS)
- mdadm or ZFS, no hardware RAID
NIC 1: LAN (Proxmox Web UI)
NIC 2: DMZ

It is planned to run two main VMs:
VM1: testing/dev server (Ubuntu, connected to LAN) running SMB shares, database, tools, ...
VM2: cloud storage server (Debian, connected to DMZ/public access) running Seafile, Baikal, Apache, ...

Issue #1: Storage
I used to set up different (logical or LVM) partitions for OS and storage. With virtualization, I assume it makes more sense to create different virtual disks?

My plan is to install Proxmox on the Samsung SSD and use the 2 WD Reds for RAID 1 mirrored storage.
Does Proxmox support software RAID by now? During installation I can only select one drive. How can I add the missing drives?

What is the best way to format the virtual disks? Should I use Ext4 or set up LVM inside the guest OS?

Please give me an overview (recommended procedure?) which filesystem etc. must be used where to achive maximum performance and security.

Issue #2: Networking
I am concerned about connecting both my LAN & DMZ to the server. It is important to ensure that these two networks aren't bridged or traffic somehow bypasses the router.
https://pve.proxmox.com/wiki/Network_Model
If I understand that correctly, I can add a second bridge and assign my DMZ interface to it.
Another possibility would be to use PCI passtrough?

Thanks in advance!

 

[alexskysilk]

Re: Issue #1
Proxmox storage support is virtually universal. With your setup, what I would suggest is as follows:
1. Proxmox itself doesnt need much space at all. Your HP Microserver has an internal SD card slot- 8GB would be perfect for proxmox, and larger is a bonus. I'd pony up for a fast card such as a sandisk extreme pro - which is only like $5 more then a "regular" one.
2. primary storage- ZFS mirror across your 3TB drives.
3. your ssd can be used for L2ARC/ZIL offload (plenty of documentation available online.)

tl;dr- SDcard for boot, HDDs for datastore, SSD for cache. All your VMs would deploy their virtual storage to your primary storage pool.

Re: Issue #2
The proxmox network model gives you a lot of flexibility. In your case, since you already have two nics and you'd like to use the physical separation- simple create two bridges, 1 per nic; connect your internal vm(s) to one bridge, and your external VMs to the other. Its perfectly acceptable to have your internal VMs also have a second virtual NIC to attach to your external bridge- with all the security vulnerabilities that this entails you'd need to address it with your VMs firewall. BONUS- create a VM that would act as your router (using pfsense, untangle, vyos, or ?) and have IT control what goes where.

 

[Dobbler]

Thank you alexskysilk for your fast response, and my apologies for not replying earlier!

In the meantime, I upgraded my setup and did some testing:
-Xeon E3-1230 V2
-SanDisk Ultrag 16GB USB boot drive

The performance is amazing. I created a Debian container and set up samba shares. Read speeds easily saturate my 1G LAN. Writing to the array is slower at 60M/sec but I have not yet added the cache SSD.


I noticed the installer created a 1.75GB Swap partition on the USB drive. Can I install PVE with swapsize=0 and manually add Swap later?
Is it a good idea to set up Swap on the ZFS pool?

Are there any other settings that should I change when I install on the USB drive?

 

[udo]

Thank you alexskysilk for your fast response, and my apologies for not replying earlier!

In the meantime, I upgraded my setup and did some testing:
-Xeon E3-1230 V2
-SanDisk Ultrag 16GB USB boot drive

The performance is amazing. I created a Debian container and set up samba shares. Read speeds easily saturate my 1G LAN. Writing to the array is slower at 60M/sec but I have not yet added the cache SSD.


I noticed the installer created a 1.75GB Swap partition on the USB drive. Can I install PVE with swapsize=0 and manually add Swap later?
Is it a good idea to set up Swap on the ZFS pool?

Are there any other settings that should I change when I install on the USB drive?

Hi,
I don't know the "SanDisk Ultrag 16GB USB boot drive", but is it reliable enough!? With SD-Cards I would say no (because of logging/swap).

swapsize=0: If you have enough ram it should work - but I'm not sure if you must set the swap for all CTs to 0 also. Of course you can change your swap-settings later - it's linux!

Swap on ZFS: AFAIK there is an issue with that. Use the search option aout this.



Udo

 

[alexskysilk]

Glad its working for you.

I noticed the installer created a 1.75GB Swap partition on the USB drive. Can I install PVE with swapsize=0 and manually add Swap later?

As Udo explained, of course. In reality, you dont need to reinstall anything, you can kill/move the cache partition and edit /etc/fstab at any time.

As for swap on zfs: on a "production" system you wouldnt do this. Swap usage can and will affect the storage pool in various ways, and it can affect performance of storage objects used by guests. if its just for your own use- sure. create a zvol of a fixed size and use mkswap on it. full instructions here:

https://github.com/zfsonlinux/pkg-zfs/wiki/HOWTO-use-a-zvol-as-a-swap-device

Swap on ZFS: AFAIK there is an issue with that. Use the search option aout this.

While I'm not aware of any issues, Its definitely worthwhile doing some research on this before committing. It can save you pain later.

 

[udo]

Swap on ZFS: AFAIK there is an issue with that. Use the search option aout this.

Hi,
found the post: https://forum.proxmox.com/threads/zfs-swap-crashes-system.25208/#post-126215

Udo

 

[alexskysilk]

Having read through that thread, I'd have to say dont use swap on a zvol as your first option. It might be usable when tuned, and it might be/has been corrected with a kernel update (I dont know, I havent tried) but in the meantime you're exposed to instability.

Depending on your workload, running swapless may be an option. alternatively, it may be more advantageous both in terms of performance and stability to simply add more ram.

 

[Dobbler]

Hi, thank you, that are helpful tips.

I bought the "SanDisk Ultra CZ48 16GB USB 3.0 Flash Drive" instead of a SD Card because it's easier to format/copy.

I want to use Swap, sorry I haven't made that clear. But I agree with Udo, Swap on an USB drive is probably a bad idea, and slow.
The HP Microserver only supports up to 16GB RAM.

There is an unused SSD installed. Can I use it for Swap and Zil?

 

[alexskysilk]

There is an unused SSD installed. Can I use it for Swap and Zil?

Sure. Just make sure that wear levelling is supported and functioning or you'll kill your drive (I know Samsung has a tool for this called Samsung Magician or something) If its an 840/850 family drive it should be.