Guidance on coming from VMware; ZFS datastore & file server

[smalltimemsp]

Hi all. VMware refugee here and new to Proxmox.

I'm building a small business server and originally was going to configure the server like this:

• Install ESXi on a small SSD.
• Create Ubuntu VM that lives on the default datastore.
• Passthrough a SAS HBA to the Ubuntu VM.
• Create a ZFS pool and share it back to ESXi over NFS and create a second datastore.
• Create additional VMs on the second datastore, use the ZFS VM as a file server in addition to being an NFS server for ESXi.

The need is for a few VMs (Samba AD, Samba file server, Linux DNS server, VPN server, Windows Server)

After a bit of researching I think I have at least these options with Proxmox:

1. Install Samba file server directly on Proxmox.
2. Passthrough HBA to a VM and share back like I would've done with ESXi.
3. Use Zamba LXC Toolbox.
4. Install LXC containers manually for the file server part and possibly for the other Linux VMs.

Any recommendations and best practices?

Option 1 doesn't appeal to me as I'd like to keep the installation as stock as possible and isolate management to a separate network. Zamba LXC Toolbox has services I don't need so feels a bit bloated and it would probably be better just to learn to do things manually anyway.

I'm not familiar with LXC containers. Are there any limitations or security and stability implications that need to be considered if running Samba AD, file server and DNS server in a container?

 

[Dunuin]

I'm not familiar with LXC containers. Are there any limitations or security and stability implications that need to be considered if running Samba AD, file server and DNS server in a container?

Its not fully isolated and shares kernel/hardware with the host. For better isolation use VMs.

Unlike VMware you usually use block devices instead of image files on filesystems to reduce the overhead.
So for example directly using the hosts ZFS pool with zvols for the virtual disks.

And PVE writes a lot. I highly recommend to mirror your system SSDs/HDDs. Especially as there is no host backup yet.

 

[smalltimemsp]

Its not fully isolated and shares kernel/hardware with the host. For better isolation use VMs.

Unlike VMware you usually use block devices instead of image files on filesystems to reduce the overhead.
So for example directly using the hosts ZFS pool with zvols for the virtual disks.

And PVE writes a lot. I highly recommend to mirror your system SSDs/HDDs. Especially as there is no host backup yet.

End users won't be logging in to the containers, so I guess the isolation isn't really an issue.

Using zvols for virtual disks in VMs sound good, but for containers ZFS subvolumes sound better. Any caveats there? For example container being bound to Samba AD and subvolume shared over SMB with owners and permissions set for the AD users. How will it look from the hypervisor side and any risks involved? Obviously touching the files shouldn't be done from the hypervisor side, but anything to be aware of?

I have a second SSD so I can setup a mirrored boot drive, which from what I looked is supported with regular onboard SATA. Although they are identical drives so have the same durability.

 

[Dunuin]

but for containers ZFS subvolumes sound better

LXCs use datasets. VMs use zvols. In both cases you directly use the hosts ZFS pool to store your data.

Any caveats there? For example container being bound to Samba AD and subvolume shared over SMB with owners and permissions set for the AD users.

You for example can't mount SMB/NFS shares in an unprivileged LXC. That only works in VMs and privileged LXCs. And privileged LXCs are privileged, so the LXCs root user is also the hosts root user.

 

[UdoB]

• Passthrough a SAS HBA to the Ubuntu VM.
• Create a ZFS pool and share it back to ESXi over NFS and create a second datastore.

One thing I hate in this computer-infrastructure context is "dependencies".

The only thing I hate more is circular dependencies. Some day in the future it will bite you - it is the exact opposite to the KISS principle. But... yes, some people went successfully this road and Passthrough of a complete controller is the correct way to do this.

I am using Zamba for my local file server requirements at home. (Without offering its service back to PVE itself - but possibly to some VMs.)

Just my 2€¢...

 

[smalltimemsp]

LXCs use datasets. VMs use zvols. In both cases you directly use the hosts ZFS pool to store your data.

The documentation talks about ZFS subvolumes, but I guess we are talking about the same thing. It also talks about directories, which seems similar but without a dataset created.

You for example can't mount SMB/NFS shares in an unprivileged LXC. That only works in VMs and privileged LXCs. And privileged LXCs are privileged, so the LXCs root user is also the hosts root user.

Luckily there shouldn't be any need to mount anything inside the container, or let end users log into it. But I read from the documentation that "With unprivileged containers you might run into permission problems caused by the user mapping and cannot use ACLs." I would need to set ACLs and permissions for Samba shares, so I guess I need to use privileged containers for this.

 

[smalltimemsp]

One thing I hate in this computer-infrastructure context is "dependencies".

The only thing I hate more is circular dependencies. Some day in the future it will bite you - it is the exact opposite to the KISS principle. But... yes, some people went successfully this road and Passthrough of a complete controller is the correct way to do this.

I am using Zamba for my local file server requirements at home. (Without offering its service back to PVE itself - but possibly to some VMs.)

Just my 2€¢...

I haven't run into any problems yet, but I also haven't run systems setup that way for many years yet so it's surely possible something can come up. I've used this in a two server setup where the ZFS data is replicated to a spare server every few minutes. It needs some setup to be able to handle power loss shutdowns and startups correctly and without intervention, but once that is configured it works pretty well.

For actual shared storage setups I've used Starwind VSAN which uses the same basic idea. I think they have Proxmox support now as well.

I agree that simpler is better. That's why I've avoided even more complicated Corosync/Pacemaker setups. There's also the idea that high availability should be built into the software product itself and bolted-on HA clustering shouldn't be needed. But "legacy" software still exists and isn't going away. Although cloud providers have outages that last for hours too and one would hope they have people supporting the system that know how it's built. But still they have outages. Just goes to show that more HA/clustering/failover solutions == more complexity == longer outages. So is it actually exchanging short outages more often for longer outages but maybe less frequently?