First thoughts on proxmox as a home user looking for virtualisation support

[sjmudd]

I'm not sure if this is the best way to make a comment, but I've been using proxmox for home usage trying it for various use cases.

• Intended usage: adhoc "development", "testing" of VMs or containers or maybe k8s or similar as needed
• I run various services at home such as webserver, DNS, email and have a NAS to provide permanent storage. The services are expected to work and ideally require least effort to keep them running. e.g. "tiny scale" production (from my perspective).
• I've been using Unix or Linux for years so am comfortable with that and learning the more "modern" container technologies. They don't seem to be easily built at home without a lot of care.
• I don't need clustering, but I don't want to lose data

So with this background I thought of using proxmox vs some VMs and adhoc LXD on a ubuntu box with a small 5-disk ZFS redundant storage. I've been playing for a few weeks and like things, but find some other things rather frustrating.

Here are my thoughts:

• The good
  • the GUI is great. Not completely intuitive but easy to use and gives you a pretty good overview of what's going on
  • you provide an API for driving proxmox. That's also good so in theory a proxmox box/cluster can be "driven" remotely without logging on to the system
  • There's a lot of good work done here and proxmox looks to be a solid product if your usage fits it's intended user base.
• The bad
  • pvesh seems hard to build if I want to run it adhoc on any say ubuntu / debian / centos / fedora box outside of the proxmox server/cluster. If there are instructions please point them to me. pvesh seems to be "Action based" not "intended state" based (like kubectl with yaml config).
  • there's no direct docker / k8s support in proxmox. I can understand why, but there's a lot of interest there. While you support containers LXC containers are not the same as docker containers and a lot of people would like an easy way to manage their containers "directly" in a proxmox cluster. I've seen similar comments from others. So maybe making docker images and related network storage management integration work there too would bring a lot more people.
  • getting hold of images, either VM or LXC images seems hard. It's not 100% clear to me exactly how to do this. You have to find them in the appropriate format, download them and can then use them, but the process while it works feels clumsy. With docker I can do docker run .... or docker pull .... and it's pulled from a remote repository and downloaded. If it's tagged against "latest" I'll get an update when one is available.
    • I think that providing a way of linking to external image sources would be good so that we can simply search them, and potentially do this automatically mentioning the appropriate image.
    • so make it easy to find both iso and VM or container images from remote repositories so they can be easily found. Even if you don't add these by default as some people may not want to provide a list of repos for the major distributions so the VMs or their appropriate cloud images can be used with less effort
    • I use CentOS 8 atm. I'm aware of the debate with the switch to CentOS 8 stream. I've now moved to stream, others have chosen differently and I am surprised that you don't provide a recent (the least) CentOS 8 iso or runnable VM image or any CentOS 8 stream ones, or even CentOS 9 stream which has been out for some time now. Again making it easy to add a url so they can be found would be really good but I don't see clear documentation of how to do this.
    • The VM or container images (LXC) work well but these days the good thing is to keep as much state out of the "container image" but to link in via mounts (overlay or simply extra mounted disks) on external storage. Technically you can do that but I don't see an easy way to upgrade an LXC container image when updates happen which leads me to the same inconveniences of managing a VM, even if it's lightweight. Coming up with a good way to link into the image the "user/app data" then allows for the base image to be updated more easily and this would give something closer to a docker experience which would be good.
  • storage is confusing. I installed with local zfs (from a single HD) and that seems to work fine and tried to attach my external zpool to the proxmox server. That works but I see there are multiple storage types and zfs seems not to support all types of usage (I really wanted to have all "user data" on my external zpool with absolute minimum data on the rpool that holds the OS). Then I read you can get fuller storage support if you export the zpool via NFS and load it back onto the proxmox box (e.g. self mount). Very confusing and unclear.
    • I wasn't able to download container images and use them to the external zfs pool. Again reasoning was not clear.
    • if you can please simplify the storage layer as seen by the user as much as possible so the container/vm images can be downloaded easily, and can be moved to the right place to be used. I couldn't use Ceph on a single box. That may be the recommended clustered storage solution but does not seem to be appropriate in my case.

So as things stand I'm planning on giving up as I find that while the base setup for my intended usage is there proxmox seems to be just a bit too hard to use when actually it looks like it should be the solution to my problems. Perhaps it requires more time and effort from me or perhaps what I'm looking for just does not exist. I'm not sure.

Have others had similar experiences but found that proxmox did solve their problems? Thanks for sharing your thoughts.

Simon

 

[dcsapak]

Hi,

thanks for the feedback, that's always helpful.

I'd like to comment on your findings:

The good

• the GUI is great. Not completely intuitive but easy to use and gives you a pretty good overview of what's going on
• you provide an API for driving proxmox. That's also good so in theory a proxmox box/cluster can be "driven" remotely without logging on to the system
• There's a lot of good work done here and proxmox looks to be a solid product if your usage fits it's intended user base.

It's always nice to hear such things

pvesh seems hard to build if I want to run it adhoc on any say ubuntu / debian / centos / fedora box outside of the proxmox server/cluster. If there are instructions please point them to me. pvesh seems to be "Action based" not "intended state" based (like kubectl with yaml config).

pvesh is more of a debugging tool than a general-purpose client (it only properly works for a locally installed pve). If you want to drive PVE from outside, i'd recommend using one of the many api clients:
https://pve.proxmox.com/wiki/Proxmox_VE_API#Clients

there's no direct docker / k8s support in proxmox. I can understand why, but there's a lot of interest there. While you support containers LXC containers are not the same as docker containers and a lot of people would like an easy way to manage their containers "directly" in a proxmox cluster. I've seen similar comments from others. So maybe making docker images and related network storage management integration work there too would bring a lot more people.

PVE is intended to be used for OS level virtualization, not application level. If you want to run docker in a PVE environment we recommend using them inside VMs or Containers (see e.g. our FAQ: https://pve.proxmox.com/pve-docs/chapter-pve-faq.html)
This is more of a Design decision more than anything else. (Also it's basically impossible to manage LXC and Docker at the same time due to the way both expect the system to behave and be configured)

getting hold of images, either VM or LXC images seems hard. It's not 100% clear to me exactly how to do this. You have to find them in the appropriate format, download them and can then use them, but the process while it works feels clumsy.

for containers there is pveam (also available on the gui) which provides many os level templates and even many turnkey templates with preconfigured/installed applications
for VMs, it is expected you bring your own ISOs and create the templates yourself, or use vendor-generated images (e.g. for cloud-init)

storage is confusing. I installed with local zfs (from a single HD) and that seems to work fine and tried to attach my external zpool to the proxmox server. That works but I see there are multiple storage types and zfs seems not to support all types of usage (I really wanted to have all "user data" on my external zpool with absolute minimum data on the rpool that holds the OS). Then I read you can get fuller storage support if you export the zpool via NFS and load it back onto the proxmox box (e.g. self mount). Very confusing and unclear.

each storage type has it's defined usage types. see https://pve.proxmox.com/wiki/Storage
that said, a single storage is not limited to one storage definition in pve. for example you can configure your zpool as zfs storage, but also create a subvolume which you configure as directory storage. that way you can also
store isos/backups/etc. on your zpool
please check out the docs, and tell us if you find something concrete missing

 

[sjmudd]

Hi Dominik,

Thanks for the response. I think the pvesh could be a good easy to use tool. It's a shame you only build it for internal usage. Perhaps it's worth considering providing it for usage outside of the pve boxes. In any case I'll look at the alternatives you suggest. So thanks for that.

So while I understand that proxmox may have been originally intended just for VM like usage, I personally would like to use it for that but also for docker style usage. Yes you can use it as suggested but that's still not really providing the direct solution which I think many people might find useful. Sometimes we want a VM sometimes a lxc container may be good but these days there's a HUGE number of pre-built, ready to be used docker containers and using them directly on proxmox would be really good. Maybe that just means you provide a turnkey docker container with a prettyish frontend, similar to docker desktop or whatever, and with an easy way to provide the docker files or docker-compose.yaml files. Direct support would be nice.

Support for external images is probably why docker is so useful and popular. They are built to be customised, and are easy to update, with a brief update cycle keeping container code away from application required data.

The LXC images don't seem to be in that state of maturity and even if you guys can not provide all the appropriate runnable images making it easy to find them would save people learning proxmox a lot of time.

As explained having quite old images of CentOS which is still pretty popular I believe is a case in point. It wasn't 100% clear to me which of the centos images provided by RedHat for CentOS 8 stream or CentOS 9 stream can be used as runnable lxc images, if any, nor what is needed to create them if they are missing. Ideally that info can be provided once (e.g. a link to an external repo?) and everyone can use them without having to go through that learning process. I may have an interest in CentOS, others may have an interest in other Linux distributions and from the VM side such links to different distributions whether linux or not just saves people time and simplifies the process of finding, downloading and installing versions.

Ubuntu's lxd I think provides quite a range of OS images to download, though as you say you can also download your own. The more options you provide the easier proxmox is to use.

Related to the storage I'll try and see what I find confusing and report back. My intention had been really for proxmox to not store almost any local storage but to use my external zpool for ALL data, basically everything except the OS, and maybe even including the OS configuration. Why? because then I can throw it away and I don't lose any of my data, just the OS that's running it and if I store enough of the proxmox config I could move the VMs or LXC containers to another platform without issue. Or do that and find I want to go back and still not have a problem. However, the locally mounted zfs pool did not seem to allow me to do everything in one place.

In fact looking at the zfs mount rpool setup of the local storage I notice:

root@mad19:/etc/pve# cat storage.cfg
dir: local
path /var/lib/vz
content iso,vztmpl,backup

zfspool: local-zfs
pool rpool/data
content images,rootdir
sparse 1

(this system is booted off zfs) so why is there a need to have 2 storage definitions here and why can't all image types be on zfs directly?
I didn't check to see why /var/lib/vz is setup differently to the base rpool, and even the rpool datasets look a bit weird:

root@mad19:/etc/pve# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 3.26G 1.75T 104K /rpool
rpool/ROOT 2.00G 1.75T 96K /rpool/ROOT
rpool/ROOT/pve-1 2.00G 1.75T 2.00G /
rpool/data 1.24G 1.75T 104K /rpool/data
rpool/data/subvol-100-disk-0 532M 7.48G 532M /rpool/data/subvol-100-disk-0
rpool/data/subvol-101-disk-0 658M 7.36G 658M /rpool/data/subvol-101-disk-0
rpool/data/subvol-102-disk-0 75.3M 19.9G 75.3M /rpool/data/subvol-102-disk-0

Perhaps if I were to understand this part a bit better I could build my own external zpool in a similar way and thus keep all my data on this external array?

Anyway I guess I need to investigate further.

Thanks,

Simon