Retired Systems Engineer Installing Proxmox for Home Lab and Learning

[Desert Yote]

I am a self taught systems/test engineer and software architect. I have been retired for 4 years. I have been involved with the latest and greatest since 1975 (high school gifted program, and dad an EE). Just because I am retired, I don't plan on slowing down. I have big projects in the pipeline, including designing a programming language for test and measurement that is based on information and category theory. My old Ubuntu is garbage and broken. I was going to install Debian, but it seems that installing Proxmox on my workstation and going virtual with a VLAN would click all the boxes, e.g. a Virtual Spectrum Analyzer and Signal Generator in LXCs

I have a plan worked up, but after 45 years of engineering, I have become a big fan of "design review". I would be thrilled if someone could look it over and tell me what I have messed up. I find that the Proxmox documentation is not very useful. In the spirit of keeping up with technology, I used Grok AI to generate a procedure for me. I have been verifying all of the steps. Some things I can't find, e.g "-cpu host,+rt". I understand what this does, it is just that I find no reference to the +rt flag anywhere to verify.

I am building on a Dell Precision 7820 with a Intel Xeon Silver 4215R 3.2 GHz cpu. I have 64GB RAM, 2 512GB NVMe SSDs for storage and a 256GB SATA for my Ubuntu boot, plus NVIDIA graphics. I am adding a 2TB NVMe. I have normal home Xfinity internet, 10.0.0.1.

My plan is to leave the SATA drive alone and to make my first 512 SSD the Proxmox boot drive. with the 2TB and the second 512GB SSDs used in a RAID1 configuration. This will be using ZFS. I love ZFS, I almost went for Open Indiana/illumos as my OS. Next I want to create a "Lab" VPN (vmbr1)

My planed initial layout:

VM0: "Internet", Debian on vmbr0 for Email and Web browsing.
VM1: "Dev", Debian on vmbr1 (lab) for software development. I use C, C++, Python, Java, Go Lang, Mono, and soon Julia.
VM2: "Art", Debian on vmbr1 (lab) for art (GIMP, Krita, Inkscape), document authoring (LibraOffice, Scribus), CAD (FreeCAD?), and later animation.
VM3: "Music", Debian, with low-latency kernel, on vmbr1 (lab) for MIDI and my Roland drum kit and keyboard.
LXC0: "Firewall", Debian on vmbr0.
LXC1: "GIT-DB", Debian on vmbr1 (lab) with a GIT server and SQlite, later a graph DB and maybe Artifactory.
LXC2: "Media", Debian on vmbr0 as my home media server.

The internet "world mind", according to Grok AI suggests the following, BTW, I am very comfortable with the command line:

1. Plug in 2TB NVMe and boot into BIOS.
Advanced -> NVMe Configuration : Enable both slots
Boot -> CSM : Disabled
And it said nothing about setting the first NVMe to a boot drive, so I am a bit confused.

2. Boot Proxmox USB (Grok said nothing about using F12 to change boot drive to USB)
Target: /dev/nvme0n1
Filesystem: ext4
Network: Static IP 10.0.0.1/24
Password: My WIFI Network IP

3. Post installation in Proxmox shell:
>apt update && apt full-upgrade -y

4. Create ZFS mirror and Datasets:
>zpool create -O compression=lz4 \
-O acltype=posixacl \
-O xattr=sa \
fastpool mirror /dev/nvme1n1 /dev/nvme2n1
>zpool status # to verify
>zfs create -o compression=lz4 fastpool/dev
>zfs create -o compression=lz4 fastpool/art
>zfs create -o compression=lz4 fastpool/music
>zfs create -o compression=lz4 fastpool/gitdb
>zfs create -o compression=zstd-3 fastpool/media
>zfs create -o compression=off fastpool/vms # QCOW2
>zfs create -o compression=lz4 fastpool/backup

5. Lab VPN (10.0.2.0/24) This is were I am a bit confused because the documentation is not clear at all. The documentation is nothing like the way that I document a GUI or Web Service at all. This is what Grok said. It is wrong.
Using the Proxmox Web: Datacenter -> Network -> Create Linux Bridge
Name: vmbr1
IP: 10.0.2.1/24
VLAN aware: No
Bridge ports: none
Autostart: Yes

Because the GUI path from Grok is wrong, I did a search online. I got NodeName -> Create -> Linux Bridge. What is NodeName? I don't like needing to run to YouTube videos or Stackoverflow to learn stuff that the documentation should provide. I don't like not
being able to verify this in the documentation.

6. Create the VMs and LXCs:
VM0 (Web + Mail)
qm create 100 --name internet --cores 4 --memory 8192 \
--net0 virtio,bridge=vmbr0 \
--cdrom local:iso/debian-12.7.0-amd64-netinst.iso # The docs don't explain this at all. Btw, I have Debian 13.2
qm set 100 --scsi0 fastpool/vms:100

VM1 (Software Development)
qm create 101 --name dev --cores 24 --memory 98304 \
--net0 virtio,bridge=vmbr1 \
--scsi0 fastpool/dev:250
qm set 101 --vga qx1 --args "-cpu host" # Finding out what this meant was a PITA.

VM2 (Art)
qm create 102 --name art --cores 8 --memory 32768 \
--net0 virtio,bridge=vmbr1 \
--scsi0 fastpool/art:400
qm set 102 --vga qx1 # No --args "-cpu host" ???

VM3 (Music) RT
qm create 103 --name music --cores 8 --memory 32768 \
--net0 virtio,bridge=vmbr1 \
--scsi0 fastpool/music:150 \
--args "-cpu host,+rt" # The documentation references the nvm(1) manpage with no link to it! That is less then helpful.
qm set 103 --usb0 host=1234.5678 # Roland V-Drums, but 1234.5678 ???

LCX0 (Firewall)
pct create 200 local:vztmpl/debian-12-standard_... \
--cores=1 --memory 1024 --rootfs fastpool/firewall:20 \ # Grok didn't tell me to make a fastpool/firewall dataset
--net0 name=eth0,bridge=vmbr0,ip=dhcp

LCX1 (GIT-DB) Lab VPN
pct create 201 local:vztmpl/debian-12-standard_... \
--cores=2 --memory 4096 --rootfs fastpool/gitdb:200 \
--net0 name=eth0,bridge=vmbr1,ip=10.0.2.10/24

LCX2 (Media)
pct create 202 local:vztmpl/debian-12-standard_... \
--cores=2 --memory 4096 --rootfs fastpool/media:1000 \
--net0 name=eth0,bridge=vmbr0

7. Install Debian in VMs:
For each VM: Start -> Console -> Install
Root password: My Proxmox root pasword ?
Partition: Use entire disk
Packages; SSH server, standard utilities

8. Post-Install for each VM:
apt update && apt install -y \
build-essentials git curl zsh tmux neovim python3-pip openjdk-17-jdk julia i3 wmctrl xdotool htop nvtop # VM1: Dev
gimp scribus krita libraoffice inkscape # VM2: Art
ardour hydrogen qjackctl linux-lowlatency # VM3: Music
gitea sqlite3 # LXC1: GIT-DB

9: Lab VPN Firewall:
In firewall LXC (OPNsense or iptables) # Groks instructions aren't very helpful here. Do I need to install OPNsense? Is this in a gui?
Allow:
vmbr0 -> Internet
vmbr1 -> 10.0.2.0/24
Block:
10.0.2.0/24 -> Internet # I am unsure about this

I want to be sure of everything before I start. I need to understand everything that I am doing. I have only one workstation and I don't want to screw it up with a failed Proxmox installation attempt. This needs to work first time. I could really use some advice on this.

thx, Bob

 

[danmac]

i had a quick scroll through, a few brief observations

"NodeName" simply means whatever you named your proxmox server lol ... just use the web GUI unless you have a super good reason not to, trust me ... i think the Proxmox documentation is very good myself, outstanding compared to some OSS projects, and it's available locally from the host's web interface, too
OPNsense is not Linux, it's based on FreeBSD, so will never work in an LXC container, you will need a full VM if you want to do that
Others may have experience of using PVE as a desktop machine with nvidia graphics which seems to be your plan (?) but i've seen from others this can be a hassle particularly when updating the host to a newer kernel so expect to have to keep on top of that from time to time
Regarding music in particular, do make sure latency won't be a problem as running inside a VM can be susceptible to those kinds of issues, also make sure your art software won't be too laggy
I'm kinda unsure why all the network separation if im honest but i do run a similar setup with 1 physical & 1 VM OPNsense

"this needs to work first time" sir with respect it will not work first time and any expectation of such will lead you to having a bad time

i recommend setting up another machine, even a basic laptop or something, using that to scratch your immediate computering itches like web and email, then use the xeon as a testbed for all this stuff. You will need another machine to interact with the Proxmox web UI (at least initially) and to test the networking, do the usual internet stuff and suchlike, and for situations where the xeon decides it's not gonna play with your nvidia graphics after an update. In general logical order i would:

1. install the proxmox system
2. get familiar with the basic operations via the web interface
3. figure out how to run it as a desktop (if that's your goal) and if you are happy with the trade-offs
4. set up your additional storage
5. try setting up one of your debian desktop VMs and get comfy with all that, ensure it will do what you want it to do, document whatever you do to make it nice and performant as you will be doing that with all the other desktop VMs too
6. set up your fancy networking (if you haven't used OPNsense before, you are in for another realm of complexity)
7. set up your auxilliary LXCs (i guess you can do this any time after #4 should be easy enough)

Regarding use of Grok, as with most AI, they are too agreeable and willing to please, and often "hand wavey" and light on the details. I use Grok occasionally and I find it useful but be aware of how you're prompting it. In a nutshell Grok is more eager to have you like him, and for you to have a nice time chatting with him, than for him to be precise. Try drilling him on the details, have him reconsider what he's already told you.

Good luck, Bob

 

[Desert Yote]

Hi Danmac,

Thank you very much for your advice. I really appreciate your willingness to wade through my autistic wall-o-words writing style. I am going to heed your advice and take things slow. I surely don't need to do this all at once. I will be saving the implementation of a lab VPN for another day.

Grok acts like a child that is anxious to please. It is too willing to agree. I have been playing with it for about a month, learning how to leverage it for research. I figured that I would see if it could get it to design a procedure to implement the system that I want. This then could be used as a starting point to get into the documentation.

(I got frustrated when I had to do a great deal of searching to find a reference for "-cpu host", and needed to resort to Google to get info on "-cpu host,+rt". The reference to kvm(1) man page should be a link to that man page.)

I didn't trust Grok very much to begin with. It would agree with my plan even if it was silly. That is the main reason that I started this thread, not documentation. Do my goals make sense? From what I gather, not so much. I need to take things one step at a time. And the music VM can wait until I do more research.

My plan was to use a VM for my general purpose Desktop (Email, web), not Proxmox directly. Is this going to be a problem?

The main goal for the other VMs is to have dedicated machines with only the software that is needed. My current desktop is a bloated pig. The LXCs don't need a GUi. I envision these as terminal only. I'm a minimalist when it comes to system design. In this spirit, on my "dev" system, I will be doing all my coding with NeoVim. I won't be installing any IDEs unless absolutely necessary.

I guess I should stay away from OPNsense (and firewall) for now. I have some experience with iptables (back in the late Bronze Age). Would that be a better option for me?

I do want to get experience with setting up VPNs. That is one reason for creating a "lab" network. I have other reasons. E.g., I have been developing a programming language targeting test automation and instrument control. It is quite unlike anything that has been done (NI better watch out ). During the test phase, I will be creating virtual test racks with virtual O-scopes, Signal Sources, Spectrum Analyzers, etc. I was planning on implementing these in LXCs on a dedicated lab network. The VMs for Music and Art don't need to be on the lab network, only my software Development system and the GIT-DB LXC.

Btw, when I wrote "this needs to work first time", I chose my words poorly. Piloting new technologies never goes as planed! I'm a test engineer with 40 experience building systems (Motorola, Hewlett-Packard/Agilent, Intel). Rarely have things gone right the first time through. I am expecting to encounter problems. I am sure that I will need to do some futzing around. After all, it is solving problems that gives meaning to the life of an engineer.

By "working" I meant "minimally functional", i.e. not a hosed system that I can't fix. Learning is all about working through problems. This project is mostly about giving me experience in new technologies, anyway. I don't want to go stale in my retirement. And, I am retired so I have all the time in the world

thx again,
bob

 

[alexskysilk]

My plan was to use a VM for my general purpose Desktop (Email, web), not Proxmox directly. Is this going to be a problem?

Understand that PVE is not designed or meant to operate as a desktop OS. I see two rational options depending of the answer to a simple question- are you intending to use this computer as a workstation or will you be attaching to it over the network from another machine?

IF the SOLE computer:
Simply put- dont use PVE. install your choice of workstation OS, and you can use a type 2 hypervisor (think virtualbox, but you can use QEMU, vmware workstation, etc to the same effect) to virtualize SOME of your described functions. I say some because any application that wants be accessed with a local OS/GDM APIs will never perform or have the same user experience virtualized vs native (eg, "Art" and "Music". a special note on external interfaces for music- getting good results through a virtualization layer can be quite challenging and serves no real benefit.) You can achieve all the desired functionality AND have a satisfactory workstation experience.

IF a dedicated server:
Similar to above, I would keep your local applications installed on your workstation. everything else can live here, with a caveat:

In either case, dont comingle your router with other hardware. it means that any time you want to bring the parent server down your whole network goes out- better to leave it on a dedicated piece of hardware; dd-wrt is available for many consumer grade router hardware that can be had super cheap- and if you really want pfsense/opensense grade you can buy a used little pc with two NICs and go nuts

 

[Desert Yote]

Understand that PVE is not designed or meant to operate as a desktop OS. I see two rational options depending of the answer to a simple question- are you intending to use this computer as a workstation or will you be attaching to it over the network from another machine?

Hi Alexskysilk,

Thanks a lot. I am going to wait until I get some proper hardware before I start playing with Proxmox. When I do, it will be primarily for hosting the RESTfull web services that I am planing on writing, a media server, and a git + db server. The more that I think about it, the more I feel that I really need a dedicated machine for Music. Running it in a VM is pretty silly. Btw, I do have my old workstation that I built in 2009. It has two NICs. So for now, Debian 13.2 it is, with XFCE

thx again,
bob

 

[danmac]

Yeah i agree with everything Alex said. 2009 workstation might be too old for virtualisation stuff but with those two NICs could make a decent OPNsense box if you don't mind the power consumption. the xeon sounds like it would make a great proxmox box tho

re: OPNsense if you can drive iptables it shouldn't take you long to get to grips with that, and if your plan is to set up VLANs and VPNs etc, it will do all that no problem, and it's all click click boom you won't have to get your hands dirty, comes with all the other stuff you'll need like DHCP, DNS, etc.

you could do a lot worse than a refurb laptop or mini PC to use as a desktop if you don't wanna invest too much (also have you seem RAM prices ...) particularly if you're used to older gear, a mid-range mini PC will blow your socks off for like 300 dollars

also some food for thought, you can create a VM on proxmox, install desktop linux, then connect to it over the network using RDP, i've been using this for a while: https://c-nergy.be/blog/?p=20317

i don't see you going stale for a fair while at least good luck

 

[Johannes S]

ProxmoxVE isn't really suited to be used as daily driver on a workstation or notebook except when you happen to be a Proxmox developer ( https://pve.proxmox.com/wiki/Developer_Workstations_with_Proxmox_VE_and_X11 ) For a usecase like yours it's propably more sensible to install a Linux distribution of your choice on the workstation and setup several VMs with virt-manager or Virtualbox. If you however want to have a server for stuff like a NAS, docker-hosting etc and want to learn about system administration and virtualization ProxmoxVE is a great plattform.
Another interesting alternative (but definitively not for novices) for a workstation Desktop is QubesOS, which is based on Xen: https://www.qubes-os.org/

I never used it myself though and it's main goal is to have a very secure system (their examples contain one setup for a investigative journalist or somebody who wants to retire early and is thus quite paranoid in their finance managment setup) so might have some limitations for your goals.

 

[Desert Yote]

Yeah i agree with everything Alex said. 2009 workstation might be too old for virtualisation stuff but with those two NICs could make a decent OPNsense box if you don't mind the power consumption. the xeon sounds like it would make a great proxmox box tho

Thanks. I am going to use my old 2008 system for OPNsense. I built this system with love, and I hate to see it just gathering dust. It needs a job, and I haven't played with BSD since the mid 90's. Later when I can afford more hardware, I will set up a Proxmox system. I am on a fixed income until I can get something going. Then I can use RDP from my dedicated workstation. I used to do that while working at Intel to connect with my VMs,

 

[Desert Yote]

ProxmoxVE isn't really suited to be used as daily driver on a workstation or notebook except when you happen to be a Proxmox developer ( https://pve.proxmox.com/wiki/Developer_Workstations_with_Proxmox_VE_and_X11 ) For a usecase like yours it's propably more sensible to install a Linux distribution of your choice on the workstation and setup several VMs with virt-manager or Virtualbox.

Thanks for the advice. I will be setting up my system as a workstation and use something like Virtualbox to set up some VMs. I had looked into QubesOS but thought that was a little much for me right now. I am more interested in developing RESTfull web services and virtual test racks then security, at the moment. When I get another system, I will be returning to Proxmox. Hopefully that will be soon.

thx,
bob

 

[Desert Yote]

Hi,

I have a question regarding the use of virtualization for web service testing.

Back in 2010, I was assigned the task of writing a RESTfull Banking service. This was to serve as a reference implementation and demonstration of the use of Intel's version of the ME based security features that were coming on line. This was a bit different from the test automation software that was my specialty. But I enjoyed it a lot. But as a test focused engineer, I wanted to test it, and test it hard.

This is what I came up with. It was written in BASH and was ran on my Linux laptop. The main system consisted of three executables. I started by designing a "User Script Language" that described all of the things that a user might do, including mistakes and long delays. The first executable was the interpreter for this language, duplicating a user session. The actual Test executable, wrote randomized user session scripts that started with a shebang pointing to the interpreter. It would add some metadata, add a line to a test record file, and execute it in a process. A third executable would use the test record and user session files to replay the test. The idea was too keep adding users until the service barfed.

This is missing a bunch of stuff, like monitoring the web service, visualization of what was happening, e.g. number of user sessions, and it was pretty limited. I wanted thousands of simultaneous users.

How would this be done in 2025 in a modern virtualized environment?

 

[alexskysilk]

How would this be done in 2025 in a modern virtualized environment?

same as you did it before. the only difference I'd be looking at is making it portable by deploying it as a docker container.