Proxmox for Beginners

[jo der bohrer]

The last 5 weeks I struggled countless hours to get a pair of 2 Proxmox 5.1 servers running. To get an impression of the catches I went into, here is an uncomplete list of the major timewasters:

1. Frist I tried to install PVE 5.1 on a mirrored pair of ZFS disks. As you all know, that will lead you in the desert of obscure CLI error statements, where no beginner has any chance of managing that.
2. After learning the hard way to install PVE with ext4, the next stumbling stone was: No LXC Container would start from the GUI (CLI neither). No "apt update" and "apt dist-upgrade" helped. This time I decided to have a new install with PVE 5.0. on both machines. On the first sight, this did the trick and the LXC starts every time. Until I did a update + upgrade.
3. The third major problem developed by testing the backup/restore function. I did it on a NAS, which was NFS connected. A VM with 300G Data got wonderfully backed up, I thought. I could see the *vm.lzo file on the NAS and went on to test the restore. Guess what? The restore process produced an error as early as 15% of the file has been read. And the VM was gone. Nothing of data whatsoever there. Gone to meet its maker. This would be a deadly blow in production.
Unfortunately this behaviour did not change, wether I backuped VM LXC or small files.
Now I realised the asthonishing fact, that only 2 or 3 people out there an Proxmox community hat this kind of problem. What is happening here?

Of course, now I know. But I put so much time in this project, that I decided to stress the most important points for beginners like me, to get them Proxmox running in much shorter time, without these annoying stumbling blocks.

These are the most important points to install Proxmox the right way

1. Do not install Proxmox on ZFS RAID. It is much easier to use a small singel HD in ext4.
2. Right after install, you should use CLI zpool create ..... raid /dev.. to establish the filesystem zfs on the RAID
3. Thereafter you should use CLI zfs create rpool/dataset , min. 2 items
4. Now you go GUI and give these datasets so called storage under datacenter matching names. You will need these storages for your new VMs. If you ignore these steps, then your VMs will be installed on so called "local" storage, which means on the little single HD from point 1. This way you will lose the advantages of ZFS and the main replication possibilities from Proxmox. Local Storage can not be replicated as well as backuped.
5. Before you create any LXCs or VMs is is a absolute must to correct this file:

/etc/apt/sources.list.​

Please read the instructions here: https://pve.proxmox.com/wiki/Package_Repositories
It is very very important for holders of subscription of Proxmox as well as non holders. What you will doing here is change the way, how Proxmox searches new updates. The default file in the install ISO is horrible wrong. It points to a deprecated directory with corrupted files. So when ever Proxmox calls for an update - and this happens automatically - you will get in severe trouble and will not know why.
I registered these two faults with the corrupted version: 1. LXC Containter wont start, no matter what and 2. Backup/Restore produce defect files

Had I known these facts earlier it would have saved me a lot of time.

At last I like to thank all these friendly and real knowledgeable staff of Proxmox who do their best to run this community. It is a great work and it will have a great future.

 

[NewDude]

That doesn't match my experience at all.

I installed the most recent ISO on new hardware, had the installer configure the drives as ZFS, and I've had no issues. The first machine I played with for a week was using a ZFS mirror that I added a SLOG to, then once I decided I was going to use Proxmox I subscribed. The second and third machines used ZFS via the installer as well (one as a mirror, the other as RAIDZ1); in each case I bought a subscription, updated, and joined the cluster.

The only issues I've run into are things like time differences because I misconfigured my firewall rules.

In my case I'm using new (though really cheap) Dell hardware. What are you installing onto?

 

[tom]

Please re-test, e..g. your statements regarding a wrong sources.list setup is just wrong. There is also a /etc/apt/sources.d/ directory, maybe you missed this one?

 

[guletz]

Hi,

I see many times in the linux world (and I use linux from the 20 century ) that a beginner can not do something .... and linux or whatever is bad. From my limited experience I see that most of the time a beginner user (in any IT field) try to do something that at first look is very smple(from his point of view) and he think that no knowledge is necessary.
Now at your problem regarding proxmox. Proxmox is a huge software, and you must take your time to UNDERSTAND how is working, what are the software components, what technology they use, and so on.
In my own case (with let say some basic understanding about linux, zfs, security, routing and many others - not a beginner ) I start to test for myself proxmox for many months. I test many failure situation and I try to slove them, and I let say LOST my time to understand how I can fix them. I wrote for myself what I do and was successfully and what was a bad path to solve. In case of bad results I have try to find answers on Internet - and many hors of talking with some experts friends in various IT / linux fields. And after let say 6 month of usage I consider myself that I am scratching only the surface.
The next step was to read almost dayly any proxmox forum post, and I learn many things.
I do not to say that I am the best smart guy from Romania, or the smartest Proxmox free user (I do not pay anything for my Proxmox usage, but I can say Thank You for the Proxmox team for this) but for sure I can say that Proxmox is a very good tool for any user who take some time to learn and to UNDERSTAND.
As a final note please Think Again at what you wrote. In my country we have a sentence, "think twice and cut only onve"

PostScriptum: I am sorry for my bad english, and for my very long post - nobody is perfect

 

[NewDude]

In my country we have a sentence, "think twice and cut only onve"

In English it's "measure twice, cut once." Something a carpenter would say. Nice that it translates so well.

 

[jo der bohrer]

That doesn't match my experience at all.

I installed the most recent ISO on new hardware, had the installer configure the drives as ZFS, and I've had no issues. The first machine I played with for a week was using a ZFS mirror that I added a SLOG to, then once I decided I was going to use Proxmox I subscribed. The second and third machines used ZFS via the installer as well (one as a mirror, the other as RAIDZ1); in each case I bought a subscription, updated, and joined the cluster.

The only issues I've run into are things like time differences because I misconfigured my firewall rules.

In my case I'm using new (though really cheap) Dell hardware. What are you installing onto?

So we experience different stories. I used 2 pc HP Z420 with 32g EEC and XEON CPU and the ZFS option in the install routine leads every time into the underwoods of linux. And I tested it many times in different configurations of the 4 disks, that each server has. Maybe its a HP z420 specific problem.

 

[jo der bohrer]

Please re-test, e..g. your statements regarding a wrong sources.list setup is just wrong. There is also a /etc/apt/sources.d/ directory, maybe you missed this one?

You acknowledge this line in the Proxmox Package Repositories Recommandations

deb http://download.proxmox.com/debian/pve stretch pve-no-subscription

This is the most important line of /etc/apt/sources.list. And you have to move this into that file by hand. It is not shipped with the current ISO Installer! If you dont do that, many ugly error occur out of the blue.

 

[jo der bohrer]

Hi,

I see many times in the linux world (and I use linux from the 20 century ) that a beginner can not do something .... and linux or whatever is bad. From my limited experience I see that most of the time a beginner user (in any IT field) try to do something that at first look is very smple(from his point of view) and he think that no knowledge is necessary.
Now at your problem regarding proxmox. Proxmox is a huge software, and you must take your time to UNDERSTAND how is working, what are the software components, what technology they use, and so on.
In my own case (with let say some basic understanding about linux, zfs, security, routing and many others - not a beginner ) I start to test for myself proxmox for many months. I test many failure situation and I try to slove them, and I let say LOST my time to understand how I can fix them. I wrote for myself what I do and was successfully and what was a bad path to solve. In case of bad results I have try to find answers on Internet - and many hors of talking with some experts friends in various IT / linux fields. And after let say 6 month of usage I consider myself that I am scratching only the surface.
The next step was to read almost dayly any proxmox forum post, and I learn many things.
I do not to say that I am the best smart guy from Romania, or the smartest Proxmox free user (I do not pay anything for my Proxmox usage, but I can say Thank You for the Proxmox team for this) but for sure I can say that Proxmox is a very good tool for any user who take some time to learn and to UNDERSTAND.
As a final note please Think Again at what you wrote. In my country we have a sentence, "think twice and cut only onve"

PostScriptum: I am sorry for my bad english, and for my very long post - nobody is perfect

I agree 100% with what you say. A complex thing like Proxmox with its Linux roots cannot be learned in 5 weeks. As you put it " scratching the surface".
And your english: Superb, could not tell it from a native speaker.

 

[dcsapak]

So we experience different stories. I used 2 pc HP Z420 with 32g EEC and XEON CPU and the ZFS option in the install routine leads every time into the underwoods of linux. And I tested it many times in different configurations of the 4 disks, that each server has. Maybe its a HP z420 specific problem.

what exactly were the problems? without more details, nobody can see what maybe went wrong and what we could improve

deb http://download.proxmox.com/debian/pve stretch pve-no-subscription

This is the most important line of /etc/apt/sources.list. And you have to move this into that file by hand. It is not shipped with the current ISO Installer! If you dont do that, many ugly error occur out of the blue.

this line is absolutely not necessary if you buy a subscription, which is highly recommended by us if you have important production servers

as i see it, all of your points (except the unknown zfs thing) are described and documented in our reference documentation (https://pve.proxmox.com/pve-docs/)

if you have suggestions on how to improve our documentation or tools (be it installer or otherwise) you could use our bugtracker, mailing list, etc.

edit: typo

 

[jo der bohrer]

what exactly were the problems? without more details, nobody can see what maybe went wrong and what we could improve


this line is absolutely not necessary if you buy a subscription, which is highly recommended by us if you have important production servers

as i see it, all of your points (except the unknown zfs thing) are described and documented in our reference documentation (https://pve.proxmox.com/pve-docs/)

if you have suggestions on how to improve our documentation or tools (be it installer or otherwise) you could use our bugtracker, mailing list, etc.

edit: typo

Sorry to be impertinent,

for non subscriptionists these line is a must. Maybe you can agree on that. And for your interest: I bought a subscription, but only 3 days ago, so most of the time I trained with non subscription.
It is to early for me to suggest improvements of the docu. Of course I do have some points, but I think the community here is best for testing the validity of them. Thank you

 

[fabian]

So we experience different stories. I used 2 pc HP Z420 with 32g EEC and XEON CPU and the ZFS option in the install routine leads every time into the underwoods of linux. And I tested it many times in different configurations of the 4 disks, that each server has. Maybe its a HP z420 specific problem.

it most likely is some kind of issue with the disk controller and/or bios settings. if you can provide the exact error messages we might be able to trouble-shoot it further. some things to consider:

• ZFS in PVE only supports non-UEFI/legacy boot (out of the box)
• Grub needs to see at least a sufficient number of disks to read the rpool data, so all those disks need to be presented as boot devices by the BIOS (some controllers/BIOS implementations don't support this!)
• (fake)raid controllers can cause all kinds of issues, disable as much of that functionality as possible
• some controllers lie about disk sizes, which can lead to spurious problems booting as well

 

[WhiteStarEOF]

So we experience different stories. I used 2 pc HP Z420 with 32g EEC and XEON CPU and the ZFS option in the install routine leads every time into the underwoods of linux. And I tested it many times in different configurations of the 4 disks, that each server has. Maybe its a HP z420 specific problem.

This sounds similar to a sneaky issue I had with Proxmox and ZFS that I hadn't run into until I setup a test server that was built up, torn down, and built up several times over. The first time, everything worked as expected. But every subsequent rebuild, I would install Proxmox to a ZFS mirror, and upon reboot, Grub wouldn't be able to load the OS properly.

To the best of my understanding, the Proxmox installer doesn't have a process to destroy an existing zpool. So when I told Proxmox to install on a pair of disks that already had an install of Proxmox on it, ZFS dutifully protected the original data. But Grub isn't included in that protection, and so a new Grub installation is expecting to boot a new Proxmox installation, but ZFS protected the old Proxmox installation. The new Proxmox installation doesn't actually exist.

I suppose I could test this by creating a zpool that wasn't labeled rpool, and try to install Proxmox over that. It should run into the same problem, and if it's correct, I would be left with the original non-rpool named zpool.

Anyways, in order to fix this, I boot to a Linux LiveCD like Ubuntu, install the zfsutils, and zfs destroy rpool before performing a new installation of Proxmox.

 

[jim.bond.9862]

well without proper info on what hardware was used and what was the targeted setup was (I supect a 2 node cluster of some sort since OP used 2 servers), it is difficult to troubleshoot the issues, even with details like that.

but I have been playing with Proxmox setups for the last 2 years. "Playing" is a key word here as I was doing installs and configs on real hardware and on Hyper-V setup. I was doing nested Virtualization etc.
yes not every try went smooth and easy, but 99% were successful.
I had ,FINALLY, build out my home setup last month.

an oldish SuperMicro server with an outdated MB.
the VE5.1 install went very easy and trouble free.
2xSSD in ZFS mirror for system. like a charm. select the drives and server was up and running in 15 min flat.

Granted, ZFS support is kind of lucking in GUI. but that is kind of the same issue in most other hyper-visors I have tried.
but even so, the only real problem I encounter is that installer uses the old disk designations like "/dev/sdX" and I prefer UUID.
a helpful post here walked me through changing the pool during boot.

another issue is related to radeon video driver "VGA1 missing error" also found post to fix this. took a while to find right help but solved it.

everything else worked just fine.
got additional storage added via GUI.
setup several VMs no problems.
setup LXI container from Turnkey template, even mount-bind 2 ZFS data pools into it with no issue.

all I can say is, do your research first. read the info on website, and even post on forums prior to go live.
plan on some time for unexpected issues. do several test runs to see if all is working.

 

[jim.bond.9862]

Anyways, in order to fix this, I boot to a Linux LiveCD like Ubuntu, install the zfsutils, and zfs destroy rpool before performing a new installation of Proxmox.

if you going to destroy "rpool" anyway and the drives only contain that one pool. why not simply do gdisk or is it parted? and drop and recreate the partition table?
it is a build in tool, no need to install anything and it is fast.

 

[WhiteStarEOF]

if you going to destroy "rpool" anyway and the drives only contain that one pool. why not simply do gdisk or is it parted? and drop and recreate the partition table?
it is a build in tool, no need to install anything and it is fast.

I've had trouble just reformatting ZFS'd disks, probably because ZFS is still protecting the disks. I've found it's just easier to install the utils, destroy the zpool the way ZFS wants you to, and then boot into the Proxmox installer.

 

[jim.bond.9862]

I've had trouble just reformatting ZFS'd disks, probably because ZFS is still protecting the disks. I've found it's just easier to install the utils, destroy the zpool the way ZFS wants you to, and then boot into the Proxmox installer.

yes reformatting the disk is a problem. I meant simply that if you boot from LiveCD or recovery CD, this way the pools are not mounted/imported gdisk should be able to destroy and recreate the partition table with no issues, and you do not need to install any additional software. but your way works too

 

[GadgetPig]

@jo der bohrer
That you decided to keep working with ProxMox, learn how to setup and further troubleshoot and seek help, shows you are in fact growing your knowledge, and creating your own steps and best practices. Keep up the great work! ProxMox is VERY powerful (and flexible) software, and yes it does take time to learn. Of course some hardware/configuration combos may cause issues with ProxMox, but that is true for other hypervisors/software as well.

Regarding the Z420, have you tried putting BIOS storage mode in AHCI instead of RAID/AHCI? Otherwise you can use RAID/AHCI mode and do raid1 with ProxMox for the main system disk, and add a new PCI-E/HBA 4 disk controller to use for the ZFS data pool.

Out of curiosity, the NAS device that failed backups for you, was it using freenas or was it a third party raid box?

thnx

 

[jo der bohrer]

----> Important Notice <-----
In March 2018 I wrote about the many obstacles installing Proxmox on a HP Z420, see my very first input in this thread.
Now - 7 months and a lot of hardware shuffling later - I can see it clear.

It was entirely my fault. I tried to install Proxmox on a unused but 4 year old workstation. And now we know: Don't do that.
The reason: HP build and delivered that workstation with a BIOS version 1.0 to a dealer, where it only gathered dust. Nobody cared that this BIOS was full of faults causing kernel panics not only with Linux but also Windows.
After installing Proxmox on a Dell precision T3600 (same CPU, same disks) without any issues at all, I realised what a different BIOS is worth. After a BIOS update on the Z420 everything's roses.

 

[dcsapak]

----> Important Notice <-----
In March 2018 I wrote about the many obstacles installing Proxmox on a HP Z420, see my very first input in this thread.
Now - 7 months and a lot of hardware shuffling later - I can see it clear.

It was entirely my fault. I tried to install Proxmox on a unused but 4 year old workstation. And now we know: Don't do that.
The reason: HP build and delivered that workstation with a BIOS version 1.0 to a dealer, where it only gathered dust. Nobody cared that this BIOS was full of faults causing kernel panics not only with Linux but also Windows.
After installing Proxmox on a Dell precision T3600 (same CPU, same disks) without any issues at all, I realised what a different BIOS is worth. After a BIOS update on the Z420 everything's roses.

great that it worked out in the end