Proxmox on SD card

You could wrap the tar -czf ... in a systemd unit and add a systemd drop-in configuration for whatever service that needs /var/lib/pve-cluster (I guess it's just pve-cluster.service) containing something along the lines of
Code: 
[Unit]
Requires=the-tar-backup.service
After=the-tar-backup.service
 
I bought a few Kingston Industrial SD cards to use as my boot drives for machines which don't support booting from NVMe (Dell R710):
https://www.kingston.com/datasheets/sdit_en.pdf

They're run in pSLC mode which greatly increases endurance, "up to" 1920 TBW which applies to the largest capacity (64 GB). Since TBW is proportionate to capacity, my 16 GB cards should have an endurance of 480 TBW... more than enough for a disk which will almost never be written to (only used to boot the box and pass control to the ZFS NVMe mirror).

Be warned with buying these on Amazon etc; since AFAIK Amazon stock is commingled with third-party FBA stock, you might wind up with counterfeit cards even if you choose the item which is listed as being sold by Amazon themselves. To avoid this, I bought them from Mouser, who guarantees in writing on the packing slip that they're either purchased directly from either the manufacturer or their authorised supply chain.
 
Last edited:
  • Like
Reactions: Dunuin
no-usernames-left said:
I bought a few Kingston Industrial SD cards to use as my boot drives for machines which don't support booting from NVMe (Dell R710):
Click to expand...
I used an old usb stick I had lying around for the boot device on one machine with the same problem as you described. The boot device does not have problems with too many writes, just the ordinary PVE root.
 
  • Like
Reactions: no-usernames-left
Digging this back up.

A standard ubuntu installation allows one to use a Dell IDSDM(basically an internal SD card mirror) exclusively for the boot partition (/boot), with a seperate file system (on NVME or whatever) mounted at / .

Can this not be done with proxmox? Can you not put the boot partition on the SD Card with the root file system placed elsewhere, like say on a zfs pool?
 
  • Like
Reactions: no-usernames-left
crashtest said:
Digging this back up.

A standard ubuntu installation allows one to use a Dell IDSDM(basically an internal SD card mirror) exclusively for the boot partition (/boot), with a seperate file system (on NVME or whatever) mounted at / .

Can this not be done with proxmox? Can you not put the boot partition on the SD Card with the root file system placed elsewhere, like say on a zfs pool?
Click to expand...
you can install proxmox on a internal nvme m2 drive (enterprise/dc grade, with a superpacitor).
If you have DEll, look a boss
https://www.dell.com/support/manuals/en-us/poweredge-r7425/boss_s1_ug_publication

proxmox is continuously writing small write (for stats, graphs, cluster exchange,....).

if you use sdcard,you are going to burn them after 1 or 2 months.

(even vmware don't support to use sdcard anymore for install)
 
crashtest said:
Digging this back up.

A standard ubuntu installation allows one to use a Dell IDSDM(basically an internal SD card mirror) exclusively for the boot partition (/boot), with a seperate file system (on NVME or whatever) mounted at / .

Can this not be done with proxmox? Can you not put the boot partition on the SD Card with the root file system placed elsewhere, like say on a zfs pool?
Click to expand...
This is absolutely possible because I am doing it today on an R710 which cannot boot from NVMe.

I bought a high-quality industrial pSLC SD card, but you can use a USB flash drive if you want since it is never written to other than for kernel updates.

I set up EFI and boot partitions and initialized it with proxmox-boot-tool, and the R710 boots to that internal SD card which then passes execution to the root partition which is a ZFS mirror on a pair of NVMe SSDs on a PCI card equipped with a PCIe switch (because no bifurcation either).

When Proxmox has a new kernel to install, it updates the boot partitions on the SSDs (which are never used) and also the SD card (which is).
 
Last edited:
spirit said:
you can install proxmox on a internal nvme m2 drive (enterprise/dc grade, with a superpacitor).
If you have DEll, look a boss
https://www.dell.com/support/manuals/en-us/poweredge-r7425/boss_s1_ug_publication

proxmox is continuously writing small write (for stats, graphs, cluster exchange,....).

if you use sdcard,you are going to burn them after 1 or 2 months.

(even vmware don't support to use sdcard anymore for install)
Click to expand...
Re-read the post to which you replied: they want to use it for /boot only. Proxmox never writes there except to install a kernel update.
 
no-usernames-left said:
This is absolutely possible because I am doing it today on an R710 which cannot boot from NVMe.

I bought a high-quality industrial pSLC SD card, but you can use a USB flash drive if you want since it is never written to other than for kernel updates.

I set up EFI and boot partitions and initialized it with proxmox-boot-tool, and the R710 boots to that internal SD card which then passes execution to the root partition which is a ZFS mirror on a pair of NVMe SSDs on a PCI card equipped with a PCIe switch (because no bifurcation either).

When Proxmox has a new kernel to install, it updates the boot partitions on the SSDs (which are never used) and also the SD card (which is).
Click to expand...
It seemed weird that so many were saying it totally shouldn’t be done without ever acknowledging that only the boot partition goes on the SD card.

So it seems it takes a few extra steps to make it happen though? Is the documentation pretty straight forward for it or, if not, are you aware of a decent guide?
 
crashtest said:
It seemed weird that so many were saying it totally shouldn’t be done without ever acknowledging that only the boot partition goes on the SD card.
Click to expand...
It seems weird to me to do this in the first place. Why split your system on 4 devices instead of 2 without any upside. I suspect that therefore a lot of people misunderstand this setup. It makes absolutely no sense to me.
 
crashtest said:
It seemed weird that so many were saying it totally shouldn’t be done without ever acknowledging that only the boot partition goes on the SD card.
Click to expand...
Rule number one is that users don't read. We've all been guilty of it from time to time.

crashtest said:
So it seems it takes a few extra steps to make it happen though? Is the documentation pretty straight forward for it or, if not, are you aware of a decent guide?
Click to expand...
I wrote it up here.
 
Last edited:
  • Like
Reactions: crashtest
LnxBil said:
It seems weird to me to do this in the first place. Why split your system on 4 devices instead of 2 without any upside. I suspect that therefore a lot of people misunderstand this setup. It makes absolutely no sense to me.
Click to expand...
The box cannot boot from NVMe because the BIOS is too old to support it. (It can apparently be modded and reflashed but fuck that noise.) Therefore, some other intermediate boot device is required.

The "system" is not split, there is simply another copy of the boot partition, which is managed entirely automatically with proxmox-boot-tool.
 
Last edited:
no-usernames-left said:
The box cannot boot from NVMe because the BIOS is too old to support it. (It can apparently be modded and reflashed but fuck that noise.) Therefore, some other intermediate boot device is required.
Click to expand...
Oh yes, that's the usecase I missed ... and yes, we often don't read (correctly) or just don't reread the whole thread.
 
no-usernames-left said:
Rule number one is that users don't read. We've all been guilty of it from time to time.


I wrote it up here.
Click to expand...
Sweet thanks!

As it turns out, I’m in the research phase of implementing proxmox in my home lab…. On a dell T430 with the IDSDM and a couple NVMEs thrown in on pcie add-in cards.

Thanks everyone for the input
 
no-usernames-left said:
This is absolutely possible because I am doing it today on an R710 which cannot boot from NVMe.

I bought a high-quality industrial pSLC SD card, but you can use a USB flash drive if you want since it is never written to other than for kernel updates.

I set up EFI and boot partitions and initialized it with proxmox-boot-tool, and the R710 boots to that internal SD card which then passes execution to the root partition which is a ZFS mirror on a pair of NVMe SSDs on a PCI card equipped with a PCIe switch (because no bifurcation either).

When Proxmox has a new kernel to install, it updates the boot partitions on the SSDs (which are never used) and also the SD card (which is).
Click to expand...
@no-usernames-left Thanks for this and the guide in the other thread. Gonna try this.

Mind if I ask why you went with a switch instead of 2 separate PCI cards?

Also, is 16GB cutting it close if PVE uses 12-14GB (from another thread)?
 
Last edited:
surveyor said:
@no-usernames-left Thanks for this and the guide in the other thread. Gonna try this.

Mind if I ask why you went with a switch instead of 2 separate PCI cards?

Also, is 16GB cutting it close if PVE uses 12-14GB (from another thread)?
Click to expand...
I ended up using a PCI switch so I could fit 4 NVMEs on a 16x slot(board doesn’t support bifrucation). All my other slots are taken up by either SAS controller cards or the vid card and the one separate NVME on a pci card that holds the PVE install.
 
crashtest said:
I ended up using a PCI switch so I could fit 4 NVMEs on a 16x slot(board doesn’t support bifrucation). All my other slots are taken up by either SAS controller cards or the vid card and the one separate NVME on a pci card that holds the PVE install.
Click to expand...
Which PCIe switch do you use?
 
Roger Simpson said:
Which PCIe switch do you use?
Click to expand...
I'm not the person you were asking, but they were asking me, so I'm replying to you.

I went with the Startech PEX8M2E2, which can be found on Amazon. I'm quite happy with it, but make sure you have decent airflow as the heatsink on the PCIe switch chip can get pretty toasty. (Your SSDs will appreciate the airflow as well.)

The two SSDs on that card are configured as a ZFS mirror which Proxmox runs from (root-on-ZFS).
 
Last edited:
  • Like
Reactions: Roger Simpson
crashtest said:
the one separate NVME on a pci card that holds the PVE install.
Click to expand...
Why have it in its own slot? According to your post, all the SSDs in your box will be seen as separate devices, so there's no benefit to having the SSD on which Proxmox is running installed in its own slot.
 
You must log in or register to reply here.
Top Bottom