Proxmox on SD card

You could wrap the tar -czf ... in a systemd unit and add a systemd drop-in configuration for whatever service that needs /var/lib/pve-cluster (I guess it's just pve-cluster.service) containing something along the lines of
Code:
[Unit]
Requires=the-tar-backup.service
After=the-tar-backup.service
 
I bought a few Kingston Industrial SD cards to use as my boot drives for machines which don't support booting from NVMe (Dell R710):
https://www.kingston.com/datasheets/sdit_en.pdf

They're run in pSLC mode which greatly increases endurance, "up to" 1920 TBW which applies to the largest capacity (64 GB). Since TBW is proportionate to capacity, my 16 GB cards should have an endurance of 480 TBW... more than enough for a disk which will almost never be written to (only used to boot the box and pass control to the ZFS NVMe mirror).

Be warned with buying these on Amazon etc; since AFAIK Amazon stock is commingled with third-party FBA stock, you might wind up with counterfeit cards even if you choose the item which is listed as being sold by Amazon themselves. To avoid this, I bought them from Mouser, who guarantees in writing on the packing slip that they're either purchased directly from either the manufacturer or their authorised supply chain.
 
Last edited:
  • Like
Reactions: Dunuin
I bought a few Kingston Industrial SD cards to use as my boot drives for machines which don't support booting from NVMe (Dell R710):
I used an old usb stick I had lying around for the boot device on one machine with the same problem as you described. The boot device does not have problems with too many writes, just the ordinary PVE root.
 
  • Like
Reactions: no-usernames-left
Digging this back up.

A standard ubuntu installation allows one to use a Dell IDSDM(basically an internal SD card mirror) exclusively for the boot partition (/boot), with a seperate file system (on NVME or whatever) mounted at / .

Can this not be done with proxmox? Can you not put the boot partition on the SD Card with the root file system placed elsewhere, like say on a zfs pool?
 
  • Like
Reactions: no-usernames-left
Digging this back up.

A standard ubuntu installation allows one to use a Dell IDSDM(basically an internal SD card mirror) exclusively for the boot partition (/boot), with a seperate file system (on NVME or whatever) mounted at / .

Can this not be done with proxmox? Can you not put the boot partition on the SD Card with the root file system placed elsewhere, like say on a zfs pool?
you can install proxmox on a internal nvme m2 drive (enterprise/dc grade, with a superpacitor).
If you have DEll, look a boss
https://www.dell.com/support/manuals/en-us/poweredge-r7425/boss_s1_ug_publication

proxmox is continuously writing small write (for stats, graphs, cluster exchange,....).

if you use sdcard,you are going to burn them after 1 or 2 months.

(even vmware don't support to use sdcard anymore for install)
 
Digging this back up.

A standard ubuntu installation allows one to use a Dell IDSDM(basically an internal SD card mirror) exclusively for the boot partition (/boot), with a seperate file system (on NVME or whatever) mounted at / .

Can this not be done with proxmox? Can you not put the boot partition on the SD Card with the root file system placed elsewhere, like say on a zfs pool?
This is absolutely possible because I am doing it today on an R710 which cannot boot from NVMe.

I bought a high-quality industrial pSLC SD card, but you can use a USB flash drive if you want since it is never written to other than for kernel updates.

I set up EFI and boot partitions and initialized it with proxmox-boot-tool, and the R710 boots to that internal SD card which then passes execution to the root partition which is a ZFS mirror on a pair of NVMe SSDs on a PCI card equipped with a PCIe switch (because no bifurcation either).

When Proxmox has a new kernel to install, it updates the boot partitions on the SSDs (which are never used) and also the SD card (which is).
 
Last edited:
you can install proxmox on a internal nvme m2 drive (enterprise/dc grade, with a superpacitor).
If you have DEll, look a boss
https://www.dell.com/support/manuals/en-us/poweredge-r7425/boss_s1_ug_publication

proxmox is continuously writing small write (for stats, graphs, cluster exchange,....).

if you use sdcard,you are going to burn them after 1 or 2 months.

(even vmware don't support to use sdcard anymore for install)
Re-read the post to which you replied: they want to use it for /boot only. Proxmox never writes there except to install a kernel update.
 
This is absolutely possible because I am doing it today on an R710 which cannot boot from NVMe.

I bought a high-quality industrial pSLC SD card, but you can use a USB flash drive if you want since it is never written to other than for kernel updates.

I set up EFI and boot partitions and initialized it with proxmox-boot-tool, and the R710 boots to that internal SD card which then passes execution to the root partition which is a ZFS mirror on a pair of NVMe SSDs on a PCI card equipped with a PCIe switch (because no bifurcation either).

When Proxmox has a new kernel to install, it updates the boot partitions on the SSDs (which are never used) and also the SD card (which is).
It seemed weird that so many were saying it totally shouldn’t be done without ever acknowledging that only the boot partition goes on the SD card.

So it seems it takes a few extra steps to make it happen though? Is the documentation pretty straight forward for it or, if not, are you aware of a decent guide?
 
It seemed weird that so many were saying it totally shouldn’t be done without ever acknowledging that only the boot partition goes on the SD card.
It seems weird to me to do this in the first place. Why split your system on 4 devices instead of 2 without any upside. I suspect that therefore a lot of people misunderstand this setup. It makes absolutely no sense to me.
 
It seemed weird that so many were saying it totally shouldn’t be done without ever acknowledging that only the boot partition goes on the SD card.
Rule number one is that users don't read. We've all been guilty of it from time to time.

So it seems it takes a few extra steps to make it happen though? Is the documentation pretty straight forward for it or, if not, are you aware of a decent guide?
I wrote it up here.
 
Last edited:
  • Like
Reactions: crashtest
It seems weird to me to do this in the first place. Why split your system on 4 devices instead of 2 without any upside. I suspect that therefore a lot of people misunderstand this setup. It makes absolutely no sense to me.
The box cannot boot from NVMe because the BIOS is too old to support it. (It can apparently be modded and reflashed but fuck that noise.) Therefore, some other intermediate boot device is required.

The "system" is not split, there is simply another copy of the boot partition, which is managed entirely automatically with proxmox-boot-tool.
 
Last edited:
The box cannot boot from NVMe because the BIOS is too old to support it. (It can apparently be modded and reflashed but fuck that noise.) Therefore, some other intermediate boot device is required.
Oh yes, that's the usecase I missed ... and yes, we often don't read (correctly) or just don't reread the whole thread.
 
Rule number one is that users don't read. We've all been guilty of it from time to time.


I wrote it up here.
Sweet thanks!

As it turns out, I’m in the research phase of implementing proxmox in my home lab…. On a dell T430 with the IDSDM and a couple NVMEs thrown in on pcie add-in cards.

Thanks everyone for the input
 
This is absolutely possible because I am doing it today on an R710 which cannot boot from NVMe.

I bought a high-quality industrial pSLC SD card, but you can use a USB flash drive if you want since it is never written to other than for kernel updates.

I set up EFI and boot partitions and initialized it with proxmox-boot-tool, and the R710 boots to that internal SD card which then passes execution to the root partition which is a ZFS mirror on a pair of NVMe SSDs on a PCI card equipped with a PCIe switch (because no bifurcation either).

When Proxmox has a new kernel to install, it updates the boot partitions on the SSDs (which are never used) and also the SD card (which is).
@no-usernames-left Thanks for this and the guide in the other thread. Gonna try this.

Mind if I ask why you went with a switch instead of 2 separate PCI cards?

Also, is 16GB cutting it close if PVE uses 12-14GB (from another thread)?
 
Last edited:
@no-usernames-left Thanks for this and the guide in the other thread. Gonna try this.

Mind if I ask why you went with a switch instead of 2 separate PCI cards?

Also, is 16GB cutting it close if PVE uses 12-14GB (from another thread)?
I ended up using a PCI switch so I could fit 4 NVMEs on a 16x slot(board doesn’t support bifrucation). All my other slots are taken up by either SAS controller cards or the vid card and the one separate NVME on a pci card that holds the PVE install.
 
I ended up using a PCI switch so I could fit 4 NVMEs on a 16x slot(board doesn’t support bifrucation). All my other slots are taken up by either SAS controller cards or the vid card and the one separate NVME on a pci card that holds the PVE install.
Which PCIe switch do you use?
 
Which PCIe switch do you use?
I'm not the person you were asking, but they were asking me, so I'm replying to you.

I went with the Startech PEX8M2E2, which can be found on Amazon. I'm quite happy with it, but make sure you have decent airflow as the heatsink on the PCIe switch chip can get pretty toasty. (Your SSDs will appreciate the airflow as well.)

The two SSDs on that card are configured as a ZFS mirror which Proxmox runs from (root-on-ZFS).
 
Last edited:
  • Like
Reactions: Roger Simpson
the one separate NVME on a pci card that holds the PVE install.
Why have it in its own slot? According to your post, all the SSDs in your box will be seen as separate devices, so there's no benefit to having the SSD on which Proxmox is running installed in its own slot.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!