Host backup - zfs snapshot

L

listhor

Member
Nov 14, 2023
53
3
13
I have PVE host with 2 nvme drives, both are formatted in zfs and one is bootable - so extra bios and efi partitions. 2nd, non bootable drive is used as PBS storage.
I have also an offline backup copies of VMs, including PBS lxc (stored on main, bootable drive) done by PBS synchronisation jobs from other host.
I would like to make a backup of host’s rpool by utilisation of zfs send/receive commands - as rescue option for future failed drive (currently it has wear out at 114%).
The main issue is to restore bios and efi partitions; is there any kind of recommended procedure to follow?
Or maybe I should install Proxmox from scratch and then overwrite it with zfs receive? Or maybe copy over only /etc folder?
 
I couldn't find one, matching my needs tutorial so, here it is my own, in markdown. It's for my own reference and maybe will be useful for somebody else.
My use case is rpool/data contains all VMs with exception of PBS lxc which is stored on other pool (second drive): bpool together with VMs' backups. Once proxmox boot partition is restored and proxmox itself is running - I start PBS and restore all my VMs...

Markdown (GitHub flavored): 
# Proxmox ZFS Root Partition — Backup & Restore Guide

---

## Placeholders

| Placeholder | Description | Example |
|---|---|---|
| `<SOURCE_DISK>` | Drive being backed up | `nvme0n1` |
| `<TARGET_DISK>` | New/replacement drive for restore | `nvme1n1` |
| `<BACKUP_HOST>` | Remote backup server hostname | `backup.example.com` |
| `<BACKUP_USER>` | SSH user on backup server | `backup-user` |
| `<SSH_PORT>` | SSH port on backup server | `22` |
| `<BOOT_BACKUP_PATH>` | Remote path for boot partition backups | `/mnt/backups/pve/BOOT` |
| `<REMOTE_ZFS_DATASET>` | ZFS dataset on backup server containing rpool snapshot | `pool/pve/rpool-root` |
| `<SNAPSHOT_NAME>` | Name of ZFS snapshot to restore | `2024-01-01_12.00.00--3d` |
| `<SUDO_USER>` | Non-root user allowed to run ZFS commands | `zfs-backup-user` |
| `<HOST_ID>` | Output of `hostid` on the original host | `4f42e3cf` |

---

## Part 1: Backup

> Run on the **live Proxmox host**.

### 1. Install required tools

```bash
apt install -y gdisk pv zfsnap
mkdir /root/host_drive_backup
```

---

### 2. Back up partition table

Saves the full GPT layout to a binary file (a few KB). Keep this updated after any partition changes.

```bash
sgdisk --backup=/root/host_drive_backup/<SOURCE_DISK>-partition-table.bin /dev/<SOURCE_DISK>
```

| Option | Description |
|---|---|
| `--backup=<file>` | Write partition table backup to the specified file |

---

### 3. Back up BIOS Boot partition (`p1`, ~1 MB)

```bash
dd if=/dev/<SOURCE_DISK>p1 bs=1M status=progress \
   | gzip > /root/host_drive_backup/<SOURCE_DISK>p1-bios-boot.img.gz
```

| Option | Description |
|---|---|
| `if=` | Input file (source partition) |
| `bs=1M` | Block size — 1 MiB; suitable for small partitions |
| `status=progress` | Show transfer progress |
| pipe to `gzip` | Compress output to save space |

---

### 4. Back up ESP/EFI partition (`p2`, ~1 GB — bootloader, kernel, initrd)

```bash
dd if=/dev/<SOURCE_DISK>p2 bs=4M status=progress \
   | gzip > /root/host_drive_backup/<SOURCE_DISK>p2-esp.img.gz
```

Same options as above; larger block size (`4M`) is more efficient for bigger partitions.

---

### 5. Transfer boot backups to remote server

```bash
scp -l 409600 -P <SSH_PORT> \
    /root/host_drive_backup/<SOURCE_DISK>-partition-table.bin \
    /root/host_drive_backup/<SOURCE_DISK>p1-bios-boot.img.gz \
    /root/host_drive_backup/<SOURCE_DISK>p2-esp.img.gz \
    <BACKUP_USER>@<BACKUP_HOST>:<BOOT_BACKUP_PATH>/
```

| Option | Description |
|---|---|
| `-l 409600` | Bandwidth limit in Kbit/s (~400 Mbit/s) |
| `-P <SSH_PORT>` | SSH port |

---

### 6. Configure sudoers for non-root ZFS operations

Allows `<SUDO_USER>` to run ZFS commands without a password — required for automated snapshot/send tasks.

Create `/etc/sudoers.d/<SUDO_USER>-zfs`:

```
# /etc/sudoers.d/<SUDO_USER>-zfs

# Recursive snapshot of rpool/ROOT
<SUDO_USER> ALL=(ALL) NOPASSWD: /usr/sbin/zfs snapshot -r rpool/ROOT@*

# Full ZFS send
<SUDO_USER> ALL=(ALL) NOPASSWD: /usr/sbin/zfs send *

# List snapshots (used to find the previous snapshot in scripts)
<SUDO_USER> ALL=(ALL) NOPASSWD: /usr/sbin/zfs list *

# Read dataset properties
<SUDO_USER> ALL=(ALL) NOPASSWD: /usr/sbin/zfs get *

# Destroy old snapshots (pruning)
<SUDO_USER> ALL=(ALL) NOPASSWD: /usr/sbin/zfs destroy -r rpool/ROOT@*

# zfSnap automation
<SUDO_USER> ALL=(ALL) NOPASSWD: /usr/sbin/zfSnap *
```

> **Note:** For TrueNAS `zfs receive` setup, see: https://wiki.familybrown.org/pve-replication 
> When configuring the replication task on TrueNAS, set override properties `mountpoint=none` and `canmount=off`
> to prevent auto-mounting of received datasets.

---

## Part 2: Restore

> Run from a **rescue / live Linux environment**.

### 1. Install required tools

```bash
apt update && apt install -y zfsutils-linux gdisk pv
mkdir -p /root/host_drive_backup
```

---

### 2. Identify drives

```bash
lsblk
```

Confirm `<SOURCE_DISK>` and `<TARGET_DISK>` device names before proceeding.

---

### 3. Download boot backups from remote server

```bash
scp -P <SSH_PORT> \
    <BACKUP_USER>@<BACKUP_HOST>:<BOOT_BACKUP_PATH>/<SOURCE_DISK>-partition-table.bin \
    <BACKUP_USER>@<BACKUP_HOST>:<BOOT_BACKUP_PATH>/<SOURCE_DISK>p1-bios-boot.img.gz \
    <BACKUP_USER>@<BACKUP_HOST>:<BOOT_BACKUP_PATH>/<SOURCE_DISK>p2-esp.img.gz \
    /root/host_drive_backup/
```

---

### 4. Restore partition table to new drive

```bash
sgdisk --load-backup=/root/host_drive_backup/<SOURCE_DISK>-partition-table.bin /dev/<TARGET_DISK>
```

| Option | Description |
|---|---|
| `--load-backup=<file>` | Restore GPT layout from a previously saved backup file |

---

### 5. Assign new partition GUIDs

Prevents GUID conflicts if the original drive is still present in the system.

```bash
sgdisk -G /dev/<TARGET_DISK>
```

| Option | Description |
|---|---|
| `-G` / `--randomize-guids` | Generate new random GUIDs for the disk and all partitions |

---

### 6. Verify partition layout

```bash
lsblk /dev/<TARGET_DISK>
```

Confirm that `p1`, `p2`, and `p3` are present and correctly sized.

---

### 7. Restore BIOS Boot partition

```bash
gunzip -c /root/host_drive_backup/<SOURCE_DISK>p1-bios-boot.img.gz \
  | dd of=/dev/<TARGET_DISK>p1 bs=1M status=progress
```

---

### 8. Restore EFI partition

```bash
gunzip -c /root/host_drive_backup/<SOURCE_DISK>p2-esp.img.gz \
  | dd of=/dev/<TARGET_DISK>p2 bs=4M status=progress
```

> **Alternative — low RAM in live environment:** Stream directly from the remote server to avoid writing to local storage:
> ```bash
> ssh <BACKUP_USER>@<BACKUP_HOST> \
>   "cat <BOOT_BACKUP_PATH>/<SOURCE_DISK>p2-esp.img.gz" \
>   | gunzip -c \
>   | dd of=/dev/<TARGET_DISK>p2 bs=4M status=progress
> ```

---

### 9. Create rpool on the new drive

```bash
zpool create -f \
  -o ashift=12 \
  -o autotrim=on \
  -O compression=on \
  rpool /dev/<TARGET_DISK>p3
```

| Option | Description |
|---|---|
| `-f` | Force creation even if the drive contains existing data or signatures |
| `-o ashift=12` | Pool-level: 4K sector alignment — optimal for NVMe/SSD drives |
| `-o autotrim=on` | Pool-level: enable automatic TRIM for SSDs |
| `-O compression=on` | Dataset-level: enable LZ4 compression on all datasets |

---

### 10. Receive ZFS ROOT dataset from remote server

```bash
ssh -p <SSH_PORT> <BACKUP_USER>@<BACKUP_HOST> \
  "sudo /sbin/zfs send -bRv <REMOTE_ZFS_DATASET>@<SNAPSHOT_NAME>" \
  | zfs recv -Fv rpool/ROOT
```

| Option | Description |
|---|---|
| **`zfs send`** | |
| `-b` | Send backup stream — preserve received properties from the original pool |
| `-R` | Replicate recursively — includes all child datasets and their snapshots |
| `-v` | Verbose output |
| **`zfs recv`** | |
| `-F` | Force rollback/destroy of conflicting local data to match the incoming stream |
| `-v` | Verbose output |

---

### 11. Create data dataset

```bash
zfs create -o mountpoint=/rpool/data rpool/data
```

---

### 12. Check readonly status

```bash
zfs get readonly rpool rpool/ROOT rpool/ROOT/pve-1 rpool/data
```

If any dataset is read-only, disable it:

```bash
zfs set readonly=off rpool/ROOT/pve-1
```

---

### 13. Export pool

Required before re-importing with an alternate root for chroot.

```bash
zpool export -f rpool
```

| Option | Description |
|---|---|
| `-f` | Force export even if the pool is currently busy |

---

### 14. Import pool with alternate root

```bash
zpool import -f -R /media/RESCUE rpool
```

| Option | Description |
|---|---|
| `-f` | Force import (pool may not have been cleanly exported) |
| `-R <altroot>` | Mount all datasets under this path prefix instead of their canonical mountpoints |

---

### 15. Verify and set bootfs

```bash
zpool get bootfs
zpool set bootfs=rpool/ROOT/pve-1 rpool
```

---

### 16. Bind-mount system filesystems for chroot

```bash
mount -o rbind /proc /media/RESCUE/proc
mount -o rbind /sys  /media/RESCUE/sys
mount -o rbind /dev  /media/RESCUE/dev
mount -o rbind /run  /media/RESCUE/run
chroot /media/RESCUE
```

| Option | Description |
|---|---|
| `-o rbind` | Recursively bind-mount, including all submounts |

---

### 17. Fix hostid and regenerate initramfs

If ZFS refuses to import the pool due to a hostid mismatch (pool was previously mounted on a different system):

```bash
hostid
# note the output value, replace <HOST_ID> below
zgenhostid -f <HOST_ID>
zpool set cachefile=/etc/zfs/zpool.cache rpool
update-initramfs -u -k all
```

| Option | Description |
|---|---|
| `zgenhostid -f` | Force overwrite of `/etc/hostid` with the given value |
| `update-initramfs -u` | Update (rather than create) existing initramfs images |
| `-k all` | Apply to all installed kernel versions |

---

### 18. Reinitialize Proxmox boot tools

```bash
proxmox-boot-tool reinit
exit
```

See also: https://pve.proxmox.com/wiki/Recover_From_Grub_Failure

---

### 19. Unmount and export (back in live/rescue Linux)

```bash
umount -R /media/RESCUE/run
umount -R /media/RESCUE/dev
umount -R /media/RESCUE/sys
umount -R /media/RESCUE/proc
```

If a process is keeping the mountpoint busy:

```bash
fuser -km /media/RESCUE
umount -lR /media/RESCUE
```

| Option | Description |
|---|---|
| `fuser -k` | Kill all processes using the mountpoint |
| `fuser -m` | Target all processes accessing the given mount |
| `umount -R` | Recursively unmount all submounts |
| `umount -l` | Lazy unmount — detach filesystem even if still busy |

---

### 20. Export pool and reboot

```bash
zpool export rpool
reboot
```
 
  • Like
Reactions: UdoB
You must log in or register to reply here.
Top Bottom
Back