vzdump with snapshot on BTRFS

TomFreudenberg · Nov 5, 2023

While currently not supported I have patched the perl module for LXC.

Code:

/usr/share/perl5/PVE/LXC.vm

You may apply the patch via

Code:

patch /usr/share/perl5/PVE/LXC.pm LXC.pm.patch

This is the content of LXC.pm.patch:

Code:

--- /usr/share/perl5/PVE/LXC.pm 2023-06-24 15:08:37.000000000 +0200
+++ /usr/share/perl5/PVE/LXC.pm 2023-11-02 19:13:10.820735410 +0100
@@ -1793,6 +1793,24 @@
                    $name .= "\@$snapname";
                    if ($scfg->{type} eq 'zfspool') {
                        PVE::Tools::run_command(['mount', '-o', 'ro', @extra_opts, '-t', 'zfs', "$scfg->{pool}/$name", $mount_path]);
+                   } elsif ($scfg->{type} eq 'btrfs') {
+                       # extract subvol path for snapshot
+                       my $snap_subvol_path = $path =~ s/$scfg->{path}\///r;
+
+                       # parse result from btrfs for device uuid
+                       my $device_uuid;
+
+                       my $parser = sub {
+                           my $line = shift;
+                           if ($line =~ m/^.*\s+uuid:\s+([0-9a-f-]+)(\s+.*)?$/) {
+                               $device_uuid = $1;
+                           }
+                       };
+
+                       # get uuid for block device from btrfs volume
+                       PVE::Tools::run_command(['btrfs', 'filesystem', 'show', '-m', $scfg->{path}], outfunc => $parser, timeout => 10);
+
+                       PVE::Tools::run_command(['mount', '-o', 'ro', @extra_opts, '-t', 'btrfs', '-o', "subvol=$snap_subvol_path", "UUID=$device_uuid", $mount_path]);
                    } else {
                        die "cannot mount subvol snapshots for storage type '$scfg->{type}'\n";
                    }

This was done on PVE 8 but should be the same on PVE 7 as well.

I also sent in a request to the development channel if they like to apply the patch into PVE.

Cheers
Tom

Deleted member 205422 · Nov 5, 2023

This should be featured.

I just struggled to find some overview what's supported and unsupported ZFS vs BTRFS, it's trial and error for me.

TomFreudenberg · Nov 5, 2023

When using BTRFS you can't create a Container from within the Webgui on the BTRFS storage.

Bash:

#!/usr/bin/env bash

# save some values
APP=$(basename "$0")

# check given action

CT_ID=$1 && shift
CT_TEMPL=$1 && shift
CT_HOSTNAME=$1 && shift
CT_CORES=$1 && shift
CT_MEMORY=$1 && shift

CT_IPV4=$1 && shift
CT_IPV4_GW=$1 && shift

CT_IPV6=$1 && shift
CT_IPV6_GW=$1 && shift

CT_DATA=$1 && shift

FLAG_UNPRIVILEGED=1

while [[ "$1" != "" ]]
do
  PARAM=$1
  case "${PARAM}" in
    --privileged)
      FLAG_UNPRIVILEGED=0
      ;;
  esac
  shift
done

# simple help output
__help () {

  echo -e "Usage:\n"
  echo -e "${APP} ID TEMPLATE HOSTNAME CORES MEM_MB IPV4/CIDR GW_V4 IPV6/PFX GW_V6 DATA_VOL\n"
  echo -e "  * IPV4, IPV6 and DATA_VOL params could be entered as dash (-) and will be ignored on config\n"

  TEMPLATES=$( ls /var/lib/vz/template/cache/ )
  echo -e "  * Templates:"
  for t in ${TEMPLATES}; do echo "    - $t"; done
  echo

  IP_ADDRS=$( ip a s vmbr0 scope global | grep 'inet' | sed 's/\s\+/ /g' | cut -d ' ' -f 3 )
  echo -e "  * Host addresses IPv4, IPv6:"
  for t in ${IP_ADDRS}; do echo "    - $t"; done
  echo

}

if [[ "${CT_ID}" == "--help" || "${CT_ID}" == "-h" || "${CT_ID}" == "" ]]
then
  __help
  exit 0
fi

# test given params before create CT
__check_param () {

  if [[ "${1}" == "" ]]
  then
    echo -e "${2} to create CT is missing!\n"
    __help
    exit 1
  fi

}

__check_param "${CT_ID}" "ID"
__check_param "${CT_TEMPL}" "Template"
__check_param "${CT_HOSTNAME}" "Hostname"
__check_param "${CT_CORES}" "Cores"
__check_param "${CT_MEMORY}" "Memory"
__check_param "${CT_IPV4}" "IPv4 address/CIDR"
__check_param "${CT_IPV4_GW}" "IPv4 Gateway"
__check_param "${CT_IPV6}" "IPv6 address/PFX"
__check_param "${CT_IPV6_GW}" "IPv6 Gateway"
__check_param "${CT_DATA}" "Data volume"

# create container from template
pct create "${CT_ID}" local:vztmpl/${CT_TEMPL} --rootfs pve:0 --password --ostype debian --hostname "${CT_HOSTNAME}" --feature nesting=1 --cores ${CT_CORES} --memory "${CT_MEMORY}" --swap 0 --timezone host --unprivileged ${FLAG_UNPRIVILEGED}

  # allow access of CT to BTRFS
  chmod +rx "/vol/pve/images/${CT_ID}/subvol-${CT_ID}-disk-0.subvol"

# append network interface
if [[ "${CT_IPV4}" != "-" && "${CT_IPV6}" != "-" ]]
then
  pct set "${CT_ID}" --net0 name=eth0,bridge=vmbr0,firewall=0,ip="${CT_IPV4}",gw="${CT_IPV4_GW}",ip6="${CT_IPV6}",gw6="${CT_IPV6_GW}"
fi
if [[ "${CT_IPV4}" != "-" && "${CT_IPV6}" == "-" ]]
then
  pct set "${CT_ID}" --net0 name=eth0,bridge=vmbr0,firewall=0,ip="${CT_IPV4}",gw="${CT_IPV4_GW}"
fi
if [[ "${CT_IPV4}" == "-" && "${CT_IPV6}" != "-" ]]
then
  pct set "${CT_ID}" --net0 name=eth0,bridge=vmbr0,firewall=0,ip6="${CT_IPV6}",gw6="${CT_IPV6_GW}"
fi

# append data store for file-store, db-store etc.
if [[ "${CT_DATA}" != "-" ]]
then
  pct set "${CT_ID}" --mp0 volume=data:0,mp=/vol/data
  # allow access of CT to BTRFS
  chmod +rx "/vol/data/images/${CT_ID}/subvol-${CT_ID}-disk-0.subvol"
fi

echo "CT ${CT_ID} created"

exit 0

The above script allows to create a container on a BTRFS volume.

Except this, all should work - it's just Linux

Deleted member 205422 · Nov 5, 2023

TomFreudenberg said:

When using BTRFS you can't create a Container from within the Webgui on the BTRFS storage.

Bash:

#!/usr/bin/env bash

# save some values
APP=$(basename "$0")

# check given action

CT_ID=$1 && shift
CT_TEMPL=$1 && shift
CT_HOSTNAME=$1 && shift
CT_CORES=$1 && shift
CT_MEMORY=$1 && shift

CT_IPV4=$1 && shift
CT_IPV4_GW=$1 && shift

CT_IPV6=$1 && shift
CT_IPV6_GW=$1 && shift

CT_DATA=$1 && shift

FLAG_UNPRIVILEGED=1

while [[ "$1" != "" ]]
do
  PARAM=$1
  case "${PARAM}" in
    --privileged)
      FLAG_UNPRIVILEGED=0
      ;;
  esac
  shift
done

# simple help output
__help () {

  echo -e "Usage:\n"
  echo -e "${APP} ID TEMPLATE HOSTNAME CORES MEM_MB IPV4/CIDR GW_V4 IPV6/PFX GW_V6 DATA_VOL\n"
  echo -e "  * IPV4, IPV6 and DATA_VOL params could be entered as dash (-) and will be ignored on config\n"

  TEMPLATES=$( ls /var/lib/vz/template/cache/ )
  echo -e "  * Templates:"
  for t in ${TEMPLATES}; do echo "    - $t"; done
  echo

  IP_ADDRS=$( ip a s vmbr0 scope global | grep 'inet' | sed 's/\s\+/ /g' | cut -d ' ' -f 3 )
  echo -e "  * Host addresses IPv4, IPv6:"
  for t in ${IP_ADDRS}; do echo "    - $t"; done
  echo

}

if [[ "${CT_ID}" == "--help" || "${CT_ID}" == "-h" || "${CT_ID}" == "" ]]
then
  __help
  exit 0
fi

# test given params before create CT
__check_param () {

  if [[ "${1}" == "" ]]
  then
    echo -e "${2} to create CT is missing!\n"
    __help
    exit 1
  fi

}

__check_param "${CT_ID}" "ID"
__check_param "${CT_TEMPL}" "Template"
__check_param "${CT_HOSTNAME}" "Hostname"
__check_param "${CT_CORES}" "Cores"
__check_param "${CT_MEMORY}" "Memory"
__check_param "${CT_IPV4}" "IPv4 address/CIDR"
__check_param "${CT_IPV4_GW}" "IPv4 Gateway"
__check_param "${CT_IPV6}" "IPv6 address/PFX"
__check_param "${CT_IPV6_GW}" "IPv6 Gateway"
__check_param "${CT_DATA}" "Data volume"

# create container from template
pct create "${CT_ID}" local:vztmpl/${CT_TEMPL} --rootfs pve:0 --password --ostype debian --hostname "${CT_HOSTNAME}" --feature nesting=1 --cores ${CT_CORES} --memory "${CT_MEMORY}" --swap 0 --timezone host --unprivileged ${FLAG_UNPRIVILEGED}

  # allow access of CT to BTRFS
  chmod +rx "/vol/pve/images/${CT_ID}/subvol-${CT_ID}-disk-0.subvol"

# append network interface
if [[ "${CT_IPV4}" != "-" && "${CT_IPV6}" != "-" ]]
then
  pct set "${CT_ID}" --net0 name=eth0,bridge=vmbr0,firewall=0,ip="${CT_IPV4}",gw="${CT_IPV4_GW}",ip6="${CT_IPV6}",gw6="${CT_IPV6_GW}"
fi
if [[ "${CT_IPV4}" != "-" && "${CT_IPV6}" == "-" ]]
then
  pct set "${CT_ID}" --net0 name=eth0,bridge=vmbr0,firewall=0,ip="${CT_IPV4}",gw="${CT_IPV4_GW}"
fi
if [[ "${CT_IPV4}" == "-" && "${CT_IPV6}" != "-" ]]
then
  pct set "${CT_ID}" --net0 name=eth0,bridge=vmbr0,firewall=0,ip6="${CT_IPV6}",gw6="${CT_IPV6_GW}"
fi

# append data store for file-store, db-store etc.
if [[ "${CT_DATA}" != "-" ]]
then
  pct set "${CT_ID}" --mp0 volume=data:0,mp=/vol/data
  # allow access of CT to BTRFS
  chmod +rx "/vol/data/images/${CT_ID}/subvol-${CT_ID}-disk-0.subvol"
fi

echo "CT ${CT_ID} created"

exit 0

The above script allows to create a container on a BTRFS volume.

Except this, all should work - it's just Linux

This is so neat! So I just starting experimenting with PVE, but came with experience with LXCs from plain LXD. I am not sure I would stay as there are many options to choose from nowadays, but many concepts are confusing to me, the way vzdumps are done (as opposed to plain keeping the image on the dataset/subvolume). In your script there is reference to:

vol/pve/images/${CT_ID}/subvol-${CT_ID}-disk-0.subvol

So this is using just files on actual subvolume in vol/pve/images/${CT_ID}, why is that so?

I mosly find BTRFS better option to test out because it can reference just blocks of the originating subvolume, not entire dataset like in ZFS (which cannot be discared). It seems to be a second-class citizen in PVE. Must have to do with demand which might be because if the RAID fiasco, but I never use 5/6 raids anyhow.

PS: I also find --privileged much more logical than unprivileged 0.

Deleted member 205422 · Nov 6, 2023

Thankfully my post was eventually published, but by now I am afraid it's been forgotten by everyone.

TomFreudenberg · Nov 6, 2023

Esiy said:
So this is using just files on actual subvolume in vol/pve/images/${CT_ID}, why is that so?

Could you explain your question, please.

I do not understand what you mean by that.

intrax · Feb 21, 2024

This should be in a pve release asap as well as snapshotting vm's on btrfs subvols !

Search

Search

vzdump with snapshot on BTRFS

TomFreudenberg

Member

Deleted member 205422

Guest

TomFreudenberg

Member

Deleted member 205422

Guest

Deleted member 205422

Guest

TomFreudenberg

Member

intrax

Member