Question on PVE Linux CTs and FUSE with replication/backups

[copec]

Is there a way to disable fsfreeze? I need to use JuiceFS inside of containers, but I also use replication to manually intervene if a hardware node totally fails. Before PVE I used ZFS snapshot to replicate and recover multiple times for many years without fsfreeze. Everything that is critical on disk I have alternative means of replication (e.g. mysql xtradb cluster). I would strongly prefer to have the fuse mounts inside of the containers rather than map them into the containers.

 

[copec]

Update: on my testing cluster I found every instance of fsfreeze and freeze that specifically causing issues during replication and backup to PBS:

- /usr/share/perl5/PVE/LXC.pm

sub freeze($) {
    my ($vmid) = @_;
# It doesn't seem like PBS backup is doing a total freeze, but commenting out this caused the container to stop freezing with juiceFS mount
#    if (PVE::CGroup::cgroup_mode() == 2) {
#       PVE::LXC::Command::freeze($vmid, 30);
#    } else {
#       PVE::LXC::CGroup->new($vmid)->freeze_thaw(1);
#    }
}

- /usr/share/perl5/PVE/LXC/Config.pm

# implements similar functionality to fsfreeze(8)
sub fsfreeze_mountpoint {
my ($path, $thaw) = @_;

my $op = $thaw ? 'thaw' : 'freeze';
my $ioctl = $thaw ? FITHAW : FIFREEZE;

# Causes issues with Replication and PBS backups with JuiceFS mounts
#    sysopen my $fd, $path, O_RDONLY or die "failed to open $path: $!\n";
#    my $ioctl_err;
#    if (!ioctl($fd, $ioctl, 0)) {
#       $ioctl_err = "$!";
#    }
#    close($fd);
#    die "fs$op '$path' failed - $ioctl_err\n" if defined $ioctl_err;
}

- /usr/share/perl5/PVE/Replication.pm

# freeze filesystem for data consistency
# Causes problems with JuiceFS FUSE mounts
#    if ($freezefs) {
#       $logfunc->("freeze guest filesystem");
#       $guest_class->__snapshot_freeze($vmid, 0);
#    }

# make snapshot of all volumes
my $sync_snapname =
PVE::ReplicationState::replication_snapshot_name($jobid, $start_time);

my $replicate_snapshots = {};
eval {
foreach my $volid (@$sorted_volids) {
$logfunc->("create snapshot '${sync_snapname}' on $volid");
PVE::Storage::volume_snapshot($storecfg, $volid, $sync_snapname);
$replicate_snapshots->{$volid} = 1;
}
};
my $err = $@;

# thaw immediately
# Probably don't thaw if not frozen above^
#    if ($freezefs) {
#       $logfunc->("thaw guest filesystem");
#       $guest_class->__snapshot_freeze($vmid, 1);
#    }

I had to restart pvescheduler.service and pvedaemon.service

This isn't a viable long term solution, because the perl gets changed with package updates and simply blocking the files from being overwritten will inevitably cause widespread breakage. I thought it would just put it here for reference.

I wish there were an option whether to use fsfreeze or not for CTs. I've been using filesystem based snapshot techniques for years and IMO fsfreeze isn't useful for all but a handful of things on the application level - Almost no applications factor write boundaries for a consistent persistent state anyways.

This is useful to make a traditional filesystems consistent internally (although not the applications storing data in it for the most part). When a ZFS filesystem dataset is used directly in a container, the zfs snapshots already capture it in a consistent state (although again, not most applications on top of it). For a real VM this is still useful as there is usually a traditional filesystem inside the VM. I understand "traditional" filesystems will always be part of the storage equation, but there should be a "unsafe! know what you are doing!" option to disable fsfreeze all together for special cases.