First of all, kudos to Wolfgang Bumiller for the “hot” summer btrfs patches. It surpassed my expectations, it really works, except several issues, perfectly normal for an experimental phase.
Related proxmox forum posts
https://forum.proxmox.com/threads/proxmox-ve-5-0-beta1-released.33731/page-2#post-165741
https://forum.proxmox.com/threads/p...possible-licensing-issues-zfs-on-linux.32561/
I imagine the release of 5.0 is very resource consuming for Proxmox team, but if the below permission denied issue could be solved, this would unlock the extensive testing on btrfs LXC.
I am aware that I should split this in several issues, but I don’t know where to report, in the devel mail list ( as separate mails) or in https://bugzilla.proxmox.com/ ? For the moment, I prefer to report bulk, in proxmox forum.
ENVIRONMENT
The implementation was done on PVE 4.4-13
For compatibitily with btrfs-progs, btrfs-tools 4.6.1-1~bpo8+1, the latest pach v3 was used https://www.mail-archive.com/pve-devel@pve.proxmox.com/msg18776.html
Patches:
https://www.mail-archive.com/pve-devel@pve.proxmox.com/msg18777.html v3 storage backend
https://www.mail-archive.com/pve-devel@pve.proxmox.com/msg18747.html LXC subvolume
https://github.com/Blub/pve-manager/compare/btrfs for GUI
After applying the GUI patches with make dinstall, the btrfs options showed at Datacenter > Storage > Add, but not working.
BTW, “apt install pve-doc-generator “ should be added to instructions https://git.proxmox.com/?p=pve-common.git;a=blob_plain;f=README.dev;hb=HEAD
No problem, I could add the storage from cli. ( afterwards GUI btrfs storage edit works)
WHAT’S WORKING
Creating + starting LXC/ KVM works. Clean up subvolumes upon deletion ( monitoring with “btrfs subvolume list -a /unt” )
MY FIRST POTENTIAL PROXMOX LUCKY PATCH
Since adding storage from interface failed, I considered there was a missing GUI patch for snapshots, so I have tried from command line.
Later on, I was pleasantly surprised that the Proxmox plugin system is so well designed, that you don’t need any other patches, for snapshots interface to work.
Comparing with ZFS implementation at /usr/share/perl5/PVE/Storage/ZFSPoolPlugin.pm, I have tried a lucky patch, that worked.
WHAT’S WORKING RELOADED
After the dummy patch, all the snapshots features are working, inclusive from GUI: Take snapshot, rollback, edit description, backup in snapshot mode. Removing snapshots from GUI properly removes subvolumes from btrfs backend. ( tested tracking dummy files)
BTW, I love the ability of BTRFS to restore from older snapshots (not only latest as ZFS limitation) . Confirmed by https://github.com/lxc/lxd/blob/master/doc/storage-backends.md#feature-comparison
The GUI listings for images ( raw + subvolumes) works
Backup and restore to/from btrfs storage backend works.
LXC mount points works.
Tested both KVM/LXC.
OTHER LOW PRIORITY ISSUES
Deleting lxc container doesn’t delete the his snapshots volumes
LXC resize rootfs from GUI fails with
LXC RANDOM PERMISSION DENIED ISSUE - RELEASE BLOCKING
Only after setting up the public network for lxc container and start playing with it, random permission issues appeared.
First , in Ubuntu Xenial LXC apt update triggers
Second, Debian Jessie 8.7 logs
Installing Apache on debian works, but fails to serve from browser. This is not a firewall/port issue, because ssh remote and nginx works in the same container.
Even further, after backing up the very same debian container and restoring to a ZFS backend, apache works on public port, without any modifications.
INVESTIGATING SECURITY OPTIONS
Privileged container won’t work.
lxc.aa_profile = unconfined and even systemctl stop apparmor won’t work
Commenting out from /usr/share/lxc/config/common.conf the Seccomp policy won’t work.
THE RANDOM PERMISSION DENIED ISSUE AFFECTS ONLY LXC SUBVOL, FOR KVM RAW WORKS. MOUNT OPTIONS
On host
Mount Inside KVM
Inside LXC
BTW, is it ok to have all this mount options with a subvol mount ? See https://btrfs.wiki.kernel.org/index.php/FAQ#Can_I_mount_subvolumes_with_different_mount_options.3F, some of them are in development.
On the other hand, testing further with LXD showed the same mount options.
COMPARING WITH UBUNTU LXD
Simply mount a Btrfs filesystem under /var/lib/lxd and it works.
Inside LXC
The seccomp/apparmor profiles seems to be the same.
The LXC versions
On LXD, all the previous tests apt/apache work flawless.
Thank you!
Related proxmox forum posts
https://forum.proxmox.com/threads/proxmox-ve-5-0-beta1-released.33731/page-2#post-165741
https://forum.proxmox.com/threads/p...possible-licensing-issues-zfs-on-linux.32561/
I imagine the release of 5.0 is very resource consuming for Proxmox team, but if the below permission denied issue could be solved, this would unlock the extensive testing on btrfs LXC.
I am aware that I should split this in several issues, but I don’t know where to report, in the devel mail list ( as separate mails) or in https://bugzilla.proxmox.com/ ? For the moment, I prefer to report bulk, in proxmox forum.
ENVIRONMENT
The implementation was done on PVE 4.4-13
For compatibitily with btrfs-progs, btrfs-tools 4.6.1-1~bpo8+1, the latest pach v3 was used https://www.mail-archive.com/pve-devel@pve.proxmox.com/msg18776.html
Patches:
https://www.mail-archive.com/pve-devel@pve.proxmox.com/msg18777.html v3 storage backend
https://www.mail-archive.com/pve-devel@pve.proxmox.com/msg18747.html LXC subvolume
https://github.com/Blub/pve-manager/compare/btrfs for GUI
After applying the GUI patches with make dinstall, the btrfs options showed at Datacenter > Storage > Add, but not working.
BTW, “apt install pve-doc-generator “ should be added to instructions https://git.proxmox.com/?p=pve-common.git;a=blob_plain;f=README.dev;hb=HEAD
No problem, I could add the storage from cli. ( afterwards GUI btrfs storage edit works)
Code:
cat /etc/pve/storage.cfg
btrfs: btrfspatch
path /unt
content rootdir,iso,backup,vztmpl,images
maxfiles 0
shared 0WHAT’S WORKING
Creating + starting LXC/ KVM works. Clean up subvolumes upon deletion ( monitoring with “btrfs subvolume list -a /unt” )
MY FIRST POTENTIAL PROXMOX LUCKY PATCH
Since adding storage from interface failed, I considered there was a missing GUI patch for snapshots, so I have tried from command line.
Later on, I was pleasantly surprised that the Proxmox plugin system is so well designed, that you don’t need any other patches, for snapshots interface to work.
Code:
pct snapshot 101 snap1
Can't use string ("1") as a HASH ref while "strict refs" in use at /usr/share/perl5/PVE/Storage/BTRFSPlugin.pm line 451Comparing with ZFS implementation at /usr/share/perl5/PVE/Storage/ZFSPoolPlugin.pm, I have tried a lucky patch, that worked.
Code:
diff --git a/PVE/Storage/BTRFSPlugin.pm b/PVE/Storage/BTRFSPlugin.pm
index 25c2555..69cdc30 100644
--- a/PVE/Storage/BTRFSPlugin.pm
+++ b/PVE/Storage/BTRFSPlugin.pm
@@ -448,7 +448,7 @@ sub volume_has_feature {
$key = $isBase ? 'base' : 'current';
}
- return 1 if defined($features->{$feature}->{$key}->{$format});
+ return 1 if defined($features->{$feature}->{$key});
return undef;
}WHAT’S WORKING RELOADED
After the dummy patch, all the snapshots features are working, inclusive from GUI: Take snapshot, rollback, edit description, backup in snapshot mode. Removing snapshots from GUI properly removes subvolumes from btrfs backend. ( tested tracking dummy files)
BTW, I love the ability of BTRFS to restore from older snapshots (not only latest as ZFS limitation) . Confirmed by https://github.com/lxc/lxd/blob/master/doc/storage-backends.md#feature-comparison
The GUI listings for images ( raw + subvolumes) works
Backup and restore to/from btrfs storage backend works.
LXC mount points works.
Tested both KVM/LXC.
OTHER LOW PRIORITY ISSUES
Deleting lxc container doesn’t delete the his snapshots volumes
LXC resize rootfs from GUI fails with
Code:
TASK ERROR: btrfs error: ERROR: invalid qgroupid or subvolume path: 0/0/289LXC RANDOM PERMISSION DENIED ISSUE - RELEASE BLOCKING
Only after setting up the public network for lxc container and start playing with it, random permission issues appeared.
First , in Ubuntu Xenial LXC apt update triggers
Code:
W: Can't drop privileges for downloading as file '/var/lib/apt/lists/partial/eu.archive.ubuntu.com_ubuntu_dists_xenial_InRelease' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)Second, Debian Jessie 8.7 logs
Code:
Mar 29 05:53:53 bpatchdebian rsyslogd: imklog: cannot open kernel log(/proc/kmsg): Permission denied.
Mar 29 05:53:53 bpatchdebian rsyslogd-2145: activation of module imklog failed [try http://www.rsyslog.com/e/2145 ]
Mar 29 05:53:53 bpatchdebian cron[959]: (CRON) INFO (pidfile fd = 3)
Mar 29 05:53:53 bpatchdebian cron[964]: (CRON) STARTUP (fork ok)
Mar 29 05:53:53 bpatchdebian cron[964]: (CRON) INFO (Running @reboot jobs)
Mar 29 05:53:53 bpatchdebian dbus[974]: [system] org.freedesktop.DBus.Error.AccessDenied: Failed to set fd limit to 65536: Operation not permitted
Mar 29 05:53:54 bpatchdebian postfix/master[1124]: fatal: open lock file /var/lib/postfix/master.lock: cannot open file: Permission denied
Mar 29 05:54:13 bpatchdebian init: Trying to re-exec init
Mar 29 05:54:23 bpatchdebian dbus[974]: [system] Unable to reload configuration: Failed to open "/etc/dbus-1/system.conf": Permission denied
Mar 29 06:08:52 bpatchdebian rsyslogd: [origin software="rsyslogd" swVersion="8.4.2" x-pid="878" x-info="http://www.rsyslog.com"] start
Mar 29 06:08:52 bpatchdebian rsyslogd: imklog: cannot open kernel log(/proc/kmsg): Permission denied.
Mar 29 06:08:52 bpatchdebian rsyslogd-2145: activation of module imklog failed [try http://www.rsyslog.com/e/2145 ]
Mar 29 06:08:53 bpatchdebian dbus[1062]: [system] org.freedesktop.DBus.Error.AccessDenied: Failed to set fd limit to 65536: Operation not permitted
Mar 29 06:08:53 bpatchdebian cron[1073]: (CRON) INFO (pidfile fd = 3)
Mar 29 06:08:53 bpatchdebian cron[1076]: (CRON) STARTUP (fork ok)
Mar 29 06:08:53 bpatchdebian cron[1076]: (CRON) INFO (Running @reboot jobs)
Mar 29 06:08:53 bpatchdebian postfix/master[1232]: fatal: open lock file /var/lib/postfix/master.lock: cannot open file: Permission denied
Mar 29 06:13:01 bpatchdebian CRON[5713]: (root) CMD (test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ))
Mar 29 06:13:01 bpatchdebian postfix/postdrop[5839]: warning: unable to look up public/pickup: No such file or directoryInstalling Apache on debian works, but fails to serve from browser. This is not a firewall/port issue, because ssh remote and nginx works in the same container.
Even further, after backing up the very same debian container and restoring to a ZFS backend, apache works on public port, without any modifications.
INVESTIGATING SECURITY OPTIONS
Privileged container won’t work.
lxc.aa_profile = unconfined and even systemctl stop apparmor won’t work
Commenting out from /usr/share/lxc/config/common.conf the Seccomp policy won’t work.
Code:
#lxc.seccomp = /usr/share/lxc/config/common.seccompTHE RANDOM PERMISSION DENIED ISSUE AFFECTS ONLY LXC SUBVOL, FOR KVM RAW WORKS. MOUNT OPTIONS
On host
Code:
cat /etc/fstab | grep btrfs
UUID="4e228036-92cb-4e07-805a-624063818047" /unt btrfs defaults,autodefrag,noatime 0 0Mount Inside KVM
Code:
/bssd/images/115/vm-115-disk-1.raw on / type ext4 (rw,relatime,data=ordered)Inside LXC
Code:
mount | grep btrfs
/dev/sdb on / type btrfs (rw,noatime,space_cache,autodefrag,subvolid=257,subvol=/images/101/vm-101-disk-1.subvol)BTW, is it ok to have all this mount options with a subvol mount ? See https://btrfs.wiki.kernel.org/index.php/FAQ#Can_I_mount_subvolumes_with_different_mount_options.3F, some of them are in development.
On the other hand, testing further with LXD showed the same mount options.
COMPARING WITH UBUNTU LXD
Simply mount a Btrfs filesystem under /var/lib/lxd and it works.
Inside LXC
Code:
root@myxenial:~# mount | grep btrfs
/dev/sdb on / type btrfs (rw,noatime,space_cache,autodefrag,subvolid=259,subvol=/containers/myxenial/rootfs)
/dev/sdb on /dev/lxd type btrfs (rw,noatime,space_cache,autodefrag,subvolid=5,subvol=/devlxd)
/dev/sdb on /dev/.lxd-mounts type btrfs (rw,noatime,space_cache,autodefrag,subvolid=5,subvol=/shmounts/myxenial)The seccomp/apparmor profiles seems to be the same.
The LXC versions
Code:
lxd 2.0.7-0ubuntu1~16.04.2
lxc-pve 2.0.7-4
lxcfs 2.0.6-pve1On LXD, all the previous tests apt/apache work flawless.
Thank you!
