Packages kept back on upgrade to 6.0 with glusterfs official repo

[robynhub]

Hello,

after upgrading i have some packages kept back:

~# apt-get dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
pve-qemu-kvm qemu-server spiceterm
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.

seems to be related with the official gluster repo:

# apt-get upgrade pve-qemu-kvm
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
glusterfs-common : Conflicts: libglusterfs0 but 5.5-3 is to be installed
pve-libspice-server1 : Conflicts: libspice-server1
E: Broken packages

~# dpkg -l | grep gluster
ii glusterfs-client 6.4-1 amd64 clustered file-system (client package)
ii glusterfs-common 6.4-1 amd64 GlusterFS common libraries and translator modules

any suggestion how to fix this?
I have enterprise repo and subscription.

 

[robynhub]

Downgrading to gluster 5.5 solved the issue but now I can't access to my repo.

I really NEED to upgrade glusterfs to 6.4 with proxmox 6. With the PVE 5.3 aren't any issues.

I just followed the official Glusterfs install guide for Debian buster.

# aptitude dist-upgrade
The following packages will be REMOVED:
libgfchangelog0{u} libgfdb0{u} libglusterfs-dev{u} libpython3.7{u} python3-jwt{u}
The following packages will be upgraded:
glusterfs-client glusterfs-common{b}
2 packages upgraded, 0 newly installed, 5 to remove and 0 not upgraded.
Need to get 8,434 kB of archives. After unpacking 9,181 kB will be freed.
The following packages have unmet dependencies:
glusterfs-common : Conflicts: libglusterfs0 but 5.5-3 is installed

 

[Dominic]

Thank you for reporting!
We are aware of problems with GlusterFS 6 on PVE6. Unfortunately, this is not something we could repair with a small patch. You can follow the exact progress on this problem on Bugzilla.

now I can't access to my repo.

I am not sure about your goals. Is refraining from a converged setup and running a VM as storage server with for example plain Debian as Gluster server an option instead?

 

[robynhub]

Hello,

Thank you for your answer. I have this setup:

4 X NAS with Debian 10 and Gluster 6.4 as servers
10 X ProxmoxVE Nodes Enterprise 6.0-2
10 Gbit networking.
about 350 VMs

Before upgrading to PVE 6.0 I had all nodes upgraded to Gluster 6.4 and the Gluster op.version of every share is 60000.
Now with Proxmox 6.0 I had to downgrade the glusterfs clients to 5.5 with clients with op.version of 55000

After a few test i managed to mount a gluster 6.4 share with a 5.5 client but I can't confirm the stability of this solution.
I'm already suprised that the gluster server allowed the mount with a client with an op.version lower than the server!
This should not work by the glusterfs documentation.

I will keep an eye on bugzilla page.

 

[malevolent]

It seems some pve packages are tied to gluster 5.5, so we cannot upgrade gluster through gluster official repositories... and obviously, if you upgrade/install a gluster version higher than 5.5 before upgrading/installing proxmox 6, installation fails.

 

[Dominic]

I'm already suprised that the gluster server allowed the mount with a client with an op.version lower than the server!
This should not work by the glusterfs documentation.

I cannot say much about stability yet, but I installed Gluster 6.4-1 (OpVersion 60000) on 2 Debian 10 VMs and added it as storage on the PVE 6 host running Gluster 5.5-3 (OpVersion 50400) without any problems.

 

[Apollon77]

I also run intothis problem now witj gluster 5.8 installed ... PVE 5.4 was accepting this without any problem and my setup is, as also described in the documentation so that I have installed the glusterfs on all my 7 PVE nodes directly ... especially glusterfs 5.5 had big issues with increased network usage that are only solved by 5.6, so reverting my installation back to 5.5 would be the worst option somehow ...

In this case PVE 6 misses a very important feature :-(

It do not help me really, and I expect that true for others too, that it works to connect to a "remote" GlusterFS because all my nodes have it installed locally.

Why glusterfs 5.5 is tied to PVE packages in that way this time? It was working without any issues in PVE 5.4 where it was not tied that hard :-(

Beside the important fact that this should be added asap to the 5-to-6 Upgrade page, because if it would had been there yesterday I would not have started to upgrade.

So my main question is: What are my options now?

Is there a fix/change forseeable in the way how the packages are tied together? Or upgraded to at least 5.6 so that I could think of downgrading (or tied to 6.x so that I could upgrade)

I already upgraded my complete cluster to corosync3 ... how can I revert that?
Or can I have my PVE 5.4 cluster run with corosync3 till a fix comes? Information from other threads let me think this is no good option...

I only have one node upgraded to buster so far, so I need to see what happens if I reboot that host now ... I expect running KVMs is broken because the packages "pve-qemu-kvm qemu-server spiceterm" are not updated ... Else I could re-setup that one on stretch - what would bring me back to the corosync-downgrade question above.

Thank you very much for ideas how to deal with this situation now...

Ingo

 

[Apollon77]

Ok, as short update ... I was able to downgrade corosync and will see if my system in general stabilize again (it was a very bad day today with 2 kernel crashes and spontanous reboots and a lot of corosync problems ...), so I now only need to find a way how to handle the one broken host.

But ith this I'm back to 5.4 ... no idea how to continue because of this topic and all those corosync3 issues that I had

 

[Dominic]

Beside the important fact that this should be added asap to the 5-to-6 Upgrade page, because if it would had been there yesterday I would not have started to upgrade.

I've just sent a patch.

The problem is that Gluster 5.5-3 does not conflict with the package libglusterfs0 but more recent versions like 6.4-1 do.

dpkg -s glusterfs-common | egrep "Version|Conflict"

Please take a look at Dominik's comment in Bugzilla for a more detailed description.

 

[Apollon77]

I've just sent a patch.

Patch for the webpage? Thank you!

Thank you very much

PS: On my system the above command shows:

root@pm7:~# dpkg -s glusterfs-common | egrep "Version|Conflict"
Version: 5.8-1
Conflicts: libglusterfs-dev, libglusterfs0

 

[Apollon77]

I have send the glusterfs "packager" an email and also pointed to the bug tracker ticket ... let's hope he can support

 

[Dominic]

5-to-6 Upgrade page

Patch for the webpage?

I overlooked the word "page"... The patch is for pve5to6, the small checklist program and can be seen on the pve-devel mailing list.

I have send the glusterfs "packager" an email and also pointed to the bug tracker ticket

Thank you!

 

[Apollon77]

I overlooked the word "page"... The patch is for pve5to6, the small checklist program and can be seen on the pve-devel mailing list.

I think it makes much sense to have it in both

 

[Dominic]

Added it to the wiki, hoping it's sufficiently visible.

 

[Apollon77]

Maybe add the link to that section as" check jknown upgrade issues if they affect you" in the "preconditions too with a link down to the konwon issues? (so easier for people that follwo checklists ;-) )

 

[Dark26]

On my own config @home, everyworking fine with gluster v6, but promox was installed over a debian insatllation and i add manually the gluster repository

deb [arch=amd64] https://download.gluster.org/pub/gluster/glusterfs/6/LATEST/Debian/buster/amd64/apt buster main

before the uprgrade it was stretch

so far so good

Edit :

After verification, it seem that i have the problem too ....

 

[Apollon77]

@Dominik But one question is still in my head: In PVE 5 everything works up to current glusterfs packages. They can simply be upgraded and the "gluster-common" and "gluster.client" gets updated too and so PVE is using the newer glusterfs version also as client. As we know this works with all glusterfs package up to 6.x current ones. So also the glusterfs packaging changes have no negative effect here.

But not with PVE 6 ... So beside the glusterfs packaging changes there needs to be a difference on the PVE packaging side! It seems that the dependecies to glusterfs packages are done differently in PVE6 packages ... So I would more think about looking into this because you (=Proxmox) have this side of the packaging under your control!
Simply do it the same as in PVE5 here and the problem should be gone ... Or at least please provide (also in the Bugzilla) information on thos packaging differences and why they are needed and can not be done as earlier.

Thank you

 

[Dominic]

We plan not to wait for Gluster but to adapt our dependencies so that the issue gets resolved.

 

[malevolent]

After upgrading all PVE nodes to v6, I realized that gluster 5.5 is damn slow compared with 3.12.15 ... I've readed somewhere it's solved on 5.8 onwards... I had to put critical (DBs) VMs on plain ZFS in order make them work, so I have no redundancy at all besides a simple raidz1

@Dominic would you be son kind to tell us if you have any ETA for the pve dependencies, so we can upgrade gluster (or downgrade it to the old and good 3.12.15)?

 

[Apollon77]

@malevolent 3.x is EOL ... same as 4.0 ...