Install Ceph on newly built 5-node Proxmox VE 8.0.3

[madscientist5500]

Hello guys,

I am trying to do a PoC of Ceph. Installed PVE no subscription and updated the repos to non subscription on 5 node cluster but Ceph will not install. It outputs the attached errors for dependencies. However, when I do a group for ceph, it shows it is already installed. Same error on all 5 nodes.

Any ideas how to fix? Thank you.

 

[jdancer]

I used the following steps on Promox 7. Should be same for Proxmox 8.


1. Make sure disk controller is in IT/HBA-mode
2. Put servers in cluster
3. Click "Install Ceph" button on each node
5. Create OSDs, Monitors, MDS as needed
4. $$$

 

[madscientist5500]

Hello. Thank you. Yes, I did that. However, the problem seems to be with the Ceph dependencies. It does not install the dependencies.

 

[Neobin]

Not sure, but maybe this helps?:
https://forum.proxmox.com/threads/p...itching-to-enterprise-repo.134577/post-594589

Otherwise, the full output in code-tags each of: grep -r '' /etc/apt/sources.list* (do not forget the: * at the end!) and: apt update might be helpful.

 

[alexskysilk]

Hello guys,

I am trying to do a PoC of Ceph. Installed PVE no subscription and updated the repos to non subscription on 5 node cluster but Ceph will not install. It outputs the attached errors for dependencies. However, when I do a group for ceph, it shows it is already installed. Same error on all 5 nodes.

Any ideas how to fix? Thank you.

have a closer look at the screencap you sent. its telling you what the problem is:
"you have held broken packages"

the first order of business is to drop to root shell.
next, apt update
after that, apt dist-upgrade, which will probably fail and inform you that you need to run dpkg --configure -a
if it does, do as instructed.
perform apt dist-upgrade again. if it returns a positive result, perform pveceph install again (either from cli or gui.)

 

[madscientist5500]

Thank you for the info, everyone. I will perform those steps and report back.

 

[madscientist5500]

Here is the output of grep -r '' /etc/apt/sources.list*

root@lepve00:~# grep -r '' /etc/apt/sources.list*
/etc/apt/sources.list:deb http://download.proxmox.com/debian/ceph-quincy bookworm no-subscription
/etc/apt/sources.list.d/ceph.list:deb http://download.proxmox.com/debian/ceph-quincy bookworm no-subscription
/etc/apt/sources.list.d/pve-enterprise.list:deb https://download.proxmox.com/debian/pve bookworm pve-no-subscription

 

[madscientist5500]

I have the same problem described here on all 5 nodes. Should I rebuild each node one at a time with a clean installation on each? I have not done anything yet on them so they have no data other than the default install and cluster creation. The way I did was I installed Proxmox from USB stick on each and then added to cluster. Should I do a vanilla install on each again, install Ceph, then add them to PVE cluster? Thank you.

 

[madscientist5500]

Here is an update: After 2 hours of troubleshooting, I decided to re-install Proxmox on all nodes. This worked. I don't recall having done anything differently the first time, but apparently I did. As the second try was successful. I appreciate the help provided, as I definitely learned new stuff with the pointers given here.

 

[Neobin]

Here is the output of grep -r '' /etc/apt/sources.list*

root@lepve00:~# grep -r '' /etc/apt/sources.list*
/etc/apt/sources.list:deb http://download.proxmox.com/debian/ceph-quincy bookworm no-subscription
/etc/apt/sources.list.d/ceph.list:deb http://download.proxmox.com/debian/ceph-quincy bookworm no-subscription
/etc/apt/sources.list.d/pve-enterprise.list:deb https://download.proxmox.com/debian/pve bookworm pve-no-subscription

The problem most likely was, that the whole Debian repositories were missing...

 

[madscientist5500]

Yes, thank you. I am now running into a different problem. Ceph is working and I am trying to get one VM on it. But I am seeing timeout errors in the log. Each host has one 2 TB NVMe dedicated to Ceph. I am unable to create a VM. There is a timeout error message. My goal in this minor project is very basic. I just want to get one VM running and I don't care about the performance of it. I know the hardware I am using is not adequate. Could the issues be caused by using 1 Gbit network for this PoC? I just want to see how it works. We just hired a new senior storage engineer. He is suggesting we look into Proxmox with Ceph to replace our VMware vSAN environment. According to him this will represent savings in costs. I just want to see this work in its very basic form before approving a purchase for new hardware to run this Proxmox environment.

The error creating the VM is


Task viewer: VM 100 - Create

OutputStatus

Stop

Download
TASK ERROR: unable to create VM 100 - rbd error: 'storage-CephPool00'-locked command timed out - aborting

 

[Amol.pande]

Hi Team,

I trying to run proxmox in my Test Lab VM demo. when i trying to install ceph this error will showing anyone guide me please every node getting same issue please see the attachment. after success this demo then we plan to purchase every things regarding subscription

Thanks
Amol

 

[UdoB]

when i trying to install ceph

Did you read and follow https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pve_ceph_install_wizard ?

 

[Amol.pande]

Did you read and follow https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pve_ceph_install_wizard ?

Dear Sir,

Let me tell you how and what I installed.
1st I am doing this demo on vm ware
Then I selected 64 GB RAM 300 sxi or nvme and by doing the rest of the process, the server was installed.
further i disabled the repos enterprise wall and selected no subscription but still this issue is coming i am not able to install it in ceph
i made new server install and try but facing same issue

HOw to unselect enterprise repos i need to demo purpose temporary after done we purchase this subscription i need to show this proxmox product how to works and scalable i want to show manage team member

Kindly See the Attachment

 

[UdoB]

In your first screenshot I see "enterprise" enabled(!) and "No ... repo is enabled" at the same time. You need to clean that up.

Use the "Add" -button to enable "Ceph Reef No-Subscription" and "No-Subscription" (for the "normal" PVE-part).

 

[Amol.pande]

In your first screenshot I see "enterprise" enabled(!) and "No ... repo is enabled" at the same time. You need to clean that up.

Use the "Add" -button to enable "Ceph Reef No-Subscription" and "No-Subscription" (for the "normal" PVE-part).

Dear UDOB,

I am trying do same things which you share me but still i am facing same issue not install Ceph and also i am ping all site
i provided full access internet

Please Help me?
And Resolve me issue after this done we purchase thhis Product i am trying to do this installation last 2/3 month but no one provide proper way please anyone who know this please guide me step

Please see the attachment

Thanks
AMol

 

[UdoB]

The repository screen looks fine to me.

The IP address (51....) belongs to the Proxmox CDN. Of course it must be reachable to get updates.

Analyze your own network and search for firewall settings preventing connections to that server on port 80. As you can ping it, the routing is fine - but "someone" blocks traffic to port 80.

In a company: contact your network admin ;-)
In a homelab: describe more of your setup.

I do not use the PVE firewall mechanism, so I have no idea if that could block traffic to its own repositories. But try to disable all firewall you can administrate. (And document this, so you can enable them again later on...)

Are there any other routers involved?