Install Ceph on newly built 5-node Proxmox VE 8.0.3

madscientist5500

New Member
Oct 15, 2023
7
0
1
Hello guys,

I am trying to do a PoC of Ceph. Installed PVE no subscription and updated the repos to non subscription on 5 node cluster but Ceph will not install. It outputs the attached errors for dependencies. However, when I do a group for ceph, it shows it is already installed. Same error on all 5 nodes.

Any ideas how to fix? Thank you.
 

Attachments

  • Error_Ceph.png
    Error_Ceph.png
    96.2 KB · Views: 49
I used the following steps on Promox 7. Should be same for Proxmox 8.


1. Make sure disk controller is in IT/HBA-mode
2. Put servers in cluster
3. Click "Install Ceph" button on each node
5. Create OSDs, Monitors, MDS as needed
4. $$$
 
Hello. Thank you. Yes, I did that. However, the problem seems to be with the Ceph dependencies. It does not install the dependencies.
 
Hello guys,

I am trying to do a PoC of Ceph. Installed PVE no subscription and updated the repos to non subscription on 5 node cluster but Ceph will not install. It outputs the attached errors for dependencies. However, when I do a group for ceph, it shows it is already installed. Same error on all 5 nodes.

Any ideas how to fix? Thank you.
have a closer look at the screencap you sent. its telling you what the problem is:
"you have held broken packages"

the first order of business is to drop to root shell.
next, apt update
after that, apt dist-upgrade, which will probably fail and inform you that you need to run dpkg --configure -a
if it does, do as instructed.
perform apt dist-upgrade again. if it returns a positive result, perform pveceph install again (either from cli or gui.)
 
I have the same problem described here on all 5 nodes. Should I rebuild each node one at a time with a clean installation on each? I have not done anything yet on them so they have no data other than the default install and cluster creation. The way I did was I installed Proxmox from USB stick on each and then added to cluster. Should I do a vanilla install on each again, install Ceph, then add them to PVE cluster? Thank you.
 
Here is an update: After 2 hours of troubleshooting, I decided to re-install Proxmox on all nodes. This worked. I don't recall having done anything differently the first time, but apparently I did. As the second try was successful. I appreciate the help provided, as I definitely learned new stuff with the pointers given here.
 
Here is the output of grep -r '' /etc/apt/sources.list*

root@lepve00:~# grep -r '' /etc/apt/sources.list*
/etc/apt/sources.list:deb http://download.proxmox.com/debian/ceph-quincy bookworm no-subscription
/etc/apt/sources.list.d/ceph.list:deb http://download.proxmox.com/debian/ceph-quincy bookworm no-subscription
/etc/apt/sources.list.d/pve-enterprise.list:deb https://download.proxmox.com/debian/pve bookworm pve-no-subscription

The problem most likely was, that the whole Debian repositories were missing...
 
Yes, thank you. I am now running into a different problem. Ceph is working and I am trying to get one VM on it. But I am seeing timeout errors in the log. Each host has one 2 TB NVMe dedicated to Ceph. I am unable to create a VM. There is a timeout error message. My goal in this minor project is very basic. I just want to get one VM running and I don't care about the performance of it. I know the hardware I am using is not adequate. Could the issues be caused by using 1 Gbit network for this PoC? I just want to see how it works. We just hired a new senior storage engineer. He is suggesting we look into Proxmox with Ceph to replace our VMware vSAN environment. According to him this will represent savings in costs. I just want to see this work in its very basic form before approving a purchase for new hardware to run this Proxmox environment.

The error creating the VM is


Task viewer: VM 100 - Create

OutputStatus

Stop

Download
TASK ERROR: unable to create VM 100 - rbd error: 'storage-CephPool00'-locked command timed out - aborting
 
Hi Team,

I trying to run proxmox in my Test Lab VM demo. when i trying to install ceph this error will showing anyone guide me please every node getting same issue please see the attachment. after success this demo then we plan to purchase every things regarding subscription

Thanks
Amol
 

Attachments

  • install error.jpg
    install error.jpg
    137.8 KB · Views: 6
Dear Sir,

Let me tell you how and what I installed.
1st I am doing this demo on vm ware
Then I selected 64 GB RAM 300 sxi or nvme and by doing the rest of the process, the server was installed.
further i disabled the repos enterprise wall and selected no subscription but still this issue is coming i am not able to install it in ceph
i made new server install and try but facing same issue

HOw to unselect enterprise repos i need to demo purpose temporary after done we purchase this subscription i need to show this proxmox product how to works and scalable i want to show manage team member

Kindly See the Attachment
 

Attachments

  • 1.jpg
    1.jpg
    240.9 KB · Views: 4
  • 2.jpg
    2.jpg
    323.3 KB · Views: 4
  • 3.jpg
    3.jpg
    220.4 KB · Views: 4
Last edited:
In your first screenshot I see "enterprise" enabled(!) and "No ... repo is enabled" at the same time. You need to clean that up.

Use the "Add" -button to enable "Ceph Reef No-Subscription" and "No-Subscription" (for the "normal" PVE-part).
 
In your first screenshot I see "enterprise" enabled(!) and "No ... repo is enabled" at the same time. You need to clean that up.

Use the "Add" -button to enable "Ceph Reef No-Subscription" and "No-Subscription" (for the "normal" PVE-part).

Dear UDOB,

I am trying do same things which you share me but still i am facing same issue not install Ceph and also i am ping all site
i provided full access internet

Please Help me?
And Resolve me issue after this done we purchase thhis Product i am trying to do this installation last 2/3 month but no one provide proper way please anyone who know this please guide me step

Please see the attachment

Thanks
AMol
 

Attachments

  • ping success.jpg
    ping success.jpg
    324.7 KB · Views: 4
  • 3.jpg
    3.jpg
    298.6 KB · Views: 4
  • 2.jpg
    2.jpg
    250.6 KB · Views: 4
  • 1.jpg
    1.jpg
    236.9 KB · Views: 4
  • pacakage.jpg
    pacakage.jpg
    259.3 KB · Views: 4
The repository screen looks fine to me.

The IP address (51....) belongs to the Proxmox CDN. Of course it must be reachable to get updates.

Analyze your own network and search for firewall settings preventing connections to that server on port 80. As you can ping it, the routing is fine - but "someone" blocks traffic to port 80.

In a company: contact your network admin ;-)
In a homelab: describe more of your setup.

I do not use the PVE firewall mechanism, so I have no idea if that could block traffic to its own repositories. But try to disable all firewall you can administrate. (And document this, so you can enable them again later on...)

Are there any other routers involved?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!