Hi guys first post here. I have the latest proxmox up and running 8.4.1 impressed with it. However, I cannot get the entire ceph installed. The monitors get setup okay. But I can't get the ceph-mgr to install or configure. I tried manually installing it and it's asking for so many dependencies with python. I am in air gapped environment. Is there a way to get through this and a smoother way down the road everytime I have an update? Thank so much
Assistance with proxmox Ceph-reef or quincy install.
- Thread starter polarbear
- Start date
Hi,
how did you try to install Ceph?
Did you configure the correct apt repositories?
Did you try using our tooling, i.e. through the web UI under Datacenter -> <node> -> Ceph or using
how did you try to install Ceph?
Did you configure the correct apt repositories?
Did you try using our tooling, i.e. through the web UI under Datacenter -> <node> -> Ceph or using
pveceph CLI?Hi thanks for your response. So what I did was I mounted the dvd debian full dvd and mounted the proxmox pve and mounted both of them. The are in the sources.list as deb [trusted=yes] file:///storage/mnt1 and mnt2.
Then I did apt update
apt upgrade.
Since I am in airgapped I use powershell and wget to download all contents from repos for ceph reef and got all the deb files.
I copied the deb files into a dir locally on one of the nodes.
Then did dpkg -i *.deb and ran a fix-install too.
That got me into the gui on the UI then it said to configure. I was like oh cool so I go all the way to monitor and they were working and healthy then when I try to "create manager" This is where I am stuck.
So now I am in constant loops because of dependencies after installing manually all the python deb and tried to grab all their dependencies. I have ceph-mgrxxxxx.deb a bunch of them but if I try to install as if I am going down rabbit hole over and over stuck at when I try to dpkg -i libpython3.13-stdlib then says however package libpython3.13-std.lib is not installed.
I wish there was a repo I can just install whatever is needed to install ceph-mgr or whatever else after that.
Thanks again
Then I did apt update
apt upgrade.
Since I am in airgapped I use powershell and wget to download all contents from repos for ceph reef and got all the deb files.
I copied the deb files into a dir locally on one of the nodes.
Then did dpkg -i *.deb and ran a fix-install too.
That got me into the gui on the UI then it said to configure. I was like oh cool so I go all the way to monitor and they were working and healthy then when I try to "create manager" This is where I am stuck.
So now I am in constant loops because of dependencies after installing manually all the python deb and tried to grab all their dependencies. I have ceph-mgrxxxxx.deb a bunch of them but if I try to install as if I am going down rabbit hole over and over stuck at when I try to dpkg -i libpython3.13-stdlib then says however package libpython3.13-std.lib is not installed.
I wish there was a repo I can just install whatever is needed to install ceph-mgr or whatever else after that.
Thanks again
hi again,
So if I use this repo grab everything and set up my sources.list and ceph.list files accordingly should work and I should get past the python and bunch of dependency issues?
We have share LUN storage a bunch of big ones like 16 tb about 15 of them. Trying to put something on proxmox cluster aware. Looks like ceph is the right choice but complex. I did try ZFS and did the whole multipath thing and got my nodes to see the LUN just fine. But seems I have to replicate the vms to each lun to migrate it. I was hoping the vm would exist and be seen by all nodes from the same lun. I even tried LVM-thin seems I gotta replicate that too.
Just trying to architect this correctly this is the last thing I can't get going.
Thanks
So if I use this repo grab everything and set up my sources.list and ceph.list files accordingly should work and I should get past the python and bunch of dependency issues?
We have share LUN storage a bunch of big ones like 16 tb about 15 of them. Trying to put something on proxmox cluster aware. Looks like ceph is the right choice but complex. I did try ZFS and did the whole multipath thing and got my nodes to see the LUN just fine. But seems I have to replicate the vms to each lun to migrate it. I was hoping the vm would exist and be seen by all nodes from the same lun. I even tried LVM-thin seems I gotta replicate that too.
Just trying to architect this correctly this is the last thing I can't get going.
Thanks
Sorry I missed out some stuff
This repo here http://download.proxmox.com/debian/
Also further explain I attached a 4tb lun and it was seen by all 3 nodes. When i moved the vm on node1 that was sitting on the 4tb lun to node 2 4tb lun (same lun) It copied the entire 100 gigs of windows. But I guess zfs and lvm-thin are not clusteraware. only ceph. Thanks again
This repo here http://download.proxmox.com/debian/
Also further explain I attached a 4tb lun and it was seen by all 3 nodes. When i moved the vm on node1 that was sitting on the 4tb lun to node 2 4tb lun (same lun) It copied the entire 100 gigs of windows. But I guess zfs and lvm-thin are not clusteraware. only ceph. Thanks again
Hi again, I got around the issue of offline mirror to get ceph reef installed. Now that I have that installed. I can present like a 10TB lun to all my proxmox nodes and it's going to share the same LUN and moving vms from node to node should be easy now. I just wanted to make sure it's not like if I have 5 nodes and I have to provide 5x10TB like one lun to each node and it replicates. It's actually shared storage if I am correct? Thanks
You have a LUN from a SAN presented to all Proxmox nodes via FC or iSCSI?
This is something that cannot be used with Ceph. Ceph uses local disks and replicates over the network.
With a LUN you can use LVM as storage for the VMs.
This is something that cannot be used with Ceph. Ceph uses local disks and replicates over the network.
With a LUN you can use LVM as storage for the VMs.
Oh wow - I thought the matrix sheet with all the filesystems that was posted said ceph is File system cluster aware? It's over FC. I already did the multipath and can see the LUN we presented to it.
So for example you are saying that I can't use cephFS like 1 5TB lun attached to all nodes. I can't float a vm from node to node if this type of file system is cluster aware?
I been working hard at this. Wanted to not replicate and only move the state memory of the vm to another node on the same LUN. If that make sense. Just like VMWARE VMFS.
Thanks again
So for example you are saying that I can't use cephFS like 1 5TB lun attached to all nodes. I can't float a vm from node to node if this type of file system is cluster aware?
I been working hard at this. Wanted to not replicate and only move the state memory of the vm to another node on the same LUN. If that make sense. Just like VMWARE VMFS.
Thanks again
Last edited:
So after reading a bit more I guess still going to lose storage space because it's replicating with all other osd/pools.
What's the best way here with many VMs to not lose so much storage with replication (This mean for example if I have 5 nodes I will have 5 vms of every virtual machine there). A dedup / thin provisioning option to offset the way this is designed to have vm on every lun presented to all proxmox nodes.
Thanks
What's the best way here with many VMs to not lose so much storage with replication (This mean for example if I have 5 nodes I will have 5 vms of every virtual machine there). A dedup / thin provisioning option to offset the way this is designed to have vm on every lun presented to all proxmox nodes.
Thanks
Ok guys - thanks for the support so far. This will be the last - I know how it is to keep asking over and over in the forums.
So this is my situation. I have FC 420 TB hitache over Fibre backend over HBA cards. I would like to know the best implantation seems like that replication has to occur in all scenarios. So if we involve perhaps whatever filesystem I was hoping Cephfs would be it. But if I implement that does it have auto deduplication and thin provisioning which I believe it does. If I combo that with clone disk with parent template would that be an acceptable solutions to not waste so much space due to replicating the VMS over every LUN?
Right now I have 10 x 16 tb LUNS provisioned to ESXI VMFS and it's cluster aware each esxi node I can share the lun you guys already know this I am sure.
What is my best solution here? My networking on the LACP side is 9000 mtu. So in combo with FC LUNs from Hitachi over Fiber and plus LACP networking what would be my best way to not lose so much disk space due to replication?
I hopefully explained it well. If any questions please let me know. I really do appreciate the responses and I am the guy in the shop that would make proxmox a great brand name for us if I can just figure out not how to lose so much space due to replication requirements.
I have a long way to go with containers and plugins. I just need this last hump to be figured out. Because whatever you guys recommend will be the way moving forward in many environments in my office.
Thanks much!!!!
So this is my situation. I have FC 420 TB hitache over Fibre backend over HBA cards. I would like to know the best implantation seems like that replication has to occur in all scenarios. So if we involve perhaps whatever filesystem I was hoping Cephfs would be it. But if I implement that does it have auto deduplication and thin provisioning which I believe it does. If I combo that with clone disk with parent template would that be an acceptable solutions to not waste so much space due to replicating the VMS over every LUN?
Right now I have 10 x 16 tb LUNS provisioned to ESXI VMFS and it's cluster aware each esxi node I can share the lun you guys already know this I am sure.
What is my best solution here? My networking on the LACP side is 9000 mtu. So in combo with FC LUNs from Hitachi over Fiber and plus LACP networking what would be my best way to not lose so much disk space due to replication?
I hopefully explained it well. If any questions please let me know. I really do appreciate the responses and I am the guy in the shop that would make proxmox a great brand name for us if I can just figure out not how to lose so much space due to replication requirements.
I have a long way to go with containers and plugins. I just need this last hump to be figured out. Because whatever you guys recommend will be the way moving forward in many environments in my office.
Thanks much!!!!
That’s not how Ceph works. Each disk is its own bucket. A server has a bunch of buckets. If Ceph is set to 3 copies it will by default keep one copy of each data block on at least 3 different servers, in whatever bucket it wants. So a VM hard disk file may (likely) be scattered across all the disks. How many servers you have is not particularly relevant, except you need 3, preferably more.
Once set up, all servers will see “the VM disk” and not care “where” it is.
With a SAN and LUNs presented to each Proxmox host you do not use Ceph as the SAN already has its internal replication (RAID).
You can use LVM on top of the LUN as written in the Proxmox documentation.
Or if you want to have something similar to VMFS you can use a cluster filesystem like OCFS2 on top of the LUN. But this is not officially supported by Proxmox. you have to setup it yourself. There are several discussions in this forum about OCFS2.
You can use LVM on top of the LUN as written in the Proxmox documentation.
Or if you want to have something similar to VMFS you can use a cluster filesystem like OCFS2 on top of the LUN. But this is not officially supported by Proxmox. you have to setup it yourself. There are several discussions in this forum about OCFS2.
So let's say I have 1 x 10 tb lun. I can present this to all proxmox nodes? With OCFS2? Then it will only migrate the state's memory? I can have like 50 vms on proxmox node 1 and migrate all 50 vms to proxmox node 2 and it's only moving the state's memory? No replicating in the background for the disks.
According to this
https://pve.proxmox.com/wiki/Storage
Only Ceph is yes/yes/yes across the board.
Before I move forward just want to make sure OCFS2 is the best choice. Is that free? I want to get closely as I can to migrate off vmware because of the broadcomm mess. Or anything that I can do "BETTER" like I hear about ZFS self healing and does a good job repairing itself. If I can make it better that's what I want. However, I have to do ZFS different LUNs to every proxmox node and does replicating the virtual machines will I worry affects the performance.
We definitely need the snapshot option before patching or messing with servers.
I'm going to rehash one more time see if we can decide a good obvious solution
1. I am using LACP on all networks and I can segregate the traffic/vlans
2. I am using FC with storage over HBA. Yes it's definitely raided in the configuration on back end and I can present it to each proxmode. However when I presented it to the proxmox nodes and executed migration it migrated the entire disk over to the same LUN acting like it's entirely a different LUN. Was weird.
3. A filesystem in proxmox however I present the luns can do snapshots and migrate to another node without having replication for the disks just migrating the states memory over.
4. Any suggestions with file system that would include thin provisioning with snapshots capability?
4. Any suggestions would be greatly appreciated.
Thanks again.
According to this
https://pve.proxmox.com/wiki/Storage
Only Ceph is yes/yes/yes across the board.
Before I move forward just want to make sure OCFS2 is the best choice. Is that free? I want to get closely as I can to migrate off vmware because of the broadcomm mess. Or anything that I can do "BETTER" like I hear about ZFS self healing and does a good job repairing itself. If I can make it better that's what I want. However, I have to do ZFS different LUNs to every proxmox node and does replicating the virtual machines will I worry affects the performance.
We definitely need the snapshot option before patching or messing with servers.
I'm going to rehash one more time see if we can decide a good obvious solution
1. I am using LACP on all networks and I can segregate the traffic/vlans
2. I am using FC with storage over HBA. Yes it's definitely raided in the configuration on back end and I can present it to each proxmode. However when I presented it to the proxmox nodes and executed migration it migrated the entire disk over to the same LUN acting like it's entirely a different LUN. Was weird.
3. A filesystem in proxmox however I present the luns can do snapshots and migrate to another node without having replication for the disks just migrating the states memory over.
4. Any suggestions with file system that would include thin provisioning with snapshots capability?
4. Any suggestions would be greatly appreciated.
Thanks again.
Last edited:
For 4. you need OCFS2. It can be used by Proxmax as shared directory storage like NFS. With qcow2 for the VM images you get thin provisioning and snapshots.
But this setup is not officially supported by the company that makes Proxmox. You have to configure OCFS2 yourself.
But this setup is not officially supported by the company that makes Proxmox. You have to configure OCFS2 yourself.