Recent content by Nathan Stratton

I have a VM with four exported 3090 GPUs. The GPUs work and I can run things like gpuburn, but when I try to train my models with NCCL I run into errors. I don't have a ACS option in bios (I believe its off now so no option) Supermicro H12SSL, but I do have IOMMU on so I can export the cards to...

So a bit more info, the 4 GPUs are one two x16 slots that are bifurcated into two x8 slots for each GPU. When I boot without the pcie_acs_override=downstream the first two cards are in Group 13 and the last two Group 49, so that wont work with 4 VMs each using 1 card. With the...

Sorry, your right I was not clear. It is still in the same state as the original post, I can start one or the other, but not both if they are not in the same VM. I am using pcie_acs_override, and see them each in a different group. /sys/kernel/iommu_groups/48/devices/0000:81:00.0...

Thanks, they are now in different groups (with pcie_acs_override), but without this they are not. Yes, plenty of RAM, its something with passthrough.

System Setup Proxmox 8.0.4 Supermicro H12SSL 1 Nvidia 4090 3 Nvidia 3080 Machine q35 virt101 - 3080 PCI Device 0000:02:00 virt103 - 4090 PCI Device 0000:01:00 I had virt 105 with two 3080s, PCI Device 0000:81:00 and 0000:82:00 Everything works great with this setup; I shut down 105, cloned...

Figured it out, you need to add a serial port. :)

Ever get past this? I am seeing the same issue.

root@virt01:/var/lib/ceph# pveceph createmon --monid virt01 --mon-address 10.0.0.101 monitor 'virt01' already exists

I am having a similar problem, I have proxmox sees a monitor, but it has been removed by ceph: root@virt01:/var/lib/ceph# pveceph mon destroy virt01 no such monitor id 'virt01' root@virt01:/var/lib/ceph# ceph mon remove virt01 mon.virt01 does not exist or has already been removed...

Yep, they are both on 49, going to try a few BIOS settings if that does not work I found: https://gitlab.com/Queuecumber/linux-acs-override But rather see if I can do this without a custom kernel.

I have 2 Nvidia 3080s, on PCI 0000:01:00.0 and 0000:02:00.0, if I put them both on a VM, with x-vga=on and multifunction=on, it works, I nvidia-smi shows two GPUs. However, if I start vm1 with one of the GPUs say 0000:01:00.0, it will start fine, if I then try to start vm2 with GPU 0000:02:00.0...

I currently have a 21 server ceph cluster with 105 OSDs and I need to enable ceph authentication because Kubernetes can't mount a ceph volume without auth! I have looked at: https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/ Is that the only way with proxmox, or is there an...

I also believe this is correct, I have not been able to create tagged and untagged in the same bridge.

I checked systemctl list-units on all hosts and don't see anything that does not belong, you mentioned directory, what should I look for there? Where does that GUI pannel pull its information?

Not sure if it is because I am in the middle of rebalancing, but when I restart OSDs I get a bunch of "active+undersized+degraded+remapped+backfill_wait:" PGs, so I stoped restarting them.