Help for another noob! How do I add a new pcie?

[CancunManny]

I have an old computer set up with proxmox running homeassistnat, frigate and OPNsense. All working fine!

Today I finally got my two pcie adapters for my TPU dual coral chips. Figured it would be an easy plug and play thing, but far from the truth.

When I plug in one of the pcie adapters with the corals, this is what I've noticed happens thus far:

1) If I connect a monitor to the actual server, after boot it says to access proxmox visit "https://10.0.0.189:8006", but turns out that is the address given when I was using the TP link router (before installing OPNsense). Following instructions via google searches, I have switched the static new IP of proxmox to the new OPNsense IP. And in fact once everything is done booting properly I am able to access via theh OPNsense IP address. The current address of proxmox is a 10.69.x.x IP. I guess when everything working fine I wouldn't have a monitor connected to the proxmox itself, so I wouldn't see the "https://10.0.0.189:8006" address, and everything would work fine. However would like to know if there is a way to update that IP address? (This happens with the PCIe coral adapter on or off. So basically when the system boots it defaults to the original IP it got)

2) If I connect a new PCIe card, the PCIe card I already have set up and configured (two ethernet ports) stop responding. Basically if I bootup with the new PCIe card attached I lose 100% connectivity to proxmox, except via the attached monitor and keyboard. I found this post https://forum.proxmox.com/threads/help-with-adding-new-m-2-ssd-pcie-drive.121144/ Which basically says when a new pcie card is installed it changes the names of the NIC names. Unfortunately I am too much of a noob to understand what I should do. I think i understand what is going on, but after running ip addr not sure what I am looking for, nor am I sure which is the network config file

I am confident many have ran into this issue and that there are a few posts on how to resolve the issue, i just seem to be missing the keywords to do a good search. When I search for "add pcei to proxmox" pretty much all the results I get are about pass-through. If I am understanding correctly I first need to be able to boot up my machine with both the coral TPU pcie and the 2 Ethernet port PCIe ports being recognized by the system before I can do any passthrough to a specific VM correct?

As it stands, if i take out the pcie with coral and boot, everything works as expected. My home assistant, frigate, and OPNsense all work great. How can I add a PCIe with the coral chip to then passthrough to frigate?

 

[CancunManny]

So I continued playing around last night and this morning. Here is some more info that might be helpful for someone to help me out.

The motherboard has an integrated ethernet port. On my original setup that was the only NIC available (vmbr0). I first installed proxmox, then home assistant, then frigate. I am guessing all 3 of these services are using the vmbr0 to connect.

I then installed a two NIC PCIe, (vmbr1 and vmbr2). I forget which is which, but on one of those ports I connect to my modem (internet in), and the other NIC I connect to an access point/switch.

I now want to install a new PCIe with coral TPU for frigate. Last night I put it in the available PCI slot but when I boot both the integrated NIC and the two other NICs via PCIe don't show any activity (lights don't blink at all). I am unable to access the machine so i am forced to do a shutdown via connected keyboard and monitor to the server.

If I pull the new PCIe out and boot, everything works as expected. Today as I am running out of things to try I swapped the PCIe slots. I put the NIC where I had the Coral TPU and vice versa. This time when I boot the PCIe with the NICs does show activity. In fact after a full boot OPNsense works as expected and does assign IPs to connected devices. Now it seems the integrated NIC is the only one that isn't responding. I can't access proxmox web gui nor homeassistant.

I am guessing that when I boot with the PCIe for the TPU connected it somehow blocks the integrated NIC from running, and since both the homeassistant and proxmox web use that NIC to get IP address they don't work.

I feel I am so close yet so far! Can anyone point me in the right direction?

 

[Dunuin]

When adding or removing PCIe devices the names of your NICs will change and then won't match your old network config any longer. So when adding PCIe devices check the new interfaces names with ip addr and then edit your network config with nano /etc/network/interfaces changing the old NIC names to the new ones and restart networking with systemctl restart networking to get PVEs network working again.

 

[CancunManny]

When adding or removing PCIe devices the names of your NICs will change and then won't match your old network config any longer. So when adding PCIe devices check the new interfaces names with ip addr and then edit your network config with nano /etc/network/interfaces changing the old NIC names to the new ones and restart networking with systemctl restart networking to get PVEs network working again.

Thank you. You gave a very similar response here https://forum.proxmox.com/threads/help-with-adding-new-m-2-ssd-pcie-drive.121144/ which is what has gotten me thus far.

After doing some research I was able to find /etc/network/interfaces which would be the file I should update correct? However not sure what i am supposed to change. Do you know of a post that goes more into details?

(edit to make things even a bit more frustrating for me, I only have one old old VGA monitor that randomly turns on/off (bad monitor). I can't use the onboard HDMI as after it starts booting screen goes blue. )

 

[Dunuin]

After doing some research I was able to find /etc/network/interfaces which would be the file I should update correct?

Yes.

? However not sure what i am supposed to change. Do you know of a post that goes more into details?

Show us the output of ip addr and cat /etc/network/interfaces and we can tell you what re replace.

 

[CancunManny]

So I figured it out! Seems I am up and running.

So the integrated NIC was enp4s0 and with new PCIe card it became enp8s0 .. On previous post Dunuin had mentioned there were at least two spots on the config file where the NIC name had to be updated.

First change is on the iface enpxxx inet manual and the second change is on the iface vmbrX bridge ports.

Ty so much for your help Dunuin!

 

[CancunManny]

Yes.


Show us the output of ip addr and cat /etc/network/interfaces and we can tell you what re replace.

Yeah showing you the outputs would have been TRICKY as hell since only way I had to run ip addr was via the keyboard and monitor connected directly to the server. Would have had NO IDEA on how to copy and paste the results I got. I did copy the network file when I booted "without" the TPU PCIe card, so when I ran ip addr with the card installed I was able to see the one name change.

 

[Dunuin]

Taking a photo via smartphone is usually a good way. Always better to have actual text output but if ssh into the server isn't possible pictures are fine too.
But good to here that it is working now.

 

[CancunManny]

Taking a foto via smartphone is usually a good way. Always better to have actual text output but if ssh into the server isn't possible pictures are fine too.
But good to here that it is working now.

See, something so basic I wouldn't have thought about myself, just take a damn picture!

Now trying to figure out how to do the passthrough. The good news is that it seems the system is recognizing the two coral chips. If I go to OPNSense or HA VM's, if I want to "add hardware" if I select "raw" I am able to see two Coral Tpu's on the list. But since it seems I always end up going the more difficult route, my Frigate is running on a container, so from what I understand I won't be able to add the coral's via a GUI, it will have to be done via command line and updating files.

Hopefully I am on the last hurdle! Again thank you for your help

 

[Dunuin]

Best you search the forum for "coral". There are tons of threads discussing problems with those coral accelerators.