Need help with my ethernet card

[warloxian]

I was following a video tutorial to install pfsense on one of my severs in cluster. I never finished the install because I lost network communication. I am a NOOB and this is kicking my butt. I was passing through my ports, which i don't fully understand yet, after I passed through all 4 ports I lost connection to LAN and WAN, i hooked keyboard up to the server, deleted the pfsense VM that I was passing through on, because I assumed this would restore my network connection, WRONG!! SO when I run ip addr I only see l0 and vmbr0, brctl tells me that enp3s0f0 doesn't exist. 4 days, 200 tabs open and still have-not found a way to restore my ethernet ports. Unfortunately, because I have not had anything on my servers that I couldn't replace easily, I did not have any back ups of my system. I thought about just reinstalling Proxmox 7.0 on this server, but if I do things that way then I'm not learning. The ports are on the motherboard (HP dl380G7). I hate asking for help, but when I've spent days and hours trying to just find a way to restore my ports. I thought I should ask and learn. If I just reinstall the system them I feel like a modern computer repair technician that just replaces parts and never does a real diagnosis. When I have most failures on a regular PC I do a component level diagnosis , if I need a capacitor then I replace a capacitor, not the entire board. If I reinstall Proxmox, then I have replaced the whole board, so to speak. Please , suggestions? If someone could also explain why this happened then I have a full understanding and I can prevent this from happening again
Thanks for giving me a small piece of your valuable time
Warlox

 

[leesteken]

Please allow me to ask some questions to make sure I understand your situation. You have 4 ethernet ports (and four network devices?) on your system? Do did a PCI passthrough of all ethernet ports and/or network devices to a VM with PFSense installed? If all network devices are passed to a VM, then the host does not have a network device anymore, as expected. You'll need to route your traffic to and from the Proxmox host via PFSense, which will require changes to the Proxmox host network configuration (/etc/network/interfaces) as well as PFSense (which I have no experience with). Such a configuration is possible (I used to have it like that years ago) but not automatic, and it rquires that the Proxmox host and PFSense VM are both connected to the vmbr0.

If you want your previous network setup back, can't you just undo the PCI passthrough and reboot? Maybe I'm missing something here. Can you share the output of cat /etc/network/interfaces and lspci -k and the VM configuration from /etc/pve/qemu-server/?

 

[Dunuin]

To make it clear:
If you passthrough any device to a VM only that single VM will be able to use that device. Neighter the host nor any other guest can use it anymore.

So like avw already said you now got a host that has no physical NIC so you would need to use virtual NICs and let your pfsense VM route between your physical switch and your host.

 

[warloxian]

That makes sense to me, the problem is that once I lost connectivity I was not abkle to get back into proxmox to finish setting up pfsense. So i hooked up the keyboard and monitor to the server, I manually deleted the pfsense vm, hoping to undo the pass-through. Now I cant get this servr to connect to anything. I have tried every method I can find to no avail. It sound slike I need to either go into BIOS and make some changes there? or just reinstall proxmox on this server? I was just at the beginning of setting up my home lab. I still hate throwing this extreme repair at a problem I think can be solved another way. I guess i am trying to understand pass-through. If I configured pass-through on the pfsense vm, why , if I may ask, when i deleted the VM didnt my server just restore the system and throw out the configuration that was attached to the VM? If those changwes are made at the BIOS level shouldn't I be able to fix it in BIOS? and if those changes were purely a software configuration, why weren't those setting purged when I deleted the VM? I really do apologize about the NOOB questions. I rarely ask for help as tyou can see, but I am so close to being able to use my cyber range Ive been working on that I want to learn how this works so I can work this problem out, instead of starting over on this server? I know your time is valuable and I really do appreciate you taking a small amount of that time to help me.

 

[leesteken]

You are not sure if you did PCI passthrough or not? Did you make changes or add files to /etc/modprobe.d/? Did you add kernel parameters to /etc/default/grub or /etc/kernel/cmdline? Those changes will remain after you remove the VM. No changes to the configuration of Proxmox (or the underlying Debian GNU/Linux) will change your BIOS settings. What changes to did you make in the BIOS? Can you show any configuration files, using photos if necessary?

Sorry for being blunt, but if you don't know what you changed or cannot share configuration files for us to see what might be wrong, it is indeed hard to troubleshoot. Since you already deleted the PFSense VM, I guess there is no point in trying to route your network via that VM, and you might as well start from scratch and reinstall Proxmox to get a "known baseline".

You're posting a lot of text but very little concrete information (such as specific configuration files), which is understandable because you might not yet know what information is of critical importance. I might be asking a lot of your patience, but please believe me that it's only because I need to understand the exact condition your system is in before I can give specific suggestion to throubleshoot or fix things. Maybe starting over is a good idea and you could explain about what you want to achieve and people here can comment more clearly on how to achieve that (on a working system)?

 

[Dunuin]

I'm also not sure if it is in general a good idea to passthrough all of your NICs. Like already said your PVE webUI and also any network connections will only work indirectly through your pfsense VM. So if whyever there would be a problem with this VM you are basically locked out and can't use SSH/webUI to fix it or to restore a working backup. I personally would:
1.) only passthrough 3 of the 4 NICs so your host can still use 1 NIC for management and maybe internet access and cluster communication or
2.) you dont passthrough NICs at all and just use 4 bridges and virtio virtual NICs. Shouldn't be much slower if you only got Gbit and your host and pfsense VM (and other VMs too) could share these 4 NICs.

 

[warloxian]

I'm not 100% what I did either, I was following this video https://www.youtube.com/watch?v=hdoBQNI_Ab8 and at the 3:31 second mark , he said lets add all of them, so I did. I then lost communication , enp3s0f0, enp3s0f1, enp4s0f0 and enp4sof1. Now all tests say that device doesn't exist, but when I run lshw it shows my ports , all ports are on motherboard , I have added two usb ethernet adapters and can ping google.com, but not 8.8.8.8, and I can update and install from cli now. but I can't access from network, only cli with keyboard and monitor attached to the server, So the output of lspci -k is quit long . I will get better specifics tomorrow and post because I am headed to work right now, but maybe the video will help clear u[ what I did. I have come a long way in 1 year from knowing nothing about Linux, but servers , VM's are still quit foreign to me .

 

[leesteken]

Indeed he did PCI passthrough and set the VM to automatically start at every boot of the host. Please don't do that without testing it first!
If you passed all ethernet devices to that VM, your host won't have any network connection. Also, the VM has no connection to the vmbr0, so routing your host network traffic via that VM is not possible.
I did not see him making any changes to the Proxmox host that would have any effect when not starting that VM. Are you sure you deleted that VM? Otherwise remove the hostpci settings and set onboot: 1 to onboot: 0.

EDIT1: Did you make multiple VMs maybe? Please also check all other VMs configuration files for hostpci settings in the directory /etc/pve/qemu-server/.
EDIT2: The lspci -k is only for checking if the vfio-pci driver is used for the ethernet devices (which means they are passed through). If another driver is used, then the problem is somewhere else.