[SOLVED] Help: Network problem, not able to set same IP on new NIC

[kobemtl]

Before I start, I want say thank you for the community first. And I did search and tried the solutions from others before. But I still got problem. Here its:

My PVE host has 2 NICs before. The /etc/network/interfaces` config shows below.

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.0.141/24
        gateway 192.168.0.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address 192.168.0.142/24
        bridge-ports eno2
        bridge-stp off
        bridge-fd 0

I wanna try 10GBE recently and got some used equipment including Brocade ICX 6450-48P and 2 Intel 10GBE NICs. Plan is using 10GB on my PVE and PBS host. So installed both NICs on the hosts. No trouble Debian shows them right after. Then started changing `/etc/network/interfaces` based on what I read. My plan is replacing the old NIC by the new one. Here is what I tried. `enp2s0f0` and `enp2s0f1` is the 2 interfaces from new NIC. For my test only `enp2s0f1` is connected. I will use only this one for my test.

### Try out #1:
First I just tried to replace interface for `vmbr0` by `enp2s0f1`, its not working at all. Either `192.168.0.141` or `192.168.0.142` can not reach the host. The solution comes from this thread

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

iface enp2s0f0 inet manual

iface enp2s0f1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.0.141/24
        gateway 192.168.0.1
        bridge-ports enp2s0f1
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address 192.168.0.142/24
        bridge-ports eno2
        bridge-stp off
        bridge-fd 0

### Try out #2:
Then I tried to use different IP for `vmbr0`. And it works, I can reach both `192.168.0.151` and `192.168.0.142`.

...

auto vmbr0
iface vmbr0 inet static
        address 192.168.0.151/24
        gateway 192.168.0.1
        bridge-ports enp2s0f1
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet static
        address 192.168.0.142/24
        bridge-ports eno2
        bridge-stp off
        bridge-fd 0

Then I could login WebUI change the `vmbr0` IP back to `192.168.0.141`.
I was happy then I realized non of the VM's network is working.

### Try out #3:

Now I am thinking just keeping the old `vmbr0` and `vmbr1` and adding new `vmbr2`. Then somehow set the default interface using by Debian to `vmbr2`. Then later I can change all VMs to use `vmbr2`. This way everything works normally except I really can't find a way to set the default interface to `vmbr1`. I tried `ip route add default`, I tried putting the set the default gateway to `vmbr2`. None of them works. `iperf3` test still see the traffic going through `192.168.0.141`.


### Conclusion:
I don't what to do now. It will be best if anyone can tell me how can completely replace the old gigabit NIC. Otherwise a working solution to set all traffic through new `vmbr2` would be fine as well I guess. Thank you so much for reading the long post first. And I wish some one could help me on this.

PS: I am using `pvesh set /nodes/pve-dellr330/network` every time I change `/etc/network/interfaces`.