Same issue here, weird thing is i have 2 nodes where one of them can properly use the qlcnic drivers and the other cant, with identical cards.
Same version of pve-firmware: 3.2-2
Not working one on the 5.4.106-1-pve kernel and the working one on 5.4.103-1-pve
Shows up on my 5.4.103-1-pve node...