Apperently the packets still go through. Here's the output of tcpdump -i vmbr0 port 8006 when the web UI is not working:
11:47:43.060004 IP 192.168.1.122.47842 > 192.168.1.20.8006: Flags [S], seq 1334214748, win 64240, options [mss 1460,sackOK,TS val 2141245646 ecr 0,nop,wscale 7], length 0...