Ok solution, hopefully proxmox make this easier in the future.
edit /usr/sbin/proxmox-boot-tool
This is the tool (bash script) that writes the loader.conf which has the timeout.
line 159 sets the value.
run 'proxmox-boot-tool init' on EFI partition.
For me I had two boot drives with rpool on mirrored, so I did it on one only first, booted from it, then when confirmed ok did the other, for me its sda2 and sdb2, you can confirm in the disks on the node in the proxmox UI or something like fdisk.
Only init writes the value so I assume even with an update to proxmox which edits the script back it wont matter as the refresh command wont rewrite the value.