Are you sure it is just after restart, or is it after shutdown + start?
The fast PBS backups are based on Qemu dirty bitmap, witch tracks the changes on virtual disk, if you stop your VM the bitmap is reset, so backup needs to reread the whole disk image again. Still backup does write only...