Unexpected backup failures

[vaschthestampede]

How reproducible is the issue?

It's been happening every day lately.

You could try to check if using a local datastore not backed by iSCSI also produces the timeout errors.

I can't; the locally attached disks will be used for other backups.
I'll be able to check if those fail as well.

What I can tell you is that it's not the first time something like this has happened but on much less performing hardware and I blamed that.
And I didn't use iSCSI there, the drivers were local.

 

[fiona]

Hi @vaschthestampede,
do you have IO thread enabled for your VM disks? If not, it's highly recommended to do so. It's also highly recommended to not start backups from all nodes at the very same time, since PBS might get overloaded with handling the initial setup for each at the same time. Is the network for the storage and for PBS separate? It's also recommended to be.

 

[vaschthestampede]

do you have IO thread enabled for your VM disks?

Yes, It's my practice to have it turned on practically all the time.

Is the network for the storage and for PBS separate?

For PBS I have the network for backup and the one for iSCSI, two separate NICs.
For PVE I only have one ceph cluster and it has completely separate networks and NICs.

 

[vaschthestampede]

It's also highly recommended to not start backups from all nodes at the very same time, since PBS might get overloaded with handling the initial setup for each at the same time.

My requirement is to have all the backups at that time.
In fact, the goal is to have all the backups of all the VMs at the same time!

 

[vaschthestampede]

I have a new item to analyze, in the meantime there are backups that fail the metrics are not being sent.