[SOLVED] Backup fails with exit code 255

K

krikey

Well-Known Member
Aug 15, 2018
115
21
58
UK
I'm trying to create a backup of a CT using a new PBS. The PBS Storage is shown correctly in the Proxmox interface, so I assume that I have entered the correct credentials, however when trying to run a manual backup to the PBS I get a 255 exit code (see below with personal details redacted for security):

INFO: starting new backup job: vzdump 102 --remove 0 --mode snapshot --node nodename --storage PBS
INFO: Starting Backup of VM 102 (lxc)
INFO: Backup started at 2020-09-16 18:30:48
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: CT Name: test
INFO: including mount point rootfs ('/') in backup
INFO: creating Proxmox Backup Server archive 'ct/102/2020-09-16T17:30:48Z'
INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp41253/etc/vzdump/pct.conf fw.conf:/var/tmp/vzdumptmp41253/etc/vzdump/pct.fw root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --backup-type ct --backup-id 102 --backup-time 1600277448 --repository root@pam@IPaddress:backup
INFO: Starting backup: ct/102/2020-09-16T17:30:48Z
INFO: Client name: nodename
INFO: Starting protocol: 2020-09-16T18:30:48+01:00
INFO: Error: parameter verification errors
INFO: parameter 'benchmark': schema does not allow additional properties.
ERROR: Backup of VM 102 failed - command '/usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp41253/etc/vzdump/pct.conf fw.conf:/var/tmp/vzdumptmp41253/etc/vzdump/pct.fw root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --backup-type ct --backup-id 102 --backup-time 1600277448 --repository root@pam@IPaddress:backup' failed: exit code 255
INFO: Failed at 2020-09-16 18:30:48
INFO: Backup job finished with errors
TASK ERROR: job errors
 
krikey said:
INFO: parameter 'benchmark': schema does not allow additional properties.
Click to expand...
please upgrade both PVE and PBS to the latest available versions - this has been fixed recently

I hope this helps!
 
Stoiko Ivanov said:
please upgrade both PVE and PBS to the latest available versions - this has been fixed recently

I hope this helps!
Click to expand...
Hello,

we updated both, the PVE and the PBS to the latest versions, but still get this err, code 255 (client details redacted to *** for privacy):

Code: 
INFO: starting new backup job: vzdump --prune-backups 'keep-last=3' --storage *** --node *** --all 1 --mode snapshot --notes-template '{{guestname}}' --mailnotification always --mailto [***]
INFO: Starting Backup of VM 222 (lxc)
INFO: Backup started at 2023-01-17 16:05:58
INFO: status = running
INFO: CT Name: [***]
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
INFO: creating Proxmox Backup Server archive 'ct/222/2023-01-17T15:05:58Z'
INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp2955704_222/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 222 --backup-time 1673967958 --repository ***@pbs@10.1.0.0:*** --ns Root
INFO: Starting backup: [Root]:ct/222/2023-01-17T15:05:58Z
INFO: Client name: ***
INFO: Starting backup protocol: Tue Jan 17 16:05:58 2023
INFO: storing login ticket failed: $XDG_RUNTIME_DIR must be set
INFO: Error: namespace not found
INFO: cleanup temporary 'vzdump' snapshot
ERROR: Backup of VM 222 failed - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp2955704_222/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 222 --backup-time 1673967958 --repository ***@pbs@10.1.0.0:*** --ns Root' failed: exit code 255
INFO: Failed at 2023-01-17 16:05:58
INFO: Starting Backup of VM 311 (lxc)

[err repeats for ALL CT's]

INFO: Failed at 2023-01-17 16:06:09
INFO: Backup job finished with errors
TASK ERROR: job errors

Code: 
INFO: storing login ticket failed: $XDG_RUNTIME_DIR must be set
Is the the cause?

Cheers,
~R.
 
Last edited:
  • Like
Reactions: Riesling.Dry
dcsapak said:
seems either the namespace does not exist or the user has no access to it
Click to expand...
namespace exist, we tried user "DatastoreAdmin" and "Admin" (=full rights)

But it indeed seems to be about permissions, as we cannot even list content of storage (PBS) on node (PVE):
Code: 
error listing snapshots - 400 Bad Request (500)

Does the node (PVE) need a User w. Permissions to write to storage (PBS)?
 
Last edited:
RollMops said:
Does the node (PVE) need a User w. Permissions to write to storage (PBS)?
Click to expand...
YES! We had to add an extra LOCAL Proxmox-VE-PAM User to PVE for the backup jobs - Linix-PAM-User "root" does NOT work here (and can't even be added).
Worx now :)
 
Last edited:
RollMops said:
YES! We had to add an extra LOCAL Proxmox-VE-PAM User to PVE for the backup jobs - Linix-PAM-User "root" does NOT work here (and can't even be added).
Click to expand...
i don't completely understand, where did you need to add a user? on the PVE side? that should really not be necessary...
and what do you mean with 'can't even be added' ?
 
dcsapak said:
i don't completely understand, where did you need to add a user? on the PVE side?
Click to expand...

Yes, on the PVE in Datacenter > Permissions > Users

dcsapak said:
that should really not be necessary...
Click to expand...

Dunno, but after we did, it works :)

dcsapak said:
and what do you mean with 'can't even be added' ?
Click to expand...

as far as I remember, it is not possible to add PVE User root to w. Linux PAM access (which is the only user on this particular PVE) to access the PBS?
 
i don't completely understand what you mean, in general it works like this:

to add a PBS storage in PVE you must enter the credentials of a avalid user of PBS into the edit window in PVE
that user must be configured in PBS with the correct permissions to access the given datastore

so the pve users don't really matter at all there, except the user you are logged in with in pve must have the correct pve permissions to add a storage
 
dcsapak said:
to add a PBS storage in PVE you must enter the credentials of a avalid user of PBS into the edit window in PVE
that user must be configured in PBS with the correct permissions to access the given datastore
Click to expand...

That is exactly what we did, but we got the errs mentioned in this thread...

dcsapak said:
so the pve users don't really matter at all there, except the user you are logged in with in pve must have the correct pve permissions to add a storage
Click to expand...

We added storage as root, but things didnt work - see descr. above - after we added a local user to PVE, things work...
 
Riesling.Dry said:
That is exactly what we did, but we got the errs mentioned in this thread...



We added storage as root, but things didnt work - see descr. above - after we added a local user to PVE, things work...
Click to expand...
I'm having the same issue. I wonder if you could be a bit more specific as to what you exactly done so I can replicate. Thank you so much, Jens
 
XPOST

I'm facing the same "failed: exit code 255" error, from 24 LXCs, only one backup task is failing. I have this setup:
  • I've got PVE and PBS running on the same host.
  • I backup to a *brand new* 4TB Seagate BarraCuda HDD in an external (powered) USB3 enclosure

Code: 
>#pveversion -v
proxmox-ve: 7.4-1 (running kernel: 5.15.131-2-pve)
pve-manager: 7.4-17 (running version: 7.4-17/513c62be)
pve-kernel-5.15: 7.4-9
pve-kernel-5.15.131-2-pve: 5.15.131-3
pve-kernel-5.15.131-1-pve: 5.15.131-2
pve-kernel-5.15.74-1-pve: 5.15.74-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4.1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.4-2
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.7
libpve-storage-perl: 7.4-3
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.4-1
proxmox-backup-file-restore: 2.4.4-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.7.3
pve-cluster: 7.3-3
pve-container: 4.4-6
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-4~bpo11+1
pve-firewall: 4.3-5
pve-firmware: 3.6-6
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-2
qemu-server: 7.4-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.14-pve1

Code: 
proxmox-backup-client v2.4.4-1
proxmox-backup-server v2.4.4-1

Here my logs:
Code: 
INFO: starting new backup job: vzdump 212 --notes-template '{{guestname}}, {{node}}, {{vmid}}' --node tundra --remove 0 --mode stop --storage prxmxbs
INFO: Starting Backup of VM 212 (lxc)
INFO: Backup started at 2024-01-09 10:31:34
INFO: status = running
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: CT Name: bazarr
INFO: including mount point rootfs ('/') in backup
INFO: stopping virtual guest
INFO: creating Proxmox Backup Server archive 'ct/212/2024-01-09T15:31:34Z'
INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp507409_212/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 212 --backup-time 1704814294 --repository user@pbs@192.168.99.51:pbs
INFO: Starting backup: ct/212/2024-01-09T15:31:34Z
INFO: Client name: tundra
INFO: Starting backup protocol: Tue Jan  9 10:31:37 2024
INFO: No previous manifest available.
INFO: Upload config file '/var/tmp/vzdumptmp507409_212/etc/vzdump/pct.conf' to 'user@pbs@192.168.99.51:8007:pbs' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'user@pbs@192.168.99.51:8007:pbs' as root.pxar.didx
INFO: catalog upload error - channel closed
INFO: Error: error at "usr/share/i18n/charmaps/CP1258.gz": Input/output error (os error 5)
INFO: restarting vm
INFO: guest is online again after 17 seconds
ERROR: Backup of VM 212 failed - command '/usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp507409_212/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 212 --backup-time 1704814294 --repository user@pbs@192.168.99.51:pbs' failed: exit code 255
INFO: Failed at 2024-01-09 10:31:51
INFO: Backup job finished with errors
TASK ERROR: job errors

Code: 
Jan 09 03:05:41 tundra pvescheduler[340086]: ERROR: Backup of VM 212 failed - command '/usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp340086_212/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 212 --backup-time 1704787523 --repository user@pbs@192.168.99.51:pbs' failed: exit code 255

Code: 
Jan 09 03:05:39 prxmxpbs proxmox-backup-proxy[179]: successfully added chunk 9215ed90d568f2ae4c4d3142fb177736d0187d44ecb319f89f2533bb8014c6e9 to dynamic index 2 (offset 1127777112, size 4217411)
Jan 09 03:05:40 prxmxpbs proxmox-backup-proxy[179]: backup ended and finish failed: backup ended but finished flag is not set.
Jan 09 03:05:40 prxmxpbs proxmox-backup-proxy[179]: removing unfinished backup
Jan 09 03:05:40 prxmxpbs proxmox-backup-proxy[179]: TASK ERROR: backup ended but finished flag is not set.
Jan 09 03:05:40 prxmxpbs proxmox-backup-[179]: prxmxpbs proxmox-backup-proxy[179]: removing backup snapshot "/media/usb-hdd/pbs/ct/212/2024-01-09T08:05:23Z"

I have looked into this file's timestamp and attributes, but could see anything out of place:
INFO: catalog upload error - channel closed
INFO: Error: error at "usr/share/i18n/charmaps/CP1258.gz": Input/output error (os error 5)

Code: 
>/usr/share/i18n/charmaps#stat CP1258.gz
  File: CP1258.gz
  Size: 2728          Blocks: 8          IO Block: 4096   regular file
Device: fd0ah/64778d    Inode: 6765        Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-01-08 12:12:43.492673572 -0500
Modify: 2023-10-02 15:22:57.000000000 -0500
Change: 2024-01-08 13:06:11.644043667 -0500
 Birth: 2023-10-04 06:50:04.977858418 -0500

for comparison
>/usr/share/i18n/charmaps#stat CP1257.gz
  File: CP1257.gz
  Size: 2813          Blocks: 8          IO Block: 4096   regular file
Device: fd0ah/64778d    Inode: 6764        Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-01-08 12:12:43.492673572 -0500
Modify: 2023-10-02 15:22:57.000000000 -0500
Change: 2023-10-04 06:50:05.369860211 -0500
 Birth: 2023-10-04 06:50:04.977858418 -0500

Any help is greatly appreciated!
 
StackIOI said:
I'm facing the same "failed: exit code 255" error, from 24 LXCs, only one backup task is failing. I have this setup:
Click to expand...

it's not really the same error, but only the same symptom (the backup fails)

StackIOI said:
I have looked into this file's timestamp and attributes, but could see anything out of place:
INFO: catalog upload error - channel closed
INFO: Error: error at "usr/share/i18n/charmaps/CP1258.gz": Input/output error (os error 5)
Click to expand...
can you read that file without an error? e.g. do a 'cat /path/to/file > /dev/null'

input/output errors are normally hardware or filesystem corruptions
 
dcsapak said:
it's not really the same error, but only the same symptom (the backup fails)


can you read that file without an error? e.g. do a 'cat /path/to/file > /dev/null'

input/output errors are normally hardware or filesystem corruptions
Click to expand...

@dcsapak thanks so much for the reply.. I was focusing on the file status and missed the input/output side of the error... I fixed this by reinstalling locales, here the step-by-step, following your suggestion:

  1. sudo cat CP1258.gz > /dev/null
    cat: CP1258.gz: Input/output error

  2. sudo apt update
    sudo apt install --reinstall locales

  3. sudo cat CP1258.gz > /dev/null
    no error this time

  4. triggered a backup, and was done successfully
Thanks again
 
  • Like
Reactions: dcsapak
just fyi, such errors can indicate a hardware problem, so i'd check the health of the underlying storage (e.g. smart output, syslogs, etc.)
 
Yes, thank you, I went over the smartctl tests/outputs and no errors were detected... will take a peek at the syslog just to make sure...
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back