Can't start backup process on PBS

R

Rikkert

New Member
Jun 26, 2023
9
0
1
Netherlands
ezdomain.nl
Hi,

Im trying to configure PBS for my PVE. I added the Datastore and connected it to PVE.
But here comes the problem everytime i want to start the backup proces i get this error:

NFO: starting new backup job: vzdump 101 --mode snapshot --storage PBS --node pve --remove 0 --notes-template '{{guestname}}'
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2024-09-18 21:11:05
INFO: status = running
INFO: VM Name: EZdomain
INFO: include disk 'scsi0' 'EZdomainStorage:vm-101-disk-0' 463G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: snapshots found (not included into backup)
INFO: creating Proxmox Backup Server archive 'vm/101/2024-09-18T19:11:05Z'
ERROR: VM 101 qmp command 'backup' failed - backup register image failed: command error: EINVAL: Invalid argument
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 101 failed - VM 101 qmp command 'backup' failed - backup register image failed: command error: EINVAL: Invalid argument
INFO: Failed at 2024-09-18 21:11:05
INFO: Backup job finished with errors
TASK ERROR: job errors

Anyone who knows why?

Im running Proxmox Backup Server on a Raspberry Pi 5 with Docker.

Thanks.
 

Attachments

  • Screenshot 2024-09-18 at 21-57-01 raspberrypi - Proxmox Backup Server.png
    Screenshot 2024-09-18 at 21-57-01 raspberrypi - Proxmox Backup Server.png
    388 KB · Views: 7
  • Screenshot 2024-09-18 at 21-57-41 pve - Proxmox Virtual Environment.png
    Screenshot 2024-09-18 at 21-57-41 pve - Proxmox Virtual Environment.png
    4.4 KB · Views: 7
Rikkert said:
Im running Proxmox Backup Server on a Raspberry Pi 5 with Docker.
Click to expand...
Just to note this straight away, that is not an officially supported setup, as there is at the time of writing no official ARM support.

Rikkert said:
ERROR: VM 101 qmp command 'backup' failed - backup register image failed: command error: EINVAL: Invalid argument
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 101 failed - VM 101 qmp command 'backup' failed - backup register image failed: command error: EINVAL: Invalid argument
Click to expand...
What versions of PVE and PBS are involved here? Please post the output of pveversion -v and proxmox-backup-manager version --verbose. Best in code tags for better readability.
 
Last edited:
Thank you for your reply,
Chris said:
Just to note this straight away, that is not an officially supported setup, as there is at the time of writing no official ARM support.
Click to expand...
I know the setup is not supported, but i know it worked for other people.

Chris said:
What versions of PVE and PBS are involved here? Please post the output of pveversion -v and proxmox-backup-manager version --verbose. Best in code tags for better readability.
Click to expand...
Output for pveversion -v:

Code: 
proxmox-ve: 7.4-1 (running kernel: 5.15.102-1-pve)
pve-manager: 7.4-18 (running version: 7.4-18/b1f94095)
pve-kernel-5.15: 7.4-15
pve-kernel-5.15.158-2-pve: 5.15.158-2
pve-kernel-5.15.143-1-pve: 5.15.143-1
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4.3
libpve-apiclient-perl: 3.2-2
libpve-common-perl: 7.4-2
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.7
libpve-storage-perl: 7.4-3
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.7-1
proxmox-backup-file-restore: 2.4.7-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.7.4
pve-cluster: 7.3-3
pve-container: 4.4-7
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-4~bpo11+3
pve-firewall: 4.3-5
pve-firmware: 3.6-6
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.10-1
pve-xtermjs: 4.16.0-2
qemu-server: 7.4-6
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.15-pve1

Output for proxmox-backup-manager version --verbose:
Code: 
proxmox-backup                unknown              running kernel: 6.6.47+rpt-rp
proxmox-backup-server         3.2.7-1              running version: 3.2.7       
ifupdown2                     3.0.0-1.1                                         
libjs-extjs                   7.0.0-4                                           
proxmox-backup-docs           3.2.7-1                                           
proxmox-backup-client         unknown                                           
proxmox-mail-forward          unknown                                           
proxmox-mini-journalreader    1.4.0                                             
proxmox-offline-mirror-helper unknown                                           
proxmox-widget-toolkit        4.2.3                                             
pve-xtermjs                   5.3.0-3                                           
smartmontools                 7.3-1+b1                                         
zfsutils-linux                2.2.3-1~bpo12+1~rpt1
 
Rikkert said:
I made an ZFS mirror with proxmox.

Filesystem Type 1K-blocks Used Available Use% Mounted on
PBS zfs 1885863424 24960 1885838464 1% /mnt/datastore1
Click to expand...
Is this inside the docker? Does your kernel have ZFS support? Please explain a bit more, as I am not at all familiar with your setup.
 
Chris said:
Is this inside the docker? Does your kernel have ZFS support? Please explain a bit more, as I am not at all familiar with your setup.
Click to expand...
On the Raspberry Pi 5 is PiOS installed, I installed PBS via a docker and both docker, PiOS and the Pi itself support ZFS. The ZFS pool is working correctly.

But when i start the backup process it gives me the error above.

If you want specifiek logsor data please tell me.

I placed a screenshot withthe kernel version etc.

Task Viewer: Datastore BackupsEZ vm/101
Code: 
2024-09-19T11:05:37+01:00: starting new backup on datastore 'BackupsEZ' from ::ffff:192.168.2.1: "vm/101/2024-09-19T10:05:36Z"
2024-09-19T11:05:37+01:00: GET /previous: 400 Bad Request: no valid previous backup
2024-09-19T11:05:37+01:00: POST /fixed_index: 400 Bad Request: EINVAL: Invalid argument
2024-09-19T11:05:37+01:00: backup ended and finish failed: backup ended but finished flag is not set.
2024-09-19T11:05:37+01:00: removing unfinished backup
2024-09-19T11:05:37+01:00: TASK ERROR: backup ended but finished flag is not set.
 

Attachments

  • Screenshot 2024-09-19 at 14-07-59 raspberrypi - Proxmox Backup Server.png
    Screenshot 2024-09-19 at 14-07-59 raspberrypi - Proxmox Backup Server.png
    359 KB · Views: 4
Rikkert said:
On the Raspberry Pi 5 is PiOS installed, I installed PBS via a docker and both docker, PiOS and the Pi itself support ZFS. The ZFS pool is working correctly.
Click to expand...
But how are you sharing the ZFS dataset with the PBS inside the container? I suspect that layer is the root of your problem. Can you share some configs?
 
Rikkert said:
I have a Radxa Penta SATA Hat installed on my raspberry pi, so it has a sata controller with 4 sata ports.

What configs would you like?
Click to expand...
I mean how do you make the storage accessible to the PBS running inside the docker container? Is this a bind-mount, is this a docker volume, do you pass trough the block device? Please share your docker commands and or docker-compose files... Also, what docker image are you using?

Further, did you check that this issue is also present on a datastore not backed by the ZFS storage? Which version of ZFS is the kernel module and userspace tools?
 
Chris said:
I mean how do you make the storage accessible to the PBS running inside the docker container? Is this a bind-mount, is this a docker volume, do you pass trough the block device? Please share your docker commands and or docker-compose files... Also, what docker image are you using?

Further, did you check that this issue is also present on a datastore not backed by the ZFS storage? Which version of ZFS is the kernel module and userspace tools?
Click to expand...
I will share some config files i hope it is enough:

Dockerfile:
Code: 
# Gebruik een basis image zoals Debian
FROM debian:bookworm-slim

# Zet debconf in non-interactive modus om frontend issues te vermijden
ENV DEBIAN_FRONTEND=noninteractive

# Installeer de benodigde tools, inclusief supervisord en andere afhankelijkheden
RUN apt-get update && apt-get install -y supervisor wget gnupg lsb-release curl fonts-font-awesome libjs-mathjax gdisk lvm2 openssh-server postfix sg3-utils smartmontools

# Voeg de Proxmox repository toe (optioneel)
RUN echo "deb [signed-by=/usr/share/keyrings/proxmox-release-bookworm.gpg] http://download.proxmox.com/debian/pbs bookworm pbs-no-subscription" > /etc/apt/sources.list.d/proxmox.list

# Voeg de Proxmox GPG-sleutel toe
RUN wget -qO- http://download.proxmox.com/debian/proxmox-release-bookworm.gpg | gpg --dearmor -o /usr/share/keyrings/proxmox-release-bookworm.gpg

# Blokkeer het starten van services tijdens de installatie van pakketten
RUN echo '#!/bin/sh\nexit 101' > /usr/sbin/policy-rc.d && chmod +x /usr/sbin/policy-rc.d

# Kopieer de .deb pakketten naar de container
COPY *.deb /tmp/

# Installeer de Proxmox Backup Server pakketten, zonder services te starten
RUN apt-get update && dpkg -i /tmp/*.deb || apt-get install -f -y

# Verwijder policy-rc.d zodat supervisord de services kan beheren
RUN rm /usr/sbin/policy-rc.d

# Exposeer poort 8006 voor toegang tot de webinterface
EXPOSE 8006

# Maak de supervisor configuratie directory
RUN mkdir -p /etc/supervisor/conf.d

# Voeg het proxmox.conf bestand toe voor supervisord configuratie
COPY proxmox.conf /etc/supervisor/conf.d/proxmox.conf

# Start supervisord als het init-proces
CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/conf.d/proxmox.conf"]

The ZFS is mounted in the docker.

Code: 
datastore: BackupsEZ
        comment
        gc-schedule daily
        notification-mode notification-system
        path /mnt/datastore1

Docker image:
Code: 
REPOSITORY              TAG             IMAGE ID       CREATED       SIZE
proxmox-backup-server   latest          c4cb630fff47   2 days ago    530MB
debian                  bookworm-slim   4d3492ddbe3a   2 weeks ago   97.1MB

[
    {
        "Id": "fa93fa78225d15b5a8a19b4fd76821695c68406ea412756cedc1837b07ef0b0b",
        "Created": "2024-09-18T18:15:15.778527324Z",
        "Path": "/usr/bin/supervisord",
        "Args": [
            "-c",
            "/etc/supervisor/conf.d/proxmox.conf"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 3611,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2024-09-18T18:15:16.314157029Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:c4cb630fff47d74ad97ea2f4adfc013147bc938d0856f9882609a57d00b3061f",
        "ResolvConfPath": "/var/lib/docker/containers/fa93fa78225d15b5a8a19b4fd76821695c68406ea412756cedc1837b07ef0b0b/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/fa93fa78225d15b5a8a19b4fd76821695c68406ea412756cedc1837b07ef0b0b/hostname",
        "HostsPath": "/var/lib/docker/containers/fa93fa78225d15b5a8a19b4fd76821695c68406ea412756cedc1837b07ef0b0b/hosts",
        "LogPath": "/var/lib/docker/containers/fa93fa78225d15b5a8a19b4fd76821695c68406ea412756cedc1837b07ef0b0b/fa93fa78225d15b5a8a19b4fd76821695c68406ea412756cedc1837b07ef0b0b-json.log",
        "Name": "/proxmox-backup-server",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {
                "8006/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "8006"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "private",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/da4733412a75b82088c5a7ebb7d34542d8fac5af725c107e6cbe9f2a8a44e231-init/diff:/var/lib/docker/overlay2/19622fdd015883aa706dd61d46fdadf1ba6a7ac52ecd981f65e6b15ebb8aeada/diff:/var/lib/docker/overlay2/a021af762d54072abf5cea505ce6d646d12db5948ed830ca489e91f16aa50d13/diff:/var/lib/docker/overlay2/f7c445da9b2b208dff725b56b9dbbe13e64a94dec32a471c429f5afda3e2e16d/diff:/var/lib/docker/overlay2/4cfdbba07acb0d0f684c87d855bbb575a270fed23934b3b52970fa4e635a7e73/diff:/var/lib/docker/overlay2/f6ab9acfdf45a9e9892a046982de0e4c78fa5a949f081c6cb428b8f201a26c96/diff:/var/lib/docker/overlay2/260654b76390c1568b501d079a189ad021131959711bb9d7856bfe2e490a2568/diff:/var/lib/docker/overlay2/876eb33b1eba1d4af15252168ab5a54fca0ca6b619b48458cadd3d3af44bf349/diff:/var/lib/docker/overlay2/d5511ff268aed078f0a88800a91d741e390ea89b65b15d5f6aa22e9bf0dc6b56/diff:/var/lib/docker/overlay2/1a5edeafbff4dc4dd3f1bce2445939a581a0e2f49231e5cb1f552ef320cdd467/diff",
                "MergedDir": "/var/lib/docker/overlay2/da4733412a75b82088c5a7ebb7d34542d8fac5af725c107e6cbe9f2a8a44e231/merged",
                "UpperDir": "/var/lib/docker/overlay2/da4733412a75b82088c5a7ebb7d34542d8fac5af725c107e6cbe9f2a8a44e231/diff",
                "WorkDir": "/var/lib/docker/overlay2/da4733412a75b82088c5a7ebb7d34542d8fac5af725c107e6cbe9f2a8a44e231/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [],
        "Config": {
            "Hostname": "fa93fa78225d",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "8006/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "DEBIAN_FRONTEND=noninteractive"
            ],
            "Cmd": [
                "/usr/bin/supervisord",
                "-c",
                "/etc/supervisor/conf.d/proxmox.conf"
            ],
            "Image": "proxmox-backup-server:latest",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "29379b9faa4c37658ad61fbf6e360b045b1586efbd445fc749d9c9ec08946fcb",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "8006/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "8006"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "8006"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/29379b9faa4c",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "56b954832961c7778061eaccf1f1c9632029a58646b44e1d43d1a06bdc2570c5",
            "Gateway": "172.17.0.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.17.0.2",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:11:00:02",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "0e08b3a5b1396665d19ea5a178b7b72cc48eae4870299d13667816110b344524",
                    "EndpointID": "56b954832961c7778061eaccf1f1c9632029a58646b44e1d43d1a06bdc2570c5",
                    "Gateway": "172.17.0.1",
                    "IPAddress": "172.17.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:11:00:02",
                    "DriverOpts": null
                }
            }
        }
    }
]

ZFS version:

Code: 
zfs-2.2.3-1~bpo12+1~rpt1
zfs-kmod-2.2.3-1~bpo12+1~rpt1

ii  libzfs4linux                         2.2.3-1~bpo12+1~rpt1                arm64        OpenZFS filesystem library for Linux - general support
ii  zfs-dkms                             2.2.3-1~bpo12+1~rpt1                all          OpenZFS filesystem kernel modules for Linux
ii  zfs-zed                              2.2.3-1~bpo12+1~rpt1                arm64        OpenZFS Event Daemon
ii  zfsutils-linux                       2.2.3-1~bpo12+1~rpt1                arm64        command-line tools to manage OpenZFS filesystems
 
Rikkert said:
"Driver": "overlay2",
Click to expand...
According to this, you do not use the ZFS driver however, probably overlay2 does not support some features required by the PBS.
Also, it seems you are not using any bind mount or volume? Note that the docker containers are ephemeral, so any data which should be persisted must be written to such a persistent storage.

Rikkert said:
"PortBindings": { "8006/tcp": [ { "HostIp": "", "HostPort": "8006" } ] },
Click to expand...
On a side note: PBS used port 8007, not 8006
 
Chris said:
According to this, you do not use the ZFS driver however, probably overlay2 does not support some features required by the PBS.
Also, it seems you are not using any bind mount or volume? Note that the docker containers are ephemeral, so any data which should be persisted must be written to such a persistent storage.


On a side note: PBS used port 8007, not 8006
Click to expand...
Do you know how i can fix this?
 
You must log in or register to reply here.
Top Bottom