Backup job error 'gzip: stdout: No space left on device'

ihasamoose · Jan 5, 2021

Hey all,

I'm having some issues with my backup job. I have a dedicated HDD I'm trying to use for backups. I formatted the disk to ext4, created a directory for that disk, and entered parameters in fstab. Then I mounted it and set up the backup job once I added the directory to my datacenter. I set it to only contain VZdump files.

My backup job consists of backing up three VM's. One of them backs up fine, the other two fail.

Here is df -h (SDA being my backup drive)

Code:

root@local:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
udev                   16G     0   16G   0% /dev
tmpfs                 3.2G   26M  3.1G   1% /run
/dev/mapper/pve-root  7.6G  3.3G  4.0G  46% /
tmpfs                  16G   43M   16G   1% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
tmpfs                  16G     0   16G   0% /sys/fs/cgroup
samsung-ssd            89G  128K   89G   1% /samsung-ssd
nvme                  252G  128K  252G   1% /nvme
/dev/fuse              30M   20K   30M   1% /etc/pve
/dev/sda              3.6T  2.6G  3.4T   1% /backup
tmpfs                 3.2G     0  3.2G   0% /run/user/0

Code:

INFO: starting new backup job: vzdump 100 102 103 --quiet 1 --mailnotification always --compress gzip --mode snapshot --storage backups
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2021-01-01 03:00:01
INFO: status = running
INFO: VM Name: NextCloud
INFO: include disk 'scsi0' 'nvme:vm-100-disk-0' 500G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/backups/dump/vzdump-qemu-100-2021_01_01-03_00_01.vma.gz'
INFO: started backup task '64e41186-2d52-44cc-8eac-c35b7ce64b71'
INFO: resuming VM again
INFO:   0% (195.8 MiB of 500.0 GiB) in  3s, read: 65.2 MiB/s, write: 21.1 MiB/s
INFO:   1% (6.0 GiB of 500.0 GiB) in  1m 10s, read: 88.3 MiB/s, write: 21.9 MiB/s
INFO:   2% (10.4 GiB of 500.0 GiB) in  3m 18s, read: 35.7 MiB/s, write: 20.4 MiB/s
INFO:   3% (15.3 GiB of 500.0 GiB) in  3m 23s, read: 992.7 MiB/s, write: 53.6 KiB/s
INFO:   4% (20.1 GiB of 500.0 GiB) in  3m 28s, read: 996.7 MiB/s, write: 52.8 KiB/s
INFO:   5% (25.0 GiB of 500.0 GiB) in  3m 33s, read: 997.7 MiB/s, write: 2.4 KiB/s
INFO:   6% (30.7 GiB of 500.0 GiB) in  3m 39s, read: 974.2 MiB/s, write: 891.3 KiB/s
INFO:   7% (35.1 GiB of 500.0 GiB) in  3m 56s, read: 266.2 MiB/s, write: 25.2 MiB/s
INFO:   8% (40.2 GiB of 500.0 GiB) in  4m  5s, read: 573.4 MiB/s, write: 15.3 MiB/s
INFO:   9% (45.6 GiB of 500.0 GiB) in  4m 11s, read: 925.3 MiB/s, write: 3.0 MiB/s
INFO:  10% (50.4 GiB of 500.0 GiB) in  4m 16s, read: 990.7 MiB/s, write: 1.6 KiB/s
INFO:  11% (55.3 GiB of 500.0 GiB) in  4m 21s, read: 992.0 MiB/s, write: 2.4 KiB/s
INFO:  12% (60.1 GiB of 500.0 GiB) in  4m 26s, read: 980.3 MiB/s, write: 429.6 KiB/s
INFO:  13% (65.9 GiB of 500.0 GiB) in  4m 32s, read: 1000.4 MiB/s, write: 2.0 KiB/s
INFO:  14% (70.8 GiB of 500.0 GiB) in  4m 38s, read: 834.2 MiB/s, write: 1.3 KiB/s
INFO:  15% (75.7 GiB of 500.0 GiB) in  4m 43s, read: 997.1 MiB/s, write: 2.4 KiB/s
INFO:  16% (80.3 GiB of 500.0 GiB) in  4m 48s, read: 939.1 MiB/s, write: 2.8 MiB/s
INFO:  17% (85.2 GiB of 500.0 GiB) in  4m 53s, read: 998.4 MiB/s, write: 2.4 KiB/s
INFO:  18% (90.0 GiB of 500.0 GiB) in  4m 58s, read: 1001.4 MiB/s, write: 1.6 KiB/s
INFO:  19% (95.9 GiB of 500.0 GiB) in  5m  4s, read: 992.2 MiB/s, write: 55.3 KiB/s
INFO:  20% (100.6 GiB of 500.0 GiB) in  5m  9s, read: 971.4 MiB/s, write: 1.0 MiB/s
INFO:  21% (105.8 GiB of 500.0 GiB) in  5m 19s, read: 531.9 MiB/s, write: 25.8 MiB/s
INFO:  22% (110.7 GiB of 500.0 GiB) in  5m 24s, read: 995.3 MiB/s, write: 1.6 KiB/s
INFO:  23% (115.5 GiB of 500.0 GiB) in  5m 29s, read: 993.7 MiB/s, write: 2.4 KiB/s
INFO:  24% (120.4 GiB of 500.0 GiB) in  5m 34s, read: 991.4 MiB/s, write: 1.6 KiB/s
INFO:  25% (125.2 GiB of 500.0 GiB) in  5m 39s, read: 983.4 MiB/s, write: 2.4 KiB/s
INFO:  26% (130.0 GiB of 500.0 GiB) in  5m 44s, read: 993.9 MiB/s, write: 4.0 KiB/s
INFO:  27% (135.8 GiB of 500.0 GiB) in  5m 50s, read: 992.6 MiB/s, write: 2.0 KiB/s
INFO:  28% (140.7 GiB of 500.0 GiB) in  5m 55s, read: 989.0 MiB/s, write: 4.0 KiB/s
INFO:  29% (145.5 GiB of 500.0 GiB) in  6m  0s, read: 994.4 MiB/s, write: 2.4 KiB/s
INFO:  30% (150.3 GiB of 500.0 GiB) in  6m  5s, read: 989.0 MiB/s, write: 23.2 KiB/s
INFO:  31% (155.1 GiB of 500.0 GiB) in  6m 24s, read: 259.4 MiB/s, write: 17.6 MiB/s
INFO:  32% (160.0 GiB of 500.0 GiB) in  6m 33s, read: 552.2 MiB/s, write: 16.9 MiB/s
INFO:  33% (165.0 GiB of 500.0 GiB) in  6m 43s, read: 515.5 MiB/s, write: 15.7 MiB/s
INFO:  34% (170.4 GiB of 500.0 GiB) in  6m 53s, read: 551.1 MiB/s, write: 14.7 MiB/s
INFO:  35% (175.5 GiB of 500.0 GiB) in  6m 59s, read: 861.6 MiB/s, write: 7.9 MiB/s
INFO:  36% (180.6 GiB of 500.0 GiB) in  7m  5s, read: 882.6 MiB/s, write: 5.8 MiB/s
INFO:  37% (185.4 GiB of 500.0 GiB) in  7m 10s, read: 976.0 MiB/s, write: 1.8 MiB/s
INFO:  38% (190.4 GiB of 500.0 GiB) in  7m 17s, read: 724.9 MiB/s, write: 10.9 MiB/s
INFO:  39% (195.6 GiB of 500.0 GiB) in  7m 23s, read: 901.7 MiB/s, write: 2.9 MiB/s
INFO:  40% (200.5 GiB of 500.0 GiB) in  7m 28s, read: 992.4 MiB/s, write: 9.6 KiB/s
INFO:  41% (205.3 GiB of 500.0 GiB) in  7m 33s, read: 991.9 MiB/s, write: 2.4 KiB/s
INFO:  42% (210.2 GiB of 500.0 GiB) in  7m 38s, read: 993.9 MiB/s, write: 1.6 KiB/s
INFO:  43% (215.0 GiB of 500.0 GiB) in  7m 43s, read: 994.4 MiB/s, write: 2.4 KiB/s
INFO:  44% (220.9 GiB of 500.0 GiB) in  7m 49s, read: 994.9 MiB/s, write: 1.3 KiB/s
INFO:  45% (225.7 GiB of 500.0 GiB) in  7m 54s, read: 989.3 MiB/s, write: 2.4 KiB/s
INFO:  46% (230.5 GiB of 500.0 GiB) in  7m 59s, read: 991.6 MiB/s, write: 1.6 KiB/s
INFO:  47% (235.4 GiB of 500.0 GiB) in  8m  4s, read: 991.9 MiB/s, write: 2.4 KiB/s
INFO:  48% (240.2 GiB of 500.0 GiB) in  8m  9s, read: 992.9 MiB/s, write: 1.6 KiB/s
INFO:  49% (245.1 GiB of 500.0 GiB) in  8m 14s, read: 995.3 MiB/s, write: 2.4 KiB/s
INFO:  50% (250.9 GiB of 500.0 GiB) in  8m 20s, read: 990.3 MiB/s, write: 1.3 KiB/s
INFO:  51% (255.7 GiB of 500.0 GiB) in  8m 25s, read: 989.4 MiB/s, write: 2.4 KiB/s
INFO:  52% (260.6 GiB of 500.0 GiB) in  8m 30s, read: 994.9 MiB/s, write: 1.6 KiB/s
INFO:  53% (265.4 GiB of 500.0 GiB) in  8m 35s, read: 990.7 MiB/s, write: 2.4 KiB/s
INFO:  54% (270.3 GiB of 500.0 GiB) in  8m 40s, read: 990.5 MiB/s, write: 1.6 KiB/s
INFO:  55% (275.1 GiB of 500.0 GiB) in  8m 45s, read: 990.2 MiB/s, write: 53.6 KiB/s
INFO:  56% (280.9 GiB of 500.0 GiB) in  8m 51s, read: 992.9 MiB/s, write: 1.3 KiB/s
INFO:  57% (285.8 GiB of 500.0 GiB) in  8m 56s, read: 994.7 MiB/s, write: 2.4 KiB/s
INFO:  58% (290.6 GiB of 500.0 GiB) in  9m  1s, read: 995.0 MiB/s, write: 1.6 KiB/s
INFO:  59% (295.5 GiB of 500.0 GiB) in  9m  6s, read: 990.7 MiB/s, write: 2.4 KiB/s
INFO:  60% (300.3 GiB of 500.0 GiB) in  9m 11s, read: 991.7 MiB/s, write: 1.6 KiB/s
INFO:  61% (305.2 GiB of 500.0 GiB) in  9m 16s, read: 992.9 MiB/s, write: 53.6 KiB/s
INFO:  62% (310.0 GiB of 500.0 GiB) in  9m 21s, read: 997.2 MiB/s, write: 1.6 KiB/s
INFO:  63% (315.8 GiB of 500.0 GiB) in  9m 27s, read: 989.8 MiB/s, write: 2.0 KiB/s
INFO:  64% (320.6 GiB of 500.0 GiB) in  9m 32s, read: 987.8 MiB/s, write: 1.6 KiB/s
INFO:  65% (325.5 GiB of 500.0 GiB) in  9m 37s, read: 989.7 MiB/s, write: 2.4 KiB/s
INFO:  66% (330.3 GiB of 500.0 GiB) in  9m 42s, read: 990.9 MiB/s, write: 1.6 KiB/s
INFO:  67% (335.2 GiB of 500.0 GiB) in  9m 47s, read: 992.3 MiB/s, write: 2.4 KiB/s
INFO:  68% (341.0 GiB of 500.0 GiB) in  9m 53s, read: 991.3 MiB/s, write: 1.3 KiB/s
INFO:  69% (345.8 GiB of 500.0 GiB) in  9m 58s, read: 990.8 MiB/s, write: 2.4 KiB/s
INFO:  70% (350.6 GiB of 500.0 GiB) in 10m  3s, read: 982.4 MiB/s, write: 1.6 KiB/s
INFO:  71% (355.4 GiB of 500.0 GiB) in 10m  8s, read: 983.2 MiB/s, write: 2.4 KiB/s
INFO:  72% (360.2 GiB of 500.0 GiB) in 10m 13s, read: 988.9 MiB/s, write: 1.6 KiB/s
INFO:  73% (365.1 GiB of 500.0 GiB) in 10m 18s, read: 991.8 MiB/s, write: 2.4 KiB/s
INFO:  74% (370.9 GiB of 500.0 GiB) in 10m 24s, read: 985.1 MiB/s, write: 1.3 KiB/s
INFO:  75% (375.7 GiB of 500.0 GiB) in 10m 29s, read: 988.5 MiB/s, write: 2.4 KiB/s
INFO:  76% (380.5 GiB of 500.0 GiB) in 10m 34s, read: 987.7 MiB/s, write: 1.6 KiB/s
INFO:  77% (385.3 GiB of 500.0 GiB) in 10m 39s, read: 991.8 MiB/s, write: 2.4 KiB/s
INFO:  78% (390.2 GiB of 500.0 GiB) in 10m 44s, read: 987.9 MiB/s, write: 1.6 KiB/s
INFO:  79% (395.9 GiB of 500.0 GiB) in 10m 50s, read: 985.7 MiB/s, write: 44.7 KiB/s
INFO:  80% (400.8 GiB of 500.0 GiB) in 10m 55s, read: 990.2 MiB/s, write: 1.6 KiB/s
INFO:  81% (405.6 GiB of 500.0 GiB) in 11m  0s, read: 985.3 MiB/s, write: 2.4 KiB/s
INFO:  82% (410.4 GiB of 500.0 GiB) in 11m  5s, read: 982.6 MiB/s, write: 1.6 KiB/s
INFO:  83% (415.2 GiB of 500.0 GiB) in 11m 10s, read: 981.5 MiB/s, write: 2.4 KiB/s
INFO:  84% (420.9 GiB of 500.0 GiB) in 11m 16s, read: 981.4 MiB/s, write: 1.3 KiB/s
INFO:  85% (425.8 GiB of 500.0 GiB) in 11m 21s, read: 987.9 MiB/s, write: 2.4 KiB/s
INFO:  86% (430.6 GiB of 500.0 GiB) in 11m 26s, read: 983.9 MiB/s, write: 1.6 KiB/s
INFO:  87% (435.4 GiB of 500.0 GiB) in 11m 31s, read: 987.4 MiB/s, write: 2.4 KiB/s
INFO:  88% (440.2 GiB of 500.0 GiB) in 11m 36s, read: 991.4 MiB/s, write: 1.6 KiB/s
INFO:  89% (445.0 GiB of 500.0 GiB) in 11m 41s, read: 984.8 MiB/s, write: 2.4 KiB/s
INFO:  90% (450.8 GiB of 500.0 GiB) in 11m 47s, read: 984.1 MiB/s, write: 1.3 KiB/s
INFO:  91% (455.6 GiB of 500.0 GiB) in 11m 52s, read: 984.1 MiB/s, write: 2.4 KiB/s
INFO:  92% (460.4 GiB of 500.0 GiB) in 11m 57s, read: 989.2 MiB/s, write: 1.6 KiB/s
INFO:  93% (465.3 GiB of 500.0 GiB) in 12m  2s, read: 989.3 MiB/s, write: 2.4 KiB/s
INFO:  94% (470.1 GiB of 500.0 GiB) in 12m  7s, read: 983.7 MiB/s, write: 1.6 KiB/s
INFO:  95% (475.8 GiB of 500.0 GiB) in 12m 13s, read: 986.4 MiB/s, write: 2.0 KiB/s
INFO:  96% (480.7 GiB of 500.0 GiB) in 12m 18s, read: 986.1 MiB/s, write: 1.6 KiB/s
INFO:  97% (485.4 GiB of 500.0 GiB) in 12m 23s, read: 979.9 MiB/s, write: 2.4 KiB/s
INFO:  98% (490.2 GiB of 500.0 GiB) in 12m 28s, read: 983.8 MiB/s, write: 1.6 KiB/s
INFO:  99% (495.1 GiB of 500.0 GiB) in 12m 33s, read: 986.7 MiB/s, write: 2.4 KiB/s
INFO: 100% (500.0 GiB of 500.0 GiB) in 12m 39s, read: 842.5 MiB/s, write: 3.3 KiB/s
INFO: backup is sparse: 494.16 GiB (98%) total zero data
INFO: transferred 500.00 GiB in 759 seconds (674.6 MiB/s)
INFO: archive file size: 2.50GB
INFO: Finished Backup of VM 100 (00:12:39)
INFO: Backup finished at 2021-01-01 03:12:40
INFO: Starting Backup of VM 102 (qemu)
INFO: Backup started at 2021-01-01 03:12:40
INFO: status = running
INFO: VM Name: unifi-controller
INFO: include disk 'scsi0' 'samsung-ssd:vm-102-disk-0' 32G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/backups/dump/vzdump-qemu-102-2021_01_01-03_12_40.vma.gz'
INFO: started backup task '930aab56-96e1-42dd-8ba4-a885aa857ef0'
INFO: resuming VM again
INFO:   0% (192.1 MiB of 32.0 GiB) in  3s, read: 64.0 MiB/s, write: 20.0 MiB/s
INFO:   1% (333.5 MiB of 32.0 GiB) in 10s, read: 20.2 MiB/s, write: 19.2 MiB/s
INFO:   3% (1.1 GiB of 32.0 GiB) in 15s, read: 166.0 MiB/s, write: 21.1 MiB/s
INFO:   4% (1.3 GiB of 32.0 GiB) in 23s, read: 20.8 MiB/s, write: 20.8 MiB/s
INFO:   5% (1.6 GiB of 32.0 GiB) in 34s, read: 29.0 MiB/s, write: 28.2 MiB/s
INFO:   6% (1.9 GiB of 32.0 GiB) in 47s, read: 24.5 MiB/s, write: 22.9 MiB/s
INFO:   7% (2.3 GiB of 32.0 GiB) in 57s, read: 33.6 MiB/s, write: 21.4 MiB/s
INFO:   8% (2.6 GiB of 32.0 GiB) in  1m  6s, read: 36.4 MiB/s, write: 24.6 MiB/s
INFO:   9% (3.1 GiB of 32.0 GiB) in  1m 11s, read: 100.0 MiB/s, write: 23.0 MiB/s
INFO:  10% (3.2 GiB of 32.0 GiB) in  1m 14s, read: 55.6 MiB/s, write: 32.9 MiB/s
INFO:  14% (4.7 GiB of 32.0 GiB) in  1m 17s, read: 519.7 MiB/s, write: 15.9 MiB/s
INFO:  16% (5.2 GiB of 32.0 GiB) in  1m 20s, read: 142.5 MiB/s, write: 31.5 MiB/s
INFO:  17% (5.5 GiB of 32.0 GiB) in  1m 32s, read: 24.8 MiB/s, write: 24.2 MiB/s
INFO:  18% (5.8 GiB of 32.0 GiB) in  1m 45s, read: 25.7 MiB/s, write: 25.6 MiB/s
INFO:  19% (6.1 GiB of 32.0 GiB) in  2m  1s, read: 20.6 MiB/s, write: 20.5 MiB/s
INFO:  20% (6.4 GiB of 32.0 GiB) in  2m 14s, read: 25.1 MiB/s, write: 25.0 MiB/s

gzip: stdout: No space left on device
INFO:  20% (6.7 GiB of 32.0 GiB) in  2m 28s, read: 21.0 MiB/s, write: 20.6 MiB/s
ERROR: vma_queue_write: write error - Broken pipe
INFO: aborting backup job
ERROR: Backup of VM 102 failed - vma_queue_write: write error - Broken pipe
INFO: Failed at 2021-01-01 03:15:08
INFO: Starting Backup of VM 103 (qemu)
INFO: Backup started at 2021-01-01 03:15:08
INFO: status = running
INFO: VM Name: pihole-nginx
INFO: include disk 'scsi0' 'samsung-ssd:vm-103-disk-0' 100G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/backups/dump/vzdump-qemu-103-2021_01_01-03_15_08.vma.gz'
INFO: started backup task '0490bb20-814c-4674-bfea-7cafe5fe06f8'
INFO: resuming VM again
INFO:   0% (192.1 MiB of 100.0 GiB) in  3s, read: 64.0 MiB/s, write: 19.7 MiB/s
INFO:   1% (1.1 GiB of 100.0 GiB) in 15s, read: 81.0 MiB/s, write: 18.9 MiB/s
INFO:   2% (2.0 GiB of 100.0 GiB) in 47s, read: 27.9 MiB/s, write: 23.3 MiB/s
INFO:   3% (3.1 GiB of 100.0 GiB) in  1m  5s, read: 59.4 MiB/s, write: 23.5 MiB/s
INFO:   5% (5.9 GiB of 100.0 GiB) in  1m  8s, read: 963.0 MiB/s, write: 46.7 KiB/s
INFO:   6% (6.8 GiB of 100.0 GiB) in  1m 11s, read: 299.7 MiB/s, write: 20.1 MiB/s
INFO:   7% (7.0 GiB of 100.0 GiB) in  1m 23s, read: 21.4 MiB/s, write: 20.6 MiB/s
INFO:   8% (8.0 GiB of 100.0 GiB) in  2m  7s, read: 23.6 MiB/s, write: 20.1 MiB/s
INFO:   9% (9.1 GiB of 100.0 GiB) in  3m  0s, read: 20.6 MiB/s, write: 18.3 MiB/s

gzip: stdout: No space left on device
INFO:   9% (9.4 GiB of 100.0 GiB) in  3m 15s, read: 21.0 MiB/s, write: 17.1 MiB/s
ERROR: vma_queue_write: write error - Broken pipe
INFO: aborting backup job
ERROR: Backup of VM 103 failed - vma_queue_write: write error - Broken pipe
INFO: Failed at 2021-01-01 03:18:23
INFO: Backup job finished with errors

TASK ERROR: job errors

Here is my backup job:

Here is my directory:

Looks like it says it's mounted:
root@local:~# mount /dev/sda
mount: /backup: /dev/sda already mounted on /backup.

findmnt returns:
TARGET SOURCE FSTYPE OPTIONS
/backup /dev/sda ext4 rw,relatime

Any help much appreciated.

UdoB · Jan 6, 2021

Hi,

theres is a typo: you've mounted `backup` but you do backup to backups (with a trailing s). That folder is on root "/" which is too small.

Happens to all of us ;-)

ihasamoose · Jan 6, 2021

UdoB said:
Hi,

theres is a typo: you've mounted `backup` but you do backup to backups (with a trailing s). That folder is on root "/" which is too small.

Happens to all of us ;-)

Thanks for checking that out.. I do see in the actual script logs it says 'backups' now.

However, my storage.cfg is pointing to /backup/, so not sure why it's trying to throw it there.. hmm..

My storage.cfg

Code:

dir: backup
    path /backup
    content backup
    prune-backups keep-all=1
    shared 0

Maybe I'll try regenerating the backup script.

Search

Search

Backup job error 'gzip: stdout: No space left on device'

ihasamoose

New Member

UdoB

Famous Member

ihasamoose

New Member