pve-root 100% after backup failed

D

danitfk

Member
Apr 20, 2015
28
1
23
Tehran
blog.gordi.ir
Hello all ,
I did run a backup task for a container with this status:
Code: 
[COLOR=#000000][FONT=tahoma]INFO: starting new backup job: vzdump 109 --remove 0 --mode snapshot --compress lzo --storage ExternalDisk --node node1[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: Starting Backup of VM 109 (openvz)[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: CTID 109 exist mounted running[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status = running[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: backup mode: snapshot[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: ionice priority: 7[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: creating lvm snapshot of /dev/mapper/pve-data ('/dev/pve/vzsnap-node1-0')[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO:   Logical volume "vzsnap-node1-0" created[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: creating archive '/ehdd/dump/vzdump-openvz-109-2015_07_28-08_54_34.tar.lzo'
The storage of backup is /ehdd (External Hard disk , attached via USB to server) but now pve-root is full.
Code: 
[/SIZE][/FONT][/COLOR]root@node1:/# df -h
Filesystem                        Size  Used Avail Use% Mounted on
udev                               10M     0   10M   0% /dev
tmpfs                             7.1G  432K  7.1G   1% /run
[B]/dev/mapper/pve-root               95G   94G     0 100% /[/B]
tmpfs                             5.0M     0  5.0M   0% /run/lock
tmpfs                              15G   25M   15G   1% /run/shm
/dev/mapper/pve-data              1.7T  380G  1.3T  23% /var/lib/vz
/dev/sdb1                         1.8T  1.6T  215G  89% /sdb
tmpfs                              15G     0   15G   0% /tmp
/dev/fuse                          30M   24K   30M   1% /etc/pve
/var/lib/vz/private/109           100G   44G   57G  44% /var/lib/vz/root/109
none                               16G  4.0K   16G   1% /var/lib/vz/root/109/dev
none                              4.0K     0  4.0K   0% /var/lib/vz/root/109/sys/fs/cgroup
none                              3.3G  104K  3.2G   1% /var/lib/vz/root/109/run
none                              5.0M     0  5.0M   0% /var/lib/vz/root/109/run/lock
none                               16G     0   16G   0% /var/lib/vz/root/109/run/shm
none                              100M     0  100M   0% /var/lib/vz/root/109/run/user
[B]/dev/mapper/pve-vzsnap--node1--0  1.7T  381G  1.3T  23% /mnt/vzsnap0
[/B]
I did backup from vz109 , Also There is a pve-vzsnap-node1--0 , What is this? a temporary lv for snapshots?

Also , I killed backup process via
Code: 
kill -9 [BACKUP-UID]
Also maybe it's useful:
Code: 
[B]root@node1:/# lvdisplay[/B]
File descriptor 3 (/usr/share/bash-completion/completions) leaked on lvdisplay invocation. Parent PID 684760: -bash
File descriptor 4 (/usr/share/bash-completion/completions) leaked on lvdisplay invocation. Parent PID 684760: -bash
File descriptor 5 (/usr/share/bash-completion/completions) leaked on lvdisplay invocation. Parent PID 684760: -bash
File descriptor 6 (/usr/share/bash-completion/completions) leaked on lvdisplay invocation. Parent PID 684760: -bash
File descriptor 7 (/usr/share/bash-completion/completions) leaked on lvdisplay invocation. Parent PID 684760: -bash
File descriptor 8 (/usr/share/bash-completion/completions) leaked on lvdisplay invocation. Parent PID 684760: -bash
File descriptor 9 (/usr/share/bash-completion/completions) leaked on lvdisplay invocation. Parent PID 684760: -bash
File descriptor 12 (/usr/share/bash-completion/completions) leaked on lvdisplay invocation. Parent PID 684760: -bash
File descriptor 13 (/usr/share/bash-completion/completions) leaked on lvdisplay invocation. Parent PID 684760: -bash
File descriptor 14 (/usr/share/bash-completion/completions) leaked on lvdisplay invocation. Parent PID 684760: -bash
  /dev/pve/vzsnap-node1-0: read failed after 0 of 4096 at 1809141661696: Input/output error
  /dev/pve/vzsnap-node1-0: read failed after 0 of 4096 at 1809141719040: Input/output error
  /dev/pve/vzsnap-node1-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-node1-0: read failed after 0 of 4096 at 4096: Input/output error
  --- Logical volume ---
  LV Path                /dev/pve/swap
  LV Name                swap
  VG Name                pve
  LV UUID                qY8M35-Gj73-1grd-Nf5B-VVRI-PQtV-zLyhCW
  LV Write Access        read/write
  LV Creation host, time proxmox, 2015-06-11 23:24:20 +0430
  LV Status              available
  # open                 0
  LV Size                66.00 GiB
  Current LE             16896
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1


  --- Logical volume ---
  LV Path                /dev/pve/root
  LV Name                root
  VG Name                pve
  LV UUID                5BGmiU-DQ4c-3Pi6-bTEt-1ok2-60ak-hzLdX2
  LV Write Access        read/write
  LV Creation host, time proxmox, 2015-06-11 23:24:20 +0430
  LV Status              available
  # open                 1
  LV Size                96.00 GiB
  Current LE             24576
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0


  --- Logical volume ---
  LV Path                /dev/pve/data
  LV Name                data
  VG Name                pve
  LV UUID                pSr3ui-tZSb-RaVK-y4Y9-cHcR-4gmS-UHevVJ
  LV Write Access        read/write
  LV Creation host, time proxmox, 2015-06-11 23:24:20 +0430
  LV snapshot status     source of
                         vzsnap-node1-0 [INACTIVE]
  LV Status              available
  # open                 1
  LV Size                1.65 TiB
  Current LE             431333
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2


  --- Logical volume ---
  LV Path                /dev/pve/vzsnap-node1-0
  LV Name                vzsnap-node1-0
  VG Name                pve
  LV UUID                fAgq9t-Sddg-VyA2-OFGv-HeqB-qMxi-SQrGVp
  LV Write Access        read/write
  LV Creation host, time node1, 2015-07-28 08:54:40 +0430
  LV snapshot status     INACTIVE destination for data
  LV Status              available
  # open                 1
  LV Size                1.65 TiB
  Current LE             431333
  COW-table size         1.00 GiB
  COW-table LE           256
  Snapshot chunk size    4.00 KiB
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:3

and fdisk
Code: 
[B]root@node1:/# fdisk -l[/B]


WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.




Disk /dev/sda: 2000.4 GB, 2000398934016 bytes
256 heads, 63 sectors/track, 242251 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1  3907029167  1953514583+  ee  GPT
Partition 1 does not start on physical sector boundary.


Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
81 heads, 63 sectors/track, 765633 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048  3907029167  1953513560   83  Linux


Disk /dev/mapper/pve-root: 103.1 GB, 103079215104 bytes
255 heads, 63 sectors/track, 12532 cylinders, total 201326592 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/pve-root doesn't contain a valid partition table


Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes
1 heads, 63 sectors/track, 31008336 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0008c07a


   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1   *          63  1953520127   976760032+   7  HPFS/NTFS/exFAT


Disk /dev/mapper/pve-swap: 70.9 GB, 70866960384 bytes
255 heads, 63 sectors/track, 8615 cylinders, total 138412032 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/pve-swap doesn't contain a valid partition table


Disk /dev/mapper/pve-data: 1809.1 GB, 1809141727232 bytes
255 heads, 63 sectors/track, 219948 cylinders, total 3533479936 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/pve-data doesn't contain a valid partition table


Disk /dev/mapper/pve-vzsnap--node1--0: 1809.1 GB, 1809141727232 bytes
255 heads, 63 sectors/track, 219948 cylinders, total 3533479936 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/pve-vzsnap--node1--0 doesn't contain a valid partition table


Disk /dev/mapper/pve-data-real: 1809.1 GB, 1809141727232 bytes
255 heads, 63 sectors/track, 219948 cylinders, total 3533479936 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/pve-data-real doesn't contain a valid partition table


Disk /dev/mapper/pve-vzsnap--node1--0-cow: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/pve-vzsnap--node1--0-cow doesn't contain a valid partition table
 
What does
Code: 
findmnt
return? Is your external disk mounted correctly?

VZDump makes some temporary files when you're backing up a running vm, to look if that's the problem use the vzdump command in a shell, you can take the command from the log output:
Code: 
vzdump 109 --remove 0 --mode snapshot --compress lzo --storage ExternalDisk --node node1 --tmpdir /ehdd
note the added -tmpdir option, which points to your external disk.
 
Last edited by a moderator:
It's seems not mounted correctly , Right?
Code: 
root@node1:~# findmnt
TARGET                                               SOURCE                                FSTYPE      OPTIONS
/                                                    /dev/mapper/pve-root                  ext3        rw,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered
├─/sys                                               sysfs                                 sysfs       rw,nosuid,nodev,noexec,relatime
├─/proc                                              proc                                  proc        rw,nosuid,nodev,noexec,relatime
│ ├─/proc/vz/beancounter                             beancounter                           cgroup      rw,relatime,blkio,name=beancounter
│ ├─/proc/vz/container                               container                             cgroup      rw,relatime,freezer,devices,name=container
│ └─/proc/vz/fairsched                               fairsched                             cgroup      rw,relatime,cpuacct,cpu,cpuset,name=fairsched
├─/dev                                               udev                                  devtmpfs    rw,relatime,size=10240k,nr_inodes=9274667,mode=755
│ └─/dev/pts                                         devpts                                devpts      rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
├─/run                                               tmpfs                                 tmpfs       rw,nosuid,noexec,relatime,size=7422044k,mode=755
│ ├─/run/lock                                        tmpfs                                 tmpfs       rw,nosuid,nodev,noexec,relatime,size=5120k
│ └─/run/shm                                         tmpfs                                 tmpfs       rw,nosuid,nodev,noexec,relatime,size=14844080k
├─/var/lib/vz                                        /dev/mapper/pve-data                  ext3        rw,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered
│ └─/var/lib/vz/root/109                             /var/lib/vz/private/109[/private/109] simfs       rw,relatime
│   ├─/var/lib/vz/root/109/proc                      proc                                  proc        rw,relatime
│   │ └─/var/lib/vz/root/109/proc/sys/fs/binfmt_misc                                       binfmt_misc rw,nosuid,nodev,noexec,relatime
│   ├─/var/lib/vz/root/109/sys                       sysfs                                 sysfs       rw,relatime
│   │ └─/var/lib/vz/root/109/sys/fs/cgroup                                                 tmpfs       rw,relatime,size=4k,nr_inodes=4194304,mode=755
│   ├─/var/lib/vz/root/109/dev                                                             devtmpfs    rw,nosuid,noexec,relatime,size=16777216k,nr_inodes=4194304,mode=755
│   │ └─/var/lib/vz/root/109/dev/pts                                                       devpts      rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
│   └─/var/lib/vz/root/109/run                                                             tmpfs       rw,nosuid,noexec,relatime,size=3355444k,nr_inodes=4194304,mode=755
│     ├─/var/lib/vz/root/109/run/lock                                                      tmpfs       rw,nosuid,nodev,noexec,relatime,size=5120k,nr_inodes=4194304
│     ├─/var/lib/vz/root/109/run/shm                                                       tmpfs       rw,relatime,size=16777216k,nr_inodes=4194304
│     └─/var/lib/vz/root/109/run/user                                                      tmpfs       rw,nosuid,nodev,noexec,relatime,size=102400k,nr_inodes=4194304,mode=755
├─/sdb                                               /dev/sdb1                             ext4        rw,relatime,barrier=1,data=ordered
├─/tmp                                               tmpfs                                 tmpfs       rw,nosuid,nodev,relatime,size=14844080k
├─/var/lib/nfs/rpc_pipefs                            rpc_pipefs                            rpc_pipefs  rw,relatime
├─/etc/pve                                           /dev/fuse                             fuse        rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
└─/mnt/vzsnap0                                       /dev/mapper/pve-vzsnap--node1--0      ext3        rw,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered
 
toni.patroni said:
No, I also doesn't see it. How do you mounted your disk in /ehdd?

http://www.debiantutorials.com/how-...-or-partition-using-uuid-and-ext4-filesystem/ may help, you can also skip the formatting part if you already have a valid formatted external disk (jump right to step 4).
Click to expand...
Also It's not in mounted disks.
Code: 
root@node1:~# df -h
Filesystem                        Size  Used Avail Use% Mounted on
udev                               10M     0   10M   0% /dev
tmpfs                             7.1G  432K  7.1G   1% /run
/dev/mapper/pve-root               95G   94G     0 100% /
tmpfs                             5.0M     0  5.0M   0% /run/lock
tmpfs                              15G   25M   15G   1% /run/shm
/dev/mapper/pve-data              1.7T  381G  1.3T  23% /var/lib/vz
/dev/sdb1                         1.8T  1.6T  215G  89% /sdb
tmpfs                              15G     0   15G   0% /tmp
/dev/fuse                          30M   24K   30M   1% /etc/pve
/var/lib/vz/private/109           100G   45G   56G  45% /var/lib/vz/root/109
none                               16G  4.0K   16G   1% /var/lib/vz/root/109/dev
none                              4.0K     0  4.0K   0% /var/lib/vz/root/109/sys/fs/cgroup
none                              3.3G  104K  3.2G   1% /var/lib/vz/root/109/run
none                              5.0M     0  5.0M   0% /var/lib/vz/root/109/run/lock
none                               16G     0   16G   0% /var/lib/vz/root/109/run/shm
none                              100M     0  100M   0% /var/lib/vz/root/109/run/user
/dev/mapper/pve-vzsnap--node1--0  1.7T  381G  1.3T  23% /mnt/vzsnap0
I must mention , My External hdd has ntfs format So I mounted with ntfs-3g type with this in fstab:
Code: 
/dev/sdc1               /ehdd           ntfs-3g defaults        1       1
 
Looks good to me.
So what does
Code: 
mount /ehdd
output?

-------
EDIT:
For NTFS you maybe shouldn't use the "defaults" option in fstab, use "permission" instead.
Or to be really specific use : uid=1000,umask=0022
Where uid is the user id from the owner of the disk and umask is the right mask for files and directories.
Use this eg.:
Code: 
/dev/sdc1               /ehdd           ntfs-3g uid=1000,umask=0022        0       0
 
Last edited by a moderator:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back