pve-root 100% after backup failed

D

danitfk

Member
Apr 20, 2015
28
1
23
Tehran
blog.gordi.ir
Hello all ,
I did run a backup task for a container with this status:
Code: 
[COLOR=#000000][FONT=tahoma]INFO: starting new backup job: vzdump 109 --remove 0 --mode snapshot --compress lzo --storage ExternalDisk --node node1[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: Starting Backup of VM 109 (openvz)[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: CTID 109 exist mounted running[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status = running[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: backup mode: snapshot[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: ionice priority: 7[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: creating lvm snapshot of /dev/mapper/pve-data ('/dev/pve/vzsnap-node1-0')[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO:   Logical volume "vzsnap-node1-0" created[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: creating archive '/ehdd/dump/vzdump-openvz-109-2015_07_28-08_54_34.tar.lzo'
The storage of backup is /ehdd (External Hard disk , attached via USB to server) but now pve-root is full.
Code: 
[/SIZE][/FONT][/COLOR]root@node1:/# df -h
Filesystem                        Size  Used Avail Use% Mounted on
udev                               10M     0   10M   0% /dev
tmpfs                             7.1G  432K  7.1G   1% /run
[B]/dev/mapper/pve-root               95G   94G     0 100% /[/B]
tmpfs                             5.0M     0  5.0M   0% /run/lock
tmpfs                              15G   25M   15G   1% /run/shm
/dev/mapper/pve-data              1.7T  380G  1.3T  23% /var/lib/vz
/dev/sdb1                         1.8T  1.6T  215G  89% /sdb
tmpfs                              15G     0   15G   0% /tmp
/dev/fuse                          30M   24K   30M   1% /etc/pve
/var/lib/vz/private/109           100G   44G   57G  44% /var/lib/vz/root/109
none                               16G  4.0K   16G   1% /var/lib/vz/root/109/dev
none                              4.0K     0  4.0K   0% /var/lib/vz/root/109/sys/fs/cgroup
none                              3.3G  104K  3.2G   1% /var/lib/vz/root/109/run
none                              5.0M     0  5.0M   0% /var/lib/vz/root/109/run/lock
none                               16G     0   16G   0% /var/lib/vz/root/109/run/shm
none                              100M     0  100M   0% /var/lib/vz/root/109/run/user
[B]/dev/mapper/pve-vzsnap--node1--0  1.7T  381G  1.3T  23% /mnt/vzsnap0
[/B]
I did backup from vz109 , Also There is a pve-vzsnap-node1--0 , What is this? a temporary lv for snapshots?

Also , I killed backup process via
Code: 
kill -9 [BACKUP-UID]
Also maybe it's useful:
Code: 
[B]root@node1:/# lvdisplay[/B]
File descriptor 3 (/usr/share/bash-completion/completions) leaked on lvdisplay invocation. Parent PID 684760: -bash
File descriptor 4 (/usr/share/bash-completion/completions) leaked on lvdisplay invocation. Parent PID 684760: -bash
File descriptor 5 (/usr/share/bash-completion/completions) leaked on lvdisplay invocation. Parent PID 684760: -bash
File descriptor 6 (/usr/share/bash-completion/completions) leaked on lvdisplay invocation. Parent PID 684760: -bash
File descriptor 7 (/usr/share/bash-completion/completions) leaked on lvdisplay invocation. Parent PID 684760: -bash
File descriptor 8 (/usr/share/bash-completion/completions) leaked on lvdisplay invocation. Parent PID 684760: -bash
File descriptor 9 (/usr/share/bash-completion/completions) leaked on lvdisplay invocation. Parent PID 684760: -bash
File descriptor 12 (/usr/share/bash-completion/completions) leaked on lvdisplay invocation. Parent PID 684760: -bash
File descriptor 13 (/usr/share/bash-completion/completions) leaked on lvdisplay invocation. Parent PID 684760: -bash
File descriptor 14 (/usr/share/bash-completion/completions) leaked on lvdisplay invocation. Parent PID 684760: -bash
  /dev/pve/vzsnap-node1-0: read failed after 0 of 4096 at 1809141661696: Input/output error
  /dev/pve/vzsnap-node1-0: read failed after 0 of 4096 at 1809141719040: Input/output error
  /dev/pve/vzsnap-node1-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-node1-0: read failed after 0 of 4096 at 4096: Input/output error
  --- Logical volume ---
  LV Path                /dev/pve/swap
  LV Name                swap
  VG Name                pve
  LV UUID                qY8M35-Gj73-1grd-Nf5B-VVRI-PQtV-zLyhCW
  LV Write Access        read/write
  LV Creation host, time proxmox, 2015-06-11 23:24:20 +0430
  LV Status              available
  # open                 0
  LV Size                66.00 GiB
  Current LE             16896
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1


  --- Logical volume ---
  LV Path                /dev/pve/root
  LV Name                root
  VG Name                pve
  LV UUID                5BGmiU-DQ4c-3Pi6-bTEt-1ok2-60ak-hzLdX2
  LV Write Access        read/write
  LV Creation host, time proxmox, 2015-06-11 23:24:20 +0430
  LV Status              available
  # open                 1
  LV Size                96.00 GiB
  Current LE             24576
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0


  --- Logical volume ---
  LV Path                /dev/pve/data
  LV Name                data
  VG Name                pve
  LV UUID                pSr3ui-tZSb-RaVK-y4Y9-cHcR-4gmS-UHevVJ
  LV Write Access        read/write
  LV Creation host, time proxmox, 2015-06-11 23:24:20 +0430
  LV snapshot status     source of
                         vzsnap-node1-0 [INACTIVE]
  LV Status              available
  # open                 1
  LV Size                1.65 TiB
  Current LE             431333
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2


  --- Logical volume ---
  LV Path                /dev/pve/vzsnap-node1-0
  LV Name                vzsnap-node1-0
  VG Name                pve
  LV UUID                fAgq9t-Sddg-VyA2-OFGv-HeqB-qMxi-SQrGVp
  LV Write Access        read/write
  LV Creation host, time node1, 2015-07-28 08:54:40 +0430
  LV snapshot status     INACTIVE destination for data
  LV Status              available
  # open                 1
  LV Size                1.65 TiB
  Current LE             431333
  COW-table size         1.00 GiB
  COW-table LE           256
  Snapshot chunk size    4.00 KiB
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:3

and fdisk
Code: 
[B]root@node1:/# fdisk -l[/B]


WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.




Disk /dev/sda: 2000.4 GB, 2000398934016 bytes
256 heads, 63 sectors/track, 242251 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1  3907029167  1953514583+  ee  GPT
Partition 1 does not start on physical sector boundary.


Disk /dev/sdb: 2000.4 GB, 2000398934016 bytes
81 heads, 63 sectors/track, 765633 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048  3907029167  1953513560   83  Linux


Disk /dev/mapper/pve-root: 103.1 GB, 103079215104 bytes
255 heads, 63 sectors/track, 12532 cylinders, total 201326592 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/pve-root doesn't contain a valid partition table


Disk /dev/sdd: 1000.2 GB, 1000204886016 bytes
1 heads, 63 sectors/track, 31008336 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0008c07a


   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1   *          63  1953520127   976760032+   7  HPFS/NTFS/exFAT


Disk /dev/mapper/pve-swap: 70.9 GB, 70866960384 bytes
255 heads, 63 sectors/track, 8615 cylinders, total 138412032 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/pve-swap doesn't contain a valid partition table


Disk /dev/mapper/pve-data: 1809.1 GB, 1809141727232 bytes
255 heads, 63 sectors/track, 219948 cylinders, total 3533479936 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/pve-data doesn't contain a valid partition table


Disk /dev/mapper/pve-vzsnap--node1--0: 1809.1 GB, 1809141727232 bytes
255 heads, 63 sectors/track, 219948 cylinders, total 3533479936 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/pve-vzsnap--node1--0 doesn't contain a valid partition table


Disk /dev/mapper/pve-data-real: 1809.1 GB, 1809141727232 bytes
255 heads, 63 sectors/track, 219948 cylinders, total 3533479936 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/pve-data-real doesn't contain a valid partition table


Disk /dev/mapper/pve-vzsnap--node1--0-cow: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/pve-vzsnap--node1--0-cow doesn't contain a valid partition table
 
What does
Code: 
findmnt
return? Is your external disk mounted correctly?

VZDump makes some temporary files when you're backing up a running vm, to look if that's the problem use the vzdump command in a shell, you can take the command from the log output:
Code: 
vzdump 109 --remove 0 --mode snapshot --compress lzo --storage ExternalDisk --node node1 --tmpdir /ehdd
note the added -tmpdir option, which points to your external disk.
 
Last edited by a moderator:
It's seems not mounted correctly , Right?
Code: 
root@node1:~# findmnt
TARGET                                               SOURCE                                FSTYPE      OPTIONS
/                                                    /dev/mapper/pve-root                  ext3        rw,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered
├─/sys                                               sysfs                                 sysfs       rw,nosuid,nodev,noexec,relatime
├─/proc                                              proc                                  proc        rw,nosuid,nodev,noexec,relatime
│ ├─/proc/vz/beancounter                             beancounter                           cgroup      rw,relatime,blkio,name=beancounter
│ ├─/proc/vz/container                               container                             cgroup      rw,relatime,freezer,devices,name=container
│ └─/proc/vz/fairsched                               fairsched                             cgroup      rw,relatime,cpuacct,cpu,cpuset,name=fairsched
├─/dev                                               udev                                  devtmpfs    rw,relatime,size=10240k,nr_inodes=9274667,mode=755
│ └─/dev/pts                                         devpts                                devpts      rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
├─/run                                               tmpfs                                 tmpfs       rw,nosuid,noexec,relatime,size=7422044k,mode=755
│ ├─/run/lock                                        tmpfs                                 tmpfs       rw,nosuid,nodev,noexec,relatime,size=5120k
│ └─/run/shm                                         tmpfs                                 tmpfs       rw,nosuid,nodev,noexec,relatime,size=14844080k
├─/var/lib/vz                                        /dev/mapper/pve-data                  ext3        rw,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered
│ └─/var/lib/vz/root/109                             /var/lib/vz/private/109[/private/109] simfs       rw,relatime
│   ├─/var/lib/vz/root/109/proc                      proc                                  proc        rw,relatime
│   │ └─/var/lib/vz/root/109/proc/sys/fs/binfmt_misc                                       binfmt_misc rw,nosuid,nodev,noexec,relatime
│   ├─/var/lib/vz/root/109/sys                       sysfs                                 sysfs       rw,relatime
│   │ └─/var/lib/vz/root/109/sys/fs/cgroup                                                 tmpfs       rw,relatime,size=4k,nr_inodes=4194304,mode=755
│   ├─/var/lib/vz/root/109/dev                                                             devtmpfs    rw,nosuid,noexec,relatime,size=16777216k,nr_inodes=4194304,mode=755
│   │ └─/var/lib/vz/root/109/dev/pts                                                       devpts      rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
│   └─/var/lib/vz/root/109/run                                                             tmpfs       rw,nosuid,noexec,relatime,size=3355444k,nr_inodes=4194304,mode=755
│     ├─/var/lib/vz/root/109/run/lock                                                      tmpfs       rw,nosuid,nodev,noexec,relatime,size=5120k,nr_inodes=4194304
│     ├─/var/lib/vz/root/109/run/shm                                                       tmpfs       rw,relatime,size=16777216k,nr_inodes=4194304
│     └─/var/lib/vz/root/109/run/user                                                      tmpfs       rw,nosuid,nodev,noexec,relatime,size=102400k,nr_inodes=4194304,mode=755
├─/sdb                                               /dev/sdb1                             ext4        rw,relatime,barrier=1,data=ordered
├─/tmp                                               tmpfs                                 tmpfs       rw,nosuid,nodev,relatime,size=14844080k
├─/var/lib/nfs/rpc_pipefs                            rpc_pipefs                            rpc_pipefs  rw,relatime
├─/etc/pve                                           /dev/fuse                             fuse        rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other
└─/mnt/vzsnap0                                       /dev/mapper/pve-vzsnap--node1--0      ext3        rw,relatime,errors=continue,user_xattr,acl,barrier=0,data=ordered
 
toni.patroni said:
No, I also doesn't see it. How do you mounted your disk in /ehdd?

http://www.debiantutorials.com/how-...-or-partition-using-uuid-and-ext4-filesystem/ may help, you can also skip the formatting part if you already have a valid formatted external disk (jump right to step 4).
Click to expand...
Also It's not in mounted disks.
Code: 
root@node1:~# df -h
Filesystem                        Size  Used Avail Use% Mounted on
udev                               10M     0   10M   0% /dev
tmpfs                             7.1G  432K  7.1G   1% /run
/dev/mapper/pve-root               95G   94G     0 100% /
tmpfs                             5.0M     0  5.0M   0% /run/lock
tmpfs                              15G   25M   15G   1% /run/shm
/dev/mapper/pve-data              1.7T  381G  1.3T  23% /var/lib/vz
/dev/sdb1                         1.8T  1.6T  215G  89% /sdb
tmpfs                              15G     0   15G   0% /tmp
/dev/fuse                          30M   24K   30M   1% /etc/pve
/var/lib/vz/private/109           100G   45G   56G  45% /var/lib/vz/root/109
none                               16G  4.0K   16G   1% /var/lib/vz/root/109/dev
none                              4.0K     0  4.0K   0% /var/lib/vz/root/109/sys/fs/cgroup
none                              3.3G  104K  3.2G   1% /var/lib/vz/root/109/run
none                              5.0M     0  5.0M   0% /var/lib/vz/root/109/run/lock
none                               16G     0   16G   0% /var/lib/vz/root/109/run/shm
none                              100M     0  100M   0% /var/lib/vz/root/109/run/user
/dev/mapper/pve-vzsnap--node1--0  1.7T  381G  1.3T  23% /mnt/vzsnap0
I must mention , My External hdd has ntfs format So I mounted with ntfs-3g type with this in fstab:
Code: 
/dev/sdc1               /ehdd           ntfs-3g defaults        1       1
 
Looks good to me.
So what does
Code: 
mount /ehdd
output?

-------
EDIT:
For NTFS you maybe shouldn't use the "defaults" option in fstab, use "permission" instead.
Or to be really specific use : uid=1000,umask=0022
Where uid is the user id from the owner of the disk and umask is the right mask for files and directories.
Use this eg.:
Code: 
/dev/sdc1               /ehdd           ntfs-3g uid=1000,umask=0022        0       0
 
Last edited by a moderator:
You must log in or register to reply here.
Top Bottom