Problem with backup on lvm

Melanxolik · Dec 3, 2015

Hello, I have two works cluster and three other nodes.
When I do backups from VM at this nodes, I see problem:

INFO: starting new backup job: vzdump 317 --storage backup02 --node cluster-1-5
INFO: Starting Backup of VM 317 (qemu)
INFO: status = stopped
INFO: update VM 317: -lock backup
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/backup02/dump/vzdump-qemu-317-2015_12_03-14_18_12.vma'
INFO: starting kvm to execute backup task
INFO: started backup task '733b5d23-d635-4c9b-a032-d4313ab2a070'
INFO: status: 0% (1048576/18874368000), sparse 0% (1044480), duration 3, 0/0 MB/s

After a long time, 1 or 2 days - nothing, backup freeze... I got to gui and stop this task.
This problems repeats often with any vmid disk.

Could you help me diag this problem?

fireon · Dec 3, 2015

Same problem here with pve4. Sometimes. So... every 5 wheeks... on some vm's. Maybe an Problem with the storage (Qnap). We have this problems only when we have an qnap as nfsserver. On us proxmox or ubuntu's Backupserver we never had this problem before.

Melanxolik · Dec 4, 2015

it's interesting, I have backup storage on gluster and problem repeat on pve3

Melanxolik · Dec 15, 2015

I try create backup to local storage, and repeat problem:

Code:

root@cluster-1-1:/mnt/pve/backup02/dump# vzdump 108 --remove 1 --mode snapshot --compress gzip --storage backup01 --remove 1
INFO: starting new backup job: vzdump 108 --remove 1 --mode snapshot --compress gzip --storage backup01
INFO: Starting Backup of VM 108 (qemu)
INFO: status = stopped
INFO: update VM 108: -lock backup
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/backup01/dump/vzdump-qemu-108-2015_12_15-14_17_32.vma.gz'
INFO: starting kvm to execute backup task
INFO: started backup task 'ad5a8d55-13a4-4262-96e2-13e694229317'
INFO: status: 0% (5308416/18874368000), sparse 0% (1040384), duration 3, 1/1 MB/s
^CERROR: interrupted by signal
INFO: aborting backup job

after 1 hour, I canceled this task.

and enter in console qm unlock 108

and start task backup again:

Code:

# vzdump 108 --remove 1 --mode snapshot --compress gzip --storage local --remove 1
INFO: starting new backup job: vzdump 108 --remove 1 --mode snapshot --compress gzip --storage local
INFO: Starting Backup of VM 108 (qemu)
INFO: status = running
INFO: update VM 108: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/var/lib/vz/dump/vzdump-qemu-108-2015_12_15-14_33_12.vma.gz'
ERROR: unable to connect to VM 108 qmp socket - timeout after 31 retries
INFO: aborting backup job

strace

Code:

write(7, "INFO: Starting Backup of VM 108 "..., 39) = 39
write(1, "INFO: status = stopped\n", 23INFO: status = stopped
) = 23
write(7, "INFO: status = stopped\n", 23) = 23
select(8, [6], NULL, NULL, {1, 0})  = 1 (in [6], left {0, 677164})
read(6, "INFO: update VM 108: -lock backu"..., 4096) = 34
write(1, "INFO: update VM 108: -lock backu"..., 34INFO: update VM 108: -lock backup
) = 34
write(7, "INFO: update VM 108: -lock backu"..., 34) = 34
select(8, [6], NULL, NULL, {1, 0})  = 1 (in [6], left {0, 994405})
read(6, "INFO: backup mode: stop\nINFO: io"..., 4096) = 49
write(1, "INFO: backup mode: stop\n", 24INFO: backup mode: stop
) = 24
write(7, "INFO: backup mode: stop\n", 24) = 24
write(1, "INFO: ionice priority: 7\n", 25INFO: ionice priority: 7
) = 25
write(7, "INFO: ionice priority: 7\n", 25) = 25
select(8, [6], NULL, NULL, {1, 0})  = 0 (Timeout)
open("/proc/127198/stat", O_RDONLY)  = 10
ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff0e91e000) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(10, 0, SEEK_CUR)  = 0
fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
fcntl(10, F_SETFD, FD_CLOEXEC)  = 0
brk(0x339a000)  = 0x339a000
read(10, "127198 (task UPID:clust) S 12718"..., 8192) = 352
close(10)  = 0
brk(0x3398000)  = 0x3398000
select(8, [6], NULL, NULL, {1, 0})  = 0 (Timeout)
open("/proc/127198/stat", O_RDONLY)  = 10
ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff0e91e000) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(10, 0, SEEK_CUR)  = 0
fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
fcntl(10, F_SETFD, FD_CLOEXEC)  = 0
read(10, "127198 (task UPID:clust) S 12718"..., 8192) = 352
close(10)  = 0
select(8, [6], NULL, NULL, {1, 0})  = 0 (Timeout)
open("/proc/127198/stat", O_RDONLY)  = 10
ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff0e91e000) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(10, 0, SEEK_CUR)  = 0
fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
fcntl(10, F_SETFD, FD_CLOEXEC)  = 0
read(10, "127198 (task UPID:clust) S 12718"..., 8192) = 352
close(10)  = 0
select(8, [6], NULL, NULL, {1, 0})  = 0 (Timeout)
open("/proc/127198/stat", O_RDONLY)  = 10
ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff0e91e000) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(10, 0, SEEK_CUR)  = 0
fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
fcntl(10, F_SETFD, FD_CLOEXEC)  = 0
read(10, "127198 (task UPID:clust) S 12718"..., 8192) = 352
close(10)  = 0
select(8, [6], NULL, NULL, {1, 0})  = 0 (Timeout)
open("/proc/127198/stat", O_RDONLY)  = 10
ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff0e91e000) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(10, 0, SEEK_CUR)  = 0
fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
fcntl(10, F_SETFD, FD_CLOEXEC)  = 0
read(10, "127198 (task UPID:clust) S 12718"..., 8192) = 352
close(10)  = 0
select(8, [6], NULL, NULL, {1, 0})  = 0 (Timeout)
open("/proc/127198/stat", O_RDONLY)  = 10
ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff0e91e000) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(10, 0, SEEK_CUR)  = 0
fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
fcntl(10, F_SETFD, FD_CLOEXEC)  = 0
read(10, "127198 (task UPID:clust) S 12718"..., 8192) = 352
close(10)  = 0
select(8, [6], NULL, NULL, {1, 0})  = 0 (Timeout)
open("/proc/127198/stat", O_RDONLY)  = 10
ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff0e91e000) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(10, 0, SEEK_CUR)  = 0
fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
fcntl(10, F_SETFD, FD_CLOEXEC)  = 0
read(10, "127198 (task UPID:clust) S 12718"..., 8192) = 352
close(10)  = 0
select(8, [6], NULL, NULL, {1, 0})  = 0 (Timeout)
open("/proc/127198/stat", O_RDONLY)  = 10
ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff0e91e000) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(10, 0, SEEK_CUR)  = 0
fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
fcntl(10, F_SETFD, FD_CLOEXEC)  = 0
read(10, "127198 (task UPID:clust) S 12718"..., 8192) = 353
close(10)  = 0
select(8, [6], NULL, NULL, {1, 0})  = 0 (Timeout)
open("/proc/127198/stat", O_RDONLY)  = 10
ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff0e91e000) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(10, 0, SEEK_CUR)  = 0
fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
fcntl(10, F_SETFD, FD_CLOEXEC)  = 0
read(10, "127198 (task UPID:clust) S 12718"..., 8192) = 353
close(10)  = 0
select(8, [6], NULL, NULL, {1, 0})  = 0 (Timeout)
open("/proc/127198/stat", O_RDONLY)  = 10
ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff0e91e000) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(10, 0, SEEK_CUR)  = 0
fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
fcntl(10, F_SETFD, FD_CLOEXEC)  = 0
read(10, "127198 (task UPID:clust) S 12718"..., 8192) = 353
close(10)  = 0
select(8, [6], NULL, NULL, {1, 0})  = 0 (Timeout)
open("/proc/127198/stat", O_RDONLY)  = 10
ioctl(10, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff0e91e000) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(10, 0, SEEK_CUR)  = 0
fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
fcntl(10, F_SETFD, FD_CLOEXEC)  = 0
read(10, "127198 (task UPID:clust) S 12718"..., 8192) = 353

I watched this text from 1 hour.

Could you please diagnostic this trouble? This problem repeats with many VM on cluster.

Search

Search

Problem with backup on lvm

Melanxolik

Well-Known Member

fireon

Distinguished Member

Melanxolik

Well-Known Member

Melanxolik

Well-Known Member