Cloning KVM-Machine stuck at 0 bytes

[SPQRInc]

Hello,

I would like to create a clone of a KVM machine - yesterday I was able to create a clone of the same machine within some minutes. Now it takes hours and is still stuck an "0%":

create full clone of drive ide0 (local:100/vm-100-disk-1.raw)
Formatting '/var/lib/vz/images/111/vm-111-disk-1.raw', fmt=raw size=536870912000
transferred: 0 bytes remaining: 536870912000 bytes total: 536870912000 bytes progression: 0.00 %

I wonder why this is the case. The task is running (i restarted it some minutes ago) - http://d.pr/tJn4

Any ideas what is going wrong again?

Edit: Disk I/O is low as the task is not writing any data to hdd (http://d.pr/1furj).


If I trace the process ID which is given in the overview, I'm getting this result:

select(24, [16 19], NULL, NULL, {0, 553078}) = 0 (Timeout)

select(24, [16 19], NULL, NULL, {1, 0}) = 0 (Timeout)

select(24, [16 19], NULL, NULL, {1, 0}) = 0 (Timeout)

select(24, [16 19], NULL, NULL, {1, 0}) = 0 (Timeout)

select(24, [16 19], NULL, NULL, {1, 0}) = 0 (Timeout)

select(24, [16 19], NULL, NULL, {1, 0}) = 0 (Timeout)

select(24, [16 19], NULL, NULL, {1, 0}) = 0 (Timeout)

select(24, [16 19], NULL, NULL, {1, 0}) = 0 (Timeout)

select(24, [16 19], NULL, NULL, {1, 0}) = 0 (Timeout)

select(24, [16 19], NULL, NULL, {1, 0}) = 0 (Timeout)

select(24, [16 19], NULL, NULL, {1, 0}) = 0 (Timeout)

select(24, [16 19], NULL, NULL, {1, 0}) = 0 (Timeout)

select(24, [16 19], NULL, NULL, {1, 0}) = 0 (Timeout)

select(24, [16 19], NULL, NULL, {1, 0}) = 0 (Timeout)

select(24, [16 19], NULL, NULL, {1, 0}) = 0 (Timeout)

If I trace the process 233113 (which is actually working) I'm getting this result:

lseek(7, 49650073600, SEEK_DATA) = 49650073600

lseek(7, 49650073600, SEEK_HOLE) = 49650077696

lseek(7, 49650077696, SEEK_DATA) = 51797557248

lseek(7, 51797557248, SEEK_DATA) = 51797557248

lseek(7, 51797557248, SEEK_HOLE) = 51797565440

lseek(7, 51797565440, SEEK_DATA) = 51797622784

lseek(7, 51797622784, SEEK_DATA) = 51797622784

lseek(7, 51797622784, SEEK_HOLE) = 51797630976

lseek(7, 51797630976, SEEK_DATA) = 51797688320

lseek(7, 51797688320, SEEK_DATA) = 51797688320

lseek(7, 51797688320, SEEK_HOLE) = 51801108480

lseek(7, 51801108480, SEEK_DATA) = 51831242752

lseek(7, 51831242752, SEEK_DATA) = 51831242752

lseek(7, 51831242752, SEEK_HOLE) = 51833991168

lseek(7, 51833991168, SEEK_DATA) = 51931774976

lseek(7, 51931774976, SEEK_DATA) = 51931774976

lseek(7, 51931774976, SEEK_HOLE) = 51962368000

lseek(7, 51962368000, SEEK_DATA) = 51962380288

lseek(7, 51962380288, SEEK_DATA) = 51962380288

lseek(7, 51962380288, SEEK_HOLE) = 51963707392

lseek(7, 51963707392, SEEK_DATA) = 51963756544

lseek(7, 51963756544, SEEK_DATA) = 51963756544

lseek(7, 51963756544, SEEK_HOLE) = 51963973632

lseek(7, 51963973632, SEEK_DATA) = 53945040896

lseek(7, 53945040896, SEEK_DATA) = 53945040896

lseek(7, 53945040896, SEEK_HOLE) = 53945049088

lseek(7, 53945049088, SEEK_DATA) = 53945106432

lseek(7, 53945106432, SEEK_DATA) = 53945106432

lseek(7, 53945106432, SEEK_HOLE) = 53945110528

lseek(7, 53945110528, SEEK_DATA) = 53945171968

lseek(7, 53945171968, SEEK_DATA) = 53945171968

lseek(7, 53945171968, SEEK_HOLE) = 53947260928

lseek(7, 53947260928, SEEK_DATA) = 53978726400

lseek(7, 53978726400, SEEK_DATA) = 53978726400

lseek(7, 53978726400, SEEK_HOLE) = 53981294592

lseek(7, 53981294592, SEEK_DATA) = 54079258624

lseek(7, 54079258624, SEEK_DATA) = 54079258624

lseek(7, 54079258624, SEEK_HOLE) = 54096543744

lseek(7, 54096543744, SEEK_DATA) = 54096560128

lseek(7, 54096560128, SEEK_DATA) = 54096560128

lseek(7, 54096560128, SEEK_HOLE) = 54096625664

lseek(7, 54096625664, SEEK_DATA) = 56092524544

lseek(7, 56092524544, SEEK_DATA) = 56092524544

lseek(7, 56092524544, SEEK_HOLE) = 56092532736

lseek(7, 56092532736, SEEK_DATA) = 56092590080

lseek(7, 56092590080, SEEK_DATA) = 56092590080

lseek(7, 56092590080, SEEK_HOLE) = 56092594176

lseek(7, 56092594176, SEEK_DATA) = 56092655616

lseek(7, 56092655616, SEEK_DATA) = 56092655616

lseek(7, 56092655616, SEEK_HOLE) = 56094593024

lseek(7, 56094593024, SEEK_DATA) = 56126210048

lseek(7, 56126210048, SEEK_DATA) = 56126210048

lseek(7, 56126210048, SEEK_HOLE) = 56128737280

lseek(7, 56128737280, SEEK_DATA) = 56226742272

lseek(7, 56226742272, SEEK_DATA) = 56226742272

lseek(7, 56226742272, SEEK_HOLE) = 56250884096

lseek(7, 56250884096, SEEK_DATA) = 58240008192

lseek(7, 58240008192, SEEK_DATA) = 58240008192

lseek(7, 58240008192, SEEK_HOLE) = 58240016384

lseek(7, 58240016384, SEEK_DATA) = 58240073728

lseek(7, 58240073728, SEEK_DATA) = 58240073728

lseek(7, 58240073728, SEEK_HOLE) = 58240077824

lseek(7, 58240077824, SEEK_DATA) = 58240139264

lseek(7, 58240139264, SEEK_DATA) = 58240139264

lseek(7, 58240139264, SEEK_HOLE) = 58241294336

 

[spirit]

Hi,
which proxmox version do you use ?
and which qemu version the vm is currently running ? (->monitor : info version)

 

[SPQRInc]

Ah, I forgot to tell you the details:

proxmox-ve: 4.1-28 (running kernel: 4.2.6-1-pve)

pve-manager: 4.1-2 (running version: 4.1-2/78c5f4a2)

pve-kernel-4.2.6-1-pve: 4.2.6-28

pve-kernel-2.6.32-43-pve: 2.6.32-166

pve-kernel-4.2.2-1-pve: 4.2.2-16

pve-kernel-2.6.32-26-pve: 2.6.32-114

pve-kernel-4.2.3-2-pve: 4.2.3-22

lvm2: 2.02.116-pve2

corosync-pve: 2.3.5-2

libqb0: 0.17.2-1

pve-cluster: 4.0-29

qemu-server: 4.0-42

pve-firmware: 1.1-7

libpve-common-perl: 4.0-42

libpve-access-control: 4.0-10

libpve-storage-perl: 4.0-38

pve-libspice-server1: 0.12.5-2

vncterm: 1.2-1

pve-qemu-kvm: 2.4-18

pve-container: 1.0-35

pve-firewall: 2.0-14

pve-ha-manager: 1.0-16

ksm-control-daemon: 1.2-1

glusterfs-client: 3.5.2-2+deb8u1

lxc-pve: 1.1.5-5

lxcfs: 0.13-pve2

cgmanager: 0.39-pve1

criu: 1.6.0-1

This is the resource usage on this machine:

Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf

udev 10M 0 10M 0% /dev

tmpfs 26G 319M 25G 2% /run

/dev/sda4 3,6T 886G 2,6T 26% /

tmpfs 63G 43M 63G 1% /dev/shm

tmpfs 5,0M 0 5,0M 0% /run/lock

tmpfs 63G 0 63G 0% /sys/fs/cgroup

/dev/sda3 946M 188M 694M 22% /boot

cgmfs 100K 0 100K 0% /run/cgmanager/fs

/dev/fuse 30M 28K 30M 1% /etc/pve

tmpfs 100K 0 100K 0% /run/lxcfs/controllers

tmpfs 13G 0 13G 0% /run/user/1000

info version does not show anything because the VM is not running.

 

[SPQRInc]

Update:

Now the VM is cloned (I do not know why) but it does not boot properly:

create full clone of drive ide0 (local:100/vm-100-disk-1.raw)
Formatting '/var/lib/vz/images/106/vm-106-disk-1.raw', fmt=raw size=536870912000
transferred: 0 bytes remaining: 536870912000 bytes total: 536870912000 bytes progression: 0.00 %
transferred: 5368709120 bytes remaining: 531502202880 bytes total: 536870912000 bytes progression: 1.00 %
transferred: 10737418240 bytes remaining: 526133493760 bytes total: 536870912000 bytes progression: 2.00 %
transferred: 16213501542 bytes remaining: 520657410458 bytes total: 536870912000 bytes progression: 3.02 %
transferred: 21635897753 bytes remaining: 515235014247 bytes total: 536870912000 bytes progression: 4.03 %
transferred: 27165668147 bytes remaining: 509705243853 bytes total: 536870912000 bytes progression: 5.06 %
transferred: 32641751449 bytes remaining: 504229160551 bytes total: 536870912000 bytes progression: 6.08 %
transferred: 38064147660 bytes remaining: 498806764340 bytes total: 536870912000 bytes progression: 7.09 %
transferred: 43540230963 bytes remaining: 493330681037 bytes total: 536870912000 bytes progression: 8.11 %
transferred: 49016314265 bytes remaining: 487854597735 bytes total: 536870912000 bytes progression: 9.13 %
transferred: 54599771750 bytes remaining: 482271140250 bytes total: 536870912000 bytes progression: 10.17 %
transferred: 60129542144 bytes remaining: 476741369856 bytes total: 536870912000 bytes progression: 11.20 %
transferred: 65712999628 bytes remaining: 471157912372 bytes total: 536870912000 bytes progression: 12.24 %
transferred: 71135395840 bytes remaining: 465735516160 bytes total: 536870912000 bytes progression: 13.25 %
transferred: 76611479142 bytes remaining: 460259432858 bytes total: 536870912000 bytes progression: 14.27 %
transferred: 82033875353 bytes remaining: 454837036647 bytes total: 536870912000 bytes progression: 15.28 %
transferred: 87456271564 bytes remaining: 449414640436 bytes total: 536870912000 bytes progression: 16.29 %
transferred: 92932354867 bytes remaining: 443938557133 bytes total: 536870912000 bytes progression: 17.31 %
transferred: 98462125260 bytes remaining: 438408786740 bytes total: 536870912000 bytes progression: 18.34 %
transferred: 103884521472 bytes remaining: 432986390528 bytes total: 536870912000 bytes progression: 19.35 %
transferred: 109306917683 bytes remaining: 427563994317 bytes total: 536870912000 bytes progression: 20.36 %
transferred: 114783000985 bytes remaining: 422087911015 bytes total: 536870912000 bytes progression: 21.38 %
transferred: 120420145561 bytes remaining: 416450766439 bytes total: 536870912000 bytes progression: 22.43 %
transferred: 126003603046 bytes remaining: 410867308954 bytes total: 536870912000 bytes progression: 23.47 %
transferred: 131533373440 bytes remaining: 405337538560 bytes total: 536870912000 bytes progression: 24.50 %
transferred: 136902082560 bytes remaining: 399968829440 bytes total: 536870912000 bytes progression: 25.50 %
transferred: 142270791680 bytes remaining: 394600120320 bytes total: 536870912000 bytes progression: 26.50 %
transferred: 147693187891 bytes remaining: 389177724109 bytes total: 536870912000 bytes progression: 27.51 %
transferred: 153061897011 bytes remaining: 383809014989 bytes total: 536870912000 bytes progression: 28.51 %
transferred: 158699041587 bytes remaining: 378171870413 bytes total: 536870912000 bytes progression: 29.56 %
transferred: 164228811980 bytes remaining: 372642100020 bytes total: 536870912000 bytes progression: 30.59 %
transferred: 169597521100 bytes remaining: 367273390900 bytes total: 536870912000 bytes progression: 31.59 %
transferred: 175180978585 bytes remaining: 361689933415 bytes total: 536870912000 bytes progression: 32.63 %
transferred: 180710748979 bytes remaining: 356160163021 bytes total: 536870912000 bytes progression: 33.66 %
transferred: 186240519372 bytes remaining: 350630392628 bytes total: 536870912000 bytes progression: 34.69 %
transferred: 191823976857 bytes remaining: 345046935143 bytes total: 536870912000 bytes progression: 35.73 %
transferred: 197407434342 bytes remaining: 339463477658 bytes total: 536870912000 bytes progression: 36.77 %
transferred: 202829830553 bytes remaining: 334041081447 bytes total: 536870912000 bytes progression: 37.78 %
transferred: 208413288038 bytes remaining: 328457623962 bytes total: 536870912000 bytes progression: 38.82 %
transferred: 213889371340 bytes remaining: 322981540660 bytes total: 536870912000 bytes progression: 39.84 %
transferred: 219472828825 bytes remaining: 317398083175 bytes total: 536870912000 bytes progression: 40.88 %
transferred: 225056286310 bytes remaining: 311814625690 bytes total: 536870912000 bytes progression: 41.92 %
transferred: 230478682521 bytes remaining: 306392229479 bytes total: 536870912000 bytes progression: 42.93 %
transferred: 235954765824 bytes remaining: 300916146176 bytes total: 536870912000 bytes progression: 43.95 %
transferred: 241484536217 bytes remaining: 295386375783 bytes total: 536870912000 bytes progression: 44.98 %
transferred: 246906932428 bytes remaining: 289963979572 bytes total: 536870912000 bytes progression: 45.99 %
transferred: 252436702822 bytes remaining: 284434209178 bytes total: 536870912000 bytes progression: 47.02 %
transferred: 257966473216 bytes remaining: 278904438784 bytes total: 536870912000 bytes progression: 48.05 %
transferred: 263496243609 bytes remaining: 273374668391 bytes total: 536870912000 bytes progression: 49.08 %
transferred: 268918639820 bytes remaining: 267952272180 bytes total: 536870912000 bytes progression: 50.09 %
transferred: 274341036032 bytes remaining: 262529875968 bytes total: 536870912000 bytes progression: 51.10
...

transferred: 488015659008 bytes remaining: 48855252992 bytes total: 536870912000 bytes progression: 90.90 %
transferred: 493545429401 bytes remaining: 43325482599 bytes total: 536870912000 bytes progression: 91.93 %
transferred: 499128886886 bytes remaining: 37742025114 bytes total: 536870912000 bytes progression: 92.97 %
transferred: 504712344371 bytes remaining: 32158567629 bytes total: 536870912000 bytes progression: 94.01 %
transferred: 510134740582 bytes remaining: 26736171418 bytes total: 536870912000 bytes progression: 95.02 %
transferred: 515557136793 bytes remaining: 21313775207 bytes total: 536870912000 bytes progression: 96.03 %
transferred: 521140594278 bytes remaining: 15730317722 bytes total: 536870912000 bytes progression: 97.07 %
transferred: 526616677580 bytes remaining: 10254234420 bytes total: 536870912000 bytes progression: 98.09 %
transferred: 531985386700 bytes remaining: 4885525300 bytes total: 536870912000 bytes progression: 99.09 %
transferred: 536870912000 bytes remaining: 0 bytes total: 536870912000 bytes progression: 100.00 %
transferred: 536870912000 bytes remaining: 0 bytes total: 536870912000 bytes progression: 100.00 %
TASK OK

During boot up I'm getting "unexpected incensestency" (http://d.pr/1bVdt )

 

[spirit]

This is really strange that you have freeze on clone when vm is stopped.

Are you sure that the source image is not corrupted or have bad blocks/sectors ?