Volume Action "Move Volume" for a container from local to nfs never ends

[ihr]

Hello,

I'm in the process of migrating my LXC contains to NFS. And I'm not able to do it. The process starts but never ends. See this screen capture:


The disk size is 2Gb. The network activity in the NAS is zero. The entire text in the Output is generated in just a few seconds. But as you see at the bottom, the task is running and never ends.

This happens only for LXC storage. The Virtual Machine disks was migrated without having to stop them.

Looks like there is a bug somewhere?

 

[ihr]

Adding more information.

Before the problem started, I was able to submit ONE container rootfs to the share. Since then, I'm unable to send more LXC rootfs to the NFS.

The NFS should be working, as it is already in use by other Virtual Machines and one Container and they raise no errors.

When I cancel the script after a long period of inactivity, the tasks shows the following:

sent 323,608,308 bytes received 106,939 bytes 215,810,164.67 bytes/sec
total size is 323,107,725 speedup is 1.00
malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "(end of string)") at /usr/share/perl5/PVE/Tools.pm line 1011, <GEN3998> chunk 1.
TASK ERROR: interrupted by unexpected signal

And line 1011 of the Tools.pm script is doing

my $child_res = decode_json(readline_nointr($pipe_out));

But there is no complete stack trace so I don't know exactly what the script was doing other that waiting for a JSON response

 

[ihr]

Doing the copy from command line with the command:

pct move-volume 141 rootfs vms

Generates the following output in syslog:

Jan 13 11:11:42 hidra pct[2591372]: <root@pam> starting task UPID:hidra:00278A8D:0352CBF7:63C12E5E:move_volume:141:root@pam:
Jan 13 11:11:42 hidra pct[2591373]: <root@pam> move volume CT 141: move --volume rootfs --storage vms
Jan 13 11:11:42 hidra kernel: loop0: detected capacity change from 0 to 4194304
Jan 13 11:11:42 hidra kernel: EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.

But the process stops here.

Jan 13 11:13:55 hidra pct[2591373]: interrupted by unexpected signal
Jan 13 11:13:55 hidra pct[2591372]: <root@pam> end task UPID:hidra:00278A8D:0352CBF7:63C12E5E:move_volume:141:root@pam: interrupted by unexpected signal

 

[ihr]

Hm... this is getting worst!

One of the CT Containers had its disk in a local storage drive whose name does not exists on other servers and I wanted to move it to "local-zfs" in order to see if I could migrate the container to another server later or to the NFS but it is not possible.

The problem is the same as trying to transfer to NFS. The transfer process does not stop in a reasonable amount of time (2 hours for 12Gb) that normally takes only less than a minute to transfer from one disk to the other!

This is the task output:

Number of files: 20,685 (reg: 15,331, dir: 3,508, link: 1,846)
Number of created files: 20,684 (reg: 15,331, dir: 3,507, link: 1,846)
Number of deleted files: 0
Number of regular files transferred: 13,742
Total file size: 7,135,906,233 bytes
Total transferred file size: 7,075,710,142 bytes
Literal data: 7,075,710,142 bytes
Matched data: 0 bytes
File list size: 982,928
File list generation time: 0.020 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 7,078,725,839
Total bytes received: 368,153

sent 7,078,725,839 bytes  received 368,153 bytes  59,239,280.27 bytes/sec
total size is 7,135,906,233  speedup is 1.01
malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "(end of string)") at /usr/share/perl5/PVE/Tools.pm line 1011, <GEN928> chunk 1.
TASK ERROR: interrupted by unexpected signal

 

[ihr]

Te problem is solved.

Restarting the nodes did the trick! But it is worth exploring the root cause to increase services stability. I normally avoid to restart the servers as that causes undesired service interruption.