can not start the CT after restart the node

[kuwang]

Hi,

I use zfs to store the CT.

after restart the node, the zfs pool can not be imported, so I have to import it myself. Now I can not start the openvz CT anymore. It just try to mount the CT disk to the /var/lib/vz/private/ but fail due to the size of that folder is only 15GB. Do you guys know any way to debug it.

2016-10-24T19:29:33+0700 vzctl : CT 186 : Warning: distribution not specified in CT config, using defaults from /etc/vz/dists/default
2016-10-24T19:29:33+0700 vzctl : CT 186 : Starting container ...
2016-10-24T19:29:33+0700 vzctl : CT 186 : stat(/var/lib/vz/private/186): No such file or directory
2016-10-24T19:29:33+0700 vzctl : CT 186 : stat(/var/lib/vz/private/186): No such file or directory
2016-10-24T19:29:33+0700 vzctl : CT 186 : Can't umount /var/lib/vz/root/186: Invalid argument
2016-10-24T19:29:33+0700 vzctl : CT 186 : stat(/var/lib/vz/private/186): No such file or directory
2016-10-24T19:29:33+0700 vzctl : CT 186 : Adding IP address(es): 103.221.222.167
2016-10-24T19:29:34+0700 vzctl : CT 186 : Setting CPU limit: 100
2016-10-24T19:29:34+0700 vzctl : CT 186 : Setting CPU units: 1000
2016-10-24T19:29:34+0700 vzctl : CT 186 : Setting CPUs: 2
2016-10-24T19:29:34+0700 vzctl : CT 186 : Error: Unable to apply new quota values: quota not running
2016-10-24T19:29:34+0700 vzctl : CT 186 : Container start failed (try to check kernel messages, e.g. "dmesg | tail")
2016-10-24T19:29:34+0700 vzctl : CT 186 : Killing container ...
2016-10-24T19:29:34+0700 vzctl : CT 186 : stat(/var/lib/vz/private/186): No such file or directory
2016-10-24T19:29:34+0700 vzctl : CT 186 : stat(/var/lib/vz/private/186): No such file or directory
2016-10-24T19:29:35+0700 vzctl : CT 186 : Container was stopped

root@sv3:~# pveversion -V
proxmox-ve-2.6.32: 3.4-177 (running kernel: 2.6.32-46-pve)
pve-manager: 3.4-15 (running version: 3.4-15/e1daa307)
pve-kernel-2.6.32-39-pve: 2.6.32-157
pve-kernel-2.6.32-46-pve: 2.6.32-177
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.7-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.10-3
pve-cluster: 3.0-20
qemu-server: 3.4-9
pve-firmware: 1.1-5
libpve-common-perl: 3.0-27
libpve-access-control: 3.0-16
libpve-storage-perl: 3.0-35
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-8
vzctl: 4.0-1pve6
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 2.2-27
ksm-control-daemon: 1.1-1
glusterfs-client: 3.5.2-1

 

[dietmar]

I use zfs to store the CT.

This works if you use our new container toolkit (pct), but the old openvz stack never supported zfs - only ext3/ext4 and NFS.

 

[kuwang]

hmm
I've installed it and it worked very well before

 

[dietmar]

what is the output of

# df

 

[kuwang]

root@sv3:~# df
Filesystem                                   1K-blocks      Used  Available Use% Mounted on
udev                                             10240         0      10240   0% /dev
tmpfs                                          9887808     19724    9868084   1% /run
/dev/mapper/pve-root                          16382888   5080232   10463796  33% /
tmpfs                                             5120         0       5120   0% /run/lock
tmpfs                                         19775600     27972   19747628   1% /run/shm
/dev/mapper/pve-data                          15350768  12224136    3126632  80% /var/lib/vz
azdigi-zfs-pool                             7305253888 453589376 6851664512   7% /azdigi-zfs-pool
/azdigi-zfs-pool/zfs-containers/private/100   20971520   3757568   17213952  18% /var/lib/vz/root/100
none                                           1048576         4    1048572   1% /var/lib/vz/root/100/dev
none                                           1048576         0    1048576   0% /var/lib/vz/root/100/dev/shm
......
/azdigi-zfs-pool/zfs-containers/private/156   20971520   3118208   17853312  15% /var/lib/vz/root/156
none                                           1048576         4    1048572   1% /var/lib/vz/root/156/dev
none                                                 4         0          4   0% /var/lib/vz/root/156/sys/fs/cgroup
none                                            209716        72     209644   1% /var/lib/vz/root/156/run
none                                              5120         0       5120   0% /var/lib/vz/root/156/run/lock
none                                           1048576         0    1048576   0% /var/lib/vz/root/156/run/shm
none                                            102400         0     102400   0% /var/lib/vz/root/156/run/user
/azdigi-zfs-pool/zfs-containers/private/151   20971520   7565824   13405696  37% /var/lib/vz/root/151
none                                           1048576         4    1048572   1% /var/lib/vz/root/151/dev
none                                                 4         0          4   0% /var/lib/vz/root/151/sys/fs/cgroup
none                                            209716        84     209632   1% /var/lib/vz/root/151/run
none                                              5120         0       5120   0% /var/lib/vz/root/151/run/lock
none                                           1048576         0    1048576   0% /var/lib/vz/root/151/run/shm
none                                            102400         0     102400   0% /var/lib/vz/root/151/run/user
/azdigi-zfs-pool/zfs-containers/private/181   20971520       128   20971392   1% /var/lib/vz/root/181
none                                           1048576         4    1048572   1% /var/lib/vz/root/181/dev
none                                           1048576         0    1048576   0% /var/lib/vz/root/181/dev/shm
/azdigi-zfs-pool/zfs-containers/private/105   20971520       128   20971392   1% /var/lib/vz/root/105
none                                           1048576         4    1048572   1% /var/lib/vz/root/105/dev
none                                           1048576         0    1048576   0% /var/lib/vz/root/105/dev/shm
/azdigi-zfs-pool/zfs-containers/private/134   20971520   3784832   17186688  19% /var/lib/vz/root/134
none                                           1048576         4    1048572   1% /var/lib/vz/root/134/dev
none                                           1048576         0    1048576   0% /var/lib/vz/root/134/dev/shm
/azdigi-zfs-pool/zfs-containers/private/132   20971520   2582016   18389504  13% /var/lib/vz/root/132
none                                           1048576         4    1048572   1% /var/lib/vz/root/132/dev
none                                           1048576         0    1048576   0% /var/lib/vz/root/132/dev/shm
/azdigi-zfs-pool/zfs-containers/private/117   31457280       128   31457152   1% /var/lib/vz/root/117
none                                           1572864         4    1572860   1% /var/lib/vz/root/117/dev
none                                           1572864        28    1572836   1% /var/lib/vz/root/117/dev/shm
/azdigi-zfs-pool/zfs-containers/private/141   20971520   2142720   18828800  11% /var/lib/vz/root/141
devtmpfs                                       1048576         0    1048576   0% /var/lib/vz/root/141/dev
tmpfs                                          1048576         0    1048576   0% /var/lib/vz/root/141/dev/shm
tmpfs                                          1048576        80    1048496   1% /var/lib/vz/root/141/run
tmpfs                                          1048576         0    1048576   0% /var/lib/vz/root/141/sys/fs/cgroup
/azdigi-zfs-pool/zfs-containers/private/102   31457280       128   31457152   1% /var/lib/vz/root/102
none                                           1572864         4    1572860   1% /var/lib/vz/root/102/dev
none                                           1572864        28    1572836   1% /var/lib/vz/root/102/dev/shm
/azdigi-zfs-pool/zfs-containers/private/197   41943040       128   41942912   1% /var/lib/vz/root/197
none                                           2097152         4    2097148   1% /var/lib/vz/root/197/dev
none                                           2097152         0    2097152   0% /var/lib/vz/root/197/dev/shm
/azdigi-zfs-pool/zfs-containers/private/201   20971520       128   20971392   1% /var/lib/vz/root/201
devtmpfs                                       1048576         0    1048576   0% /var/lib/vz/root/201/dev
tmpfs                                          1048576         0    1048576   0% /var/lib/vz/root/201/dev/shm
tmpfs                                          1048576       124    1048452   1% /var/lib/vz/root/201/run
tmpfs                                          1048576         0    1048576   0% /var/lib/vz/root/201/sys/fs/cgroup
tmpfs                                           209716         0     209716   0% /var/lib/vz/root/201/run/user/0
/azdigi-zfs-pool/zfs-containers/private/137   31457280   3231488   28225792  11% /var/lib/vz/root/137
none                                           1572864         4    1572860   1% /var/lib/vz/root/137/dev
none                                           1572864         0    1572864   0% /var/lib/vz/root/137/dev/shm
/azdigi-zfs-pool/zfs-containers/private/138   20971520       128   20971392   1% /var/lib/vz/root/138
none                                           1048576         4    1048572   1% /var/lib/vz/root/138/dev
none                                           1048576         0    1048576   0% /var/lib/vz/root/138/dev/shm
/azdigi-zfs-pool/zfs-containers/private/101   20971520       128   20971392   1% /var/lib/vz/root/101
none                                           1048576         4    1048572   1% /var/lib/vz/root/101/dev
none                                           1048576         0    1048576   0% /var/lib/vz/root/101/dev/shm
tmpfs                                           209716         0     209716   0% /var/lib/vz/root/141/run/user/0
/dev/fuse                                        30720       156      30564   1% /etc/pve
/azdigi-zfs-pool/zfs-containers/private/199   20971520       128   20971392   1% /var/lib/vz/root/199
none                                           1048576         4    1048572   1% /var/lib/vz/root/199/dev
none                                           1048576         0    1048576   0% /var/lib/vz/root/199/dev/shm
/azdigi-zfs-pool/zfs-containers/private/103   20971520       128   20971392   1% /var/lib/vz/root/103
none                                           1048576         4    1048572   1% /var/lib/vz/root/103/dev
none                                                 4         0          4   0% /var/lib/vz/root/103/sys/fs/cgroup
none                                            209716        52     209664   1% /var/lib/vz/root/103/run
none                                              5120         0       5120   0% /var/lib/vz/root/103/run/lock
none                                           1048576         0    1048576   0% /var/lib/vz/root/103/run/shm
none                                            102400         0     102400   0% /var/lib/vz/root/103/run/user
/azdigi-zfs-pool/zfs-containers/private/195   20971520       128   20971392   1% /var/lib/vz/root/195
none                                           1048576         4    1048572   1% /var/lib/vz/root/195/dev
none                                           1048576         0    1048576   0% /var/lib/vz/root/195/dev/shm

 

[dietmar]

seems you mount your zfs volume at the wrong place (noting is mounted at /var/lib/vz/)