I can't start LXC

JohnD said:
I got it.

Removed dev dirs in subvolumes.
zfs unmount nmve1/subvol....
zfs mount nvme1/subvol....
lxc-start works fine again.

No idea what is going in there but it worked.

Thanks again,.
Click to expand...

This worked for me to, had dev folderes in my zfs subvols. deleted it and unmunt / mount, start ct and away it goes. This happened after a powercut so might be a quirk with ZFS and power loss...
 
axgio said:
I have the same problem, but i'm using Proxmox for a month.
I would try the solution of JohnD but how I 'Remove dev dirs in subvolumes.' ?
Thank you in advance.

UPDATE: I found the solution in this article: https://securityboulevard.com/2019/09/proxmox-container-with-debian-10-does-not-work-after-upgrade/

Thank you!
Click to expand...


for those that follow in our footsteps and future me...... examples below and will need to be changed to match your storage


Bash: 
rm -rf /storage/proxmox/subvol-112-disk-0/dev/
zfs mount storage/proxmox/subvol-112-disk-0
pct start 112
rm -rf /storage/proxmox/subvol-113-disk-0/dev/
zfs mount storage/proxmox/subvol-113-disk-0
pct start 113
rm -rf /storage/proxmox/subvol-115-disk-0/dev/
zfs mount storage/proxmox/subvol-115-disk-0
pct start 115
.......
......
....
 
  • Like
Reactions: marcelprox
The same thing broke for me on my upgrade from PVE 5 to PVE 6. The issue is that there are files like /dev sitting in the mount point that ZFS wants to mount the subvol at, so ZFS complains that it is unable to mount the subvol, and then of course the container cannot start without the disk. In case it helps anyone else, below is my auto fix script (as-is, ymmv). It assumes your LXC subvols are on ZFS and mounted under /rpool/data/. Check your system with zfs list to check where your subvols are mounted. This also assumes that the root disk for each LXC is named ...-disk-0. Beware that there could be other lingering files besides /dev, for some of my containers, I also had to remove /var and /mnt

Bash: 
#!/bin/bash

STOPPED_CTS=$(pct list | grep stopped | awk '{print $1}')

for CT in $STOPPED_CTS; do
  pct unmount $CT
  zfs unmount rpool/data/subvol-$CT-disk-0
  rm -rf /rpool/data/subvol-$CT-disk-0/dev/
  zfs mount rpool/data/subvol-$CT-disk-0
  pct start $CT
done
 
Last edited:
  • Like
Reactions: fodder and Riesling.Dry
to fix this in the end I backed up all my CT and machines onto another pool - destroyed the old pool and made a new one, then imported all my machines again - works fine now :)
 
Josholith said:
The same thing broke for me on my upgrade from PVE 5 to PVE 6.
Click to expand...
And for me :(

Upgrade seemd to go well, but after reboot no container will start...

Kinda stuck here - tried your script and got:
Code: 
root@pve:~# zfs list
NAME                       USED  AVAIL     REFER  MOUNTPOINT
rpool                     1.03T   741G      104K  /rpool
rpool/ROOT                 973G   741G       96K  /rpool/ROOT
rpool/ROOT/pve-1           973G   741G      973G  /
rpool/subvol-1010-disk-0   568M  7.45G      568M  /rpool/subvol-1010-disk-0
rpool/subvol-111-disk-1   4.95G  15.1G     4.95G  /rpool/subvol-111-disk-1
rpool/subvol-222-disk-2   9.46G  70.5G     9.46G  /rpool/subvol-222-disk-2
rpool/subvol-333-disk-0    724M  29.3G      724M  /rpool/subvol-333-disk-0
rpool/subvol-444-disk-2   4.71G  3.29G     4.71G  /rpool/subvol-444-disk-2
rpool/subvol-666-disk-0    741M  7.28G      741M  /rpool/subvol-666-disk-0
rpool/subvol-777-disk-1   58.4G  71.6G     58.4G  /rpool/subvol-777-disk-1
rpool/subvol-888-disk-0    496M  7.52G      496M  /rpool/subvol-888-disk-0
rpool/swap                4.25G   742G     3.58G  -
root@pve:~#
root@pve:~# sh prox.fix.sh
no lock found trying to remove 'mounted'  lock
cannot open 'rpool/data/subvol-111-disk-0': dataset does not exist
cannot open 'rpool/data/subvol-111-disk-0': dataset does not exist
Job for pve-container@111.service failed because the control process exited with error code.
See "systemctl status pve-container@111.service" and "journalctl -xe" for details.
command 'systemctl start pve-container@111' failed: exit code 1
no lock found trying to remove 'mounted'  lock
cannot open 'rpool/data/subvol-222-disk-0': dataset does not exist
cannot open 'rpool/data/subvol-222-disk-0': dataset does not exist
Job for pve-container@222.service failed because the control process exited with error code.
See "systemctl status pve-container@222.service" and "journalctl -xe" for details.
command 'systemctl start pve-container@222' failed: exit code 1
no lock found trying to remove 'mounted'  lock
cannot open 'rpool/data/subvol-333-disk-0': dataset does not exist
cannot open 'rpool/data/subvol-333-disk-0': dataset does not exist
Job for pve-container@333.service failed because the control process exited with error code.
See "systemctl status pve-container@333.service" and "journalctl -xe" for details.
command 'systemctl start pve-container@333' failed: exit code 1
no lock found trying to remove 'mounted'  lock
cannot open 'rpool/data/subvol-444-disk-0': dataset does not exist
cannot open 'rpool/data/subvol-444-disk-0': dataset does not exist
Job for pve-container@444.service failed because the control process exited with error code.
See "systemctl status pve-container@444.service" and "journalctl -xe" for details.
command 'systemctl start pve-container@444' failed: exit code 1
no lock found trying to remove 'mounted'  lock
cannot open 'rpool/data/subvol-666-disk-0': dataset does not exist
cannot open 'rpool/data/subvol-666-disk-0': dataset does not exist
Job for pve-container@666.service failed because the control process exited with error code.
See "systemctl status pve-container@666.service" and "journalctl -xe" for details.
command 'systemctl start pve-container@666' failed: exit code 1
no lock found trying to remove 'mounted'  lock
cannot open 'rpool/data/subvol-777-disk-0': dataset does not exist
cannot open 'rpool/data/subvol-777-disk-0': dataset does not exist
Job for pve-container@777.service failed because the control process exited with error code.
See "systemctl status pve-container@777.service" and "journalctl -xe" for details.
command 'systemctl start pve-container@777' failed: exit code 1
no lock found trying to remove 'mounted'  lock
cannot open 'rpool/data/subvol-888-disk-0': dataset does not exist
cannot open 'rpool/data/subvol-888-disk-0': dataset does not exist
Job for pve-container@888.service failed because the control process exited with error code.
See "systemctl status pve-container@888.service" and "journalctl -xe" for details.
command 'systemctl start pve-container@888' failed: exit code 1
no lock found trying to remove 'mounted'  lock
cannot open 'rpool/data/subvol-1010-disk-0': dataset does not exist
cannot open 'rpool/data/subvol-1010-disk-0': dataset does not exist
Job for pve-container@1010.service failed because the control process exited with error code.
See "systemctl status pve-container@1010.service" and "journalctl -xe" for details.
command 'systemctl start pve-container@1010' failed: exit code 1
root@pve:~#
All packages are up to date.
Any ideas?
~R.

EDIT:
different paths: as shown in zflist, mine were in rpool/subvol-xxx - script assumes rpool/data/subvol-xxx - hence it couldn't work for me, my bad, sry ;)
cd /rpool and rm -rf plus reboot solved it for me though, see below:
 
Last edited:
o.k. solved :)

logs are written to be read ;)

in boot.log I spotted:
Code: 
Oct 05 16:06:55 pve systemd[1]: Starting Mount ZFS filesystems...
Oct 05 16:06:55 pve zfs[1113]: cannot mount '/rpool': directory is not empty
Oct 05 16:06:55 pve systemd[1]: zfs-mount.service: Main process exited, code=exited, status=1/FAILURE
Oct 05 16:06:55 pve systemd[1]: zfs-mount.service: Failed with result 'exit-code'.
Oct 05 16:06:55 pve systemd[1]: Failed to start Mount ZFS filesystems.

so I brutally rm -rf everything from /rpool, rebooted and... tadaaa all containers up'n running :)
...they even 'survive' subsequent reboots.

Puh...

krgds,
~R.
 
axgio said:
I have the same problem, but i'm using Proxmox for a month.
I would try the solution of JohnD but how I 'Remove dev dirs in subvolumes.' ?
Thank you in advance.

UPDATE: I found the solution in this article: https://securityboulevard.com/2019/09/proxmox-container-with-debian-10-does-not-work-after-upgrade/

Thank you!
Click to expand...
Thanks, I am having this same issue. However, I am updated to latest version. I have not rebooted yet though, restarted pveproxy and pvedaemon. Anyone know what to restart to get this to work?
 
BUG? Update needed?

Installed a new phresh Proxmox Server
downloaded template "Debian 10.0 (Standard) 10.0-1" through GUI
created new CT (ID 444)
entered new CT
apt-dist upgraded
everything fine.
rebooted host
new CT wont start
deep, deep in the logs:
"lxc-start 444 20191012123719.464 DEBUG conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 444 lxc pre-start with output: unsupported debian version '10.1'"

Solution:
edit file :/rpool/data/subvol-444-disk-0/etc/debian_version
said 10.1 there, made it 10, saved: CT starts ;)

I reckon w. the next update the file /usr/share/perl5/PVE/LXC/Setup/Debian.pm will consider Deb version 10+ :)

Cheers,
~R.
 
Last edited:
RollMops said:
BUG? Update needed?

Installed a new phresh Proxmox Server
downloaded template "Debian 10.0 (Standard) 10.0-1" through GUI
created new CT (ID 444)
entered new CT
apt-dist upgraded
everything fine.
rebooted host
new CT wont start
deep, deep in the logs:
"lxc-start 444 20191012123719.464 DEBUG conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 444 lxc pre-start with output: unsupported debian version '10.1'"
;)
Click to expand...
I fixed it with above method. What version of Proxmox does your web UI show?
 
liquidox said:
Latest is actually 6.0-7, and I had the same issue on 6.0-4 as you are having.
Click to expand...
Thanks for pointing that out!
I checked my apt sources and found no proxmox (except the out-commented enterprise).
Added the PVE No-subscription repository, found 23 (!) new packages.
Shows ver. 6.0-8 in GUI now.
Obviously I missed manually adding the apt source (which IMHO should be installed by/with proxmox upon install?) after I set up the new server, but the assumption above was basically correct and the file updated meanwhile. Thank you guys.
Will verify later, whether or not that lets a 10.1 deb run w./o. problems, but I guess it will :)
 
RollMops said:
I checked my apt sources and found no proxmox (except the out-commented enterprise).
Added the PVE No-subscription repository, found 23 (!) new packages.
Shows ver. 6.0-8 in GUI now.
Obviously I missed manually adding the apt source (which IMHO should be installed by/with proxmox upon install?) after I set up the new server, but the assumption above was basically correct and the file updated meanwhile. Thank you guys.
Click to expand...


during installation the pve-enterprise repository is added as default. if you commented it out and didn't add any of the other repositories (pve-no-subscription or pvetest), you obviously won't get any updates :)
 
  • Like
Reactions: marcelprox
same problem and same cryptic exit code 1 error.
my solution:
i noticed that the lxc containers that had trouble after the update from 5.4 to 6.0.9 and a reboot had one common thing.default size of 8G and they were'nt full,only about 1-1.5G consumed space.Every other LXC with allocated custom size was fine.
So i resized their space by just 1G and they started even surviving a reboot.

Since all the solutions seemed different but had in common a storage issue,can somebody more versed in the inner workings of proxmox explain why this happened? If it is the same issue after all...

thanks!
 
I have the same problem with two independent machines. Workaround solve's it, but it is not very comfortable to think about that after every reboot :/
 
Hi,

Have you found a solution because the only one that works for me is to mount manually and every time I reboot, I have to do this manipulation.

ps: Deleting the dev folder does not work.
Recreating the cache and updating the kernel does not work
Options in storage.cfg:
mkdir no
is_mountpoint yes does not work

Thanks
 
Can the proxmox team provide an answer to this problem which seems to impact many persons.

Running : pve-manager/5.4-13/aee6f0ec (running kernel: 4.15.18-21-pve)

In my case I have 15 containers running on a server.
I have deployed a new container yesterday (Ubuntu 18)



I have stopped a container today and impossible to restart it !

From ZFS point of view everything seems "normal" and "mounted"

Code: 
root@proxmonster:/home/gregober# zfs list
NAME                             USED  AVAIL  REFER  MOUNTPOINT
monster                          497G  2.68T   622M  /monster
monster/data                     493G  2.68T   240K  /monster/data
monster/data/subvol-104-disk-1  4.68G  3.32G  4.68G  /monster/data/subvol-104-disk-1
monster/data/subvol-106-disk-0   573M  19.4G   573M  /monster/data/subvol-106-disk-0
monster/data/subvol-115-disk-0  2.21G  2.68T  2.20G  /monster/data/subvol-115-disk-0
monster/data/subvol-115-disk-1  2.20G  7.80G  2.20G  /monster/data/subvol-115-disk-1
monster/data/subvol-117-disk-1  46.3G  42.2G  45.8G  /monster/data/subvol-117-disk-1
monster/data/subvol-118-disk-0  4.31G  21.2G  3.83G  /monster/data/subvol-118-disk-0
monster/data/subvol-122-disk-1  17.2G  43.3G  16.7G  /monster/data/subvol-122-disk-1
monster/data/subvol-123-disk-1  7.85G  5.84G  2.16G  /monster/data/subvol-123-disk-1
monster/data/subvol-126-disk-1  1.29G  6.71G  1.29G  /monster/data/subvol-126-disk-1
monster/data/subvol-127-disk-1  1.44G  6.56G  1.44G  /monster/data/subvol-127-disk-1
monster/data/subvol-132-disk-1  5.96G  14.2G  5.77G  /monster/data/subvol-132-disk-1
monster/data/subvol-142-disk-1  2.05G  5.95G  2.05G  /monster/data/subvol-142-disk-1
monster/data/subvol-143-disk-1   775M  7.24G   775M  /monster/data/subvol-143-disk-1
monster/data/subvol-151-disk-0  12.9G  20.1G  11.9G  /monster/data/subvol-151-disk-0
monster/data/subvol-155-disk-0   963M  9.10G   917M  /monster/data/subvol-155-disk-0
monster/data/vm-110-disk-1       252G  2.68T   252G  -
monster/data/vm-124-disk-0      7.00G  2.68T  7.00G  -
monster/data/vm-133-disk-1      33.2G  2.68T  20.1G  -
monster/data/vm-133-state-Snap  4.10G  2.68T  4.10G  -
monster/data/vm-147-disk-0      36.0G  2.68T  36.0G  -
monster/data/vm-152-disk-1      49.9G  2.68T  49.9G  -
monster/data/vm-70000-disk-1     112K  2.68T   112K  -
monster/templates                691M  2.68T   691M  /monster/templates
rpool                           31.0G  22.8G    96K  /rpool
rpool/ROOT                      15.0G  22.8G    96K  /rpool/ROOT
rpool/ROOT/pve-1                15.0G  22.8G  15.0G  /
rpool/data                        96K  22.8G    96K  /rpool/data
rpool/swap                        16G  32.5G  6.23G  -


But if you "ls" into the /monster/data/ directory (where all disks shall be mounted)… It is quite scarry (to say the less) !
Code: 
root@proxmonster:/home/gregober# ls /monster/data
subvol-106-disk-0  subvol-115-disk-0

There is only 2 VM instead of 15:

  • The one that I have setup yesterday
  • Another one which I have stopped… desperately tried to restart and finally restored from a ZFS snapshot !
  • This does not seem to be related to the dev issue since there is no dev directory available for all containers

SO what are we supposed to do ?

  1. restore all VM with "zfs send" from backup and delete old Mount Point ?
  2. do some magic which we are not aware of


This is a strategic server and rebooting It is not really something easy, specially if I end-up in the same situation as I am now… Or even worst if all my CT turns dead after a reboot !


I think that this problem is critical enough to bring a precise answer.

  • What does the Proxmox team have to say about this ?
  • Has a patch been published on more recent versions ?
  • Shall I upgrade to a 5.4.x ?
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back