[SOLVED] Server won't boot: Failed to mount rpool/ROOT/pve-1/… on /root//… no such file or dir

[Oygron]

Hi,

This morning I found my proxmox server down. When trying to restart, I had this message:

Gave up waiting for suspend/resume device

Command: mount -o zfsutil -t zfs rpool/ROOT/pve-1/be64[...]7d /root//be64[...]7d
Message: filesystem 'rpool/ROOT/pve-1/be64[...]7d' cannot be mounted at '/root//be64[...]7d' due to canonicalization error 2.
mount: mounting rpool/ROOT/pve-1/be64[...]7d on /root//be64[...]7d failed: No such file or directory

Failed to mount rpool/ROOT/pve-1/be64[...]7d on /root//be64[...]7d.
Manually mount the filesystem and exit.

On the given console, I can access, under the root directory, to the proxmox file system. In /var/log/syslog, I have a strange message around one minute before the last recorded message that says:

pvestatd[2279]: could not activate storage 'local-zfs', zfs error: use the form 'zpool import <pool | id> <newpool>' to give it a new name

The time is just after the beginning of my weekly backup. On pve logs, it seems that VM 100 had started its backup but not finished it yet, and the others has not.

Does anyone has an idea of what happened and how to correct it ?
Which information should I add to help resolve the problem ?

Thanks in advance,

Edit: I'm on proxmox 5, and I have two disks on raid1

 

[Oygron]

Hi,
Update on my researches:
What bothers me is that the error is

cannot be mounted at '/root//be64[...]7d'

there seems to miss something between the two shahes after root. Perhaps the right path would be

/root/var/lib/docker/zfs/graph

as there seems to be folders with all the asked names.

My current hypothesis is that the file containing this path has been corrupted and the path has disappeared.
Does that tell something to anyone?

(I tried a zpool scrub, but it told me that everything was ok)

Other question: As my data seem to still be there, maybe I can copy the VMs on an external disk, reinstall proxmox on the server, and put back the VMs. But I cannot find a way to either mount an external disk with this command line (neither ntfs or fat seems to be recognized), nor access to the data on the server disk from a linux live.
If someone can help me to rescue the data that would be great.

Thanks in advance,

Oygron

 

[Oygron]

Hi again,

Since the earlier discovery, I managed to half solve my problem.
I made symbolic links in /root to all the folders listed in /root/var/lib/docker/zfs/graph and rebooted the server, and this time it booted correctly.
I will later completely reinstall proxmox to have something clean, but for the moment, it just works.

That leaves one big question: what made this problem appear, and how can I protect myself from it to happen again?

Oygron

 

[quickes]

Hi!
I have this error too and find solution.

Spoiler: zfs list

root@(none):~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 101G 1.02T 96K /rpool
rpool/ROOT 61.0G 1.02T 96K /rpool/ROOT
rpool/ROOT/pve-1 61.0G 1.02T 58.3G /
rpool/ROOT/pve-1/018c45eda2fd74499f87ffebae57e8e16329993b3765098414b23e01c8c6bb6b 160K 1.02T 71.9M legacy
rpool/ROOT/pve-1/0f80b84ed4c5165533a53aefdcabc114ee26ac1fd3da1bd7ed72642269c6a35c 78.6M 1.02T 78.6M legacy
rpool/ROOT/pve-1/14b342e56e5d50aa365a4024e1bbd031e1cc5fe801d758199abde09bdd0e4404 144K 1.02T 169M legacy
rpool/ROOT/pve-1/17611b6bad7305f663d9a513a951fdd868e40ae5b7bc0d338194318cd8b01fa3 6.04M 1.02T 124M legacy
rpool/ROOT/pve-1/1b0336f40433c92f6dbb9dad9e2b7ddfbc458578fb881b430f917a643060d779 120K 1.02T 72.2M legacy
rpool/ROOT/pve-1/1bf8e26ed78ac650e1ead0e14327d0963347aecbc504d1bde635cef3b4a121ec 408M 1.02T 577M legacy
rpool/ROOT/pve-1/1cfad10ee51a388d68c93502b2ffa9c6bc7d951743980b44fc1e0f7c244ec954 3.17M 1.02T 3.16M legacy
rpool/ROOT/pve-1/1ecdefc2f9995ed7566cde60ff2a2da1f4dd2c42110cc226550e08dfa1bf8a93 140K 1.02T 71.9M legacy
rpool/ROOT/pve-1/211c156a31164a9151f1912ee7e59da545828f75a18281a4395eaa5e06a68d02 67.8M 1.02T 169M legacy
rpool/ROOT/pve-1/2f38e4acf0d856f3ef2e216c25142bcf5c7104f429bc029e9cbbd8a464deebcb 575M 1.02T 660M legacy
rpool/ROOT/pve-1/3abbfd13b59a54aac7885d1b66d8935834e27de2893d3ad0fbfe338dc78f601e 152K 1.02T 1.09G legacy
rpool/ROOT/pve-1/3f045d1e598100c41d56f514e55a090323ee0882051e76c76414a11debfdf01b 4.56M 1.02T 58.9M legacy
rpool/ROOT/pve-1/4654fcd31096eecb9469d6e73a1031a4e45720263326d55f155b46956199002e 144M 1.02T 202M legacy
rpool/ROOT/pve-1/479d5ea2776f3cdcf94707b44a7749b3938faddd1a353ccc999b1fd38125b8f6 73.5M 1.02T 222M legacy
rpool/ROOT/pve-1/4873b82b0481d3892d9a9a73900071af242285bd6f427e11cc4372160938911e 2.42M 1.02T 203M legacy
rpool/ROOT/pve-1/4873b82b0481d3892d9a9a73900071af242285bd6f427e11cc4372160938911e-init 128K 1.02T 202M legacy
rpool/ROOT/pve-1/4967aa5deeb8ffc5af6d8c886b163481851cf30eb765bcf0993b007f2ed4ffa3 840K 1.02T 678M legacy
rpool/ROOT/pve-1/49c9685a5d63e7c1a03468f63de97370ba187a368d0196865e90badf61fa1bce 132K 1.02T 202M legacy
rpool/ROOT/pve-1/4bc0777147e8328c6e8aba0601bed357f85878e9c9972dc80bb36551991e2b04 120K 1.02T 71.9M legacy
rpool/ROOT/pve-1/5258e8460f884731d93eaea5a21dc40f4c9e1760c8e371accdc97b1fb9f47734 10.6M 1.02T 160M legacy
rpool/ROOT/pve-1/53d2fd70253fc71f492f02b40541dd2e85033dc873d8fda55086357668d5e701 104K 1.02T 160M legacy
rpool/ROOT/pve-1/54304697f98309f72b11ac71463906f959633892873aa5b66bea9f17996a7cc7 144K 1.02T 165M legacy
rpool/ROOT/pve-1/5f3f785b3b246878c9daa7c170fd3ecfba5d31ed2feaa3a8379b6c543d0b60ff 90.8M 1.02T 90.8M legacy
rpool/ROOT/pve-1/684bdbfeffbb4ec990de92ac6f66fc8b8db593022743e805797c35063c7a45dc 124K 1.02T 148M legacy
rpool/ROOT/pve-1/778999d5c9c6f6b2a6e5dcb0495c2c4fb5cca9aaaa6dcd85df432da472a6c68c 192K 1.02T 678M legacy
rpool/ROOT/pve-1/785acf92208216b45c1e317a07e992c22e0785ddd6de13e3c5ab485d4a8a554f 4.61M 1.02T 677M legacy
rpool/ROOT/pve-1/7cef621ed0239de16e32ca777995c28cc026edba71d7726c2e9be30dc5f39ca1 184K 1.02T 660M legacy
rpool/ROOT/pve-1/8198655b730a70472c8424b3b2e8f9df1bd8cf118bc2e0ff5bbdde0d85f7abd0 144K 1.02T 118M legacy
rpool/ROOT/pve-1/81e3f156a1428400ecba5a9004dc7847eabae95d0ecbffbf73a097fd987e5005 12.0M 1.02T 160M legacy
rpool/ROOT/pve-1/8af8564dfeb83e6570163d58cbdba5e434d314838a2747fcf71c184575a0fce3 144M 1.02T 148M legacy
rpool/ROOT/pve-1/9caafa661c29bb7f43f06bfa8ed2be16f691f6aaf24f6cf42b6cfb1dabd0c809 228K 1.02T 78.6M legacy
rpool/ROOT/pve-1/a4acfab34ac209aff62c3d200d2deb745ea0640c7747ff38d696fc54f19e3277 128K 1.02T 985M legacy
rpool/ROOT/pve-1/a531d2b8b169faf91853297a67c6693032fd0b92c05147811034d364793f50b4 140K 1.02T 118M legacy
rpool/ROOT/pve-1/a5b3be1b53e1732c552f8cc0b72b28bb9eb471b2d48a37a8ecb778932d2367bc 51.2M 1.02T 55.0M legacy
rpool/ROOT/pve-1/a648cf59ada2f0d059115c8ae5f661fa8055cc020fa569dfb033db65e232bcfb 144K 1.02T 118M legacy
rpool/ROOT/pve-1/a745b6e5747bb1ee670763d7c35611200cc36e2716b550fcb3dec4b9c8f5970a 79.4M 1.02T 79.4M legacy
rpool/ROOT/pve-1/a872df413f464eabca7e87a7cf2bf81d263fa7c30fdec1b41b6e3de3f6ee7e54 160K 1.02T 148M legacy
rpool/ROOT/pve-1/aabc8abc1edfb249058113cba4f23f33c534e108b86f59f7e5f02dd106e72218 86.5M 1.02T 165M legacy
rpool/ROOT/pve-1/accfc172add1460511fe740e20675ea5014d7d9363299e1b1d88cd8b7acb7842 107M 1.02T 590M legacy
rpool/ROOT/pve-1/b091847f3edcd121f400be318e2a14b354aced73089de09349cae36b0ff5c257 228K 1.02T 79.4M legacy
rpool/ROOT/pve-1/b4276212f12f3466bde247a37fd3b96d4a369708c46a58410c87e56207063c0e 1.36M 1.02T 160M legacy
rpool/ROOT/pve-1/b4e0c2b3a4bc72955639079b0eb2e284068be9c6e04807fdf180323fac7d67d3 344K 1.02T 590M legacy
rpool/ROOT/pve-1/b4e0c2b3a4bc72955639079b0eb2e284068be9c6e04807fdf180323fac7d67d3-init 160K 1.02T 590M legacy
rpool/ROOT/pve-1/b5ae566d42ff2e0506e4c162b8762889ef87a53af0b8966550bb3926f92a689c 140K 1.02T 72.2M legacy
rpool/ROOT/pve-1/b91d88edd79993098027cbb2224b404c7c7315c73caa0fe88dcc5f19a1679f03 13.4M 1.02T 673M legacy
rpool/ROOT/pve-1/ba7e8c23a7b7964973767a7907493434d60a98b5e430ffa031a4d990a4f2711a 3.17M 1.02T 3.16M legacy
rpool/ROOT/pve-1/bbbc90929a60740cfd77519d9a3dad29259cbcef1e437d9bacadc2c127ec67be 128K 1.02T 55.0M legacy
rpool/ROOT/pve-1/bccd07a3a1f17906a497db87bd0a0c8c398213461660484f2a4ff7b5f94e466d 956K 1.02T 678M legacy
rpool/ROOT/pve-1/bcec675019f269c5f73a545b2b4ba69bd226463caac43f12f203daaaa7592344 6.27M 1.02T 1.09G legacy
rpool/ROOT/pve-1/beddee9d2df010a13a88e8afc3edd7db963a090ff6233e0eaf9210876cc73f4c 1.35M 1.02T 4.32M legacy
rpool/ROOT/pve-1/c169aa0336f09dc2c8df91b5e9cf3b242c45d796b1f8f5f68c683ec0a1bcafd8 53.4M 1.02T 118M legacy
rpool/ROOT/pve-1/c21136b0292bf8487d363509112269de129835a3ceec6af313c8814d6add6554 1.05M 1.02T 166M legacy
rpool/ROOT/pve-1/c21136b0292bf8487d363509112269de129835a3ceec6af313c8814d6add6554-init 160K 1.02T 165M legacy
rpool/ROOT/pve-1/c6783924aa3db60cd19bee7a5af917bd7a61b27dad85816435f004d6a724adbc 104K 1.02T 4.33M legacy
rpool/ROOT/pve-1/c9cd4eb6c2aeee1d7cd2c3b3401151c3ebf6c79e1d2b901a894f476684ce4cb3 96K 1.02T 160M legacy
rpool/ROOT/pve-1/cd051f6ce271be17c2e5638bb8ac91d578425c16d16b90dc7feddec9829b329c 168K 1.02T 72.2M legacy
rpool/ROOT/pve-1/cf8bdbf1f2361bc116d3a17c42e3260e5e8bf34b96aae5539192c21899784673 435M 1.02T 1.09G legacy
rpool/ROOT/pve-1/dea1bcdad635e89ec634b33170cbc3cdde584094ae4f1a75bf7a63d44040773e 307M 1.02T 985M legacy
rpool/ROOT/pve-1/eb7c6b5f5d9107c61f974ff924e961a64e027dce536b58bb75a99d2dcfeb9ffe 136K 1.02T 678M legacy
rpool/ROOT/pve-1/eed11643eb9b30d917b5f10007dda8eb74acd5bc42f6ec2c158424c0a1e74c3b 37.4M 1.02T 102M legacy
rpool/ROOT/pve-1/f2c64ecf3b36b8b3859422333547154f8e7eacc6872f750ba486df4258079336 192K 1.02T 678M legacy
rpool/ROOT/pve-1/f40af1e715c52e76508bd5fb90981c016247746a41131cd1c684920a925bdf99 1.35M 1.02T 4.32M legacy
rpool/ROOT/pve-1/fd5a58c5e46971f8826d4a1413e367756d557f50c679d2b0c842b5888c81acc7 104K 1.02T 148M legacy
rpool/data 31.0G 1.02T 104K /rpool/data
rpool/data/subvol-100-disk-1 20.1G 39.9G 20.1G /rpool/data/subvol-100-disk-1
rpool/data/subvol-101-disk-1 509M 9.50G 509M /rpool/data/subvol-101-disk-1
rpool/data/subvol-102-disk-1 8.43G 21.6G 8.43G /rpool/data/subvol-102-disk-1
rpool/data/subvol-103-disk-1 395M 7.61G 395M /rpool/data/subvol-103-disk-1
rpool/data/subvol-108-disk-1 1.15G 23.9G 1.15G /rpool/data/subvol-108-disk-1
rpool/data/subvol-136-disk-1 433M 3.58G 433M /rpool/data/subvol-136-disk-1
rpool/swap 8.50G 1.02T 7.97G -

zfs mount -va
mount without errors.
zfs status
without errors.

I can boot to normal mode. You need enter "exit" and enter in busybox for each rpool/ROOT/pve-1/xxxxxxx.
When all legacy volumes ends, system will boot to normal mode.
I have this ussie after installed docker on proxmox.
This bug only on proxmox + zfs + docker. I can try reconfigure docker for overlay2 driver and remove all legacy subvolumes.
But proxmox + ext4 (lvm) + docker works perfectly.

 

[Oygron]

Hi,
Nice to know there is a solution (I've reinstalled the server since that).
And yes, I installed Docker a few days before and every VM is using zfs.

Thanks !