[SOLVED] Cannot mount any zfs pool

CheesyPete

New Member
Sep 14, 2020
12
3
3
23
Hi,

I have recently added two disks to my server, when I run fdisk -l they occupy /sdc and /sdd. Both previously occupied by disks in my apool and bpool respectively.
After installing these disks and running proxmox 3 of my containers, that used one or both of these pools for storage would not start, I narrowed the issue down to neither of these being mounted. After fiddling around trying other solutions found on these forums, all I have done is made it much worse.

Now below is the output of zfs get mounted:
Bash:
NAME                          PROPERTY  VALUE    SOURCE
apool                         mounted   no       -
apool/plex                    mounted   no       -
apool/torrents                mounted   no       -
bpool                         mounted   no       -
bpool/youtube-backup          mounted   no       -
rpool                         mounted   no       -
rpool/ROOT                    mounted   no       -
rpool/ROOT/pve-1              mounted   yes      -
rpool/data                    mounted   no       -
rpool/data/subvol-100-disk-0  mounted   no       -
rpool/data/subvol-101-disk-0  mounted   no       -
rpool/data/subvol-102-disk-0  mounted   no       -
rpool/data/subvol-102-disk-1  mounted   no       -
rpool/data/subvol-103-disk-0  mounted   no       -

This appears to show that now none of my rpool is mounted apart from the proxmox partition. None of my containers boot up.

I try to run zfs mount apool/plex for example I get cannot mount '/apool/plex': directory is not empty
This is true it contains about 1.3TB worth of stuff, which I dont want to get rid off so I can mount it.

I am able to mount some things, below after I manually try to mount everything in the list:
Bash:
NAME                          PROPERTY  VALUE    SOURCE
apool                         mounted   no       -
apool/plex                    mounted   no       -
apool/torrents                mounted   no       -
bpool                         mounted   no       -
bpool/youtube-backup          mounted   no       -
rpool                         mounted   no       -
rpool/ROOT                    mounted   yes      -
rpool/ROOT/pve-1              mounted   yes      -
rpool/data                    mounted   no       -
rpool/data/subvol-100-disk-0  mounted   yes      -
rpool/data/subvol-101-disk-0  mounted   yes      -
rpool/data/subvol-102-disk-0  mounted   yes      -
rpool/data/subvol-102-disk-1  mounted   yes      -
rpool/data/subvol-103-disk-0  mounted   yes      -

This allows me to spin up CT102 - just a simple pi-hole container, but still noting else.

Please if anyone can help with this I would greatly appreciate it, I don't really know what to do otherwise beacuse I don't want to delete the data on the drives - this was my backup.

Thanks, more info below if needed:

Bash:
                path: '/dev/disk/by-id/ata-ST4000DM004-2CV104_ZFN22DH6-part1'
                phys_path: 'pci-0000:02:00.0-ata-1'
                path: '/dev/disk/by-id/ata-ST4000DM004-2CV104_ZFN22G93-part1'
                phys_path: 'pci-0000:02:00.0-ata-1'
                path: '/dev/disk/by-id/ata-ST4000DM004-2CV104_ZFN22DEX-part1'
                phys_path: 'pci-0000:02:00.0-ata-1'
                path: '/dev/disk/by-id/ata-ST4000DM004-2CV104_ZFN20PF9-part1'
                phys_path: 'pci-0000:02:00.0-ata-2'
                path: '/dev/disk/by-id/ata-ST8000VN004-2M2101_WKD29ZAV-part1'
                phys_path: 'pci-0000:01:00.0-ata-1'
                path: '/dev/disk/by-id/ata-ST8000VN004-2M2101_WKD2VKF9-part1'
                phys_path: 'pci-0000:01:00.0-ata-1'
                path: '/dev/disk/by-id/ata-ST8000VN004-2M2101_WKD303M6-part1'
                phys_path: 'pci-0000:01:00.0-ata-2'
                path: '/dev/disk/by-id/ata-ST8000VN004-2M2101_WKD31JZF-part1'
                phys_path: 'pci-0000:01:00.0-ata-1'
                path: '/dev/disk/by-id/ata-WDC_WDS500G1R0A-68A4W0_202574800385-part3'
                path: '/dev/disk/by-id/ata-WDC_WDS500G1R0A-68A4W0_202574800257-part3'

Bash:
lxc-start: 100: lxccontainer.c: wait_on_daemonized_start: 843 No such file or directory - Failed to receive the container state
lxc-start: 100: tools/lxc_start.c: main: 308 The container failed to start
lxc-start: 100: tools/lxc_start.c: main: 311 To get more details, run the container in foreground mode
lxc-start: 100: tools/lxc_start.c: main: 314 Additional information can be obtained by setting the --logfile and --logpriority options

Bash:
lxc-start 100 20200920175341.143 ERROR    conf - conf.c:run_buffer:323 - Script exited with status 2
lxc-start 100 20200920175341.144 ERROR    start - start.c:lxc_init:804 - Failed to run lxc.hook.pre-start for container "100"
lxc-start 100 20200920175341.144 ERROR    start - start.c:__lxc_start:1903 - Failed to initialize container "100"
lxc-start 100 20200920175341.961 ERROR    conf - conf.c:run_buffer:323 - Script exited with status 1
lxc-start 100 20200920175341.961 ERROR    start - start.c:lxc_end:971 - Failed to run lxc.hook.post-stop for container "100"
lxc-start 100 20200920175341.961 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:843 - No such file or directory - Failed to receive the container state
lxc-start 100 20200920175341.961 ERROR    lxc_start - tools/lxc_start.c:main:308 - The container failed to start
lxc-start 100 20200920175341.961 ERROR    lxc_start - tools/lxc_start.c:main:311 - To get more details, run the container in foreground mode
lxc-start 100 20200920175341.961 ERROR    lxc_start - tools/lxc_start.c:main:314 - Additional information can be obtained by setting the --logfile and --logpriority options
 
Last edited:
This is pretty chunky so it didn't allow it in the original post:

Bash:
Disk /dev/sda: 465.8 GiB, 500107862016 bytes, 976773168 sectors
Disk model: WDC  WDS500G1R0A
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 3665A6CC-5556-434A-A227-FB40746C9893

Device       Start       End   Sectors   Size Type
/dev/sda1       34      2047      2014  1007K BIOS boot
/dev/sda2     2048   1050623   1048576   512M EFI System
/dev/sda3  1050624 976773134 975722511 465.3G Solaris /usr & Apple ZFS


Disk /dev/sdb: 465.8 GiB, 500107862016 bytes, 976773168 sectors
Disk model: WDC  WDS500G1R0A
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: C4802098-DCE3-47CB-B734-DB4555B9F9CF

Device       Start       End   Sectors   Size Type
/dev/sdb1       34      2047      2014  1007K BIOS boot
/dev/sdb2     2048   1050623   1048576   512M EFI System
/dev/sdb3  1050624 976773134 975722511 465.3G Solaris /usr & Apple ZFS


Disk /dev/sdc: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: ST3000VN007-2AH1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdd: 2.7 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: ST3000VN007-2AH1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sde: 7.3 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: ST8000VN004-2M21
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 9FD5D5F4-7035-574E-B57F-9D5240FFF9AE

Device           Start         End     Sectors  Size Type
/dev/sde1         2048 15628036095 15628034048  7.3T Solaris /usr & Apple ZFS
/dev/sde9  15628036096 15628052479       16384    8M Solaris reserved 1


Disk /dev/sdf: 7.3 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: ST8000VN004-2M21
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: D8FE7810-6239-894F-BC59-E947AD6EAE64

Device           Start         End     Sectors  Size Type
/dev/sdf1         2048 15628036095 15628034048  7.3T Solaris /usr & Apple ZFS
/dev/sdf9  15628036096 15628052479       16384    8M Solaris reserved 1


Disk /dev/sdg: 7.3 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: ST8000VN004-2M21
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: D2E6B06D-FCED-5E4E-8576-A2EAFA404C72

Device           Start         End     Sectors  Size Type
/dev/sdg1         2048 15628036095 15628034048  7.3T Solaris /usr & Apple ZFS
/dev/sdg9  15628036096 15628052479       16384    8M Solaris reserved 1


Disk /dev/sdi: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: ST4000DM004-2CV1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 30A2F134-B64B-2543-9816-3864AA8506A0

Device          Start        End    Sectors  Size Type
/dev/sdi1        2048 7814019071 7814017024  3.7T Solaris /usr & Apple ZFS
/dev/sdi9  7814019072 7814035455      16384    8M Solaris reserved 1


Disk /dev/sdh: 7.3 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: ST8000VN004-2M21
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: A8EAF67D-C0C9-5544-846D-DF271E001593

Device           Start         End     Sectors  Size Type
/dev/sdh1         2048 15628036095 15628034048  7.3T Solaris /usr & Apple ZFS
/dev/sdh9  15628036096 15628052479       16384    8M Solaris reserved 1


Disk /dev/sdj: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: ST4000DM004-2CV1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: D99263A9-2B8A-DB4C-8138-0E0E67F5D761

Device          Start        End    Sectors  Size Type
/dev/sdj1        2048 7814019071 7814017024  3.7T Solaris /usr & Apple ZFS
/dev/sdj9  7814019072 7814035455      16384    8M Solaris reserved 1


Disk /dev/sdk: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: ST4000DM004-2CV1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: A1A45061-21DB-034A-9B5A-867D54E41635

Device          Start        End    Sectors  Size Type
/dev/sdk1        2048 7814019071 7814017024  3.7T Solaris /usr & Apple ZFS
/dev/sdk9  7814019072 7814035455      16384    8M Solaris reserved 1


Disk /dev/sdl: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Disk model: ST4000DM004-2CV1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 86A880DA-75E0-2E42-BDA4-0C0ABF20D36A

Device          Start        End    Sectors  Size Type
/dev/sdl1        2048 7814019071 7814017024  3.7T Solaris /usr & Apple ZFS
/dev/sdl9  7814019072 7814035455      16384    8M Solaris reserved 1

Bash:
  pool: apool
 state: ONLINE
  scan: none requested
config:

        NAME                                 STATE     READ WRITE CKSUM
        apool                                ONLINE       0     0     0
          raidz1-0                           ONLINE       0     0     0
            ata-ST4000DM004-2CV104_ZFN22DH6  ONLINE       0     0     0
            ata-ST4000DM004-2CV104_ZFN22G93  ONLINE       0     0     0
            ata-ST4000DM004-2CV104_ZFN22DEX  ONLINE       0     0     0
            ata-ST4000DM004-2CV104_ZFN20PF9  ONLINE       0     0     0

errors: No known data errors

  pool: bpool
 state: ONLINE
  scan: none requested
config:

        NAME                                 STATE     READ WRITE CKSUM
        bpool                                ONLINE       0     0     0
          raidz1-0                           ONLINE       0     0     0
            ata-ST8000VN004-2M2101_WKD29ZAV  ONLINE       0     0     0
            ata-ST8000VN004-2M2101_WKD2VKF9  ONLINE       0     0     0
            ata-ST8000VN004-2M2101_WKD303M6  ONLINE       0     0     0
            ata-ST8000VN004-2M2101_WKD31JZF  ONLINE       0     0     0

errors: No known data errors

  pool: rpool
 state: ONLINE
  scan: none requested
config:

        NAME                                               STATE     READ WRITE CKSUM
        rpool                                              ONLINE       0     0     0
          mirror-0                                         ONLINE       0     0     0
            ata-WDC_WDS500G1R0A-68A4W0_202574800385-part3  ONLINE       0     0     0
            ata-WDC_WDS500G1R0A-68A4W0_202574800257-part3  ONLINE       0     0     0

errors: No known data errors
 
Directory not empty means that theres something in there, remove it then run "zfs import -a" or "systemctl restart zfs-import"

I was wondering if there was a way around that - I have about 1.5TB across the two pools, as directories, shown below:
1600633475547.png
I really really don't want to have to delete that data, if there is any way around that.

If I do have to, could I just scp the data from these to somewhere else and bring it back again?

Thanks
 
Good you didn't delete your data, i thought nothing was mounted......

What gives:

Code:
zfs list -o name,mounted,mountpoint

You can manualy mount them one by mount with "zfs mount <mountpoint>"
 
OK did some experimenting - turns out I have a misunderstanding I deleted everything in bpool/youtube-backup with rm -rf /bpool/youtube-backup after disabling the directory in the proxmox gui then I mounted, re-enabled and I could get into that container and all the data is still there, so I presume it is in some weird limbo of thinking it has data there but not really
 
This is now the out put of that command (zfs list -o name,mounted,mountpoint):

Bash:
NAME                          MOUNTED  MOUNTPOINT
apool                              no  /apool
apool/plex                         no  /apool/plex
apool/torrents                     no  /apool/torrents
bpool                              no  /bpool
bpool/youtube-backup              yes  /bpool/youtube-backup
rpool                              no  /rpool
rpool/ROOT                         no  /rpool/ROOT
rpool/ROOT/pve-1                  yes  /
rpool/data                         no  /rpool/data
rpool/data/subvol-100-disk-0       no  /rpool/data/subvol-100-disk-0
rpool/data/subvol-101-disk-0       no  /rpool/data/subvol-101-disk-0
rpool/data/subvol-102-disk-0       no  /rpool/data/subvol-102-disk-0
rpool/data/subvol-102-disk-1       no  /rpool/data/subvol-102-disk-1
rpool/data/subvol-103-disk-0      yes  /rpool/data/subvol-103-disk-0
 
so hopefully that has solved that issue, but the next question is how to get all these to automatically mount at boot?

Thanks for the help too, usually im happy to experiment with these things but it feels like one wrong move and it is all gone
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!