Restoring storage.cfg after joining cluster

D

def1ant

New Member
Apr 26, 2020
15
2
3
33
In the process of backing up my server (oh the irony) i managed to screw up my configuration, so i can't acces my data anymore.

So i've gathered that my problem probably lays in the /etc/pve/storage.cfg, as this is being overwritten when you join a cluster - wich i did, for testing purposes.
I am building a new, more energy efficient server atm that was supposed to replace my existing server.
Well, you should really read the documentation before you click on anything, lesson learned.


My setup:

Server with 5x 1tb HDDs in a ZFS Pool + 1x SSD for the OS
On the server there's an OpenMediaVault VM running (id 100), sharing files via smb.

Kernel Version Linux 5.4.65-1-pve #1 SMP PVE 5.4.65-1 (Mon, 21 Sep 2020 15:40:22 +0200)
PVE Manager Version pve-manager/6.2-6/ee1d7754


The node is called "pve", the ZFS Pool is called "Local1" (and is alive and happy, according to the Proxmox Gui).

zfspool.JPG

OMV complains on boot that it cant find its Data volume, which used to be the shared storage.

Code: 
Jan 10 13:09:17 omgomv monit[602]: 'filesystem_srv_dev-disk-by-label-omvData' unable to read filesystem '/srv/dev-disk-by-label-omvData' state
Jan 10 13:09:17 omgomv monit[602]: 'filesystem_srv_dev-disk-by-label-omvData' trying to restart
Jan 10 13:09:17 omgomv monit[602]: 'mountpoint_srv_dev-disk-by-label-omvData' status failed (1) -- /srv/dev-disk-by-label-omvData is not a mountpoint
Jan 10 13:09:47 omgomv monit[602]: Filesystem '/srv/dev-disk-by-label-omvData' not mounted

The data is on "vm-100-disk-0", the raw disk image wich can be found under
/dev/zvol/Local1
(i can see it in the Node's terminal, so it should be accessible.)


Right now my storage.cfg looks like this:

storage.cfg.JPG

Wich is basically the default config, as i gathered.

In the guide for joining a cluster it also states that you will have to re-add your storage locations manually, but i cant find an example configuration that would fit that usecase.

Anyone who has any idea what should be written in the storage.cfg exactly?

Or maybe someone running OpenMediaVault on Proxmox could share their storage.cfg?

I am really lost at the moment.


Proxmox also gives me this error over and over again during boot:

Code: 
Jan 10 12:50:41 pve rrdcached[917]: handle_request_update: Could not read RRD file.
Jan 10 12:50:41 pve pmxcfs[930]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve: -1
Jan 10 12:50:41 pve pmxcfs[930]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/pve: mmaping file '/var/lib/rrdcached/db/pve2-node/pve': Invalid argument

Not sure if thats relevant or not for this problem.


If anyone can help me: Thanks so much in advance!
 
Last edited:
Hi,
what does the configuration for the VM look like, i.e. qm config <ID>? I think the storage config should be fine.
 
  • Like
Reactions: def1ant
Code: 
root@pve:~# qm config 100
bootdisk: virtio0
cores: 2
ide2: none,media=cdrom
memory: 3072
name: OMV
net0: e1000=A2:62:9F:97:AA:83,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=2d77611d-1bc7-46ba-9737-9fe4f8cfaa7a
sockets: 1
usb0: host=174c:55aa,usb3=1
virtio0: local-lvm:vm-100-disk-0,size=14G
virtio1: Local1:vm-100-disk-0,size=3000G  //this is the virtual disk that is missing
virtio2: Local1:vm-100-disk-1,size=1G   //this i added just as a test to see if it creates any "links"
vmgenid: 28212180-54b6-4319-9a49-90e905f43dff
 
Last edited:
Okay, so you think the problem is not in the storage.cfg?

I did tinker around in the fstab file of the node aswsell, trying to setup an external hdd for backups.

Code: 
# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext4 errors=remount-ro 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0

Should there be anything else in there maybe?
 
Last edited:
Sorry for bombarding you with all this different stuff - im trying to figure out what went wrong exactly as some of the data i had on there was not backupped yet and therefore i am really trying to get it back.

If it's an option, i would gladly pay for an hour or two of tech support services!
 
Sorry, I didn't have time to answer yet. If you need guaranteed response times, you need to have a subscription and open a ticket.

But let's get back to the issue. Can you run
Code: 
pvesm list Local1
zfs list Local1 -r -t volume
the volume should show up in both cases.

Can you see the disk from within the VM, e.g. with
Code: 
lsblk

Is there anything in the startup task log (in the bottom panel in the GUI, double click on VM 100 - Start)?
 
  • Like
Reactions: def1ant
No worries whatsoever! Thanks for taking the time.

I know that if i needed 24/7 i would need a subscription - but as i got myself into a pickle here i thought offering payment might help. :)
I do understand that nobody "owes" me anything at this point, so thanks again for looking into the issue.


Here's my output.
The two commands on the nodes shell:

Code: 
root@pve:~# pvesm list Local1
file /etc/pve/storage.cfg line 14 (section 'Local1') - ignore config line:
Volid                Format  Type               Size VMID
Local1:vm-100-disk-0 raw     images    3221225472000 100
Local1:vm-100-disk-1 raw     images       1073741824 100
Local1:vm-101-disk-0 raw     images     137438953472 101
Local1:vm-102-disk-0 raw     images      34359738368 102
Local1:vm-103-disk-0 raw     images      34359738368 103
Local1:vm-104-disk-0 raw     images      34359738368 104
Local1:vm-105-disk-0 raw     images      51539607552 105
Local1:vm-106-disk-0 raw     images       8589934592 106



Code: 
root@pve:~# zfs list Local1 -r -t volume
NAME                   USED  AVAIL     REFER  MOUNTPOINT
Local1/vm-100-disk-0  3.02T  3.16T     55.3M  -
Local1/vm-100-disk-1  1.63G   148G      320K  -
Local1/vm-101-disk-0   132G   279G     89.5K  -
Local1/vm-102-disk-0  52.2G   181G     18.4G  -
Local1/vm-103-disk-0  33.0G   178G     1.99G  -
Local1/vm-104-disk-0  52.2G   184G     15.1G  -
Local1/vm-105-disk-0  78.3G   185G     39.9G  -
Local1/vm-106-disk-0  13.1G   156G     3.59G  -


And lsblk in omv (cant copy out of that shell for some reason)

lsblk.JPG


Interesting, in the startup task it says
Code: 
Task viewer: VM 100 - Start
OutputStatus
Stop
zfs error: cannot open 'Local1': no such pool
TASK OK

Cheers
Eric
 
Last edited:
def1ant said:
file /etc/pve/storage.cfg line 14 (section 'Local1') - ignore config line:
Click to expand...
Do you have white spaces on the last line? Please remove them.

def1ant said:
And lsblk in omv (cant copy out of that shell for some reason)

View attachment 22772
Click to expand...
Isn't vdb the disk you are looking for? It matches size-wise.

def1ant said:
Interesting, in the startup task it says
Code: 
Task viewer: VM 100 - Start
OutputStatus
Stop
zfs error: cannot open 'Local1': no such pool
TASK OK
Click to expand...
Could you run:
Code: 
zpool list
zpool list -o name -H "Local1"
 
  • Like
Reactions: def1ant
Fabian_E said:
Do you have white spaces on the last line? Please remove them.
Click to expand...
Did not look like it, but i deleted everything after "Local1" now and will see how it looks after a restart.


Fabian_E said:
Isn't vdb the disk you are looking for? It matches size-wise.
Click to expand...
Yes, vdb should be the disk in question.


Code: 
root@pve:~# zpool list
NAME     SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
Local1  4.55T  99.0G  4.45T        -         -     2%     2%  1.00x    ONLINE  -

This is the pool i would want, but it shouldn't be almost empty.


Code: 
root@pve:~# zpool list -o name -H "Local1"
Local1

I will restart the node and see if the "ignore config line" message will remain.

Cheers
 
I'm sorry to say, but that seems like there is no data on that disk. Any warnings/errors when you run zpool status -v?
What does zpool history Local1 say about creation time (or general history) for vm-100-disk-0?
 
  • Like
Reactions: def1ant
Hmm. But why would the data "vanish"?
I massively screwed up the configuration while joining the cluster, yes, but why should the data be overwritten?

Code: 
NAME                   USED  AVAIL     REFER  MOUNTPOINT
Local1/vm-100-disk-0  3.02T  3.16T     55.3M  -

And why can i still see the disk in question with the correct size, if there's no data on it?
Im just trying to understand what exactly is going on.

Here's my output:

Code: 
root@pve:~# zpool status -v
  pool: Local1
 state: ONLINE
  scan: resilvered 18.1M in 0 days 00:00:05 with 0 errors on Fri Jan  1 20:47:52 2021
config:

        NAME                        STATE     READ WRITE CKSUM
        Local1                      ONLINE       0     0     0
          raidz1-0                  ONLINE       0     0     0
            wwn-0x50014ee2ad85056a  ONLINE       0     0     0
            wwn-0x50024e920300204f  ONLINE       0     0     0
            wwn-0x50024e9203002066  ONLINE       0     0     0
            wwn-0x50014ee2020a1280  ONLINE       0     0     0
            wwn-0x50014ee2020a1263  ONLINE       0     0     0

errors: No known data errors


Code: 
root@pve:~# zpool history Local1
History for 'Local1':
2019-02-14.11:12:35 zpool create -o ashift=12 Local1 raidz /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf
2019-02-14.11:12:41 zfs set compression=on Local1
2019-02-14.12:11:49 zfs create -V 3145728000k Local1/vm-100-disk-0                                             
2019-02-14.11:12:07 zpool import -c /etc/zfs/zpool.cache -aN
2020-03-11.19:31:04 zpool import -c /etc/zfs/zpool.cache -aN
2020-03-11.20:08:34 zpool import -c /etc/zfs/zpool.cache -aN
2020-03-11.21:34:43 zpool import -c /etc/zfs/zpool.cache -aN
2020-03-12.11:38:16 zfs destroy -r Local1/vm-100-disk-0
2020-03-12.11:39:42 zfs create -V 3145728000k Local1/vm-100-disk-0                         <--- Theres the important disk
2020-03-12.14:49:49 zpool import -c /etc/zfs/zpool.cache -aN
2020-03-12.14:56:11 zpool import -c /etc/zfs/zpool.cache -aN
2020-03-16.14:42:54 zpool import -c /etc/zfs/zpool.cache -aN
2020-03-16.14:53:17 zpool import -c /etc/zfs/zpool.cache -aN
2020-03-16.16:51:47 zfs create -V 134217728k Local1/vm-101-disk-0
2020-03-17.13:24:44 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-03-21.13:59:56 zfs create -V 33554432k Local1/vm-102-disk-0
2020-03-22.11:29:22 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-03-22.12:07:44 zfs create -V 33554432k Local1/vm-103-disk-0
2020-03-22.16:51:06 zfs create -V 33554432k Local1/vm-104-disk-0
2020-03-22.17:28:31 zfs destroy -r Local1/vm-104-disk-0
2020-03-22.17:30:18 zfs create -V 33554432k Local1/vm-104-disk-0
2020-03-22.17:44:26 zfs destroy -r Local1/vm-104-disk-0
2020-03-22.17:45:02 zfs create -V 33554432k Local1/vm-104-disk-0
2020-03-23.01:06:18 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-03-23.17:38:25 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-03-23.19:10:04 zfs create -V 36700160k Local1/vm-104-disk-1
2020-03-24.12:10:53 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-03-24.14:29:11 zfs destroy -r Local1/vm-104-disk-1
2020-03-24.15:37:34 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-03-24.17:35:35 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-03-24.19:41:31 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-03-24.19:41:57 zfs create -V 33554432k Local1/vm-105-disk-0
2020-03-24.20:08:32 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-03-24.23:05:25 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-03-25.11:31:06 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-03-25.20:45:33 zfs create -V 8388608k Local1/vm-106-disk-0
2020-03-26.17:14:44 zfs create -V 2097152k Local1/vm-107-disk-0
2020-03-26.17:21:19 zfs destroy -r Local1/vm-107-disk-0
2020-04-02.22:11:18 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-04-12.13:07:53 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-04-12.13:12:25 zfs create -V 1048576k Local1/vm-107-disk-0
2020-04-12.13:14:20 zfs destroy -r Local1/vm-107-disk-0
2020-04-12.13:55:46 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-04-26.15:58:24 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-04-26.22:50:25 zfs destroy -r Local1/vm-102-disk-0
2020-04-26.23:00:13 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-04-26.23:04:45 zfs create -V 33554432k Local1/vm-102-disk-0
2020-04-27.10:50:43 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-04-27.14:37:43 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-04-28.17:28:20 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-04-28.18:07:41 zfs destroy -r Local1/vm-104-disk-0
2020-04-28.18:53:44 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-05-09.18:38:44 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-05-10.00:24:08 zpool scrub Local1
2020-05-10.13:15:09 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-05-11.17:41:54 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-05-11.17:45:40 zpool clear Local1
2020-05-13.17:58:23 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-06-13.10:37:42 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-06-14.00:24:04 zpool scrub Local1
2020-06-14.12:02:59 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-06-15.14:57:30 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-06-15.15:10:53 zpool clear Local1
2020-06-15.22:39:35 zpool clear Local1
2020-06-16.13:29:07 zpool clear Local1
2020-06-16.14:24:11 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-06-16.14:27:20 zpool clear Local1
2020-06-16.14:50:37 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-07-24.12:15:08 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-07-24.16:55:16 zfs set volsize=50331648k Local1/vm-105-disk-0
2020-07-27.22:50:27 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-10-27.16:39:11 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-10-27.16:42:16 zfs create -V 33554432k Local1/vm-104-disk-0
2020-10-27.18:06:57 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2021-01-10.12:50:39 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2021-01-10.12:59:24 zfs create -V 1048576k Local1/vm-100-disk-1                       <------ Thats the test disk i created to see what "links" are generated
2021-01-12.11:53:46 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2021-01-19.10:27:52 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1

Cheers
 
def1ant said:
Hmm. But why would the data "vanish"?
I massively screwed up the configuration while joining the cluster, yes, but why should the data be overwritten?
Click to expand...
Yes, it's strange. I don't think that when joining a cluster any virtual disks are touched. The storage configuration is just configuration after all.

def1ant said:
Code: 
NAME                   USED  AVAIL     REFER  MOUNTPOINT
Local1/vm-100-disk-0  3.02T  3.16T     55.3M  -

And why can i still see the disk in question with the correct size, if there's no data on it?
Im just trying to understand what exactly is going on.
Click to expand...
I'm also trying to understand, but at least here I can answer you: If you don't use the sparse 1 option for the PVE configuration of your ZFS storage, then refreservation is used to ensure there's enough space reserved for the disk to be fully writeable. See zfs list -o space Local1/vm-100-disk-0 if you want to see a detailed breakdown.

def1ant said:
Here's my output:

Code: 
root@pve:~# zpool status -v
  pool: Local1
state: ONLINE
  scan: resilvered 18.1M in 0 days 00:00:05 with 0 errors on Fri Jan  1 20:47:52 2021
config:

        NAME                        STATE     READ WRITE CKSUM
        Local1                      ONLINE       0     0     0
          raidz1-0                  ONLINE       0     0     0
            wwn-0x50014ee2ad85056a  ONLINE       0     0     0
            wwn-0x50024e920300204f  ONLINE       0     0     0
            wwn-0x50024e9203002066  ONLINE       0     0     0
            wwn-0x50014ee2020a1280  ONLINE       0     0     0
            wwn-0x50014ee2020a1263  ONLINE       0     0     0

errors: No known data errors


Code: 
root@pve:~# zpool history Local1
History for 'Local1':
2019-02-14.11:12:35 zpool create -o ashift=12 Local1 raidz /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf
2019-02-14.11:12:41 zfs set compression=on Local1
2019-02-14.12:11:49 zfs create -V 3145728000k Local1/vm-100-disk-0                                            
2019-02-14.11:12:07 zpool import -c /etc/zfs/zpool.cache -aN
2020-03-11.19:31:04 zpool import -c /etc/zfs/zpool.cache -aN
2020-03-11.20:08:34 zpool import -c /etc/zfs/zpool.cache -aN
2020-03-11.21:34:43 zpool import -c /etc/zfs/zpool.cache -aN
2020-03-12.11:38:16 zfs destroy -r Local1/vm-100-disk-0
2020-03-12.11:39:42 zfs create -V 3145728000k Local1/vm-100-disk-0                         <--- Theres the important disk
2020-03-12.14:49:49 zpool import -c /etc/zfs/zpool.cache -aN
2020-03-12.14:56:11 zpool import -c /etc/zfs/zpool.cache -aN
2020-03-16.14:42:54 zpool import -c /etc/zfs/zpool.cache -aN
2020-03-16.14:53:17 zpool import -c /etc/zfs/zpool.cache -aN
2020-03-16.16:51:47 zfs create -V 134217728k Local1/vm-101-disk-0
2020-03-17.13:24:44 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-03-21.13:59:56 zfs create -V 33554432k Local1/vm-102-disk-0
2020-03-22.11:29:22 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-03-22.12:07:44 zfs create -V 33554432k Local1/vm-103-disk-0
2020-03-22.16:51:06 zfs create -V 33554432k Local1/vm-104-disk-0
2020-03-22.17:28:31 zfs destroy -r Local1/vm-104-disk-0
2020-03-22.17:30:18 zfs create -V 33554432k Local1/vm-104-disk-0
2020-03-22.17:44:26 zfs destroy -r Local1/vm-104-disk-0
2020-03-22.17:45:02 zfs create -V 33554432k Local1/vm-104-disk-0
2020-03-23.01:06:18 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-03-23.17:38:25 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-03-23.19:10:04 zfs create -V 36700160k Local1/vm-104-disk-1
2020-03-24.12:10:53 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-03-24.14:29:11 zfs destroy -r Local1/vm-104-disk-1
2020-03-24.15:37:34 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-03-24.17:35:35 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-03-24.19:41:31 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-03-24.19:41:57 zfs create -V 33554432k Local1/vm-105-disk-0
2020-03-24.20:08:32 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-03-24.23:05:25 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-03-25.11:31:06 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-03-25.20:45:33 zfs create -V 8388608k Local1/vm-106-disk-0
2020-03-26.17:14:44 zfs create -V 2097152k Local1/vm-107-disk-0
2020-03-26.17:21:19 zfs destroy -r Local1/vm-107-disk-0
2020-04-02.22:11:18 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-04-12.13:07:53 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-04-12.13:12:25 zfs create -V 1048576k Local1/vm-107-disk-0
2020-04-12.13:14:20 zfs destroy -r Local1/vm-107-disk-0
2020-04-12.13:55:46 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-04-26.15:58:24 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-04-26.22:50:25 zfs destroy -r Local1/vm-102-disk-0
2020-04-26.23:00:13 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-04-26.23:04:45 zfs create -V 33554432k Local1/vm-102-disk-0
2020-04-27.10:50:43 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-04-27.14:37:43 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-04-28.17:28:20 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-04-28.18:07:41 zfs destroy -r Local1/vm-104-disk-0
2020-04-28.18:53:44 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-05-09.18:38:44 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-05-10.00:24:08 zpool scrub Local1
2020-05-10.13:15:09 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-05-11.17:41:54 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-05-11.17:45:40 zpool clear Local1
2020-05-13.17:58:23 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-06-13.10:37:42 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-06-14.00:24:04 zpool scrub Local1
2020-06-14.12:02:59 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-06-15.14:57:30 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-06-15.15:10:53 zpool clear Local1
2020-06-15.22:39:35 zpool clear Local1
2020-06-16.13:29:07 zpool clear Local1
2020-06-16.14:24:11 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-06-16.14:27:20 zpool clear Local1
2020-06-16.14:50:37 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-07-24.12:15:08 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-07-24.16:55:16 zfs set volsize=50331648k Local1/vm-105-disk-0
2020-07-27.22:50:27 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-10-27.16:39:11 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2020-10-27.16:42:16 zfs create -V 33554432k Local1/vm-104-disk-0
2020-10-27.18:06:57 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2021-01-10.12:50:39 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2021-01-10.12:59:24 zfs create -V 1048576k Local1/vm-100-disk-1                       <------ Thats the test disk i created to see what "links" are generated
2021-01-12.11:53:46 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1
2021-01-19.10:27:52 zpool import -d /dev/disk/by-id/ -o cachefile=none Local1

Cheers
Click to expand...
Nothing strange in those outputs AFAICT.

Could you post the output of pveversion -v, zpool get all Local1 and zpool history -i Local1 (might be lengthy, from around the time the accident happened should be enough).
 
  • Like
Reactions: def1ant
Doesn't look good then. :(

Heres the output:


Code: 
root@pve:~# pveversion -v
proxmox-ve: 6.2-1 (running kernel: 5.4.65-1-pve)
pve-manager: 6.2-6 (running version: 6.2-6/ee1d7754)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-3
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve2
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-1
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-3
libpve-guest-common-perl: 3.0-10
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve2
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-7
pve-cluster: 6.1-8
pve-container: 3.1-8
pve-docs: 6.2-4
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-3
pve-qemu-kvm: 5.0.0-4
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-3
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.4-pve1

Code: 
root@pve:~# zpool get all Local1
NAME    PROPERTY                       VALUE                          SOURCE
Local1  size                           4.55T                          -
Local1  capacity                       2%                             -
Local1  altroot                        -                              default
Local1  health                         ONLINE                         -
Local1  guid                           6692709766186344983            -
Local1  version                        -                              default
Local1  bootfs                         -                              default
Local1  delegation                     on                             default
Local1  autoreplace                    off                            default
Local1  cachefile                      none                           local
Local1  failmode                       wait                           default
Local1  listsnapshots                  off                            default
Local1  autoexpand                     off                            default
Local1  dedupditto                     0                              default
Local1  dedupratio                     1.00x                          -
Local1  free                           4.45T                          -
Local1  allocated                      99.0G                          -
Local1  readonly                       off                            -
Local1  ashift                         12                             local
Local1  comment                        -                              default
Local1  expandsize                     -                              -
Local1  freeing                        0                              -
Local1  fragmentation                  2%                             -
Local1  leaked                         0                              -
Local1  multihost                      off                            default
Local1  checkpoint                     -                              -
Local1  load_guid                      8455705221771510738            -
Local1  autotrim                       off                            default
Local1  feature@async_destroy          enabled                        local
Local1  feature@empty_bpobj            active                         local
Local1  feature@lz4_compress           active                         local
Local1  feature@multi_vdev_crash_dump  enabled                        local
Local1  feature@spacemap_histogram     active                         local
Local1  feature@enabled_txg            active                         local
Local1  feature@hole_birth             active                         local
Local1  feature@extensible_dataset     active                         local
Local1  feature@embedded_data          active                         local
Local1  feature@bookmarks              enabled                        local
Local1  feature@filesystem_limits      enabled                        local
Local1  feature@large_blocks           enabled                        local
Local1  feature@large_dnode            enabled                        local
Local1  feature@sha512                 enabled                        local
Local1  feature@skein                  enabled                        local
Local1  feature@edonr                  enabled                        local
Local1  feature@userobj_accounting     active                         local
Local1  feature@encryption             enabled                        local
Local1  feature@project_quota          active                         local
Local1  feature@device_removal         enabled                        local
Local1  feature@obsolete_counts        enabled                        local
Local1  feature@zpool_checkpoint       enabled                        local
Local1  feature@spacemap_v2            active                         local
Local1  feature@allocation_classes     enabled                        local
Local1  feature@resilver_defer         enabled                        local
Local1  feature@bookmark_v2            enabled                        local



The last output is too long, so it's in the textfile attached.

Cheers
 

Attachments

  • zpool_history.txt
    121.6 KB · Views: 2
In the history there are a few
Code: 
2021-01-06.19:47:47 ioctl reopen
    input:
scrub_restart: false
messages, and only at that date. I don't think this indicates anything by itself, but it might be a good idea to check the syslog from around that time too. When exactly did the data disappear?
 
  • Like
Reactions: def1ant
Must have been around that time.

I had the server powered off mostly, as it was comsuming too much power.

After trying and failing to join the cluster of my new server, i tried mounting an external usb harddrive to create a backup before doing anything else.
This resulted in the server not booting anymore, as i assigned it by drive id and as "automount", following a guide a found somewhere. Something went wrong there aswell, as my node couldn't find the drive after a restart and panicing.
To resolve that, i removed the entry from my fstab file again, wich let the system boot again.

After that the data was gone.

Maybe, sadly, proxmox is a bit too much for my humble learning by doing skills.

I will check the sys logs as soon as i have access to the server.

Thanks again for taking the time.
 
The oldest logs that were still there are from the 9th of January.

These lines might be of interest:

Code: 
Jan  9 15:41:02 pve rrdcached[945]: handle_request_update: Could not read RRD file.
Jan  9 15:41:02 pve pmxcfs[976]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-node/pve: -1
Jan  9 15:41:02 pve pmxcfs[976]: [status] notice: RRD update error /var/lib/rrdcached/db/pve2-node/pve: mmaping file '/var/lib/rrdcached/db/pve2-node/pve': Invalid argument


You can find the complete log in the attachments.
 

Attachments

  • 7.txt
    724.6 KB · Views: 2
Quick update, im disassembling the server now and will be giving up on restoring the data on it.
Thanks again for your help!
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back