Flash drive with PVE died and only disk are left

[epoddubniy]

Hello! Please help!
I have a problem.

Server configuration: ssd 240gb, hdd 1TB, hdd 2TB

Previously proxmox was installed on 16gb flash drive.
ssd and 2*hdd were wiped and initialized as GTP, and created storage "directory"
After that virtual machine was created. System disk of VM was created and placed on SSD (80gb) and data disk was created and placed on hdd 1TB
Suddenly flash drive crushed.
I installed proxmox again. During installation gave another name to node, turned it on and didn't find any VM or disks.
Earlier i worked with ESXI. When the flash had died i installed the ESXI of the same version, created a VM from a file (or created new one) and attached disks (existing vmdk's). It took me 30 minutes. With proxmox i got stuck.

I beg you please, write me the tutorial on my problem like if you did it for a child, because i am far from linux and know a few commands like "lsblk" and "systemctl" and work with system only through the "WinSCP"

Please ask me a question if i missed any details.

 

[Maximiliano]

Hello, proxmox is an entire operative system which needs to do lots of IO (logs, for example), this IO can wear out USB sticks (or sd cards for that matter) extremely quickly. I would advise to install pve on a HDD or SSD.

 

[Chris]

HI,
do you have a backup of the contents of /etc/pve or the proxmox cluster filesystem backing database located at /var/lib/pve-cluster/config.db? If so, you can either recreate the contents in /etc/pve and your storage and VMs/CTs should reappear or follow the procedure from the docs to change the pmxcfs database [0].

If you did not make a backup of those files, then you will have to recreate your storage and VM configs from scratch.

Alternatively, if you have a backup of all of your VMs, you can just recreate the storage config and extract the VM config from the backup or restore the VMs from backup entirely.

[0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_recovery

 

[epoddubniy]

Chris, to my dissapointment i don't have any backup.

Could you tell me please how do i have to backup directories you talked about. Will it be enough just to join the host through WinSCP, follow the directory and copy the files to another drive?

Also i would like to ask, what steps i will have to reproduce if i had backups of those directories.

And i would be very greatfull if you tell me step by step in the smallest details the procedure of recreating by scratch (i hope this way will not damage the data saved on virtual disks which were placed on hdd 1TB and SSD).

It is too difficult and too much desire to make it clear and solved...

Thank you

 

[Dunuin]

Could you tell me please how do i have to backup directories you talked about. Will it be enough just to join the host through WinSCP, follow the directory and copy the files to another drive?

Yes. But I would automate it, as people tend to forget to do such things regularly because it is annoying and when then something happens your backups are totally outdated. Create a cron to copy the files to a NAS, sync them to a cloud storage, use the proxmox-backup-client to send them to your Proxmox Bacup Server or whatever.

Also i would like to ask, what steps i will have to reproduce if i had backups of those directories.

And i would be very greatfull if you tell me step by step in the smallest details the procedure of recreating by scratch (i hope this way will not damage the data saved on virtual disks which were placed on hdd 1TB and SSD).

It is too difficult and too much desire to make it clear and solved...

Best you search the forum to see how other people did this. You would wonder how much people didn't create any backups and then fall into despair and want to rescue some data...

In short:
- backup all your disks (for example clonezilla) so you can't screw up things even more
- create a new VM without any disks using another VMID using the same configs (you will have to remember how you set them up previously or they might not work/boot)
- rename your virtual disks to the new VMID(how this works totally depends on what storage you used) to match the naming sceme
- run a qm rescan YourVMID and then a "unattached disk" should show up in the webUI for that VM
- edit that unattached disk and attach it
- repeat for next VM

 

[epoddubniy]

Finally i found a possibility to copy /etc/pve and /var/lib/pve-cluster/config.db to the cooresponding directories of the new node (the new nod has name different from the old one).

Then i opened storage.cfg and replaced all the names of old node to new one. And i did the same with config.db

But here is the result: no directories mounted. For example i tried to do following mkdir /mnt/pve/toshiba (because this directory existed) and tried to mnt /dev/sdc1 (the partition on disk toshiba) /mnt/pve/toshiba. Previously i stored there on the toshiba 2 ISO images, but after this commands there was nothing there. May be i did something wrong? Help me please to restore.

I also did this:

created file toshiba.mount in /etc/systemd/system

[Install]
WantedBy=multi-user.target


[Mount]
Options=defaults
Type=ext4
What=/dev/disk/by-uuid/15fe5a36-26f9-4098-be64-16695d68a48b
Where=/mnt/pve/toshiba

and tried to do this command systemctl start toshiba.mount

but i got some red message of following content:
Failed to start toshiba.mount: Unit toshiba.mount has a bad unit file setting.
See system logs and 'systemctl status toshiba.mount' for details.

 

[epoddubniy]

By the way, when i tried to copy the content of /etc/pve files that began with dot didnt copy

 

[Maximiliano]

Those are for tracking the internal state of Proxmox VE, I would not recommend backing those up.

 

[Chris]

Finally i found a possibility to copy /etc/pveand /var/lib/pve-cluster/config.db to the cooresponding directories of the new node (the new nod has name different from the old one).

So you had a backup nevertheless, or where did you get these files from?

Then i opened storage.cfg and replaced all the names of old node to new one. And i did the same with config.db

I am not sure I am following here. storage.cfg from an other node will not reflect the storage config of the node as you had before on this node. I mean, it might be a good starting point if these nodes had a similar storage layout. Also, config.db is the sqlite database backing /etc/pve, so also not sure what you mean by the last part.

But here is the result: no directories mounted. For example i tried to do following mkdir /mnt/pve/toshiba (because this directory existed) and tried to mnt /dev/sdc1 (the partition on disk toshiba) /mnt/pve/toshiba. Previously i stored there on the toshiba 2 ISO images, but after this commands there was nothing there. May be i did something wrong? Help me please to restore.

Could you please share the output of

lsblk -o +FSTYPE
ls -lR /mnt/pve/toshiba

 

[epoddubniy]

I am not sure I am following here. storage.cfg from an other node will not reflect the storage config of the node as you had before on this node. I mean, it might be a good starting point if these nodes had a similar storage layout. Also, config.db is the sqlite database backing /etc/pve, so also not sure what you mean by the last part.

i tried to do this

do you have a backup of the contents of /etc/pve or the proxmox cluster filesystem backing database located at /var/lib/pve-cluster/config.db? If so, you can either recreate the contents in /etc/pve and your storage and VMs/CTs should reappear or follow the procedure from the docs to change the pmxcfs database [0].

i tried to recreate the contents, but i don't know how to do it right. All the storages, partitions, VMs, virtual disks - all the same. I just put off flash drive with "oldProxmox", put in another flash and installed "newProxmox" and can't understand what shoul i do next to see storages and VMs again.

So you had a backup nevertheless, or where did you get these files from?

Fortunately It looks like "yes"

lsblk -o +FSTYPE ls -lR /mnt/pve/toshiba

 

[Chris]

HI, well there is nothing mounted on /mnt/pve/toshiba. You will have to add the drive in the fstab in order for it to get mounted on boot. You can use the following command to do that

 echo 'UUID=<PARTITION-UUID> /mnt/pve/toshiba    ext4    defaults 0   2' >> /etc/fstab

Of course you will have to set the correct partition UUID, which you can find in the output of lsblk -o +FSTYPE,UUID

Can you paste the output of cat /etc/pve/storage.cfg and ls -lah /etc/pve/qemu-server/. I am not sure you restored any configs by your actions, copying the content of /etc/pve from an fresh PVE install does not give you back your original config.

 

[epoddubniy]

Thank you Chris!
I feel i am going to success due to your help!

cat /etc/pve/storage.cfg and ls -lah /etc/pve/qemu-server/

 

[epoddubniy]

Perhaps this information on my problem will be usefull

 

[epoddubniy]

Finally i am fully exhausted and have strong disire to give up...
i have never met such amount of trouble...
i am crying. ESXi seems much easier in use

 

[Chris]

Finally i am fully exhausted and have strong disire to give up...
i have never met such amount of trouble...
i am crying. ESXi seems much easier in use

seems like you put the incorrect UUID in the fstab entry, systemd waits for the disk to mount it but is not able to find it. Did you use the UUID or PARTUUID

 

[epoddubniy]

seems like you put the incorrect UUID in the fstab entry, systemd waits for the disk to mount it but is not able to find it. Did you use the UUID or PARTUUID

Chris, would you please correct me if i did someting wrong

 

[Dunuin]

Don't put your UUID in "<" and ">". A "<REPLACE_ME>" usually means you should replace the whole thing including the "<" and ">" signs.

 

[epoddubniy]

Thank you, Chris!

Don't put your UUID in "<" and ">". A "<REPLACE_ME>" usually means you should replace the whole thing including the "<" and ">" signs.

The disk has appeared, i can see it among storages, but another problem has come.
Earlier i stored on "toshiba" 2 ISO files and now, after mounting, i can't find them there.
This thing makes me worry about the virtual disks and files of virtual machines, stored earlier on another 2 disks (when the old node named "Proxmox" existed).
Will i also loose them if i do the same procedure of mounting:

echo 'UUID=<PARTITION-UUID> /mnt/pve/toshiba ext4 defaults 0 2' >> /etc/fstab

And what should i do to avoid such result.
Thank you for help!

 

[Dunuin]

Earlier i stored on "toshiba" 2 ISO files and now, after mounting, i can't find them there.

So you stored the ISOs on the root filesystem, as the mountpoint wasn't mounted? Linux won't merge a muntpoint. If you put data in a folder that should act as a mounbtpoint but that isn't mounted and you then mount that mountpoint, your data will still be there, you just can't access it, as only the mounted data is now available. So you could unmount that disk and see if the ISOs will show up in that mountpoint again.

If they do, you should set a "is_mountpoint /mnt/pve/toshiba" for your directory storage as a storage shouldn`t be able to be used when not mounted.

 

[epoddubniy]

I am afraid it is to difficult for me to understand.
I still don't understand how does linux works? I read about it but...
For example, i have windows, i have system SSD, and HDD. I create partition on HDD, i create folders and copy there some data. When my SSD dies i just reinstall windows and can use my data again. Everything still exists.
Another example. ESXi. I have SSD, I have HDD, i create datastore-SSD, i create datastore-HDD. On SSD datastore i create vmdk, where i will install windows (systemdrive), on HDD i create vmdk, which i will use for data and for storeing ISOs.
If my ESXi dies, i just reinstall it, scan for datastores, scan for VMs and that is it. I don't need to make any backups of ESXi, i know that in the worst way i will spend 1 hour but i restore whole system and it will be working. With proxmox - one week left.
Some people told me: "friend, proxmox is the future, it is so easy! It is much easy than ESXi! It is just linux Debian!" And i decided to try. I decided to recreate the real situation that i had ran into my work (hypervisor died). To solve this problem on my work it took me 1 hour. All servers began its work. I am afraid, if i administrated proxmox and face such problem i would delete all the data (like now with ISOs on disk toshiba) and retired.
Excuse me please for the story, and making you read it with my negative emotions.
I saw video about backup system in proxmox, but i dont have NAS.
Do i understand right, if system drive with proxmox (flash or hdd) dies whole my server (all vmdk's, virtual machines, ISOs) becomes useless?

If you put data in a folder that should act as a mounbtpoint but that isn't mounted and you then mount that mountpoint, your data will still be there, you just can't access it, as only the mounted data is now available.

Really i don't now the difference beetwen "mount mountpoint" and "mount disk". I don't even now what i did, i clicked on node-name, clicked on menu "disks", chose the disk, wiped it and clicked "add directory". It seems like i mounted a mountpoint. Does it mean that i do something wrong with this command

echo 'UUID=<PARTITION-UUID> /mnt/pve/toshiba ext4 defaults 0 2' >> /etc/fstab

So you could unmount that disk and see if the ISOs will show up in that mountpoint again.

Did i understand right, that mountpiont can be mounted without mounting disk in my situation?

you should set a "is_mountpoint /mnt/pve/toshiba" for your directory storage

Tell me please, if i should copy/paste this command in shell and that is it?