PROXMOX FAILS TO BOOT AFTER POWER LOSS

[CryptoVibe]

Hello Proxmox forum, I am hoping someone can help me with this.

This is the second time this happened with Proxmox. My network went down & once I had it back up, I had to power off the Proxmox host using the power button on the front of the server. Now Proxmox won't boot up again at all. I get these errors, which are very similar to what happened about a month ago. I am using the server for Chia farming. I started learning how to setup Proxmox, TrueNAS & Ubuntu in March. I finally got Chia farming in May. This same thing happened in June & I redid everything from scratch to learn this stuff more.

Is there a way to repair Proxmox, like in Windows you can restore the OS & not lose data? I have backups of the VM's, but not the True NAS VM. That VM would not backup for some reason, perhaps because the NFS shares were in use or something similar I am thinking. I was going to shut off the other VM's & see if I could get TrueNAS to run a backup, but this happened before I was able to try that. I have 3700+ plots on drives in TrueNAS. Last time this happened I lost all the plots, because they were no longer able to be read. It took nearly two weeks to plot all of those plots, not to mention the wear on the drives to create that many plots... I'm really hoping to be able to recover this Proxmox installation with the help of forum users with more knowledge of Proxmox than I have so far.

Thank you,

These are the errors I see when booting up:

 

[Dunuin]

My network went down & once I had it back up, I had to power off the Proxmox host using the power button on the front of the server.

Power-loss = data-loss. You really shouldn't do that intentionally (and your server should be behind a UPS so this also won`t happen unintentionally...). Correct would be to attach a keyboard + display or use you IPMIs webKVM to access the console and then fix the networking problems or at least do a proper shutdown.

Is there a way to repair Proxmox, like in Windows you can restore the OS & not lose data?

No, not that easily.

I have 3700+ plots on drives in TrueNAS. Last time this happened I lost all the plots, because they were no longer able to be read.

TrueNAS is using ZFS. You could have imported that pool on any linux with ZFS support, even a Live Linux like Ubuntu instead of wiping everthing.

I have backups of the VM's, but not the True NAS VM.

Then you should have backed up the TrueNAS config file. With that it would have been very easy to create a new TrueNAS VM from scratch to then import that config file.

 

[CryptoVibe]

Power-loss = data-loss. You really shouldn't do that intentionally (and your server should be behind a UPS so this also won`t happen unintentionally...). Correct would be to attach a keyboard + display or use you IPMIs webKVM to access the console and then fix the networking problems or at least do a proper shutdown.

No, not that easily.


TrueNAS is using ZFS. You could have imported that pool on any linux with ZFS support, even a Live Linux like Ubuntu instead of wiping everthing.

I did not want to power it down that way, but it was not responsive & I could not get into it, even through the Dell iDRAC I have connected.

I have backups of the MadMax Chia Farmer, Ubuntu Chia GUI Farmer & Chia Plotter. Is there a way to recover the 3700+ plots that I had? I did import the pool last time & all the plots were there, but they were unable to be read / farmed. There was some error that said the plots could not be read.

Thank you for your fast reply.

 

[CryptoVibe]

I can reinstall Proxmox & easily restore the:

MadMax Chia Farmer
Ubuntu Chia GUI Farmer
MadMax Chia Plotter

Last time TrueNAS saw the pools I had previously created, but the MadMax Chia Farmer was unable to read the plots... That's why I re-plotted everything...

 

[Dunuin]

I did import the pool last time & all the plots were there, but they were unable to be read / farmed. There was some error that said the plots could not be read.

Then I would run a ZFS scrub and long SMART selftest (especially of /dev/sdm) and check if zpool status -v is reporting any errors.

 

[CryptoVibe]

Thank you for your reply.

How can I run that, I'm unable to get into the Proxmox installation right now... The attached screen shot is all I see when I try to boot the server up:

 

[aaronouthier]

Use a bootable USB rescue disk, such as system rescue disk. I seem to recall the Proxmox install disk has a rescue mode? Could be wrong on that last one.

 

[CryptoVibe]

Thank you for your reply, I tried that... This is the message I get when I try the "Rescue Boot":

 

[CryptoVibe]

I'm fine with re-doing the Proxmox installation, & restoring the backups, but I want to be able to recover the 3700+ plots...

Thank you,

 

[CryptoVibe]

I tried another option, Debug Mode with the Graphical Interface, I think it is called. I get this output though:

 

[CryptoVibe]

If anyone knows a way to recover Proxmox, or not lose these plots, I would VERY much be grateful for your help.

Thank you,

 

[CryptoVibe]

I ended up redoing the Proxmox host, recovering my backups, installing a fresh TrueNAS & it looks like I am able to recover the previous pools / plots. The cause of this problem looks to be the LSI00300 IT Mode LSI 9207-8E 6Gb/s Extelnal PCI-E 3.0x8 Host Controller Card I recently installed before all this started. I'm not sure why the card caused all of this yet, but if someone else experiences this too, I hope they find this post & it saves them time. I'll update this when I figure out what is going on with this Controller Card.

I have a Dell R720XD using two NVMe SSD's for system drives & I am farming Chia with this system.

 

[cirqui]

I ended up redoing the Proxmox host, recovering my backups, installing a fresh TrueNAS & it looks like I am able to recover the previous pools / plots. The cause of this problem looks to be the LSI00300 IT Mode LSI 9207-8E 6Gb/s Extelnal PCI-E 3.0x8 Host Controller Card I recently installed before all this started. I'm not sure why the card caused all of this yet, but if someone else experiences this too, I hope they find this post & it saves them time. I'll update this when I figure out what is going on with this Controller Card.

I have a Dell R720XD using two NVMe SSD's for system drives & I am farming Chia with this system.

Were you able to make any progress? I installed the same HBA card and can no longer boot it caused a boot issue. Removing the card and deleting the vm passthrough config resolved the issue. Seems like a different card might be necessary.

 

[CryptoVibe]

Were you able to make any progress? I installed the same HBA card and can no longer boot it caused a boot issue. Removing the card and deleting the vm passthrough config resolved the issue. Seems like a different card might be necessary.

I have the 2nd R730xd almost fully setup. I am trying Chia 2.0 & going to use Bladebit to see how the compressed plots go using that. I switched what I was doing from the NetApp connected to the 1st R730, so I would have a 2nd server to keep backups of my VM's on, in a cluster. The next thing will be setting up this NetApp & figuring out the correct HBA card. HOWEVER, if you look for "Art of Server" on eBay & send him an outline of what you're trying to do, what you want to connect, he MIGHT have the correct card for you. I have gotten two H710 cards in IT mode & I am able to see all 12 HDD's & able to set them up in TrueNAS then create my mount points & get them seen in the other VM's. I hope this helps you.