[SOLVED] Accidentally deleted SSH Key

[Erikaso]

**Edit**
I finally was able to fix this! Thanks to everyone who helped ^^

Here is what I did:
I found a BTC miner on my proxmox node. It was running under the name "./kswapd0" and used 100% CPU.
While I was trying to remove everything associated with it, I did the following 2 mistakes:
1. I deleted all ssh keys in ~/.ssh/authorized_keys
2. I commented out everything in /etc/fstab (because I thought that the malware would use cron automated task)

Here is how I solved it:
After I rebooted the node I wasn't able to edit almost anything. So I used mount -o remount,rw / to be able to edit /etc/fstab and uncomment it.
Next I did was adding the ssh keys located in "/var/lib/pve-cluster/config.db" back to ~/.ssh/authorized_keys (idk if that is needed)
Then finally I was able to restart the service pve-cluster with systemctl status pve-cluster

Thats it! I hope this solves the issue for other Members with the same issue.

**Edit end**
Hey

My proxmox node server had a btc miner on it.
After I removed it I also deleted all ssh keys (didn't know proxmox needs them)
Please tell me there is a way to restore them...
I can't run any proxmox program:

I really can't lose the data :/
Can I at least backup the vms and reinstall proxmox?
I can't even install or update via apt.

 

[spirit]

ssh keys are stored in /etc/pve (the are a symlink in /root/.ssh/).

seem than corosync or pve-cluster service are not running. (so /etc/pve is not mounted).

can you check:

systemctl status corosync
systemctl status pve-cluster
df

?

 

[Erikaso]

ssh keys are stored in /etc/pve (the are a symlink in /root/.ssh/).

seem than corosync or pve-cluster service are not running. (so /etc/pve is not mounted).

can you check:

systemctl status corosync
systemctl status pve-cluster
df

?

root@nfacf1d:/etc# systemctl status corosync
● corosync.service - Corosync Cluster Engine
Loaded: loaded (/lib/systemd/system/corosync.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Condition: start condition failed at Mon 2021-04-12 09:03:40 CEST; 5min ago
└─ ConditionPathExists=/etc/corosync/corosync.conf was not met
Docs: man:corosync
man:corosync.conf
man:corosync_overview

Apr 12 08:31:28 nfacf1d systemd[1]: Condition check resulted in Corosync Cluster Engine being skipped.
Apr 12 08:31:28 nfacf1d systemd[1]: Condition check resulted in Corosync Cluster Engine being skipped.
Apr 12 08:31:28 nfacf1d systemd[1]: Condition check resulted in Corosync Cluster Engine being skipped.
Apr 12 08:31:28 nfacf1d systemd[1]: Condition check resulted in Corosync Cluster Engine being skipped.
Apr 12 09:03:39 nfacf1d systemd[1]: Condition check resulted in Corosync Cluster Engine being skipped.
Apr 12 09:03:39 nfacf1d systemd[1]: Condition check resulted in Corosync Cluster Engine being skipped.
Apr 12 09:03:40 nfacf1d systemd[1]: Condition check resulted in Corosync Cluster Engine being skipped.
Apr 12 09:03:40 nfacf1d systemd[1]: Condition check resulted in Corosync Cluster Engine being skipped.
Apr 12 09:03:40 nfacf1d systemd[1]: Condition check resulted in Corosync Cluster Engine being skipped.
Apr 12 09:03:40 nfacf1d systemd[1]: Condition check resulted in Corosync Cluster Engine being skipped.
root@nfacf1d:/etc# systemctl status pve-cluster
● pve-cluster.service - The Proxmox VE cluster filesystem
Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2021-04-12 09:03:40 CEST; 5min ago
Process: 4847 ExecStart=/usr/bin/pmxcfs (code=exited, status=255/EXCEPTION)

Apr 12 09:03:40 nfacf1d systemd[1]: pve-cluster.service: Service RestartSec=100ms expired, scheduling restart.
Apr 12 09:03:40 nfacf1d systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
Apr 12 09:03:40 nfacf1d systemd[1]: Stopped The Proxmox VE cluster filesystem.
Apr 12 09:03:40 nfacf1d systemd[1]: pve-cluster.service: Start request repeated too quickly.
Apr 12 09:03:40 nfacf1d systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Apr 12 09:03:40 nfacf1d systemd[1]: Failed to start The Proxmox VE cluster filesystem.
root@nfacf1d:/etc# df
Filesystem 1K-blocks Used Available Use% Mounted on
udev 32878388 0 32878388 0% /dev
tmpfs 6587900 25676 6562224 1% /run
/dev/mapper/pve-root 98559220 28378608 65131064 31% /
tmpfs 32939488 0 32939488 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 32939488 0 32939488 0% /sys/fs/cgroup
tmpfs 6587896 0 6587896 0% /run/user/0


I deleted the key from ~/.ssh/authorized_keys.

 

[spirit]

I deleted the key from ~/.ssh/authorized_keys.

it's not a problem, you can still readd them.

(just copy/paste /root/.ssh/id_rsa.pub from each server to authorized_keys)


Do you have a cluster with multiple hosts or it's a single server ? (corosync is only used with multiple servers)


For pve-cluster: can you try to launch "/usr/bin/pmxcfs" manually , to see if you have a specific error

 

[Erikaso]

it's not a problem, you can still readd them.

(just copy/paste /root/.ssh/id_rsa.pub from each server to authorized_keys)


Do you have a cluster with multiple hosts or it's a single server ? (corosync is only used with multiple servers)


For pve-cluster: can you try to launch "/usr/bin/pmxcfs" manually , to see if you have a specific error

But the file /root/.ssh/id_rsa.pub doesn't exist.
root@nfacf1d:~/.ssh# cat /root/.ssh/id_rsa.pub
cat: /root/.ssh/id_rsa.pub: No such file or directory
I was restarting the server before maybe that wasn't good.

I have installed 2 vm's and owncloud on it.

Do you think I can back up the vm's without using proxmox?

 

[Tapio Lehtonen]

Can you restore the missing files from backups?

 

[Erikaso]

Can you restore the missing files from backups?

What files do you mean?
If you mean the ssh key then no I don't have any backup of that.
I only have old backups of the vm's

 

[spirit]

ls -lah /root/.ssh/
total 56K
drwx------  2 root root    7 Mar  7 17:16 .
drwx------ 45 root root   74 Apr 10 14:53 ..
lrwxrwxrwx  1 root root   29 Oct 20 08:06 authorized_keys -> /etc/pve/priv/authorized_keys

if you have really delete it, it's not a problem, you can recreate it manually.

but your problem is that pve-cluster (pmxcfs) is not running, and so, /etc/pve/ is not mounted.

 

[Erikaso]

ls -lah /root/.ssh/
total 56K
drwx------  2 root root    7 Mar  7 17:16 .
drwx------ 45 root root   74 Apr 10 14:53 ..
lrwxrwxrwx  1 root root   29 Oct 20 08:06 authorized_keys -> /etc/pve/priv/authorized_keys

if you have really delete it, it's not a problem, you can recreate it manually.

but your problem is that pve-cluster (pmxcfs) is not running, and so, /etc/pve/ is not mounted.

Is there anything I can do to mount it again?

 

[Erikaso]

I found a lot of ssh keys in "/var/lib/pve-cluster/config.db"
Is there any key that I can use for this to recover?