Can't unlock vm

[mmohamed2]

Hi,

i have errors when i try to backup


I tried

• qm unlock 129

but i got

• unable to open file '/etc/pve/nodes/proxmoxtiebosch/qemu-server/129.conf.tmp.22475' - Input/output error

Also i can't create folder




How can i fix it please ?
Thank you !

 

[t.lamprecht]

Hi,

First: you're on a pretty outdated version of Proxmox VE, the current version of the old 5.x release is 5.4, and 6.1 would be available already..

That said, it seems like the cluster file system hangs, this could be due to two things:

* some bug / situation which made the pmxcfs lock up

* Real hardware error, e.g., a faulty disk where the database of the pmxcfs is, or some thing similar.

Please post some additional info:

ps aux|grep pmxcfs
systemctl status pve-cluster

Check your disks and the syslog/journal for some more telling error messages.

 

[mmohamed2]

Thank you for reply,

First: you're on a pretty outdated version of Proxmox VE, the current version of the old 5.x release is 5.4, and 6.1 would be available already..

Can you guide me how to upgrade Proxmox VE please ?

ps aux|grep pmxcfs

root@proxmoxtiebosch:~# ps aux|grep pmxcfs
root 4361 0.1 0.1 1136680 59048 ? Ssl Apr08 4:14 /usr/bin/pmxcfs
root 7465 0.0 0.0 12784 852 pts/1 S+ 09:52 0:00 grep pmxcfs

systemctl status pve-cluster

root@proxmoxtiebosch:~# systemctl status pve-cluster
● pve-cluster.service - The Proxmox VE cluster filesystem
Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2020-04-08 12:18:11 CEST; 2 days ago
Main PID: 4361 (pmxcfs)
Tasks: 13 (limit: 4915)
Memory: 59.2M
CPU: 4min 15.268s
CGroup: /system.slice/pve-cluster.service
└─4361 /usr/bin/pmxcfs

Apr 09 01:32:46 proxmoxtiebosch pmxcfs[4361]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmoxtiebosch/local: -1
Apr 09 01:32:46 proxmoxtiebosch pmxcfs[4361]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmoxtiebosch/local-zfs: -1
Apr 09 01:35:48 proxmoxtiebosch pmxcfs[4361]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmoxtiebosch/rpool2: -1
Apr 09 01:35:48 proxmoxtiebosch pmxcfs[4361]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmoxtiebosch/local: -1
Apr 09 01:35:48 proxmoxtiebosch pmxcfs[4361]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmoxtiebosch/local-zfs: -1
Apr 09 01:36:38 proxmoxtiebosch pmxcfs[4361]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmoxtiebosch/local-zfs: -1
Apr 09 01:36:38 proxmoxtiebosch pmxcfs[4361]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmoxtiebosch/local: -1
Apr 09 01:36:38 proxmoxtiebosch pmxcfs[4361]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmoxtiebosch/rpool2: -1
Apr 09 01:46:26 proxmoxtiebosch pmxcfs[4361]: [database] crit: commit transaction failed: database or disk is full#010
Apr 09 01:46:26 proxmoxtiebosch pmxcfs[4361]: [database] crit: rollback transaction failed: cannot rollback - no transaction is active#010


Disk spaces

 

[mmohamed2]

Is it safe to upgrade from 5.2 to 5.4.2 without interference of the runnings 2 vm ?

i need to use

apt update

followed by

apt dist-upgrade

Correct ?

 

[spirit]

Apr 09 01:32:46 proxmoxtiebosch pmxcfs[4361]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmoxtiebosch/local: -1
Apr 09 01:32:46 proxmoxtiebosch pmxcfs[4361]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmoxtiebosch/local-zfs: -1
Apr 09 01:35:48 proxmoxtiebosch pmxcfs[4361]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmoxtiebosch/rpool2: -1
Apr 09 01:35:48 proxmoxtiebosch pmxcfs[4361]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmoxtiebosch/local: -1
Apr 09 01:35:48 proxmoxtiebosch pmxcfs[4361]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmoxtiebosch/local-zfs: -1
Apr 09 01:36:38 proxmoxtiebosch pmxcfs[4361]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmoxtiebosch/local-zfs: -1
Apr 09 01:36:38 proxmoxtiebosch pmxcfs[4361]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmoxtiebosch/local: -1
Apr 09 01:36:38 proxmoxtiebosch pmxcfs[4361]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/proxmoxtiebosch/rpool2: -1
Apr 09 01:46:26 proxmoxtiebosch pmxcfs[4361]: [database] crit: commit transaction failed: database or disk is full#010

seem that you have some kind of disk problem, and it's not able to write to /var/lib. (pmxcfs db is located in /var/lib/pve-cluster/config.db).

Are you able to create/write files in / , /var/lib/ ?

 

[mmohamed2]

seem that you have some kind of disk problem, and it's not able to write to /var/lib. (pmxcfs db is located in /var/lib/pve-cluster/config.db).

Are you able to create/write files in / , /var/lib/ ?

i did
nano test.txt
and file created successfully

 

[mmohamed2]

Is it safe to do apt dist-upgrade to upgrade from 5.2 to the latest 5 version ?
Do you think this will fix the problem ?

 

[mmohamed2]

Here's some of syslog

Apr 11 16:42:54 proxmoxtiebosch kernel: EXT4-fs error (device sdc1): ext4_find_entry:1436: inode #2: comm pvestatd: reading directory lblock 0
Apr 11 16:42:54 proxmoxtiebosch kernel: blk_partition_remap: fail for partition 1
Apr 11 16:42:54 proxmoxtiebosch kernel: EXT4-fs error (device sdc1): ext4_find_entry:1436: inode #2: comm pvestatd: reading directory lblock 0
Apr 11 16:42:57 proxmoxtiebosch pve-ha-lrm[4705]: unable to write lrm status file - unable to open file '/etc/pve/nodes/proxmoxtiebosch/lrm_status.tmp.4705' - Input/output error
Apr 11 16:43:00 proxmoxtiebosch systemd[1]: Starting Proxmox VE replication runner...
Apr 11 16:43:01 proxmoxtiebosch pvesr[9177]: trying to acquire cfs lock 'file-replication_cfg' ...
Apr 11 16:43:02 proxmoxtiebosch pvesr[9177]: trying to acquire cfs lock 'file-replication_cfg' ...
Apr 11 16:43:02 proxmoxtiebosch pve-ha-lrm[4705]: unable to write lrm status file - unable to open file '/etc/pve/nodes/proxmoxtiebosch/lrm_status.tmp.4705' - Input/output error
Apr 11 16:43:03 proxmoxtiebosch pvesr[9177]: trying to acquire cfs lock 'file-replication_cfg' ...
Apr 11 16:43:04 proxmoxtiebosch pvestatd[4513]: mkdir /media/usb-drive/private: Input/output error at /usr/share/perl5/PVE/Storage/Plugin.pm line 903.
Apr 11 16:43:04 proxmoxtiebosch kernel: blk_partition_remap: fail for partition 1
Apr 11 16:43:04 proxmoxtiebosch kernel: EXT4-fs error (device sdc1): ext4_find_entry:1436: inode #2: comm pvestatd: reading directory lblock 0
Apr 11 16:43:04 proxmoxtiebosch kernel: blk_partition_remap: fail for partition 1
Apr 11 16:43:04 proxmoxtiebosch kernel: EXT4-fs error (device sdc1): ext4_find_entry:1436: inode #2: comm pvestatd: reading directory lblock 0
Apr 11 16:43:04 proxmoxtiebosch kernel: blk_partition_remap: fail for partition 1
Apr 11 16:43:04 proxmoxtiebosch kernel: EXT4-fs error (device sdc1): ext4_find_entry:1436: inode #2: comm pvestatd: reading directory lblock 0
Apr 11 16:43:04 proxmoxtiebosch pvesr[9177]: trying to acquire cfs lock 'file-replication_cfg' ...
Apr 11 16:43:05 proxmoxtiebosch pvesr[9177]: trying to acquire cfs lock 'file-replication_cfg' ...
Apr 11 16:43:06 proxmoxtiebosch pvesr[9177]: trying to acquire cfs lock 'file-replication_cfg' ...
Apr 11 16:43:07 proxmoxtiebosch pvesr[9177]: trying to acquire cfs lock 'file-replication_cfg' ...
Apr 11 16:43:07 proxmoxtiebosch pve-ha-lrm[4705]: unable to write lrm status file - unable to open file '/etc/pve/nodes/proxmoxtiebosch/lrm_status.tmp.4705' - Input/output error
Apr 11 16:43:08 proxmoxtiebosch pvesr[9177]: trying to acquire cfs lock 'file-replication_cfg' ...
Apr 11 16:43:09 proxmoxtiebosch pvesr[9177]: trying to acquire cfs lock 'file-replication_cfg' ...
Apr 11 16:43:10 proxmoxtiebosch pvesr[9177]: error with cfs lock 'file-replication_cfg': got lock request timeout
Apr 11 16:43:10 proxmoxtiebosch systemd[1]: pvesr.service: Main process exited, code=exited, status=5/NOTINSTALLED
Apr 11 16:43:10 proxmoxtiebosch systemd[1]: Failed to start Proxmox VE replication runner.
Apr 11 16:43:10 proxmoxtiebosch systemd[1]: pvesr.service: Unit entered failed state.
Apr 11 16:43:10 proxmoxtiebosch systemd[1]: pvesr.service: Failed with result 'exit-code'.
Apr 11 16:43:12 proxmoxtiebosch pve-ha-lrm[4705]: unable to write lrm status file - unable to open file '/etc/pve/nodes/proxmoxtiebosch/lrm_status.tmp.4705' - Input/output error
Apr 11 16:43:14 proxmoxtiebosch pvestatd[4513]: mkdir /media/usb-drive/private: Input/output error at /usr/share/perl5/PVE/Storage/Plugin.pm line 903.
Apr 11 16:43:14 proxmoxtiebosch kernel: blk_partition_remap: fail for partition 1
Apr 11 16:43:14 proxmoxtiebosch kernel: EXT4-fs error (device sdc1): ext4_find_entry:1436: inode #2: comm pvestatd: reading directory lblock 0
Apr 11 16:43:14 proxmoxtiebosch kernel: blk_partition_remap: fail for partition 1
Apr 11 16:43:14 proxmoxtiebosch kernel: EXT4-fs error (device sdc1): ext4_find_entry:1436: inode #2: comm pvestatd: reading directory lblock 0
Apr 11 16:43:14 proxmoxtiebosch kernel: blk_partition_remap: fail for partition 1
Apr 11 16:43:14 proxmoxtiebosch kernel: EXT4-fs error (device sdc1): ext4_find_entry:1436: inode #2: comm pvestatd: reading directory lblock 0
Apr 11 16:43:17 proxmoxtiebosch pve-ha-lrm[4705]: unable to write lrm status file - unable to open file '/etc/pve/nodes/proxmoxtiebosch/lrm_status.tmp.4705' - Input/output error
Apr 11 16:43:21 proxmoxtiebosch sshd[9766]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=49.88.112.72  user=root
Apr 11 16:43:22 proxmoxtiebosch pve-ha-lrm[4705]: unable to write lrm status file - unable to open file '/etc/pve/nodes/proxmoxtiebosch/lrm_status.tmp.4705' - Input/output error
Apr 11 16:43:23 proxmoxtiebosch sshd[9766]: Failed password for root from 49.88.112.72 port 13958 ssh2
Apr 11 16:43:24 proxmoxtiebosch pvestatd[4513]: mkdir /media/usb-drive/private: Input/output error at /usr/share/perl5/PVE/Storage/Plugin.pm line 903.
Apr 11 16:43:24 proxmoxtiebosch kernel: blk_partition_remap: fail for partition 1
Apr 11 16:43:24 proxmoxtiebosch kernel: EXT4-fs error (device sdc1): ext4_find_entry:1436: inode #2: comm pvestatd: reading directory lblock 0
Apr 11 16:43:24 proxmoxtiebosch kernel: blk_partition_remap: fail for partition 1
Apr 11 16:43:24 proxmoxtiebosch kernel: EXT4-fs error (device sdc1): ext4_find_entry:1436: inode #2: comm pvestatd: reading directory lblock 0
Apr 11 16:43:24 proxmoxtiebosch kernel: blk_partition_remap: fail for partition 1
Apr 11 16:43:24 proxmoxtiebosch kernel: EXT4-fs error (device sdc1): ext4_find_entry:1436: inode #2: comm pvestatd: reading directory lblock 0
Apr 11 16:43:25 proxmoxtiebosch sshd[9766]: Failed password for root from 49.88.112.72 port 13958 ssh2
Apr 11 16:43:27 proxmoxtiebosch pve-ha-lrm[4705]: unable to write lrm status file - unable to open file '/etc/pve/nodes/proxmoxtiebosch/lrm_status.tmp.4705' - Input/output error
Apr 11 16:43:27 proxmoxtiebosch sshd[9766]: Failed password for root from 49.88.112.72 port 13958 ssh2
Apr 11 16:43:27 proxmoxtiebosch sshd[9766]: Received disconnect from 49.88.112.72 port 13958:11:  [preauth]
Apr 11 16:43:27 proxmoxtiebosch sshd[9766]: Disconnected from 49.88.112.72 port 13958 [preauth]
Apr 11 16:43:27 proxmoxtiebosch sshd[9766]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=49.88.112.72  user=root
Apr 11 16:43:32 proxmoxtiebosch pve-ha-lrm[4705]: unable to write lrm status file - unable to open file '/etc/pve/nodes/proxmoxtiebosch/lrm_status.tmp.4705' - Input/output error
Apr 11 16:43:34 proxmoxtiebosch pvestatd[4513]: mkdir /media/usb-drive/private: Input/output error at /usr/share/perl5/PVE/Storage/Plugin.pm line 903.
Apr 11 16:43:34 proxmoxtiebosch kernel: blk_partition_remap: fail for partition 1
Apr 11 16:43:34 proxmoxtiebosch kernel: EXT4-fs error (device sdc1): ext4_find_entry:1436: inode #2: comm pvestatd: reading directory lblock 0
Apr 11 16:43:34 proxmoxtiebosch kernel: blk_partition_remap: fail for partition 1
Apr 11 16:43:34 proxmoxtiebosch kernel: EXT4-fs error (device sdc1): ext4_find_entry:1436: inode #2: comm pvestatd: reading directory lblock 0
Apr 11 16:43:34 proxmoxtiebosch kernel: blk_partition_remap: fail for partition 1
Apr 11 16:43:34 proxmoxtiebosch kernel: EXT4-fs error (device sdc1): ext4_find_entry:1436: inode #2: comm pvestatd: reading directory lblock 0
Apr 11 16:43:37 proxmoxtiebosch pve-ha-lrm[4705]: unable to write lrm status file - unable to open file '/etc/pve/nodes/proxmoxtiebosch/lrm_status.tmp.4705' - Input/output error
Apr 11 16:43:42 proxmoxtiebosch pve-ha-lrm[4705]: unable to write lrm status file - unable to open file '/etc/pve/nodes/proxmoxtiebosch/lrm_status.tmp.4705' - Input/output error
Apr 11 16:43:44 proxmoxtiebosch kernel: blk_partition_remap: fail for partition 1
Apr 11 16:43:44 proxmoxtiebosch kernel: EXT4-fs error (device sdc1): ext4_find_entry:1436: inode #2: comm pvestatd: reading directory lblock 0
Apr 11 16:43:44 proxmoxtiebosch kernel: blk_partition_remap: fail for partition 1
Apr 11 16:43:44 proxmoxtiebosch kernel: EXT4-fs error (device sdc1): ext4_find_entry:1436: inode #2: comm pvestatd: reading directory lblock 0
Apr 11 16:43:44 proxmoxtiebosch kernel: blk_partition_remap: fail for partition 1
Apr 11 16:43:44 proxmoxtiebosch kernel: EXT4-fs error (device sdc1): ext4_find_entry:1436: inode #2: comm pvestatd: reading directory lblock 0
Apr 11 16:43:44 proxmoxtiebosch pvestatd[4513]: mkdir /media/usb-drive/private: Input/output error at /usr/share/perl5/PVE/Storage/Plugin.pm line 903.
Apr 11 16:43:47 proxmoxtiebosch pve-ha-lrm[4705]: unable to write lrm status file - unable to open file '/etc/pve/nodes/proxmoxtiebosch/lrm_status.tmp.4705' - Input/output error
Apr 11 16:43:52 proxmoxtiebosch pve-ha-lrm[4705]: unable to write lrm status file - unable to open file '/etc/pve/nodes/proxmoxtiebosch/lrm_status.tmp.4705' - Input/output error
Apr 11 16:43:54 proxmoxtiebosch pvestatd[4513]: mkdir /media/usb-drive/private: Input/output error at /usr/share/perl5/PVE/Storage/Plugin.pm line 903.

 

[mmohamed2]

@t.lamprecht i added syslog, please check