I just woke up to a crashed virtual machine.
Thanks to some help on the internet I've run the command:
journalctl --since "2022-05-20 12:00:00" --until "2022-05-21 12:00:00"
Revealing the following:
Upon investigation I see a 59GB backup image.
Is this image a full snapshot byte for byte, NOT a ZFS style snapshot (which would be much smaller and only store differences?)
I'm also a little concerned about the nvme lines (2-5) is that a physical fault?
Thanks to some help on the internet I've run the command:
journalctl --since "2022-05-20 12:00:00" --until "2022-05-21 12:00:00"
Revealing the following:
Code:
May 21 00:14:55 proxmox kernel: pcieport 0000:00:1d.0: AER: Corrected error received: 0000:01:00.0
May 21 00:14:55 proxmox kernel: nvme 0000:01:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
May 21 00:14:55 proxmox kernel: nvme 0000:01:00.0: device [1c5c:174a] error status/mask=00000001/00000000
May 21 00:14:55 proxmox kernel: nvme 0000:01:00.0: [ 0] RxErr
May 21 00:15:15 proxmox pmxcfs[1313]: [database] crit: commit transaction failed: database or disk is full#010
May 21 00:15:15 proxmox rsyslogd[1006]: file '/var/log/syslog'[8] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.2102.0 try https://www.rsyslog.com/e/2027 ]
May 21 00:15:15 proxmox pmxcfs[1313]: [database] crit: rollback transaction failed: cannot rollback - no transaction is active#010
May 21 00:15:15 proxmox rsyslogd[1006]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.2102.0 try https://www.rsyslog.com/e/2027 ]
May 21 00:15:15 proxmox pvescheduler[821177]: unable to delete old temp file: Input/output error
May 21 00:15:15 proxmox rsyslogd[1006]: file '/var/log/daemon.log'[9] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.2102.0 try https://www.rsyslog.com/e/2027 ]
May 21 00:15:15 proxmox rsyslogd[1006]: action 'action-2-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.2102.0 try https://www.rsyslog.com/e/2027 ]
May 21 00:15:15 proxmox QEMU[8366]: kvm: terminating on signal 15 from pid 1010 (/usr/sbin/qmeventd)
May 21 00:15:15 proxmox rsyslogd[1006]: file '/var/log/daemon.log'[9] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.2102.0 try https://www.rsyslog.com/e/2027 ]
May 21 00:15:15 proxmox rsyslogd[1006]: action 'action-2-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.2102.0 try https://www.rsyslog.com/e/2027 ]
May 21 00:15:16 proxmox kernel: zd0: p1 p2 p3
May 21 00:15:16 proxmox kernel: vmbr0: port 3(tap100i0) entered disabled state
May 21 00:15:16 proxmox qmeventd[1087523]: Starting cleanup for 100
May 21 00:15:16 proxmox qmeventd[1087523]: Finished cleanup for 100
May 21 00:15:15 proxmox systemd[1]: Stopping LVM event activation on device 230:3...
May 21 00:15:15 proxmox rsyslogd[1006]: file '/var/log/daemon.log'[9] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.2102.0 try https://www.rsyslog.com/e/2027 ]
May 21 00:15:16 proxmox lvm[1086073]: pvscan[1086073] /dev/zd0p3 excluded by filters: device is rejected by filter config.
May 21 00:15:15 proxmox systemd[1]: Requested transaction contradicts existing jobs: Resource deadlock avoided
May 21 00:15:15 proxmox rsyslogd[1006]: action 'action-2-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.2102.0 try https://www.rsyslog.com/e/2027 ]
May 21 00:15:15 proxmox systemd[1]: lvm2-pvscan@230:3.service: Succeeded.
May 21 00:15:15 proxmox rsyslogd[1006]: file '/var/log/daemon.log'[9] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.2102.0 try https://www.rsyslog.com/e/2027 ]
May 21 00:15:15 proxmox systemd[1]: Stopped LVM event activation on device 230:3.
May 21 00:15:15 proxmox rsyslogd[1006]: action 'action-2-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.2102.0 try https://www.rsyslog.com/e/2027 ]
May 21 00:15:16 proxmox systemd[1]: 100.scope: Succeeded.
Upon investigation I see a 59GB backup image.
Is this image a full snapshot byte for byte, NOT a ZFS style snapshot (which would be much smaller and only store differences?)
I'm also a little concerned about the nvme lines (2-5) is that a physical fault?
Last edited: