PVE Server freezes after some hours of runtime; only cold reset possible

U

user332255

Member
Sep 5, 2021
44
5
13
29
Hi,
Recently I setup a new proxmox server. However, the server freezes after some hours of runtime. Fortunately, i can access the server via IPMI, but the console is not responding. SSH connection is also not possible.
Are there any log files, which could help me narrow down this problem? Can you help me find a solution here?

My setup:
Mainboard: Gigabyte MC12-LE0
CPU: AMD Ryzen 9 3950X
RAM: 128GB DDR4 ECC UDIMM
NIC: 10G 2P X520 Adapter
HBA: ASM1166 Serial ATA Controller; 8x SSD (SATA)
SATA: 2x SATA SSD (PVE OS)

Edit:
The HBA is configured with PCIe passthrough to my Truenas VM
Code: 
root@truenas[~]# lsblk -o name,fstype,size,label,model
NAME FSTYPE SIZE LABEL MODEL
sda 32G QEMU HARDDISK
├─sda1 1M
├─sda2 vfat 512M EFI
└─sda3 zfs_member 31.5G boot-pool
sdb 1.7T INTEL SSDSC2KB019T7
├─sdb1 linux_raid_member 2G truenas:swap0
│ └─md127 2G
│ └─md127 swap 2G
└─sdb2 zfs_member 1.7T truenas_scale_pool2
sdc 1.7T MZ7KM1T9HAJM00D3
├─sdc1 linux_raid_member 2G truenas:swap1
│ └─md126 2G
│ └─md126 swap 2G
└─sdc2 zfs_member 1.7T truenas_scale_pool2
sdd 1.7T MTFDDAK1T9TCB
├─sdd1 2G
└─sdd2 zfs_member 1.7T truenas_scale_pool2
sde 1.7T MZ7KM1T9HAJM00D3
├─sde1 linux_raid_member 2G truenas:swap0
│ └─md127 2G
│ └─md127 swap 2G
└─sde2 zfs_member 1.7T truenas_scale_pool2
sdf 1.7T MZ7KM1T9HAJM00D3
├─sdf1 linux_raid_member 2G truenas:swap0
│ └─md127 2G
│ └─md127 swap 2G
└─sdf2 zfs_member 1.7T truenas_scale_pool2
sdg 1.7T MTFDDAK1T9TCB
├─sdg1 linux_raid_member 2G truenas:swap1
│ └─md126 2G
│ └─md126 swap 2G
└─sdg2 zfs_member 1.7T truenas_scale_pool2
sdh 1.7T Micron_5100_MTFDDAK1T9TCB
├─sdh1 linux_raid_member 2G truenas:swap1
│ └─md126 2G
│ └─md126 swap 2G
└─sdh2 zfs_member 1.7T truenas_scale_pool2
sdi 1.7T MTFDDAK1T9TCB
├─sdi1 linux_raid_member 2G truenas:swap1
└─sdi2 zfs_member 1.7T truenas_scale_pool2
Click to expand...
 
Last edited:
Hello,

Which kernel are you running? You can query this via

Code: 
uname -a

Is there anything interesting in the system logs before the crash? You can gather the system logs of, for example, the 19th of june, into a file via the following command:

Code: 
journalctl --since "2024-06-19" --until "2024-06-20" > $(hostname)-syslog.txt

Please modify the dates accordingly. A new boot is represented in the system logs as a line that starts with

Code: 
 -- Boot

Please find the boot after the crash and share the last 200 lines before the boot.

Does the IPMI screen show any errors when the system is frozen?
 
Maximiliano said:
uname -a
Click to expand...
Linux proxmox 6.8.8-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.8-2 (2024-06-24T09:00Z) x86_64 GNU/Linux
Click to expand...

Maximiliano said:
journalctl --since "2024-06-19" --until "2024-06-20" > $(hostname)-syslog.txt
Click to expand...
Sorry, how can I open/display this log file ?:D
Maximiliano said:
Does the IPMI screen show any errors when the system is frozen?
Click to expand...
Not really, sorry but I forget to make a screenshot. When the next freeze occurs, I will make one.
 
The file will be stored as $(hostname)-syslog.txt where the command was executed (it will run the `hostname` command so the file will be named your-hostname-syslog.txt.

You can open it with cat $(hostname)-syslog.txt or less $(hostname)-syslog.txt if you know how to navigate a file with `less`. You can also specify an hour in the `journalctl` to narrow it down even further if the file is too long, e.g. with --since "2024-06-19 15:00".
 
Maximiliano said:
Please find the boot after the crash and share the last 200 lines before the boot.
Click to expand...
1/2
Jul 10 00:00:25 proxmox pveproxy[2851]: received signal HUP
Jul 10 00:00:25 proxmox pveproxy[2851]: server closing
Jul 10 00:00:25 proxmox pveproxy[2851]: server shutdown (restart)
Jul 10 00:00:25 proxmox systemd[1]: Reloaded pveproxy.service - PVE API Proxy Server.
Jul 10 00:00:25 proxmox systemd[1]: Reloading spiceproxy.service - PVE SPICE Proxy Server...
Jul 10 00:00:25 proxmox spiceproxy[93602]: send HUP to 2857
Jul 10 00:00:25 proxmox spiceproxy[2857]: received signal HUP
Jul 10 00:00:25 proxmox spiceproxy[2857]: server closing
Jul 10 00:00:25 proxmox spiceproxy[2857]: server shutdown (restart)
Jul 10 00:00:25 proxmox systemd[1]: Reloaded spiceproxy.service - PVE SPICE Proxy Server.
Jul 10 00:00:25 proxmox pvefw-logger[2416]: received terminate request (signal)
Jul 10 00:00:25 proxmox pvefw-logger[2416]: stopping pvefw logger
Jul 10 00:00:25 proxmox systemd[1]: Stopping pvefw-logger.service - Proxmox VE firewall logger...
Jul 10 00:00:26 proxmox spiceproxy[2857]: restarting server
Jul 10 00:00:26 proxmox spiceproxy[2857]: starting 1 worker(s)
Jul 10 00:00:26 proxmox spiceproxy[2857]: worker 93610 started
Jul 10 00:00:26 proxmox systemd[1]: pvefw-logger.service: Deactivated successfully.
Jul 10 00:00:26 proxmox systemd[1]: Stopped pvefw-logger.service - Proxmox VE firewall logger.
Jul 10 00:00:26 proxmox systemd[1]: pvefw-logger.service: Consumed 1.354s CPU time.
Jul 10 00:00:26 proxmox systemd[1]: Starting pvefw-logger.service - Proxmox VE firewall logger...
Jul 10 00:00:26 proxmox pvefw-logger[93613]: starting pvefw logger
Jul 10 00:00:26 proxmox systemd[1]: Started pvefw-logger.service - Proxmox VE firewall logger.
Jul 10 00:00:26 proxmox systemd[1]: logrotate.service: Deactivated successfully.
Jul 10 00:00:26 proxmox systemd[1]: Finished logrotate.service - Rotate log files.
Jul 10 00:00:26 proxmox pveproxy[2851]: restarting server
Jul 10 00:00:26 proxmox pveproxy[2851]: starting 3 worker(s)
Jul 10 00:00:26 proxmox pveproxy[2851]: worker 93617 started
Jul 10 00:00:26 proxmox pveproxy[2851]: worker 93618 started
Jul 10 00:00:26 proxmox pveproxy[2851]: worker 93619 started
Jul 10 00:00:31 proxmox spiceproxy[2858]: worker exit
Jul 10 00:00:31 proxmox spiceproxy[2857]: worker 2858 finished
Jul 10 00:00:31 proxmox pveproxy[2854]: worker exit
Jul 10 00:00:31 proxmox pveproxy[2852]: worker exit
Jul 10 00:00:31 proxmox pveproxy[2853]: worker exit
Jul 10 00:00:31 proxmox pveproxy[2851]: worker 2853 finished
Jul 10 00:00:31 proxmox pveproxy[2851]: worker 2854 finished
Jul 10 00:00:31 proxmox pveproxy[2851]: worker 2852 finished
Jul 10 00:02:24 proxmox systemd[1]: Starting apt-daily.service - Daily apt download activities...
Jul 10 00:02:25 proxmox systemd[1]: apt-daily.service: Deactivated successfully.
Jul 10 00:02:25 proxmox systemd[1]: Finished apt-daily.service - Daily apt download activities.
Jul 10 00:08:32 proxmox postfix/qmgr[2792]: 4D476106C4: from=<root@proxmox.main>, size=739, nrcpt=1 (queue active)
Jul 10 00:08:32 proxmox postfix/qmgr[2792]: 413AB10FB4: from=<root@proxmox.main>, size=826, nrcpt=1 (queue active)
Jul 10 00:08:32 proxmox postfix/qmgr[2792]: 4D6E5106C5: from=<root@proxmox.main>, size=739, nrcpt=1 (queue active)
Jul 10 00:08:32 proxmox postfix/qmgr[2792]: 587C610D15: from=<root@proxmox.main>, size=826, nrcpt=1 (queue active)
Jul 10 00:08:32 proxmox postfix/smtp[96041]: 587C610D15: host mx03.posteo.de[185.67.36.70] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 00:08:32 proxmox postfix/smtp[96038]: 4D476106C4: host mx01.posteo.de[185.67.36.62] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 00:08:32 proxmox postfix/smtp[96039]: 413AB10FB4: host mx03.posteo.de[185.67.36.70] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 00:08:32 proxmox postfix/smtp[96040]: 4D6E5106C5: host mx04.posteo.de[185.67.36.71] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 00:08:34 proxmox postfix/smtp[96038]: 4D476106C4: to=<DUMMYADDRESS@posteo.de>, relay=mx04.posteo.de[185.67.36.64]:25, delay=225513, delays=225511/0.01/2.1/0.04, dsn=4.1.8, status=deferred (host mx04.posteo.de[185.67.36.64] sa
id: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 00:08:34 proxmox postfix/smtp[96039]: 413AB10FB4: to=<DUMMYADDRESS@posteo.de>, relay=mx01.posteo.de[185.67.36.61]:25, delay=302913, delays=302911/0.01/2.1/0.2, dsn=4.1.8, status=deferred (host mx01.posteo.de[185.67.36.61] sai
d: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 00:08:34 proxmox postfix/smtp[96041]: 587C610D15: to=<DUMMYADDRESS@posteo.de>, relay=mx04.posteo.de[185.67.36.71]:25, delay=216513, delays=216511/0.02/2.3/0.04, dsn=4.1.8, status=deferred (host mx04.posteo.de[185.67.36.71] sa
id: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 00:08:34 proxmox postfix/smtp[96040]: 4D6E5106C5: to=<DUMMYADDRESS@posteo.de>, relay=mx03.posteo.de[185.67.36.70]:25, delay=225513, delays=225511/0.02/2.3/0.02, dsn=4.1.8, status=deferred (host mx03.posteo.de[185.67.36.70] sa
id: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 00:17:01 proxmox CRON[98622]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jul 10 00:17:01 proxmox CRON[98623]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jul 10 00:17:01 proxmox CRON[98622]: pam_unix(cron:session): session closed for user root
Jul 10 00:24:01 proxmox CRON[100707]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jul 10 00:24:01 proxmox CRON[100708]: (root) CMD (if [ $(date +%w) -eq 0 ] && [ -x /usr/lib/zfs-linux/scrub ]; then /usr/lib/zfs-linux/scrub; fi)
Jul 10 00:24:01 proxmox CRON[100707]: pam_unix(cron:session): session closed for user root
Jul 10 01:17:01 proxmox CRON[116387]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jul 10 01:17:01 proxmox CRON[116389]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jul 10 01:17:01 proxmox CRON[116387]: pam_unix(cron:session): session closed for user root
Jul 10 01:18:34 proxmox postfix/qmgr[2792]: 4D476106C4: from=<root@proxmox.main>, size=739, nrcpt=1 (queue active)
Jul 10 01:18:34 proxmox postfix/qmgr[2792]: 413AB10FB4: from=<root@proxmox.main>, size=826, nrcpt=1 (queue active)
Jul 10 01:18:34 proxmox postfix/qmgr[2792]: 4D6E5106C5: from=<root@proxmox.main>, size=739, nrcpt=1 (queue active)
Jul 10 01:18:34 proxmox postfix/qmgr[2792]: 587C610D15: from=<root@proxmox.main>, size=826, nrcpt=1 (queue active)
Jul 10 01:18:35 proxmox postfix/smtp[116838]: 4D6E5106C5: host mx01.posteo.de[185.67.36.62] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 01:18:35 proxmox postfix/smtp[116837]: 413AB10FB4: host mx04.posteo.de[185.67.36.71] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 01:18:35 proxmox postfix/smtp[116839]: 587C610D15: host mx01.posteo.de[185.67.36.62] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 01:18:35 proxmox postfix/smtp[116836]: 4D476106C4: host mx04.posteo.de[185.67.36.71] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 01:18:36 proxmox postfix/smtp[116838]: 4D6E5106C5: to=<DUMMYADDRESS@posteo.de>, relay=mx03.posteo.de[185.67.36.63]:25, delay=229716, delays=229713/0.02/2.1/0.03, dsn=4.1.8, status=deferred (host mx03.posteo.de[185.67.36.63] s
aid: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 01:18:36 proxmox postfix/smtp[116837]: 413AB10FB4: to=<DUMMYADDRESS@posteo.de>, relay=mx01.posteo.de[185.67.36.62]:25, delay=307116, delays=307113/0.01/2.1/0.03, dsn=4.1.8, status=deferred (host mx01.posteo.de[185.67.36.62] s
aid: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 01:18:37 proxmox postfix/smtp[116836]: 4D476106C4: to=<DUMMYADDRESS@posteo.de>, relay=mx01.posteo.de[185.67.36.62]:25, delay=229716, delays=229713/0.01/2.3/0.03, dsn=4.1.8, status=deferred (host mx01.posteo.de[185.67.36.62] s
aid: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 01:18:37 proxmox postfix/smtp[116839]: 587C610D15: to=<DUMMYADDRESS@posteo.de>, relay=mx03.posteo.de[185.67.36.63]:25, delay=220716, delays=220713/0.02/2.3/0.03, dsn=4.1.8, status=deferred (host mx03.posteo.de[185.67.36.63] s
aid: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 02:00:08 proxmox audit[128963]: AVC apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-102_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=128963 comm="(ogrotate)" srcname
="/" flags="rw, rbind"
Jul 10 02:00:08 proxmox kernel: kauditd_printk_skb: 16 callbacks suppressed
Jul 10 02:00:08 proxmox kernel: audit: type=1400 audit(1720569608.083:54): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-102_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=
128963 comm="(ogrotate)" srcname="/" flags="rw, rbind"
Jul 10 02:17:01 proxmox CRON[134033]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jul 10 02:17:01 proxmox CRON[134034]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jul 10 02:17:01 proxmox CRON[134033]: pam_unix(cron:session): session closed for user root
Jul 10 02:21:24 proxmox systemd[1]: Starting man-db.service - Daily man-db regeneration...
Jul 10 02:21:25 proxmox systemd[1]: man-db.service: Deactivated successfully.
Jul 10 02:21:25 proxmox systemd[1]: Finished man-db.service - Daily man-db regeneration.
Jul 10 02:28:34 proxmox postfix/qmgr[2792]: 4D476106C4: from=<root@proxmox.main>, size=739, nrcpt=1 (queue active)
Jul 10 02:28:34 proxmox postfix/qmgr[2792]: 413AB10FB4: from=<root@proxmox.main>, size=826, nrcpt=1 (queue active)
Jul 10 02:28:34 proxmox postfix/qmgr[2792]: 4D6E5106C5: from=<root@proxmox.main>, size=739, nrcpt=1 (queue active)
Jul 10 02:28:34 proxmox postfix/qmgr[2792]: 587C610D15: from=<root@proxmox.main>, size=826, nrcpt=1 (queue active)
Jul 10 02:28:35 proxmox postfix/smtp[137402]: 587C610D15: host mx03.posteo.de[185.67.36.70] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 02:28:35 proxmox postfix/smtp[137400]: 413AB10FB4: host mx04.posteo.de[185.67.36.64] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 02:28:35 proxmox postfix/smtp[137401]: 4D6E5106C5: host mx04.posteo.de[185.67.36.71] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 02:28:35 proxmox postfix/smtp[137399]: 4D476106C4: host mx01.posteo.de[185.67.36.61] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Click to expand...
 
2/2
Jul 10 02:28:36 proxmox postfix/smtp[137400]: 413AB10FB4: to=<DUMMYADDRESS@posteo.de>, relay=mx04.posteo.de[185.67.36.71]:25, delay=311316, delays=311313/0.01/2.2/0.04, dsn=4.1.8, status=deferred (host mx04.posteo.de[185.67.36.71] s
aid: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 02:28:36 proxmox postfix/smtp[137402]: 587C610D15: to=<DUMMYADDRESS@posteo.de>, relay=mx01.posteo.de[185.67.36.61]:25, delay=224916, delays=224913/0.02/2.2/0.03, dsn=4.1.8, status=deferred (host mx01.posteo.de[185.67.36.61] s
aid: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 02:28:36 proxmox postfix/smtp[137399]: 4D476106C4: to=<DUMMYADDRESS@posteo.de>, relay=mx04.posteo.de[185.67.36.64]:25, delay=233916, delays=233913/0.01/2.2/0.03, dsn=4.1.8, status=deferred (host mx04.posteo.de[185.67.36.64] s
aid: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 02:28:37 proxmox postfix/smtp[137401]: 4D6E5106C5: to=<DUMMYADDRESS@posteo.de>, relay=mx01.posteo.de[185.67.36.62]:25, delay=233916, delays=233913/0.02/2.5/0.04, dsn=4.1.8, status=deferred (host mx01.posteo.de[185.67.36.62] s
aid: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 02:30:01 proxmox CRON[137809]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jul 10 02:30:01 proxmox CRON[137810]: (root) CMD (sudo shutdown -h 0)
Jul 10 02:30:01 proxmox CRON[137809]: pam_unix(cron:session): session closed for user root
Jul 10 02:30:01 proxmox postfix/pickup[131579]: 9938D10743: uid=0 from=<root>
Jul 10 02:30:01 proxmox postfix/cleanup[137813]: 9938D10743: message-id=<20240710003001.9938D10743@proxmox.main>
Jul 10 02:30:01 proxmox postfix/qmgr[2792]: 9938D10743: from=<root@proxmox.main>, size=560, nrcpt=1 (queue active)
Jul 10 02:30:01 proxmox postfix/pickup[131579]: A198310C35: uid=65534 from=<root>
Jul 10 02:30:01 proxmox postfix/cleanup[137813]: A198310C35: message-id=<20240710003001.9938D10743@proxmox.main>
Jul 10 02:30:01 proxmox proxmox-mail-fo[137815]: proxmox proxmox-mail-forward[137815]: notified via target `mail-to-root`
Jul 10 02:30:01 proxmox postfix/qmgr[2792]: A198310C35: from=<root@proxmox.main>, size=726, nrcpt=1 (queue active)
Jul 10 02:30:01 proxmox postfix/local[137814]: 9938D10743: to=<root@proxmox.main>, orig_to=<root>, relay=local, delay=0.04, delays=0.01/0.01/0/0.02, dsn=2.0.0, status=sent (delivered to command: /usr/bin/proxmox-mail-forward)
Jul 10 02:30:01 proxmox postfix/qmgr[2792]: 9938D10743: removed
Jul 10 02:30:01 proxmox postfix/smtp[137400]: A198310C35: host mx03.posteo.de[185.67.36.63] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 02:30:03 proxmox postfix/smtp[137400]: A198310C35: to=<DUMMYADDRESS@posteo.de>, relay=mx01.posteo.de[185.67.36.61]:25, delay=1.9, delays=0/0/1.8/0.03, dsn=4.1.8, status=deferred (host mx01.posteo.de[185.67.36.61] said: 450 4.
1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 02:32:54 proxmox audit[138702]: AVC apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-102_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=138702 comm="(install)" srcname=
"/" flags="rw, rbind"
Jul 10 02:32:54 proxmox kernel: audit: type=1400 audit(1720571574.371:55): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-102_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=
138702 comm="(install)" srcname="/" flags="rw, rbind"
Jul 10 02:32:54 proxmox audit[138704]: AVC apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-102_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=138704 comm="(find)" srcname="/"
flags="rw, rbind"
Jul 10 02:32:54 proxmox kernel: audit: type=1400 audit(1720571574.378:56): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-102_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=
138704 comm="(find)" srcname="/" flags="rw, rbind"
Jul 10 02:32:54 proxmox kernel: audit: type=1400 audit(1720571574.393:57): apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-102_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=
138706 comm="(mandb)" srcname="/" flags="rw, rbind"
Jul 10 02:32:54 proxmox audit[138706]: AVC apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-102_</var/lib/lxc>" name="/run/systemd/unit-root/" pid=138706 comm="(mandb)" srcname="/
" flags="rw, rbind"
Jul 10 02:38:34 proxmox postfix/qmgr[2792]: A198310C35: from=<root@proxmox.main>, size=726, nrcpt=1 (queue active)
Jul 10 02:38:34 proxmox postfix/smtp[140332]: A198310C35: host mx04.posteo.de[185.67.36.64] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 02:38:35 proxmox postfix/smtp[140332]: A198310C35: to=<DUMMYADDRESS@posteo.de>, relay=mx01.posteo.de[185.67.36.61]:25, delay=514, delays=512/0.01/1.6/0.03, dsn=4.1.8, status=deferred (host mx01.posteo.de[185.67.36.61] said: 4
50 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 02:48:34 proxmox postfix/qmgr[2792]: A198310C35: from=<root@proxmox.main>, size=726, nrcpt=1 (queue active)
Jul 10 02:48:34 proxmox postfix/smtp[143296]: A198310C35: host mx01.posteo.de[185.67.36.61] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 02:48:36 proxmox postfix/smtp[143296]: A198310C35: to=<DUMMYADDRESS@posteo.de>, relay=mx04.posteo.de[185.67.36.64]:25, delay=1114, delays=1113/0.01/1.7/0.18, dsn=4.1.8, status=deferred (host mx04.posteo.de[185.67.36.64] said:
450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 03:08:34 proxmox postfix/qmgr[2792]: A198310C35: from=<root@proxmox.main>, size=726, nrcpt=1 (queue active)
Jul 10 03:08:34 proxmox postfix/smtp[149058]: A198310C35: host mx03.posteo.de[185.67.36.63] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 03:08:36 proxmox postfix/smtp[149058]: A198310C35: to=<DUMMYADDRESS@posteo.de>, relay=mx04.posteo.de[185.67.36.64]:25, delay=2315, delays=2312/0.01/2.4/0.02, dsn=4.1.8, status=deferred (host mx04.posteo.de[185.67.36.64] said:
450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 03:10:01 proxmox CRON[149517]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jul 10 03:10:01 proxmox CRON[149518]: (root) CMD (test -e /run/systemd/system || SERVICE_MODE=1 /sbin/e2scrub_all -A -r)
Jul 10 03:10:01 proxmox CRON[149517]: pam_unix(cron:session): session closed for user root
Jul 10 03:17:01 proxmox CRON[151577]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jul 10 03:17:01 proxmox CRON[151578]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jul 10 03:17:01 proxmox CRON[151577]: pam_unix(cron:session): session closed for user root
Jul 10 03:38:34 proxmox postfix/qmgr[2792]: 4D476106C4: from=<root@proxmox.main>, size=739, nrcpt=1 (queue active)
Jul 10 03:38:34 proxmox postfix/qmgr[2792]: 413AB10FB4: from=<root@proxmox.main>, size=826, nrcpt=1 (queue active)
Jul 10 03:38:34 proxmox postfix/qmgr[2792]: 4D6E5106C5: from=<root@proxmox.main>, size=739, nrcpt=1 (queue active)
Jul 10 03:38:34 proxmox postfix/qmgr[2792]: 587C610D15: from=<root@proxmox.main>, size=826, nrcpt=1 (queue active)
Jul 10 03:38:35 proxmox postfix/smtp[157840]: 4D476106C4: host mx01.posteo.de[185.67.36.61] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 03:38:35 proxmox postfix/smtp[157841]: 413AB10FB4: host mx01.posteo.de[185.67.36.62] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 03:38:35 proxmox postfix/smtp[157843]: 587C610D15: host mx04.posteo.de[185.67.36.71] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 03:38:35 proxmox postfix/smtp[157842]: 4D6E5106C5: host mx03.posteo.de[185.67.36.63] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 03:38:36 proxmox postfix/smtp[157841]: 413AB10FB4: to=<DUMMYADDRESS@posteo.de>, relay=mx03.posteo.de[185.67.36.63]:25, delay=315515, delays=315513/0.01/2/0.03, dsn=4.1.8, status=deferred (host mx03.posteo.de[185.67.36.63] sai
d: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 03:38:36 proxmox postfix/smtp[157842]: 4D6E5106C5: to=<DUMMYADDRESS@posteo.de>, relay=mx01.posteo.de[185.67.36.62]:25, delay=238116, delays=238113/0.02/2.3/0.03, dsn=4.1.8, status=deferred (host mx01.posteo.de[185.67.36.62] s
aid: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 03:38:37 proxmox postfix/smtp[157843]: 587C610D15: to=<DUMMYADDRESS@posteo.de>, relay=mx03.posteo.de[185.67.36.70]:25, delay=229116, delays=229113/0.02/2.3/0.05, dsn=4.1.8, status=deferred (host mx03.posteo.de[185.67.36.70] s
aid: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 03:38:37 proxmox postfix/smtp[157840]: 4D476106C4: to=<DUMMYADDRESS@posteo.de>, relay=mx03.posteo.de[185.67.36.70]:25, delay=238116, delays=238113/0.01/2.2/0.33, dsn=4.1.8, status=deferred (host mx03.posteo.de[185.67.36.70] s
aid: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 03:48:34 proxmox postfix/qmgr[2792]: A198310C35: from=<root@proxmox.main>, size=726, nrcpt=1 (queue active)
Jul 10 03:48:34 proxmox postfix/smtp[160867]: A198310C35: host mx03.posteo.de[185.67.36.63] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 03:48:36 proxmox postfix/smtp[160867]: A198310C35: to=<DUMMYADDRESS@posteo.de>, relay=mx04.posteo.de[185.67.36.71]:25, delay=4714, delays=4713/0.01/1.6/0.03, dsn=4.1.8, status=deferred (host mx04.posteo.de[185.67.36.71] said:
450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 04:17:01 proxmox CRON[169965]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jul 10 04:17:01 proxmox CRON[169966]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jul 10 04:17:01 proxmox CRON[169965]: pam_unix(cron:session): session closed for user root
Jul 10 04:28:35 proxmox smartd[2499]: Device: /dev/sdj [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 60 to 59
Jul 10 04:48:34 proxmox postfix/qmgr[2792]: 4D476106C4: from=<root@proxmox.main>, size=739, nrcpt=1 (queue active)
Jul 10 04:48:34 proxmox postfix/qmgr[2792]: 413AB10FB4: from=<root@proxmox.main>, size=826, nrcpt=1 (queue active)
Jul 10 04:48:34 proxmox postfix/qmgr[2792]: 4D6E5106C5: from=<root@proxmox.main>, size=739, nrcpt=1 (queue active)
Jul 10 04:48:34 proxmox postfix/qmgr[2792]: 587C610D15: from=<root@proxmox.main>, size=826, nrcpt=1 (queue active)
Jul 10 04:48:35 proxmox postfix/smtp[179820]: 4D476106C4: host mx04.posteo.de[185.67.36.71] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 04:48:35 proxmox postfix/smtp[179821]: 413AB10FB4: host mx03.posteo.de[185.67.36.70] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 04:48:35 proxmox postfix/smtp[179823]: 587C610D15: host mx01.posteo.de[185.67.36.62] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 04:48:35 proxmox postfix/smtp[179822]: 4D6E5106C5: host mx04.posteo.de[185.67.36.64] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 04:48:37 proxmox postfix/smtp[179821]: 413AB10FB4: to=<DUMMYADDRESS@posteo.de>, relay=mx04.posteo.de[185.67.36.71]:25, delay=319716, delays=319714/0.01/2.1/0.03, dsn=4.1.8, status=deferred (host mx04.posteo.de[185.67.36.71] s
aid: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 04:48:37 proxmox postfix/smtp[179820]: 4D476106C4: to=<DUMMYADDRESS@posteo.de>, relay=mx03.posteo.de[185.67.36.63]:25, delay=242316, delays=242313/0.01/2.2/0.03, dsn=4.1.8, status=deferred (host mx03.posteo.de[185.67.36.63] s
aid: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 04:48:37 proxmox postfix/smtp[179822]: 4D6E5106C5: to=<DUMMYADDRESS@posteo.de>, relay=mx04.posteo.de[185.67.36.71]:25, delay=242316, delays=242314/0.02/2.4/0.03, dsn=4.1.8, status=deferred (host mx04.posteo.de[185.67.36.71] s
aid: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 04:48:37 proxmox postfix/smtp[179823]: 587C610D15: to=<DUMMYADDRESS@posteo.de>, relay=mx03.posteo.de[185.67.36.63]:25, delay=233316, delays=233313/0.02/2.5/0.03, dsn=4.1.8, status=deferred (host mx03.posteo.de[185.67.36.63] s
aid: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 04:58:30 proxmox smartd[2499]: Device: /dev/sdi [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 64 to 63
Jul 10 04:58:30 proxmox smartd[2499]: Device: /dev/sdi [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 64 to 63
Jul 10 04:58:34 proxmox postfix/qmgr[2792]: A198310C35: from=<root@proxmox.main>, size=726, nrcpt=1 (queue active)
Jul 10 04:58:35 proxmox postfix/smtp[182865]: A198310C35: host mx04.posteo.de[185.67.36.64] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 04:58:35 proxmox smartd[2499]: Device: /dev/sdj [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 59 to 58
Jul 10 04:58:36 proxmox postfix/smtp[182865]: A198310C35: to=<DUMMYADDRESS@posteo.de>, relay=mx03.posteo.de[185.67.36.70]:25, delay=8915, delays=8913/0.01/1.9/0.04, dsn=4.1.8, status=deferred (host mx03.posteo.de[185.67.36.70] said:
450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
-- Boot 66d748c3a03446c1a89e577c5418fca9 --
Click to expand...
 
There is nothing there that would explain the problem, but it does not look like a clean shutdown so it is probably the correct log of a crash.

Note that as per the following entries:

Code: 
Jul 10 04:48:34 proxmox postfix/qmgr[2792]: 4D476106C4: from=<root@proxmox.main>, size=739, nrcpt=1 (queue active)
Jul 10 04:48:34 proxmox postfix/qmgr[2792]: 413AB10FB4: from=<root@proxmox.main>, size=826, nrcpt=1 (queue active)
Jul 10 04:48:34 proxmox postfix/qmgr[2792]: 4D6E5106C5: from=<root@proxmox.main>, size=739, nrcpt=1 (queue active)
Jul 10 04:48:34 proxmox postfix/qmgr[2792]: 587C610D15: from=<root@proxmox.main>, size=826, nrcpt=1 (queue active)
Jul 10 04:48:35 proxmox postfix/smtp[179820]: 4D476106C4: host mx04.posteo.de[185.67.36.71] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 04:48:35 proxmox postfix/smtp[179821]: 413AB10FB4: host mx03.posteo.de[185.67.36.70] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 04:48:35 proxmox postfix/smtp[179823]: 587C610D15: host mx01.posteo.de[185.67.36.62] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 04:48:35 proxmox postfix/smtp[179822]: 4D6E5106C5: host mx04.posteo.de[185.67.36.64] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 04:48:37 proxmox postfix/smtp[179821]: 413AB10FB4: to=<DUMMYADDRESS@posteo.de>, relay=mx04.posteo.de[185.67.36.71]:25, delay=319716, delays=319714/0.01/2.1/0.03, dsn=4.1.8, status=deferred (host mx04.posteo.de[185.67.36.71] s
aid: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 04:48:37 proxmox postfix/smtp[179820]: 4D476106C4: to=<DUMMYADDRESS@posteo.de>, relay=mx03.posteo.de[185.67.36.63]:25, delay=242316, delays=242313/0.01/2.2/0.03, dsn=4.1.8, status=deferred (host mx03.posteo.de[185.67.36.63] s
aid: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 04:48:37 proxmox postfix/smtp[179822]: 4D6E5106C5: to=<DUMMYADDRESS@posteo.de>, relay=mx04.posteo.de[185.67.36.71]:25, delay=242316, delays=242314/0.02/2.4/0.03, dsn=4.1.8, status=deferred (host mx04.posteo.de[185.67.36.71] s
aid: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 04:48:37 proxmox postfix/smtp[179823]: 587C610D15: to=<DUMMYADDRESS@posteo.de>, relay=mx03.posteo.de[185.67.36.63]:25, delay=233316, delays=233313/0.02/2.5/0.03, dsn=4.1.8, status=deferred (host mx03.posteo.de[185.67.36.63] s
aid: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command))
Jul 10 04:58:34 proxmox postfix/qmgr[2792]: A198310C35: from=<root@proxmox.main>, size=726, nrcpt=1 (queue active)
Jul 10 04:58:35 proxmox postfix/smtp[182865]: A198310C35: host mx04.posteo.de[185.67.36.64] said: 450 4.1.8 <root@proxmox.main>: Sender address rejected: Domain not found (in reply to MAIL FROM command)
Jul 10 04:58:36 proxmox postfix/smtp[182865]: A198310C35: to=<DUMMYADDRESS@posteo.de>, relay=mx03.posteo.de[185.67.36.70]:25, delay=8915, delays=8913/0.01/1.9/0.04, dsn=4.1.8, status=deferred (host mx03.posteo.de[185.67.36.70] said:

your notifications are not set correctly. Please ensure that your user has a correct email set. SMTP targets might be of interest here [1].

[1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#notification_targets_smtp
 
Could you please go further up to the `-- Boot` line that corresponds to the boot which crashed? Could you please double check that indeed the version of the kernel that was running when the crash happened was the same that you have right now? The 6.8.8 version landed very recently and it is not out of the question the crash was actually on 6.8.4.
 
Sorry for beeing away.
Meanwhile I was able to make a screenshot from CLI during freeze (the console was unresponsive)
 

Attachments

  • Screenshot 2024-07-21 183316.png
    Screenshot 2024-07-21 183316.png
    586.2 KB · Views: 7
Maximiliano said:
Could you please go further up to the `-- Boot` line
Click to expand...
Before crash (crash was around 21.07 18:23):
Code: 
root@proxmox:~# journalctl --since "2024-07-21 18:00" --until "2024-07-21 18:44" > $(hostname)-syslog.txt
Code: 
root@proxmox:~# cat $(hostname)-syslog.txt
Jul 21 18:17:01 proxmox CRON[100485]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jul 21 18:17:01 proxmox CRON[100486]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Jul 21 18:17:01 proxmox CRON[100485]: pam_unix(cron:session): session closed for user root
Click to expand...

After reboot, which was around 18:45, I was not able to fully get a log file, because excatly at 18:45:03 my log history was flooded with log infos, hence I could not display them all completly (the boot sequence was not shown). Is there another way to get all log entries completely?
 
_gabriel said:
SSD Model ?
Do you use ZFS ?
Click to expand...
The HBA is configured with PCIe passthrough to my Truenas VM
Code: 
root@truenas[~]# lsblk -o name,fstype,size,label,model
NAME FSTYPE SIZE LABEL MODEL
sda 32G QEMU HARDDISK
├─sda1 1M
├─sda2 vfat 512M EFI
└─sda3 zfs_member 31.5G boot-pool
sdb 1.7T INTEL SSDSC2KB019T7
├─sdb1 linux_raid_member 2G truenas:swap0
│ └─md127 2G
│ └─md127 swap 2G
└─sdb2 zfs_member 1.7T truenas_scale_pool2
sdc 1.7T MZ7KM1T9HAJM00D3
├─sdc1 linux_raid_member 2G truenas:swap1
│ └─md126 2G
│ └─md126 swap 2G
└─sdc2 zfs_member 1.7T truenas_scale_pool2
sdd 1.7T MTFDDAK1T9TCB
├─sdd1 2G
└─sdd2 zfs_member 1.7T truenas_scale_pool2
sde 1.7T MZ7KM1T9HAJM00D3
├─sde1 linux_raid_member 2G truenas:swap0
│ └─md127 2G
│ └─md127 swap 2G
└─sde2 zfs_member 1.7T truenas_scale_pool2
sdf 1.7T MZ7KM1T9HAJM00D3
├─sdf1 linux_raid_member 2G truenas:swap0
│ └─md127 2G
│ └─md127 swap 2G
└─sdf2 zfs_member 1.7T truenas_scale_pool2
sdg 1.7T MTFDDAK1T9TCB
├─sdg1 linux_raid_member 2G truenas:swap1
│ └─md126 2G
│ └─md126 swap 2G
└─sdg2 zfs_member 1.7T truenas_scale_pool2
sdh 1.7T Micron_5100_MTFDDAK1T9TCB
├─sdh1 linux_raid_member 2G truenas:swap1
│ └─md126 2G
│ └─md126 swap 2G
└─sdh2 zfs_member 1.7T truenas_scale_pool2
sdi 1.7T MTFDDAK1T9TCB
├─sdi1 linux_raid_member 2G truenas:swap1
└─sdi2 zfs_member 1.7T truenas_scale_pool2
Click to expand...
 
Last edited:
Meanwhile I got another Freeze:
2024-08-01 17_07_41-Remote KVM [192.168.10.119] - [1024 x 768 ] - Geschäftlich – Microsoft​ Edge.png
If someone can put me in the right direction, I would be very thankful.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back