Error 500: unable to open file '/var/tmp/pve-restricted-ports.tmp.1090' - Read-only file system
- Thread starter dsandhu
- Start date
Hi @dsandhu
Do you have space on the root filesystem? Did you experience a power outage and it became corrupted?
Can you "touch file.test" in /var/tmp? Is there anything in the log: journalctl -n 100?
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Do you have space on the root filesystem? Did you experience a power outage and it became corrupted?
Can you "touch file.test" in /var/tmp? Is there anything in the log: journalctl -n 100?
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Last edited:
yes I do have the space.
No I cannot create file in /var/tmp, it says ready only file system.
I don't recall any outage.
Last edited:
Code:
root@proxmox:~# journalctl -n 100
May 10 12:29:23 proxmox kernel: ata1.00: configured for UDMA/133
May 10 12:29:23 proxmox kernel: sd 0:0:0:0: [sda] tag#29 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=5s
May 10 12:29:23 proxmox kernel: sd 0:0:0:0: [sda] tag#29 Sense Key : Medium Error [current]
May 10 12:29:23 proxmox kernel: sd 0:0:0:0: [sda] tag#29 Add. Sense: Unrecovered read error - auto reallocate failed
May 10 12:29:23 proxmox kernel: sd 0:0:0:0: [sda] tag#29 CDB: Read(10) 28 00 14 9d f6 00 00 00 80 00
May 10 12:29:23 proxmox kernel: I/O error, dev sda, sector 345896448 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
May 10 12:29:23 proxmox kernel: ata1: EH complete
May 10 12:29:24 proxmox pvestatd[1062]: can't lock file '/var/log/pve/tasks/.active.lock' - can't open file - Read-only file system
May 10 12:29:24 proxmox smartd[734]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 33 to 38
May 10 12:29:24 proxmox kernel: Buffer I/O error on dev dm-18, logical block 241777, lost async page write
May 10 12:29:25 proxmox kernel: Buffer I/O error on dev dm-18, logical block 241777, lost async page write
May 10 12:29:26 proxmox kernel: Buffer I/O error on dev dm-18, logical block 241777, lost async page write
May 10 12:29:27 proxmox kernel: Buffer I/O error on dev dm-18, logical block 241777, lost async page write
May 10 12:29:28 proxmox kernel: Buffer I/O error on dev dm-18, logical block 241777, lost async page write
May 10 12:29:34 proxmox pvestatd[1062]: can't lock file '/var/log/pve/tasks/.active.lock' - can't open file - Read-only file system
May 10 12:29:41 proxmox kernel: Buffer I/O error on dev dm-18, logical block 241777, lost async page write
May 10 12:29:42 proxmox kernel: Buffer I/O error on dev dm-18, logical block 241777, lost async page write
May 10 12:29:43 proxmox kernel: Buffer I/O error on dev dm-18, logical block 241777, lost async page write
May 10 12:29:44 proxmox pvestatd[1062]: can't lock file '/var/log/pve/tasks/.active.lock' - can't open file - Read-only file system
May 10 12:29:44 proxmox kernel: Buffer I/O error on dev dm-18, logical block 241777, lost async page write
May 10 12:29:45 proxmox kernel: Buffer I/O error on dev dm-18, logical block 241777, lost async page write
May 10 12:29:50 proxmox kernel: ata1.00: exception Emask 0x0 SAct 0x600001 SErr 0x0 action 0x0
May 10 12:29:50 proxmox kernel: ata1.00: irq_stat 0x40000008
May 10 12:29:50 proxmox kernel: ata1.00: failed command: READ FPDMA QUEUED
May 10 12:29:50 proxmox kernel: ata1.00: cmd 60/80:a8:00:f5:9d/00:00:14:00:00/40 tag 21 ncq dma 65536 in
res 41/40:a8:58:f5:9d/00:00:14:00:00/40 Emask 0x409 (media error) <F>
May 10 12:29:50 proxmox kernel: ata1.00: status: { DRDY ERR }
May 10 12:29:50 proxmox kernel: ata1.00: error: { UNC }
May 10 12:29:50 proxmox kernel: ata1.00: configured for UDMA/133
May 10 12:29:50 proxmox kernel: sd 0:0:0:0: [sda] tag#21 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=2s
May 10 12:29:50 proxmox kernel: sd 0:0:0:0: [sda] tag#21 Sense Key : Medium Error [current]
May 10 12:29:50 proxmox kernel: sd 0:0:0:0: [sda] tag#21 Add. Sense: Unrecovered read error - auto reallocate failed
May 10 12:29:50 proxmox kernel: sd 0:0:0:0: [sda] tag#21 CDB: Read(10) 28 00 14 9d f5 00 00 00 80 00
May 10 12:29:50 proxmox kernel: I/O error, dev sda, sector 345896280 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
May 10 12:29:50 proxmox kernel: ata1: EH complete
May 10 12:29:54 proxmox pvestatd[1062]: can't lock file '/var/log/pve/tasks/.active.lock' - can't open file - Read-only file system
May 10 12:29:58 proxmox kernel: Buffer I/O error on dev dm-18, logical block 241777, lost async page write
May 10 12:29:59 proxmox kernel: Buffer I/O error on dev dm-18, logical block 241777, lost async page write
May 10 12:30:00 proxmox kernel: Buffer I/O error on dev dm-18, logical block 241777, lost async page write
May 10 12:30:00 proxmox pvescheduler[9278]: replication: can't lock file '/var/lib/pve-manager/pve-replication-state.lck' - can't open file - Read-only file>
May 10 12:30:01 proxmox kernel: Buffer I/O error on dev dm-18, logical block 241777, lost async page write
May 10 12:30:02 proxmox kernel: Buffer I/O error on dev dm-18, logical block 241777, lost async page write
May 10 12:30:04 proxmox pvestatd[1062]: can't lock file '/var/log/pve/tasks/.active.lock' - can't open file - Read-only file system
May 10 12:30:14 proxmox pvestatd[1062]: can't lock file '/var/log/pve/tasks/.active.lock' - can't open file - Read-only file system
May 10 12:30:14 proxmox kernel: Buffer I/O error on dev dm-18, logical block 241777, lost async page write
May 10 12:30:15 proxmox kernel: Buffer I/O error on dev dm-18, logical block 241777, lost async page write
May 10 12:30:16 proxmox kernel: Buffer I/O error on dev dm-18, logical block 241777, lost async page write
May 10 12:30:17 proxmox kernel: Buffer I/O error on dev dm-18, logical block 241777, lost async page write
May 10 12:30:18 proxmox kernel: Buffer I/O error on dev dm-18, logical block 241777, lost async page write
May 10 12:30:20 proxmox kernel: ata1.00: exception Emask 0x0 SAct 0x90400001 SErr 0x0 action 0x0Your physical disk is failing. The system re-mounted the file system to read-only as a precaution.
Good luck
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Good luck
Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Looks like disk going bad. Backup what you can and plan on replacing the disk
Only way to fix an ext4 filesystem that has remounted itself R/O due to errors, is to reboot and hope fsck works as intended
Only way to fix an ext4 filesystem that has remounted itself R/O due to errors, is to reboot and hope fsck works as intended
haha thanks for pin pointing the issue. I guess it was not a good brand of ssd.
I've been having continuous problems with my Proxmox installation for a few months now, and I was wondering if it was possible that it could be SSD-dependent. MY system only needs to run 5 LXCs and I have been using a DELL Wyse 5070 with 32GB, quadcore Pentium CPU for 2 years now, and for low workloads it has always worked great. Unfortunately it uses M.2 Sata and there is little choice of 1TB chips.
Since yesterday I also got the connection failed error (Error 500: closing file '/var/tmp/pve-reserved-ports.tmp.2866551' failed
I have disk space so I understand that the defective SSD is indeed the cause. Since I can only connect with Solar-Putty to the shell (it gives the same error if I open the shell from the Proxmox GUI what command can I use to try to fix the SSD errors ?
Since yesterday I also got the connection failed error (Error 500: closing file '/var/tmp/pve-reserved-ports.tmp.2866551' failed
I have disk space so I understand that the defective SSD is indeed the cause. Since I can only connect with Solar-Putty to the shell (it gives the same error if I open the shell from the Proxmox GUI what command can I use to try to fix the SSD errors ?
You can try fsck from a rescue environment, but odds are you'll have to replace the disk.
If you have USB3 you could also look into a Samsung T7, but honestly long-term you'd be better off upgrading the whole server. A quadcore Pentium is probably using more power per month than a more modern CPU
If you have limited budget I would look into mini-PCs; personally I went with a Qotom firewall appliance for my 24/7 PVE server but it has limitations. VGA out, no soundcard, kinda slow for interactive-GUI VMs.
https://www.amazon.com/Qotom-Q20331G9-Pfsense-Firewall-Barebone/dp/B0CN172ZKC
Personally I went for 16GB RAM and 256GB nvme to keep it under $400 and upgraded it from there. Been running on a UPS and dead reliable since Feb 2024.
If you have USB3 you could also look into a Samsung T7, but honestly long-term you'd be better off upgrading the whole server. A quadcore Pentium is probably using more power per month than a more modern CPU
If you have limited budget I would look into mini-PCs; personally I went with a Qotom firewall appliance for my 24/7 PVE server but it has limitations. VGA out, no soundcard, kinda slow for interactive-GUI VMs.
https://www.amazon.com/Qotom-Q20331G9-Pfsense-Firewall-Barebone/dp/B0CN172ZKC
Personally I went for 16GB RAM and 256GB nvme to keep it under $400 and upgraded it from there. Been running on a UPS and dead reliable since Feb 2024.
Great choice but I could never live with only 1 USB3, I normally use 3 (older) 1TB USB HDDs for backup, have a ZigBee USB stick and a 2nd. LAN on USB. On the other side you're right my electricity bill went to the stars.