Root file system read-only

A

avn

Active Member
Mar 2, 2015
36
4
28
Hello,

We have Proxmox cluster with shared storage HPE MSA 2050. Nodes don't have local disks, so root file system located on the storage. Not every time, but rarely when hard drive on the storage dies, root file system may become read-only on one of the nodes (it's different nodes, not the same every time). If I try to remount root file system, I get this error: "cannot remount /dev/mapper/pve-root read-write, is write-protected". Is there any way to revert the node to working state without reboot? And what can be done to prevent device from becoming write-protected?
 
Last edited:
When HDD dies on HPE storage, root fs in VM can switch to ro state? Impressive.
How is HPE connected with PVE nodes?
 
All PVE nodes connected to HPE storage via SAS. Two SAS links per node, one for each controller on the storage, for redundancy. Multipath configured on all nodes.
 
You can make test.
Remove HDD from HPE what it will do. Check what HPE/PVE/VM logs/dumps about disks. I think, losing disk create some unstability with disk access, so root fs is switching to RO state because it detects errors.
 
czechsys said:
You can make test.
Remove HDD from HPE what it will do. Check what HPE/PVE/VM logs/dumps about disks. I think, losing disk create some unstability with disk access, so root fs is switching to RO state because it detects errors.
Click to expand...

I agree, apparently when HDD dies, there's some critical delay in disk IO. And it's not FS switched to RO by itself, but because underlying block device became write-protected. I don't know what precisely causing this - device mapper, multipathd, or LVM.

Removing HDD will lead to another many-hours RAID rebuild. It's production environment, and I don't feel comfortable experimenting on it. Besides, we already have one node in faulty state, I didn't restart it yet, because VMs on it works fine. We just can't do anything with them. Can't migrate, can't shutdown, nothing.
 
I would check `dmesg` on the read-only node - it probably has a hint where in the stack the RO-switch is set.
also check the journal and the output of the tools (multipath, dmsetup, pvs,vgs,lvs)
 
Stoiko Ivanov said:
I would check `dmesg` on the read-only node - it probably has a hint where in the stack the RO-switch is set.
also check the journal and the output of the tools (multipath, dmsetup, pvs,vgs,lvs)
Click to expand...

Multipath, dmsetup, pvs, vgs, lvs don't show anything out of ordinary. All seems ok.

Dmesg and jornalctl show that multipath lost one path, then another. FS was remounted read-only, then both paths were restored.

Here is cut from journal:

Code: 
янв 15 04:30:38 adm55 kernel: hpsa 0000:86:00.0: scsi 2:2:1:5: resetting logical  Direct-Access     HPE      MSA 2050 SAS     external SSDSmartPathCap- En- Exp=1
янв 15 04:30:41 adm55 kernel: hpsa 0000:86:00.0: device is ready.
янв 15 04:30:41 adm55 kernel: hpsa 0000:86:00.0: scsi 2:2:1:5: reset logical  completed successfully Direct-Access     HPE      MSA 2050 SAS     external SSDSmartPathCap- En- Exp=1
янв 15 04:30:46 adm55 kernel: sd 2:2:2:5: Power-on or device reset occurred
янв 15 04:31:09 adm55 kernel: sd 2:2:1:3: Power-on or device reset occurred
янв 15 04:31:15 adm55 rrdcached[11085]: flushing old values
янв 15 04:31:15 adm55 rrdcached[11085]: rotating journals
янв 15 04:31:38 adm55 kernel: hpsa 0000:86:00.0: scsi 2:2:1:3: resetting logical  Direct-Access     HPE      MSA 2050 SAS     external SSDSmartPathCap- En- Exp=1
янв 15 04:31:53 adm55 multipathd[7088]: ADM55_SYS: sda - directio checker reports path is down
янв 15 04:31:53 adm55 multipathd[7088]: checker failed path 8:0 in map ADM55_SYS
янв 15 04:31:53 adm55 multipathd[7088]: ADM55_SYS: remaining active paths: 1
янв 15 04:31:53 adm55 multipathd[7088]: VMS2: sdh - directio checker reports path is down
янв 15 04:31:53 adm55 multipathd[7088]: checker failed path 8:112 in map VMS2
янв 15 04:31:53 adm55 multipathd[7088]: VMS2: remaining active paths: 1
янв 15 04:31:53 adm55 multipathd[7088]: VMS3: sdi - directio checker reports path is down
янв 15 04:31:53 adm55 multipathd[7088]: checker failed path 8:128 in map VMS3
янв 15 04:31:53 adm55 multipathd[7088]: VMS3: remaining active paths: 1
янв 15 04:31:53 adm55 kernel: device-mapper: multipath: Failing path 8:0.
янв 15 04:31:53 adm55 kernel: device-mapper: multipath: Failing path 8:112.
янв 15 04:31:53 adm55 kernel: device-mapper: multipath: Failing path 8:128.
янв 15 04:31:54 adm55 multipathd[7088]: VMS1: sdg - directio checker reports path is down
янв 15 04:31:54 adm55 multipathd[7088]: checker failed path 8:96 in map VMS1
янв 15 04:31:54 adm55 multipathd[7088]: VMS1: remaining active paths: 1
янв 15 04:31:54 adm55 kernel: device-mapper: multipath: Failing path 8:96.
янв 15 04:31:55 adm55 multipathd[7088]: VMS1: sdb - directio checker reports path is down
янв 15 04:31:55 adm55 multipathd[7088]: checker failed path 8:16 in map VMS1
янв 15 04:31:55 adm55 kernel: device-mapper: multipath: Failing path 8:16.
янв 15 04:31:55 adm55 kernel: device-mapper: multipath: Failing path 8:80.
янв 15 04:31:55 adm55 multipathd[7088]: VMS1: remaining active paths: 0
янв 15 04:31:55 adm55 multipathd[7088]: ADM55_SYS: sdf - directio checker reports path is down
янв 15 04:31:55 adm55 multipathd[7088]: checker failed path 8:80 in map ADM55_SYS
янв 15 04:31:55 adm55 multipathd[7088]: ADM55_SYS: remaining active paths: 0
янв 15 04:31:55 adm55 kernel: print_req_error: I/O error, dev dm-4, sector 24728959320
янв 15 04:31:55 adm55 kernel: print_req_error: I/O error, dev dm-4, sector 24763602250
янв 15 04:31:55 adm55 kernel: print_req_error: I/O error, dev dm-4, sector 24799547552
янв 15 04:31:55 adm55 kernel: print_req_error: I/O error, dev dm-4, sector 26072408272
янв 15 04:31:55 adm55 kernel: print_req_error: I/O error, dev dm-4, sector 24706807568
янв 15 04:31:55 adm55 kernel: print_req_error: I/O error, dev dm-4, sector 24708867680
янв 15 04:31:55 adm55 kernel: print_req_error: I/O error, dev dm-4, sector 24736074984
янв 15 04:31:55 adm55 kernel: print_req_error: I/O error, dev dm-4, sector 24752216808
янв 15 04:31:55 adm55 kernel: print_req_error: I/O error, dev dm-4, sector 24825200960
янв 15 04:31:55 adm55 kernel: print_req_error: I/O error, dev dm-4, sector 25015340448
янв 15 04:31:56 adm55 multipathd[7088]: VMS2: sdc - directio checker reports path is down
янв 15 04:31:56 adm55 multipathd[7088]: checker failed path 8:32 in map VMS2
янв 15 04:31:56 adm55 multipathd[7088]: VMS2: remaining active paths: 0
янв 15 04:31:56 adm55 multipathd[7088]: VMS4: sde - directio checker reports path is down
янв 15 04:31:56 adm55 multipathd[7088]: checker failed path 8:64 in map VMS4
янв 15 04:31:56 adm55 multipathd[7088]: VMS4: remaining active paths: 1
янв 15 04:31:56 adm55 kernel: device-mapper: multipath: Failing path 8:32.
янв 15 04:31:56 adm55 kernel: device-mapper: multipath: Failing path 8:64.
янв 15 04:31:57 adm55 multipathd[7088]: VMS3: sdd - directio checker reports path is down
янв 15 04:31:57 adm55 multipathd[7088]: checker failed path 8:48 in map VMS3
янв 15 04:31:57 adm55 multipathd[7088]: VMS3: remaining active paths: 0
янв 15 04:31:57 adm55 kernel: device-mapper: multipath: Failing path 8:48.
янв 15 04:31:58 adm55 multipathd[7088]: ADM55_SYS: sda - directio checker reports path is down
янв 15 04:31:58 adm55 multipathd[7088]: VMS2: sdh - directio checker reports path is down
янв 15 04:31:58 adm55 multipathd[7088]: VMS3: sdi - directio checker reports path is down
янв 15 04:31:58 adm55 multipathd[7088]: VMS4: sdj - directio checker reports path is down
янв 15 04:31:58 adm55 multipathd[7088]: checker failed path 8:144 in map VMS4
янв 15 04:31:58 adm55 multipathd[7088]: VMS4: remaining active paths: 0
янв 15 04:31:58 adm55 kernel: device-mapper: multipath: Failing path 8:144.
янв 15 04:31:59 adm55 multipathd[7088]: VMS1: sdg - directio checker reports path is down
янв 15 04:32:00 adm55 multipathd[7088]: VMS1: sdb - directio checker reports path is down
янв 15 04:32:00 adm55 multipathd[7088]: ADM55_SYS: sdf - directio checker reports path is down
янв 15 04:32:00 adm55 kernel: print_req_error: 1395 callbacks suppressed
янв 15 04:32:00 adm55 kernel: print_req_error: I/O error, dev dm-5, sector 10752619680
янв 15 04:32:00 adm55 kernel: print_req_error: I/O error, dev dm-5, sector 0
янв 15 04:32:00 adm55 kernel: print_req_error: I/O error, dev dm-7, sector 5531718087
янв 15 04:32:00 adm55 kernel: print_req_error: I/O error, dev dm-7, sector 5620092871
янв 15 04:32:00 adm55 kernel: print_req_error: I/O error, dev dm-7, sector 911416288
янв 15 04:32:00 adm55 kernel: print_req_error: I/O error, dev dm-7, sector 913189752
янв 15 04:32:00 adm55 kernel: print_req_error: I/O error, dev dm-7, sector 917017856
янв 15 04:32:00 adm55 kernel: print_req_error: I/O error, dev dm-7, sector 917173120
янв 15 04:32:00 adm55 kernel: print_req_error: I/O error, dev dm-7, sector 917177264
янв 15 04:32:00 adm55 kernel: print_req_error: I/O error, dev dm-7, sector 919014984
янв 15 04:32:01 adm55 multipathd[7088]: VMS2: sdc - directio checker reports path is down
янв 15 04:32:01 adm55 multipathd[7088]: VMS4: sde - directio checker reports path is down
янв 15 04:32:01 adm55 kernel: hpsa 0000:86:00.0: device is ready.
янв 15 04:32:01 adm55 kernel: hpsa 0000:86:00.0: scsi 2:2:1:3: reset logical  completed successfully Direct-Access     HPE      MSA 2050 SAS     external SSDSmartPathCap- En- Exp=1
янв 15 04:32:01 adm55 kernel: hpsa 0000:86:00.0: scsi 2:2:1:5: resetting logical  Direct-Access     HPE      MSA 2050 SAS     external SSDSmartPathCap- En- Exp=1
янв 15 04:32:02 adm55 multipathd[7088]: VMS3: sdd - directio checker reports path is down
янв 15 04:32:03 adm55 multipathd[7088]: ADM55_SYS: sda - directio checker reports path is down
янв 15 04:32:03 adm55 multipathd[7088]: VMS2: sdh - directio checker reports path is down
янв 15 04:32:03 adm55 multipathd[7088]: VMS3: sdi - directio checker reports path is down
янв 15 04:32:03 adm55 multipathd[7088]: VMS4: sdj - directio checker reports path is down
янв 15 04:32:04 adm55 multipathd[7088]: VMS1: sdg - directio checker reports path is down
янв 15 04:32:04 adm55 kernel: hpsa 0000:86:00.0: device is ready.
янв 15 04:32:04 adm55 kernel: hpsa 0000:86:00.0: scsi 2:2:1:5: reset logical  completed successfully Direct-Access     HPE      MSA 2050 SAS     external SSDSmartPathCap- En- Exp=1
янв 15 04:32:04 adm55 kernel: sd 2:2:2:3: Power-on or device reset occurred
янв 15 04:32:04 adm55 kernel: sd 2:2:2:5: Power-on or device reset occurred
янв 15 04:32:04 adm55 rrdcached[11085]: started new journal /var/lib/rrdcached/journal/rrd.journal.1547515924.275550
янв 15 04:32:04 adm55 kernel: Aborting journal on device dm-333-8.
янв 15 04:32:04 adm55 kernel: Buffer I/O error on dev dm-333, logical block 262144, lost sync page write
янв 15 04:32:04 adm55 kernel: JBD2: Error -5 detected when updating journal superblock for dm-333-8.
янв 15 04:32:04 adm55 kernel: Buffer I/O error on dev dm-333, logical block 0, lost sync page write
янв 15 04:32:04 adm55 kernel: EXT4-fs error (device dm-333): ext4_journal_check_start:61: Detected aborted journal
янв 15 04:32:04 adm55 kernel: EXT4-fs (dm-333): Remounting filesystem read-only
янв 15 04:32:04 adm55 kernel: EXT4-fs (dm-333): previous I/O error to superblock detected
янв 15 04:32:04 adm55 kernel: Buffer I/O error on dev dm-333, logical block 290, lost async page write
янв 15 04:32:04 adm55 kernel: Buffer I/O error on dev dm-333, logical block 0, lost sync page write
янв 15 04:32:04 adm55 kernel: EXT4-fs error (device dm-333): ext4_journal_check_start:61: Detected aborted journal
янв 15 04:32:04 adm55 kernel: EXT4-fs (dm-333): previous I/O error to superblock detected
янв 15 04:32:04 adm55 kernel: Buffer I/O error on dev dm-333, logical block 361, lost async page write
янв 15 04:32:04 adm55 kernel: Buffer I/O error on dev dm-333, logical block 372, lost async page write
янв 15 04:32:04 adm55 kernel: Buffer I/O error on dev dm-333, logical block 384, lost async page write
янв 15 04:32:04 adm55 kernel: Buffer I/O error on dev dm-333, logical block 388, lost async page write
янв 15 04:32:04 adm55 kernel: Buffer I/O error on dev dm-333, logical block 0, lost sync page write
янв 15 04:32:04 adm55 kernel: EXT4-fs error (device dm-333): ext4_journal_check_start:61: Detected aborted journal
янв 15 04:32:04 adm55 kernel: EXT4-fs (dm-333): previous I/O error to superblock detected
янв 15 04:32:04 adm55 kernel: Buffer I/O error on dev dm-333, logical block 398, lost async page write
янв 15 04:32:04 adm55 kernel: EXT4-fs (dm-333): previous I/O error to superblock detected
янв 15 04:32:04 adm55 kernel: EXT4-fs warning (device dm-333): ext4_end_bio:323: I/O error 10 writing to inode 1205 (offset 13811712 size 61440 starting block 1535276)
янв 15 04:32:04 adm55 kernel: Buffer I/O error on device dm-333, logical block 1535276
янв 15 04:32:04 adm55 kernel: Buffer I/O error on device dm-333, logical block 1535277
янв 15 04:32:04 adm55 kernel: Buffer I/O error on device dm-333, logical block 1535278
янв 15 04:32:04 adm55 kernel: Buffer I/O error on device dm-333, logical block 1535279
янв 15 04:32:04 adm55 kernel: Buffer I/O error on device dm-333, logical block 1535280
янв 15 04:32:04 adm55 kernel: Buffer I/O error on device dm-333, logical block 1535281
янв 15 04:32:04 adm55 kernel: Buffer I/O error on device dm-333, logical block 1535282
янв 15 04:32:04 adm55 kernel: Buffer I/O error on device dm-333, logical block 1535283
янв 15 04:32:04 adm55 kernel: Buffer I/O error on device dm-333, logical block 1535284
янв 15 04:32:04 adm55 kernel: Buffer I/O error on device dm-333, logical block 1535285
янв 15 04:32:04 adm55 kernel: EXT4-fs warning (device dm-333): ext4_end_bio:323: I/O error 10 writing to inode 1205 (offset 13873152 size 4096 starting block 1535291)
янв 15 04:32:04 adm55 kernel: EXT4-fs error (device dm-333): ext4_journal_check_start:61: Detected aborted journal
янв 15 04:32:04 adm55 rrdcached[11085]: removing old journal /var/lib/rrdcached/journal/rrd.journal.1547508675.837324
янв 15 04:32:04 adm55 corosync[11233]: error   [CPG   ] *** 0x55983a6b9aa0 can't mcast to group pve_dcdb_v1 state:1, error:12
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/adm60/vms4) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-storage/ad
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/adm60/vms3) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-storage/ad
янв 15 04:32:04 adm55 pmxcfs[11132]: [database] crit: commit transaction failed: disk I/O error#010
янв 15 04:32:04 adm55 pmxcfs[11132]: [database] crit: rollback transaction failed: cannot rollback - no transaction is active#010
янв 15 04:32:04 adm55 pmxcfs[11132]: [dcdb] crit: serious internal error - stop cluster connection
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 1' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 1' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 1' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 1' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 1' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 1' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 1' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 1' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 1' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 1' suspended, next retry is Tue Jan 15 04:32:34 2019 [v8.24.0 try http://www.rsyslog.com/e/2007 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 2' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 2' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 2' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 2' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 2' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 2' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 2' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 2' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 2' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 2' suspended, next retry is Tue Jan 15 04:32:34 2019 [v8.24.0 try http://www.rsyslog.com/e/2007 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 10' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 10' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 10' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 10' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 10' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 10' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 10' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 10' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 10' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 10' suspended, next retry is Tue Jan 15 04:32:34 2019 [v8.24.0 try http://www.rsyslog.com/e/2007 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 11' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 11' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 11' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 11' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 11' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 11' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 11' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 11' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 11' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 11' suspended, next retry is Tue Jan 15 04:32:34 2019 [v8.24.0 try http://www.rsyslog.com/e/2007 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 3' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 corosync[11233]:  [CPG   ] *** 0x55983a6b9aa0 can't mcast to group pve_dcdb_v1 state:1, error:12
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 3' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 3' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 3' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 3' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 3' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 3' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 3' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 3' resumed (module 'builtin:omfile') [v8.24.0 try http://www.rsyslog.com/e/2359 ]
янв 15 04:32:04 adm55 liblogging-stdlog[10478]: action 'action 3' suspended, next retry is Tue Jan 15 04:32:34 2019 [v8.24.0 try http://www.rsyslog.com/e/2007 ]
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/139) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/139': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/255) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/255': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/321) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/321': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/280) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/280': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/167) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/167': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/219) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/219': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/288) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/288': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/135) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/135': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/157) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/157': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/191) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/191': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/253) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/253': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/145) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/145': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/212) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/212': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/149) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/149': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/342) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/342': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/230) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/230': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/324) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/324': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/144) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/144': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/213) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/213': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/156) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/156': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/246) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/246': Read-only fil
янв 15 04:32:04 adm55 kernel: JBD2: Detected IO errors while flushing file data on dm-333-8
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/186) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/186': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/217) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/217': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/182) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/182': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/311) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/311': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/223) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/223': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/180) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/180': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/343) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/343': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/229) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/229': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/112) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/112': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/198) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/198': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/279) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/279': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/266) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/266': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/214) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/214': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/332) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/332': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/125) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/125': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/209) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/209': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/252) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/252': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/340) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/340': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/124) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/124': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/120) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/120': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/268) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/268': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/138) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/138': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/346) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/346': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/265) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/265': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/250) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/250': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/248) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/248': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/312) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/312': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/315) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/315': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/159) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/159': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/313) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/313': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/105) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/105': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/161) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/161': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/165) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/165': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/205) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/205': Read-only fil
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/adm51/vms1) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-storage/ad
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/adm51/vms4) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-storage/ad
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/adm51/local) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-storage/a
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/adm51/nfs-iso) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-storage
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/adm51/nfs-vmbackup) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-st
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/adm51/vms3) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-storage/ad
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-storage/adm51/vms2) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-storage/ad
янв 15 04:32:04 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-node/adm51) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-node/adm51': Read-
янв 15 04:32:04 adm55 kernel: EXT4-fs error (device dm-333): ext4_find_entry:1436: inode #33874: comm pve-firewall: reading directory lblock 0
янв 15 04:32:04 adm55 kernel: EXT4-fs (dm-333): previous I/O error to superblock detected
янв 15 04:32:04 adm55 kernel: EXT4-fs error (device dm-333): ext4_find_entry:1436: inode #33874: comm pve-firewall: reading directory lblock 0
янв 15 04:32:04 adm55 kernel: EXT4-fs (dm-333): previous I/O error to superblock detected
янв 15 04:32:04 adm55 kernel: EXT4-fs error (device dm-333): ext4_find_entry:1436: inode #33874: comm pve-firewall: reading directory lblock 0
янв 15 04:32:04 adm55 kernel: EXT4-fs (dm-333): previous I/O error to superblock detected
янв 15 04:32:04 adm55 kernel: EXT4-fs error (device dm-333): ext4_find_entry:1436: inode #33874: comm pve-firewall: reading directory lblock 0
янв 15 04:32:04 adm55 kernel: EXT4-fs (dm-333): previous I/O error to superblock detected
янв 15 04:32:04 adm55 pve-firewall[11299]: status update error: unable to open file '/var/lib/pve-firewall/ipsetcmdlist1.tmp.11299' - Input/output error
янв 15 04:32:04 adm55 pve-firewall[11299]: firewall update time (46.520 seconds)
янв 15 04:32:04 adm55 kernel: EXT4-fs error (device dm-333): ext4_find_entry:1436: inode #33874: comm pve-firewall: reading directory lblock 0
янв 15 04:32:04 adm55 kernel: EXT4-fs (dm-333): previous I/O error to superblock detected
янв 15 04:32:04 adm55 kernel: EXT4-fs error (device dm-333): ext4_find_entry:1436: inode #33874: comm pve-firewall: reading directory lblock 0
янв 15 04:32:04 adm55 kernel: EXT4-fs error (device dm-333): ext4_find_entry:1436: inode #33874: comm pve-firewall: reading directory lblock 0
янв 15 04:32:04 adm55 kernel: EXT4-fs error (device dm-333): ext4_find_entry:1436: inode #33874: comm pve-firewall: reading directory lblock 0
янв 15 04:32:04 adm55 pve-firewall[11299]: status update error: unable to open file '/var/lib/pve-firewall/ipsetcmdlist1.tmp.11299' - Input/output error
янв 15 04:32:04 adm55 pvestatd[11292]: status update time (87.634 seconds)
янв 15 04:32:04 adm55 pvestatd[11292]: can't lock file '/var/log/pve/tasks/.active.lock' - can't open file - Read-only file system
янв 15 04:32:05 adm55 multipathd[7088]: VMS1: sdb - directio checker reports path is up
янв 15 04:32:05 adm55 multipathd[7088]: 8:16: reinstated
янв 15 04:32:05 adm55 multipathd[7088]: VMS1: remaining active paths: 1
янв 15 04:32:05 adm55 multipathd[7088]: ADM55_SYS: sdf - directio checker reports path is up
янв 15 04:32:05 adm55 multipathd[7088]: 8:80: reinstated
янв 15 04:32:05 adm55 multipathd[7088]: ADM55_SYS: remaining active paths: 1
янв 15 04:32:05 adm55 kernel: device-mapper: multipath: Reinstating path 8:16.
янв 15 04:32:05 adm55 kernel: device-mapper: multipath: Reinstating path 8:80.
янв 15 04:32:05 adm55 pmxcfs[11132]: [dcdb] crit: can't initialize service
янв 15 04:32:05 adm55 kernel: print_req_error: 11903 callbacks suppressed
янв 15 04:32:05 adm55 kernel: print_req_error: I/O error, dev dm-7, sector 7115592288
янв 15 04:32:05 adm55 kernel: print_req_error: I/O error, dev dm-7, sector 25420496528
янв 15 04:32:05 adm55 kernel: print_req_error: I/O error, dev dm-7, sector 25420649032
янв 15 04:32:05 adm55 kernel: print_req_error: I/O error, dev dm-7, sector 25447392920
янв 15 04:32:05 adm55 kernel: print_req_error: I/O error, dev dm-7, sector 3849687416
янв 15 04:32:05 adm55 kernel: print_req_error: I/O error, dev dm-7, sector 3817174664
янв 15 04:32:05 adm55 kernel: print_req_error: I/O error, dev dm-7, sector 3860493416
янв 15 04:32:05 adm55 kernel: print_req_error: I/O error, dev dm-7, sector 0
янв 15 04:32:05 adm55 kernel: print_req_error: I/O error, dev dm-7, sector 3814213904
янв 15 04:32:05 adm55 kernel: print_req_error: I/O error, dev dm-7, sector 0
янв 15 04:32:06 adm55 multipathd[7088]: VMS2: sdc - directio checker reports path is up
янв 15 04:32:06 adm55 multipathd[7088]: 8:32: reinstated
янв 15 04:32:06 adm55 multipathd[7088]: VMS2: remaining active paths: 1
янв 15 04:32:06 adm55 multipathd[7088]: VMS4: sde - directio checker reports path is up
янв 15 04:32:06 adm55 multipathd[7088]: 8:64: reinstated
янв 15 04:32:06 adm55 multipathd[7088]: VMS4: remaining active paths: 1
янв 15 04:32:06 adm55 kernel: device-mapper: multipath: Reinstating path 8:32.
янв 15 04:32:06 adm55 kernel: device-mapper: multipath: Reinstating path 8:64.
янв 15 04:32:07 adm55 multipathd[7088]: VMS3: sdd - directio checker reports path is up
янв 15 04:32:07 adm55 multipathd[7088]: 8:48: reinstated
янв 15 04:32:07 adm55 multipathd[7088]: VMS3: remaining active paths: 1
янв 15 04:32:07 adm55 kernel: device-mapper: multipath: Reinstating path 8:48.
янв 15 04:32:07 adm55 kernel: sd 2:2:1:5: Power-on or device reset occurred
янв 15 04:32:08 adm55 multipathd[7088]: ADM55_SYS: sda - directio checker reports path is up
янв 15 04:32:08 adm55 multipathd[7088]: 8:0: reinstated
янв 15 04:32:08 adm55 multipathd[7088]: ADM55_SYS: remaining active paths: 2
янв 15 04:32:08 adm55 multipathd[7088]: VMS2: sdh - directio checker reports path is up
янв 15 04:32:08 adm55 multipathd[7088]: 8:112: reinstated
янв 15 04:32:08 adm55 multipathd[7088]: VMS2: remaining active paths: 2
янв 15 04:32:08 adm55 multipathd[7088]: VMS3: sdi - directio checker reports path is up
янв 15 04:32:08 adm55 multipathd[7088]: 8:128: reinstated
янв 15 04:32:08 adm55 multipathd[7088]: VMS3: remaining active paths: 2
янв 15 04:32:08 adm55 multipathd[7088]: VMS4: sdj - directio checker reports path is up
янв 15 04:32:08 adm55 multipathd[7088]: 8:144: reinstated
янв 15 04:32:08 adm55 multipathd[7088]: VMS4: remaining active paths: 2
янв 15 04:32:08 adm55 kernel: device-mapper: multipath: Reinstating path 8:0.
янв 15 04:32:08 adm55 kernel: device-mapper: multipath: Reinstating path 8:112.
янв 15 04:32:08 adm55 kernel: device-mapper: multipath: Reinstating path 8:128.
янв 15 04:32:08 adm55 kernel: device-mapper: multipath: Reinstating path 8:144.
янв 15 04:32:09 adm55 multipathd[7088]: VMS1: sdg - directio checker reports path is up
янв 15 04:32:09 adm55 multipathd[7088]: 8:96: reinstated
янв 15 04:32:09 adm55 multipathd[7088]: VMS1: remaining active paths: 2
янв 15 04:32:09 adm55 kernel: device-mapper: multipath: Reinstating path 8:96.
янв 15 04:32:12 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/235) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/235': Read-only fil
янв 15 04:32:12 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/225) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/225': Read-only fil
янв 15 04:32:12 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/210) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/210': Read-only fil
янв 15 04:32:12 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/151) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/151': Read-only fil
янв 15 04:32:12 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/228) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/228': Read-only fil
янв 15 04:32:12 adm55 rrdcached[11085]: queue_thread_main: rrd_update_r (/var/lib/rrdcached/db/pve2-vm/211) failed with status -1. (opening '/var/lib/rrdcached/db/pve2-vm/211': Read-only fil
янв 15 04:32:13 adm55 kernel: sd 2:2:2:5: Power-on or device reset occurred
 
I've seen this in VMware hosted on Netapp when there are network blips. It's a general Linux thing, probably "errors=remount-ro" related.
 
Ismael, good thought. Maybe I just need to set errors=continue. I have backups of system volumes anyway.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back