[SOLVED] Proxmox Updates Failing

I

infinityM

Well-Known Member
Dec 7, 2019
179
1
58
32
Hey Guys,

I've been struggling for a while to get my 1 server to update correctly. It just refuses... So being at a loss, I'm hoping the proxmox community will come to my rescue...

When I run the update on our server. I am getting the below output... Does anyone have ANY advise that might help me resolve it?

Bash: 
Starting system upgrade: apt-get dist-upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following package was automatically installed and is no longer required:
  python-argcomplete
Use 'apt autoremove' to remove it.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
14 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up lxc-pve (4.0.2-1) ...
usermod: cannot open /etc/passwd
dpkg: error processing package lxc-pve (--configure):
 installed lxc-pve package post-installation script subprocess returned error exit status 1
Setting up ceph-common (14.2.10-pve1) ...
Setting system user ceph properties..usermod: no changes
usermod: cannot open /etc/passwd
dpkg: error processing package ceph-common (--configure):
 installed ceph-common package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of pve-container:
 pve-container depends on lxc-pve; however:
  Package lxc-pve is not configured yet.

dpkg: error processing package pve-container (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of pve-qemu-kvm:
 pve-qemu-kvm depends on ceph-common (>= 0.48); however:
  Package ceph-common is not configured yet.

dpkg: error processing package pve-qemu-kvm (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ceph-base:
 ceph-base depends on ceph-common (= 14.2.10-pve1); however:
  Package ceph-common is not configured yet.

dpkg: error processing package ceph-base (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of pve-manager:
 pve-manager depends on pve-container (>= 2.0-21); however:
  Package pve-container is not configured yet.

dpkg: error processing package pve-manager (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ceph-mds:
 ceph-mds depends on ceph-base (= 14.2.10-pve1); however:
  Package ceph-base is not configured yet.

dpkg: error processing package ceph-mds (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of qemu-server:
 qemu-server depends on pve-qemu-kvm (>= 3.0.1-62); however:
  Package pve-qemu-kvm is not configured yet.

dpkg: error processing package qemu-server (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ceph-mgr:
 ceph-mgr depends on ceph-base (= 14.2.10-pve1); however:
  Package ceph-base is not configured yet.

dpkg: error processing package ceph-mgr (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libpve-storage-perl:
 libpve-storage-perl depends on ceph-common (>= 12.2~); however:
  Package ceph-common is not configured yet.

dpkg: error processing package libpve-storage-perl (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ceph-osd:
 ceph-osd depends on ceph-base (= 14.2.10-pve1); however:
  Package ceph-base is not configured yet.

dpkg: error processing package ceph-osd (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ceph-mon:
 ceph-mon depends on ceph-base (= 14.2.10-pve1); however:
  Package ceph-base is not configured yet.

dpkg: error processing package ceph-mon (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ceph:
 ceph depends on ceph-mgr (= 14.2.10-pve1); however:
  Package ceph-mgr is not configured yet.
 ceph depends on ceph-mon (= 14.2.10-pve1); however:
  Package ceph-mon is not configured yet.
 ceph depends on ceph-osd (= 14.2.10-pve1); however:
  Package ceph-osd is not configured yet.

dpkg: error processing package ceph (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libpve-guest-common-perl:
 libpve-guest-common-perl depends on libpve-storage-perl (>= 6.1-6); however:
  Package libpve-storage-perl is not configured yet.

dpkg: error processing package libpve-guest-common-perl (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent processing triggers for pve-ha-manager:
 pve-ha-manager depends on pve-container; however:
  Package pve-container is not configured yet.
 pve-ha-manager depends on qemu-server (>= 6.0-15); however:
  Package qemu-server is not configured yet.

dpkg: error processing package pve-ha-manager (--configure):
 dependency problems - leaving triggers unprocessed
Processing triggers for libc-bin (2.28-10) ...
Errors were encountered while processing:
 lxc-pve
 ceph-common
 pve-container
 pve-qemu-kvm
 ceph-base
 pve-manager
 ceph-mds
 qemu-server
 ceph-mgr
 libpve-storage-perl
 ceph-osd
 ceph-mon
 ceph
 libpve-guest-common-perl
 pve-ha-manager
E: Sub-process /usr/bin/dpkg returned an error code (1)

Your System is up-to-date

starting shell
 
@infinityM Did you set up a repository? Also, it might be a good idea to issue apt-get updae before running apt-get dist-upgrade.
 
eoinkim said:
@infinityM Did you set up a repository? Also, it might be a good idea to issue apt-get updae before running apt-get dist-upgrade.
Click to expand...
Hey Eoinkim,

I've done that. But still happens though?

Bash: 
root@c5:~# apt-get update
Hit:1 http://security.debian.org/debian-security buster/updates InRelease
Hit:2 http://ftp.debian.org/debian buster InRelease
Hit:3 http://download.proxmox.com/debian/pve buster InRelease
Get:4 http://ftp.debian.org/debian buster-updates InRelease [51.9 kB]
Hit:5 http://download.proxmox.com/debian/ceph-nautilus buster InRelease
Fetched 51.9 kB in 1s (35.9 kB/s)
Reading package lists... Done
root@c5:~# apt-get dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following package was automatically installed and is no longer required:
  python-argcomplete
Use 'apt autoremove' to remove it.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
14 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up lxc-pve (4.0.2-1) ...
usermod: cannot open /etc/passwd
dpkg: error processing package lxc-pve (--configure):
 installed lxc-pve package post-installation script subprocess returned error exit status 1
Setting up ceph-common (14.2.10-pve1) ...
Setting system user ceph properties..usermod: no changes
usermod: cannot open /etc/passwd
dpkg: error processing package ceph-common (--configure):
 installed ceph-common package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of pve-container:
 pve-container depends on lxc-pve; however:
  Package lxc-pve is not configured yet.

dpkg: error processing package pve-container (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of pve-qemu-kvm:
 pve-qemu-kvm depends on ceph-common (>= 0.48); however:
  Package ceph-common is not configured yet.

dpkg: error processing package pve-qemu-kvm (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ceph-base:
 ceph-base depends on ceph-common (= 14.2.10-pve1); however:
  Package ceph-common is not configured yet.

dpkg: error processing package ceph-base (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of pve-manager:
 pve-manager depends on pve-container (>= 2.0-21); however:
  Package pve-container is not configured yet.

dpkg: error processing package pve-manager (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ceph-mds:
 ceph-mds depends on ceph-base (= 14.2.10-pve1); however:
  Package ceph-base is not configured yet.

dpkg: error processing package ceph-mds (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of qemu-server:
 qemu-server depends on pve-qemu-kvm (>= 3.0.1-62); however:
  Package pve-qemu-kvm is not configured yet.

dpkg: error processing package qemu-server (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ceph-mgr:
 ceph-mgr depends on ceph-base (= 14.2.10-pve1); however:
  Package ceph-base is not configured yet.

dpkg: error processing package ceph-mgr (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libpve-storage-perl:
 libpve-storage-perl depends on ceph-common (>= 12.2~); however:
  Package ceph-common is not configured yet.

dpkg: error processing package libpve-storage-perl (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ceph-osd:
 ceph-osd depends on ceph-base (= 14.2.10-pve1); however:
  Package ceph-base is not configured yet.

dpkg: error processing package ceph-osd (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ceph-mon:
 ceph-mon depends on ceph-base (= 14.2.10-pve1); however:
  Package ceph-base is not configured yet.

dpkg: error processing package ceph-mon (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ceph:
 ceph depends on ceph-mgr (= 14.2.10-pve1); however:
  Package ceph-mgr is not configured yet.
 ceph depends on ceph-mon (= 14.2.10-pve1); however:
  Package ceph-mon is not configured yet.
 ceph depends on ceph-osd (= 14.2.10-pve1); however:
  Package ceph-osd is not configured yet.

dpkg: error processing package ceph (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libpve-guest-common-perl:
 libpve-guest-common-perl depends on libpve-storage-perl (>= 6.1-6); however:
  Package libpve-storage-perl is not configured yet.

dpkg: error processing package libpve-guest-common-perl (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent processing triggers for pve-ha-manager:
 pve-ha-manager depends on pve-container; however:
  Package pve-container is not configured yet.
 pve-ha-manager depends on qemu-server (>= 6.0-15); however:
  Package qemu-server is not configured yet.

dpkg: error processing package pve-ha-manager (--configure):
 dependency problems - leaving triggers unprocessed
Processing triggers for libc-bin (2.28-10) ...
Errors were encountered while processing:
 lxc-pve
 ceph-common
 pve-container
 pve-qemu-kvm
 ceph-base
 pve-manager
 ceph-mds
 qemu-server
 ceph-mgr
 libpve-storage-perl
 ceph-osd
 ceph-mon
 ceph
 libpve-guest-common-perl
 pve-ha-manager
E: Sub-process /usr/bin/dpkg returned an error code (1)
 
infinityM said:
usermod: cannot open /etc/passwd
Click to expand...
that might be a hint to what is problematic...
* does the system have a sensible '/etc/passwd' file?
* is it locked in some way? (`vipw` can help in opening it)
* can you open it for writing? (open it in an editor and try to save it (without making any modifications))

I hope this helps!
 
Stoiko Ivanov said:
that might be a hint to what is problematic...
* does the system have a sensible '/etc/passwd' file?
* is it locked in some way? (`vipw` can help in opening it)
* can you open it for writing? (open it in an editor and try to save it (without making any modifications))

I hope this helps!
Click to expand...
When I open it, It is readonly
Code: 
"/etc/passwd" [readonly] 31 lines, 1661 characters
but I'm not sure why... I've checked our other servers are 100%...

How can I fix it? I want to move very carefully since I don't want to break anything.
 
infinityM said:
When I open it, It is readonly
Click to expand...
* did you open it as 'root' ?
* what are the permissions on the file? ls -la /etc/passwd
* maybe there is a problem with the filesystem where the file is on - anything interesting in dmesg output?
* please post the output of mount

I hope this helps!
 
Stoiko Ivanov said:
that might be a hint to what is problematic...
* does the system have a sensible '/etc/passwd' file?
* is it locked in some way? (`vipw` can help in opening it)
* can you open it for writing? (open it in an editor and try to save it (without making any modifications))

I hope this helps!
Click to expand...
This did also not work...

Code: 
root@c5:~# vipw

Select an editor.  To change later, run 'select-editor'.
  1. /bin/nano        <---- easiest
  2. /usr/bin/vim.tiny

Choose 1-2 [1]: 1
vipw: can't restore /etc/passwd: Operation not permitted (your changes are in /etc/passwd.edit)
vipw: /etc/passwd is unchanged
root@c5:~#
 
Stoiko Ivanov said:
* did you open it as 'root' ?
Click to expand...
Yes I am logged in as root

Stoiko Ivanov said:
* what are the permissions on the file? ls -la /etc/passwd
Click to expand...
-rw-r--r-- 1 root root 1661 Aug 18 12:17 /etc/passwd

Stoiko Ivanov said:
* please post the output of mount
Click to expand...

Bash: 
root@c5:~# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=65957520k,nr_inodes=16489380,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=13196412k,mode=755)
rpool/ROOT/pve-1 on / type zfs (rw,relatime,xattr,noacl)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=41,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=48832)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
rpool on /rpool type zfs (rw,noatime,xattr,noacl)
rpool/ROOT on /rpool/ROOT type zfs (rw,noatime,xattr,noacl)
rpool/data on /rpool/data type zfs (rw,noatime,xattr,noacl)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
tmpfs on /var/lib/ceph/osd/ceph-28 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-0 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-20 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-26 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-31 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-27 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-29 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-30 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-34 type tmpfs (rw,relatime)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
10.161.0.247:/volume1/Backups on /mnt/pve/NAS type nfs (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=1                       0.161.0.247,mountvers=3,mountport=2195,mountproto=udp,local_lock=none,addr=10.161.0.247)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=13196408k,mode=700)

Stoiko Ivanov said:
* maybe there is a problem with the filesystem where the file is on - anything interesting in dmesg output?
Click to expand...
I have pasted the output below. Only errors I can see are mount errors for an lxc...

Code: 
[521324.743860]
[1176435.794982] device fwpr111p0 left promiscuous mode
[1176435.794986] vmbr0: port 13(fwpr111p0) entered disabled state
[1177437.372606] audit: type=1400 audit(1597737303.286:28): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-107_</var/lib/lxc>" name="/tmp/" pid=1844809 comm="(sh)" flags="rw, remount, bind"
[1177724.941737] audit: type=1400 audit(1597737590.862:29): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-107_</var/lib/lxc>" name="/tmp/" pid=1857405 comm="(imedated)" flags="rw, remount, bind"
[1178815.280286] device tap116i0 entered promiscuous mode
[1178815.319201] fwbr116i0: port 1(fwln116i0) entered blocking state
[1178815.319204] fwbr116i0: port 1(fwln116i0) entered disabled state
[1178815.319369] device fwln116i0 entered promiscuous mode
[1178815.319542] fwbr116i0: port 1(fwln116i0) entered blocking state
[1178815.319544] fwbr116i0: port 1(fwln116i0) entered forwarding state
[1178815.323956] vmbr0: port 4(fwpr116p0) entered blocking state
[1178815.323960] vmbr0: port 4(fwpr116p0) entered disabled state
[1178815.324083] device fwpr116p0 entered promiscuous mode
[1178815.324137] vmbr0: port 4(fwpr116p0) entered blocking state
[1178815.324139] vmbr0: port 4(fwpr116p0) entered forwarding state
[1178815.328731] fwbr116i0: port 2(tap116i0) entered blocking state
[1178815.328734] fwbr116i0: port 2(tap116i0) entered disabled state
[1178815.329055] fwbr116i0: port 2(tap116i0) entered blocking state
[1178815.329056] fwbr116i0: port 2(tap116i0) entered forwarding state
[1178816.081967] device tap116i1 entered promiscuous mode
[1178816.120525] fwbr116i1: port 1(fwln116i1) entered blocking state
[1178816.120529] fwbr116i1: port 1(fwln116i1) entered disabled state
[1178816.120686] device fwln116i1 entered promiscuous mode
[1178816.120774] fwbr116i1: port 1(fwln116i1) entered blocking state
[1178816.120775] fwbr116i1: port 1(fwln116i1) entered forwarding state
[1178816.125307] vmbr0: port 5(fwpr116p1) entered blocking state
[1178816.125309] vmbr0: port 5(fwpr116p1) entered disabled state
[1178816.125448] device fwpr116p1 entered promiscuous mode
[1178816.125502] vmbr0: port 5(fwpr116p1) entered blocking state
[1178816.125503] vmbr0: port 5(fwpr116p1) entered forwarding state
[1178816.129888] fwbr116i1: port 2(tap116i1) entered blocking state
[1178816.129890] fwbr116i1: port 2(tap116i1) entered disabled state
[1178816.130129] fwbr116i1: port 2(tap116i1) entered blocking state
[1178816.130130] fwbr116i1: port 2(tap116i1) entered forwarding state
 
hmm - ok - filesystem looks ok, and the problem with opening it with vipw could indicate a locking error (although that does not explain the read-only when you open it with a plain editor (nano/vim)....)


please try the following:
* `touch /etc/testpasswd`
* `rm /etc/testpasswd`
(that should confirm that the filesystem is indeed writeable)
* `ls /etc/*pass*`
(to see if some left-over lockfile is there)
* `lsof -n |grep passwd`
(to see if some process has passwd open)

EDIT: please also post the output of:
`zpool status`
 
Stoiko Ivanov said:
hmm - ok - filesystem looks ok, and the problem with opening it with vipw could indicate a locking error (although that does not explain the read-only when you open it with a plain editor (nano/vim)....)


please try the following:
* `touch /etc/testpasswd`
* `rm /etc/testpasswd`
(that should confirm that the filesystem is indeed writeable)
* `ls /etc/*pass*`
(to see if some left-over lockfile is there)
* `lsof -n |grep passwd`
(to see if some process has passwd open)

EDIT: please also post the output of:
`zpool status`
Click to expand...


Looks like it might be the zpool... So 3 questions...
1: Why do we not get notifications about the degraded state for zpool?
2: How can I determine which drive has failed? It's not showing the drive name or serial number?
3: Would this then in fact stop me from editing the pass file and updating?

Code: 
root@c5:~# zpool status
  pool: rpool
state: DEGRADED
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: scrub repaired 2.64M in 0 days 00:00:54 with 0 errors on Sun Jun 14 00:24:55 2020
config:

        NAME                                              STATE     READ WRITE CKSUM
        rpool                                             DEGRADED     0     0     0
          mirror-0                                        DEGRADED     0     0     0
            scsi-3600508b1001c6ac7bc3b650ab56b87d2-part3  DEGRADED     0     0     0  too many errors
            scsi-3600508b1001c5f894366d2fc583ee4bc-part3  ONLINE       0     0     0

errors: No known data errors
 
infinityM said:
Looks like it might be the zpool... So 3 questions...
1: Why do we not get notifications about the degraded state for zpool?
2: How can I determine which drive has failed? It's not showing the drive name or serial number?
3: Would this then in fact stop me from editing the pass file and updating?
Click to expand...
@1 - check your logs for messages from zed (zfs-zed) and check its docs - I'm not sure that it sends out mails in all situations - but should if a device is faulted - also check your journal when this happened

@2 it should be - check `ls /dev/disk/by-id/` there you should find scsi-3600508b1001c6ac7bc3b650ab56b87d2 pointing to the disk in question

@3 should not seem that way :
infinityM said:
Applications are unaffected.
Click to expand...
 
Stoiko Ivanov said:
hmm - ok - filesystem looks ok, and the problem with opening it with vipw could indicate a locking error (although that does not explain the read-only when you open it with a plain editor (nano/vim)....)


please try the following:
* `touch /etc/testpasswd`
* `rm /etc/testpasswd`
(that should confirm that the filesystem is indeed writeable)
* `ls /etc/*pass*`
(to see if some left-over lockfile is there)
* `lsof -n |grep passwd`
(to see if some process has passwd open)

EDIT: please also post the output of:
`zpool status`
Click to expand...


Ok so the zpool is stable again. But the file remains readonly for me.. So I'm unsure what is the cause?
The touch worked, and the ls /etc/*pass* only shows 2 files.
I've removed the passwd- one but it keeps coming back.

Code: 
root@c5:~# ls /etc/*pass*
/etc/passwd  /etc/passwd-
 
this is rather odd - any thing relevant in the lsof output?

(/etc/passwd- is ok and present usually)

could you try to strace the vipw invocation - check which syscall creates the error
 
Stoiko Ivanov said:
could you try to strace the vipw invocation - check which syscall creates the error
Click to expand...
Just seeing the write error...

Code: 
sroot@c5:~# strace vipw
execve("/usr/sbin/vipw", ["vipw"], 0x7ffce0950250 /* 18 vars */) = 0
brk(NULL)                               = 0x55d52b8e2000
access("/etc/ld.so.preload", R_OK)      = 0
openat(AT_FDCWD, "/etc/ld.so.preload", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=24, ...}) = 0
mmap(NULL, 24, PROT_READ|PROT_WRITE, MAP_PRIVATE, 3, 0) = 0x7f769313a000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libmetadata.so", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\21\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=16888, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7693138000
mmap(NULL, 16552, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7693133000
mmap(0x7f7693134000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7f7693134000
mmap(0x7f7693135000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f7693135000
mmap(0x7f7693136000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f7693136000
close(3)                                = 0
munmap(0x7f769313a000, 24)              = 0
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=40491, ...}) = 0
mmap(NULL, 40491, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7693129000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@k\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=155296, ...}) = 0
mmap(NULL, 2259632, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7692f01000
mprotect(0x7f7692f26000, 2093056, PROT_NONE) = 0
mmap(0x7f7693125000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x24000) = 0x7f7693125000
mmap(0x7f7693127000, 6832, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7693127000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260A\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1824496, ...}) = 0
mmap(NULL, 1837056, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7692d40000
mprotect(0x7f7692d62000, 1658880, PROT_NONE) = 0
mmap(0x7f7692d62000, 1343488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x22000) = 0x7f7692d62000
mmap(0x7f7692eaa000, 311296, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16a000) = 0x7f7692eaa000
mmap(0x7f7692ef7000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b6000) = 0x7f7692ef7000
mmap(0x7f7692efd000, 14336, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7692efd000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000\21\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=14592, ...}) = 0
mmap(NULL, 16656, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7692d3b000
mmap(0x7f7692d3c000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7f7692d3c000
mmap(0x7f7692d3d000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f7692d3d000
mmap(0x7f7692d3e000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f7692d3e000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpcre.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340!\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=468944, ...}) = 0
mmap(NULL, 471304, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7692cc7000
mmap(0x7f7692cc9000, 335872, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f7692cc9000
mmap(0x7f7692d1b000, 122880, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x54000) = 0x7f7692d1b000
mmap(0x7f7692d39000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x71000) = 0x7f7692d39000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@l\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=146968, ...}) = 0
mmap(NULL, 132288, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7692ca6000
mmap(0x7f7692cac000, 61440, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7f7692cac000
mmap(0x7f7692cbb000, 24576, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15000) = 0x7f7692cbb000
mmap(0x7f7692cc1000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1a000) = 0x7f7692cc1000
mmap(0x7f7692cc3000, 13504, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7692cc3000
close(3)                                = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7692ca4000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7692ca2000
arch_prctl(ARCH_SET_FS, 0x7f7692ca5680) = 0
mprotect(0x7f7692ef7000, 16384, PROT_READ) = 0
mprotect(0x7f7692cc1000, 4096, PROT_READ) = 0
mprotect(0x7f7692d39000, 4096, PROT_READ) = 0
mprotect(0x7f7692d3e000, 4096, PROT_READ) = 0
mprotect(0x7f7693125000, 4096, PROT_READ) = 0
mprotect(0x7f7693136000, 4096, PROT_READ) = 0
mprotect(0x55d529c53000, 4096, PROT_READ) = 0
mprotect(0x7f7693162000, 4096, PROT_READ) = 0
munmap(0x7f7693129000, 40491)           = 0
set_tid_address(0x7f7692ca5950)         = 2659065
set_robust_list(0x7f7692ca5960, 24)     = 0
rt_sigaction(SIGRTMIN, {sa_handler=0x7f7692cac6b0, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7f7692cb87         30}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {sa_handler=0x7f7692cac740, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART|SA_SIGINFO, sa_restorer=0x         7f7692cb8730}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
statfs("/sys/fs/selinux", 0x7ffc2931b140) = -1 ENOENT (No such file or directory)
statfs("/selinux", 0x7ffc2931b140)      = -1 ENOENT (No such file or directory)
brk(NULL)                               = 0x55d52b8e2000
brk(0x55d52b903000)                     = 0x55d52b903000
openat(AT_FDCWD, "/proc/filesystems", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
read(3, "nodev\tsysfs\nnodev\ttmpfs\nnodev\tbd"..., 1024) = 415
read(3, "", 1024)                       = 0
close(3)                                = 0
access("/etc/selinux/config", F_OK)     = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=3031632, ...}) = 0
mmap(NULL, 3031632, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f76929bd000
close(3)                                = 0
access("/etc/passwd", F_OK)             = 0
openat(AT_FDCWD, "/etc/.pwd.lock", O_WRONLY|O_CREAT|O_CLOEXEC, 0600) = 3
rt_sigaction(SIGALRM, {sa_handler=0x7f7692e3f100, sa_mask=~[], sa_flags=SA_RESTORER, sa_restorer=0x7f7692d77840}, {sa_ha         ndler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [ALRM], [], 8) = 0
alarm(15)                               = 0
 
Stoiko Ivanov said:
could you try to strace the vipw invocation - check which syscall creates the error
Click to expand...
Sorry could not paste the entire output in 1 message...

Code: 
fcntl(3, F_SETLKW, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0

alarm(0)                                = 15

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

rt_sigaction(SIGALRM, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f7692d77840}, NULL, 8) = 0

getpid()                                = 2659065

openat(AT_FDCWD, "/etc/passwd.2659065", O_WRONLY|O_CREAT|O_EXCL, 0600) = 4

getpid()                                = 2659065

write(4, "2659065\0", 8)                = 8

close(4)                                = 0

link("/etc/passwd.2659065", "/etc/passwd.lock") = -1 EEXIST (File exists)

openat(AT_FDCWD, "/etc/passwd.lock", O_RDWR) = 4

read(4, "2657352\0", 31)                = 8

close(4)                                = 0

kill(2657352, 0)                        = -1 ESRCH (No such process)

unlink("/etc/passwd.lock")              = 0

link("/etc/passwd.2659065", "/etc/passwd.lock") = 0

stat("/etc/passwd.2659065", {st_mode=S_IFREG|0600, st_size=8, ...}) = 0

unlink("/etc/passwd.2659065")           = 0

stat("/etc/passwd", {st_mode=S_IFREG|0644, st_size=1661, ...}) = 0

openat(AT_FDCWD, "/etc/passwd", O_RDONLY) = 4

umask(077)                              = 022

openat(AT_FDCWD, "/etc/passwd.edit", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 5

umask(022)                              = 077

fstat(4, {st_mode=S_IFREG|0644, st_size=1661, ...}) = 0

lseek(4, 0, SEEK_SET)                   = 0

read(4, "root:x:0:0:root:/root:/bin/bash\n"..., 2048) = 1661

fstat(5, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

read(4, "", 2048)                       = 0

write(5, "root:x:0:0:root:/root:/bin/bash\n"..., 1661) = 1661

fsync(5)                                = 0

close(5)                                = 0

utime("/etc/passwd.edit", {actime=1597753021 /* 2020-08-18T14:17:01+0200 */, modtime=1597753021 /* 2020-08-18T14:17:01+0         200 */}) = 0

chmod("/etc/passwd.edit", 0100644)      = 0

chown("/etc/passwd.edit", 0, 0)         = 0

close(4)                                = 0

clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f7692ca5950) = 2659066

[{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WSTOPPED, NULL) = 2659066

--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2659066, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---

stat("/etc/passwd.edit", {st_mode=S_IFREG|0644, st_size=1661, ...}) = 0

unlink("/etc/passwd-")                  = 0

link("/etc/passwd", "/etc/passwd-")     = -1 EPERM (Operation not permitted)

rename("/etc/passwd.edit", "/etc/passwd") = -1 EPERM (Operation not permitted)

openat(AT_FDCWD, "/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 4

fstat(4, {st_mode=S_IFREG|0644, st_size=2995, ...}) = 0

read(4, "# Locale name alias data base.\n#"..., 3072) = 2995

read(4, "", 3072)                       = 0

close(4)                                = 0

openat(AT_FDCWD, "/usr/share/locale/en_US.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

openat(AT_FDCWD, "/usr/share/locale/en_US.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

openat(AT_FDCWD, "/usr/share/locale/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

openat(AT_FDCWD, "/usr/share/locale/en.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

openat(AT_FDCWD, "/usr/share/locale/en.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

openat(AT_FDCWD, "/usr/share/locale/en_US.UTF-8/LC_MESSAGES/shadow.mo", O_RDONLY) = -1 ENOENT (No such file or directory         )

openat(AT_FDCWD, "/usr/share/locale/en_US.utf8/LC_MESSAGES/shadow.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

openat(AT_FDCWD, "/usr/share/locale/en_US/LC_MESSAGES/shadow.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

openat(AT_FDCWD, "/usr/share/locale/en.UTF-8/LC_MESSAGES/shadow.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

openat(AT_FDCWD, "/usr/share/locale/en.utf8/LC_MESSAGES/shadow.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/shadow.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

write(2, "vipw: can't restore /etc/passwd:"..., 96vipw: can't restore /etc/passwd: Operation not permitted (your changes          are in /etc/passwd.edit)

) = 96

unlink("/etc/passwd.lock")              = 0

close(3)                                = 0

fstat(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0x1), ...}) = 0

write(1, "vipw: /etc/passwd is unchanged\n", 31vipw: /etc/passwd is unchanged

) = 31

exit_group(1)                           = ?

+++ exited with 1 +++
 
This is not getting any less odd:
* What's the permissions of '/etc'? `ls -ld /etc`
* What's the permissions of '/etc/passwd-'? `ls -la /etc/passwd-`
* What's the content of '/etc/passwd-'? `diff -s /etc/passwd /etc/passwd-`
* can you create a hardlink in '/etc/' ? `ln /etc/passwd /etc/passwd.tst`
 
Stoiko Ivanov said:
This is not getting any less odd:
Click to expand...
I'm telling you! D:

Stoiko Ivanov said:
* What's the permissions of '/etc'? `ls -ld /etc`
Click to expand...
root@c5:~# ls -ld /etc
drwxr-xr-x 92 root root 190 Aug 18 15:17 /etc

Stoiko Ivanov said:
* What's the permissions of '/etc/passwd-'? `ls -la /etc/passwd-`
Click to expand...
-rw-r--r-- 1 root root 1661 Aug 18 15:17 /etc/passwd-

Stoiko Ivanov said:
* What's the content of '/etc/passwd-'? `diff -s /etc/passwd /etc/passwd-`
Click to expand...
root@c5:~# diff -s /etc/passwd /etc/passwd-
Files /etc/passwd and /etc/passwd- are identical

Stoiko Ivanov said:
* can you create a hardlink in '/etc/' ? `ln /etc/passwd /etc/passwd.tst`
Click to expand...
root@c5:~# ln /etc/passwd /etc/passwd.tst
ln: failed to create hard link '/etc/passwd.tst' => '/etc/passwd': Operation not permitted
 
* can you create other hardlinks in '/etc/'?
`ln /etc/motd /etc/motd.tst`
* `lsattr /etc/passwd`

else - check `man 2 link` for the conditions where the call from the strace output yields a EPERM
 
Stoiko Ivanov said:
* can you create other hardlinks in '/etc/'?
`ln /etc/motd /etc/motd.tst`
* `lsattr /etc/passwd`

else - check `man 2 link` for the conditions where the call from the strace output yields a EPERM
Click to expand...
The ln /etc/motd /etc/motd.tst was successfull.

* `lsattr /etc/passwd` returned ----i-------------- /etc/passwd

I'm not sure what else I can check?
 
infinityM said:
returned ----i-------------- /etc/passwd
Click to expand...
the file is marked as immutable - see `man lsattr`, `man chattr`.

Unless you know how that happened I would suggest to take a good and close look at the system (and especially at the user accounts in /etc/passwd) - this is something I've seen happening on hacked systems (though it can also be a cautios local admin, who wishes to prevent such things)

In any case I guess this explains the problems of `usermod` initially causing the problems

I hope this helps!
 
You must log in or register to reply here.
Top Bottom