[SOLVED] Proxmox Updates Failing

I

infinityM

Well-Known Member
Dec 7, 2019
179
1
58
31
Hey Guys,

I've been struggling for a while to get my 1 server to update correctly. It just refuses... So being at a loss, I'm hoping the proxmox community will come to my rescue...

When I run the update on our server. I am getting the below output... Does anyone have ANY advise that might help me resolve it?

Bash: 
Starting system upgrade: apt-get dist-upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following package was automatically installed and is no longer required:
  python-argcomplete
Use 'apt autoremove' to remove it.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
14 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up lxc-pve (4.0.2-1) ...
usermod: cannot open /etc/passwd
dpkg: error processing package lxc-pve (--configure):
 installed lxc-pve package post-installation script subprocess returned error exit status 1
Setting up ceph-common (14.2.10-pve1) ...
Setting system user ceph properties..usermod: no changes
usermod: cannot open /etc/passwd
dpkg: error processing package ceph-common (--configure):
 installed ceph-common package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of pve-container:
 pve-container depends on lxc-pve; however:
  Package lxc-pve is not configured yet.

dpkg: error processing package pve-container (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of pve-qemu-kvm:
 pve-qemu-kvm depends on ceph-common (>= 0.48); however:
  Package ceph-common is not configured yet.

dpkg: error processing package pve-qemu-kvm (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ceph-base:
 ceph-base depends on ceph-common (= 14.2.10-pve1); however:
  Package ceph-common is not configured yet.

dpkg: error processing package ceph-base (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of pve-manager:
 pve-manager depends on pve-container (>= 2.0-21); however:
  Package pve-container is not configured yet.

dpkg: error processing package pve-manager (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ceph-mds:
 ceph-mds depends on ceph-base (= 14.2.10-pve1); however:
  Package ceph-base is not configured yet.

dpkg: error processing package ceph-mds (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of qemu-server:
 qemu-server depends on pve-qemu-kvm (>= 3.0.1-62); however:
  Package pve-qemu-kvm is not configured yet.

dpkg: error processing package qemu-server (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ceph-mgr:
 ceph-mgr depends on ceph-base (= 14.2.10-pve1); however:
  Package ceph-base is not configured yet.

dpkg: error processing package ceph-mgr (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libpve-storage-perl:
 libpve-storage-perl depends on ceph-common (>= 12.2~); however:
  Package ceph-common is not configured yet.

dpkg: error processing package libpve-storage-perl (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ceph-osd:
 ceph-osd depends on ceph-base (= 14.2.10-pve1); however:
  Package ceph-base is not configured yet.

dpkg: error processing package ceph-osd (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ceph-mon:
 ceph-mon depends on ceph-base (= 14.2.10-pve1); however:
  Package ceph-base is not configured yet.

dpkg: error processing package ceph-mon (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ceph:
 ceph depends on ceph-mgr (= 14.2.10-pve1); however:
  Package ceph-mgr is not configured yet.
 ceph depends on ceph-mon (= 14.2.10-pve1); however:
  Package ceph-mon is not configured yet.
 ceph depends on ceph-osd (= 14.2.10-pve1); however:
  Package ceph-osd is not configured yet.

dpkg: error processing package ceph (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libpve-guest-common-perl:
 libpve-guest-common-perl depends on libpve-storage-perl (>= 6.1-6); however:
  Package libpve-storage-perl is not configured yet.

dpkg: error processing package libpve-guest-common-perl (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent processing triggers for pve-ha-manager:
 pve-ha-manager depends on pve-container; however:
  Package pve-container is not configured yet.
 pve-ha-manager depends on qemu-server (>= 6.0-15); however:
  Package qemu-server is not configured yet.

dpkg: error processing package pve-ha-manager (--configure):
 dependency problems - leaving triggers unprocessed
Processing triggers for libc-bin (2.28-10) ...
Errors were encountered while processing:
 lxc-pve
 ceph-common
 pve-container
 pve-qemu-kvm
 ceph-base
 pve-manager
 ceph-mds
 qemu-server
 ceph-mgr
 libpve-storage-perl
 ceph-osd
 ceph-mon
 ceph
 libpve-guest-common-perl
 pve-ha-manager
E: Sub-process /usr/bin/dpkg returned an error code (1)

Your System is up-to-date

starting shell
 
@infinityM Did you set up a repository? Also, it might be a good idea to issue apt-get updae before running apt-get dist-upgrade.
 
eoinkim said:
@infinityM Did you set up a repository? Also, it might be a good idea to issue apt-get updae before running apt-get dist-upgrade.
Click to expand...
Hey Eoinkim,

I've done that. But still happens though?

Bash: 
root@c5:~# apt-get update
Hit:1 http://security.debian.org/debian-security buster/updates InRelease
Hit:2 http://ftp.debian.org/debian buster InRelease
Hit:3 http://download.proxmox.com/debian/pve buster InRelease
Get:4 http://ftp.debian.org/debian buster-updates InRelease [51.9 kB]
Hit:5 http://download.proxmox.com/debian/ceph-nautilus buster InRelease
Fetched 51.9 kB in 1s (35.9 kB/s)
Reading package lists... Done
root@c5:~# apt-get dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following package was automatically installed and is no longer required:
  python-argcomplete
Use 'apt autoremove' to remove it.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
14 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up lxc-pve (4.0.2-1) ...
usermod: cannot open /etc/passwd
dpkg: error processing package lxc-pve (--configure):
 installed lxc-pve package post-installation script subprocess returned error exit status 1
Setting up ceph-common (14.2.10-pve1) ...
Setting system user ceph properties..usermod: no changes
usermod: cannot open /etc/passwd
dpkg: error processing package ceph-common (--configure):
 installed ceph-common package post-installation script subprocess returned error exit status 1
dpkg: dependency problems prevent configuration of pve-container:
 pve-container depends on lxc-pve; however:
  Package lxc-pve is not configured yet.

dpkg: error processing package pve-container (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of pve-qemu-kvm:
 pve-qemu-kvm depends on ceph-common (>= 0.48); however:
  Package ceph-common is not configured yet.

dpkg: error processing package pve-qemu-kvm (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ceph-base:
 ceph-base depends on ceph-common (= 14.2.10-pve1); however:
  Package ceph-common is not configured yet.

dpkg: error processing package ceph-base (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of pve-manager:
 pve-manager depends on pve-container (>= 2.0-21); however:
  Package pve-container is not configured yet.

dpkg: error processing package pve-manager (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ceph-mds:
 ceph-mds depends on ceph-base (= 14.2.10-pve1); however:
  Package ceph-base is not configured yet.

dpkg: error processing package ceph-mds (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of qemu-server:
 qemu-server depends on pve-qemu-kvm (>= 3.0.1-62); however:
  Package pve-qemu-kvm is not configured yet.

dpkg: error processing package qemu-server (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ceph-mgr:
 ceph-mgr depends on ceph-base (= 14.2.10-pve1); however:
  Package ceph-base is not configured yet.

dpkg: error processing package ceph-mgr (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libpve-storage-perl:
 libpve-storage-perl depends on ceph-common (>= 12.2~); however:
  Package ceph-common is not configured yet.

dpkg: error processing package libpve-storage-perl (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ceph-osd:
 ceph-osd depends on ceph-base (= 14.2.10-pve1); however:
  Package ceph-base is not configured yet.

dpkg: error processing package ceph-osd (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ceph-mon:
 ceph-mon depends on ceph-base (= 14.2.10-pve1); however:
  Package ceph-base is not configured yet.

dpkg: error processing package ceph-mon (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ceph:
 ceph depends on ceph-mgr (= 14.2.10-pve1); however:
  Package ceph-mgr is not configured yet.
 ceph depends on ceph-mon (= 14.2.10-pve1); however:
  Package ceph-mon is not configured yet.
 ceph depends on ceph-osd (= 14.2.10-pve1); however:
  Package ceph-osd is not configured yet.

dpkg: error processing package ceph (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libpve-guest-common-perl:
 libpve-guest-common-perl depends on libpve-storage-perl (>= 6.1-6); however:
  Package libpve-storage-perl is not configured yet.

dpkg: error processing package libpve-guest-common-perl (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent processing triggers for pve-ha-manager:
 pve-ha-manager depends on pve-container; however:
  Package pve-container is not configured yet.
 pve-ha-manager depends on qemu-server (>= 6.0-15); however:
  Package qemu-server is not configured yet.

dpkg: error processing package pve-ha-manager (--configure):
 dependency problems - leaving triggers unprocessed
Processing triggers for libc-bin (2.28-10) ...
Errors were encountered while processing:
 lxc-pve
 ceph-common
 pve-container
 pve-qemu-kvm
 ceph-base
 pve-manager
 ceph-mds
 qemu-server
 ceph-mgr
 libpve-storage-perl
 ceph-osd
 ceph-mon
 ceph
 libpve-guest-common-perl
 pve-ha-manager
E: Sub-process /usr/bin/dpkg returned an error code (1)
 
infinityM said:
usermod: cannot open /etc/passwd
Click to expand...
that might be a hint to what is problematic...
* does the system have a sensible '/etc/passwd' file?
* is it locked in some way? (`vipw` can help in opening it)
* can you open it for writing? (open it in an editor and try to save it (without making any modifications))

I hope this helps!
 
Stoiko Ivanov said:
that might be a hint to what is problematic...
* does the system have a sensible '/etc/passwd' file?
* is it locked in some way? (`vipw` can help in opening it)
* can you open it for writing? (open it in an editor and try to save it (without making any modifications))

I hope this helps!
Click to expand...
When I open it, It is readonly
Code: 
"/etc/passwd" [readonly] 31 lines, 1661 characters
but I'm not sure why... I've checked our other servers are 100%...

How can I fix it? I want to move very carefully since I don't want to break anything.
 
infinityM said:
When I open it, It is readonly
Click to expand...
* did you open it as 'root' ?
* what are the permissions on the file? ls -la /etc/passwd
* maybe there is a problem with the filesystem where the file is on - anything interesting in dmesg output?
* please post the output of mount

I hope this helps!
 
Stoiko Ivanov said:
that might be a hint to what is problematic...
* does the system have a sensible '/etc/passwd' file?
* is it locked in some way? (`vipw` can help in opening it)
* can you open it for writing? (open it in an editor and try to save it (without making any modifications))

I hope this helps!
Click to expand...
This did also not work...

Code: 
root@c5:~# vipw

Select an editor.  To change later, run 'select-editor'.
  1. /bin/nano        <---- easiest
  2. /usr/bin/vim.tiny

Choose 1-2 [1]: 1
vipw: can't restore /etc/passwd: Operation not permitted (your changes are in /etc/passwd.edit)
vipw: /etc/passwd is unchanged
root@c5:~#
 
Stoiko Ivanov said:
* did you open it as 'root' ?
Click to expand...
Yes I am logged in as root

Stoiko Ivanov said:
* what are the permissions on the file? ls -la /etc/passwd
Click to expand...
-rw-r--r-- 1 root root 1661 Aug 18 12:17 /etc/passwd

Stoiko Ivanov said:
* please post the output of mount
Click to expand...

Bash: 
root@c5:~# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=65957520k,nr_inodes=16489380,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=13196412k,mode=755)
rpool/ROOT/pve-1 on / type zfs (rw,relatime,xattr,noacl)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=41,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=48832)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
rpool on /rpool type zfs (rw,noatime,xattr,noacl)
rpool/ROOT on /rpool/ROOT type zfs (rw,noatime,xattr,noacl)
rpool/data on /rpool/data type zfs (rw,noatime,xattr,noacl)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
tmpfs on /var/lib/ceph/osd/ceph-28 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-0 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-20 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-26 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-31 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-27 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-29 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-30 type tmpfs (rw,relatime)
tmpfs on /var/lib/ceph/osd/ceph-34 type tmpfs (rw,relatime)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
10.161.0.247:/volume1/Backups on /mnt/pve/NAS type nfs (rw,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=1                       0.161.0.247,mountvers=3,mountport=2195,mountproto=udp,local_lock=none,addr=10.161.0.247)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=13196408k,mode=700)

Stoiko Ivanov said:
* maybe there is a problem with the filesystem where the file is on - anything interesting in dmesg output?
Click to expand...
I have pasted the output below. Only errors I can see are mount errors for an lxc...

Code: 
[521324.743860]
[1176435.794982] device fwpr111p0 left promiscuous mode
[1176435.794986] vmbr0: port 13(fwpr111p0) entered disabled state
[1177437.372606] audit: type=1400 audit(1597737303.286:28): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-107_</var/lib/lxc>" name="/tmp/" pid=1844809 comm="(sh)" flags="rw, remount, bind"
[1177724.941737] audit: type=1400 audit(1597737590.862:29): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-107_</var/lib/lxc>" name="/tmp/" pid=1857405 comm="(imedated)" flags="rw, remount, bind"
[1178815.280286] device tap116i0 entered promiscuous mode
[1178815.319201] fwbr116i0: port 1(fwln116i0) entered blocking state
[1178815.319204] fwbr116i0: port 1(fwln116i0) entered disabled state
[1178815.319369] device fwln116i0 entered promiscuous mode
[1178815.319542] fwbr116i0: port 1(fwln116i0) entered blocking state
[1178815.319544] fwbr116i0: port 1(fwln116i0) entered forwarding state
[1178815.323956] vmbr0: port 4(fwpr116p0) entered blocking state
[1178815.323960] vmbr0: port 4(fwpr116p0) entered disabled state
[1178815.324083] device fwpr116p0 entered promiscuous mode
[1178815.324137] vmbr0: port 4(fwpr116p0) entered blocking state
[1178815.324139] vmbr0: port 4(fwpr116p0) entered forwarding state
[1178815.328731] fwbr116i0: port 2(tap116i0) entered blocking state
[1178815.328734] fwbr116i0: port 2(tap116i0) entered disabled state
[1178815.329055] fwbr116i0: port 2(tap116i0) entered blocking state
[1178815.329056] fwbr116i0: port 2(tap116i0) entered forwarding state
[1178816.081967] device tap116i1 entered promiscuous mode
[1178816.120525] fwbr116i1: port 1(fwln116i1) entered blocking state
[1178816.120529] fwbr116i1: port 1(fwln116i1) entered disabled state
[1178816.120686] device fwln116i1 entered promiscuous mode
[1178816.120774] fwbr116i1: port 1(fwln116i1) entered blocking state
[1178816.120775] fwbr116i1: port 1(fwln116i1) entered forwarding state
[1178816.125307] vmbr0: port 5(fwpr116p1) entered blocking state
[1178816.125309] vmbr0: port 5(fwpr116p1) entered disabled state
[1178816.125448] device fwpr116p1 entered promiscuous mode
[1178816.125502] vmbr0: port 5(fwpr116p1) entered blocking state
[1178816.125503] vmbr0: port 5(fwpr116p1) entered forwarding state
[1178816.129888] fwbr116i1: port 2(tap116i1) entered blocking state
[1178816.129890] fwbr116i1: port 2(tap116i1) entered disabled state
[1178816.130129] fwbr116i1: port 2(tap116i1) entered blocking state
[1178816.130130] fwbr116i1: port 2(tap116i1) entered forwarding state
 
hmm - ok - filesystem looks ok, and the problem with opening it with vipw could indicate a locking error (although that does not explain the read-only when you open it with a plain editor (nano/vim)....)


please try the following:
* `touch /etc/testpasswd`
* `rm /etc/testpasswd`
(that should confirm that the filesystem is indeed writeable)
* `ls /etc/*pass*`
(to see if some left-over lockfile is there)
* `lsof -n |grep passwd`
(to see if some process has passwd open)

EDIT: please also post the output of:
`zpool status`
 
Stoiko Ivanov said:
hmm - ok - filesystem looks ok, and the problem with opening it with vipw could indicate a locking error (although that does not explain the read-only when you open it with a plain editor (nano/vim)....)


please try the following:
* `touch /etc/testpasswd`
* `rm /etc/testpasswd`
(that should confirm that the filesystem is indeed writeable)
* `ls /etc/*pass*`
(to see if some left-over lockfile is there)
* `lsof -n |grep passwd`
(to see if some process has passwd open)

EDIT: please also post the output of:
`zpool status`
Click to expand...


Looks like it might be the zpool... So 3 questions...
1: Why do we not get notifications about the degraded state for zpool?
2: How can I determine which drive has failed? It's not showing the drive name or serial number?
3: Would this then in fact stop me from editing the pass file and updating?

Code: 
root@c5:~# zpool status
  pool: rpool
state: DEGRADED
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: scrub repaired 2.64M in 0 days 00:00:54 with 0 errors on Sun Jun 14 00:24:55 2020
config:

        NAME                                              STATE     READ WRITE CKSUM
        rpool                                             DEGRADED     0     0     0
          mirror-0                                        DEGRADED     0     0     0
            scsi-3600508b1001c6ac7bc3b650ab56b87d2-part3  DEGRADED     0     0     0  too many errors
            scsi-3600508b1001c5f894366d2fc583ee4bc-part3  ONLINE       0     0     0

errors: No known data errors
 
infinityM said:
Looks like it might be the zpool... So 3 questions...
1: Why do we not get notifications about the degraded state for zpool?
2: How can I determine which drive has failed? It's not showing the drive name or serial number?
3: Would this then in fact stop me from editing the pass file and updating?
Click to expand...
@1 - check your logs for messages from zed (zfs-zed) and check its docs - I'm not sure that it sends out mails in all situations - but should if a device is faulted - also check your journal when this happened

@2 it should be - check `ls /dev/disk/by-id/` there you should find scsi-3600508b1001c6ac7bc3b650ab56b87d2 pointing to the disk in question

@3 should not seem that way :
infinityM said:
Applications are unaffected.
Click to expand...
 
Stoiko Ivanov said:
hmm - ok - filesystem looks ok, and the problem with opening it with vipw could indicate a locking error (although that does not explain the read-only when you open it with a plain editor (nano/vim)....)


please try the following:
* `touch /etc/testpasswd`
* `rm /etc/testpasswd`
(that should confirm that the filesystem is indeed writeable)
* `ls /etc/*pass*`
(to see if some left-over lockfile is there)
* `lsof -n |grep passwd`
(to see if some process has passwd open)

EDIT: please also post the output of:
`zpool status`
Click to expand...


Ok so the zpool is stable again. But the file remains readonly for me.. So I'm unsure what is the cause?
The touch worked, and the ls /etc/*pass* only shows 2 files.
I've removed the passwd- one but it keeps coming back.

Code: 
root@c5:~# ls /etc/*pass*
/etc/passwd  /etc/passwd-
 
this is rather odd - any thing relevant in the lsof output?

(/etc/passwd- is ok and present usually)

could you try to strace the vipw invocation - check which syscall creates the error
 
Stoiko Ivanov said:
could you try to strace the vipw invocation - check which syscall creates the error
Click to expand...
Just seeing the write error...

Code: 
sroot@c5:~# strace vipw
execve("/usr/sbin/vipw", ["vipw"], 0x7ffce0950250 /* 18 vars */) = 0
brk(NULL)                               = 0x55d52b8e2000
access("/etc/ld.so.preload", R_OK)      = 0
openat(AT_FDCWD, "/etc/ld.so.preload", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=24, ...}) = 0
mmap(NULL, 24, PROT_READ|PROT_WRITE, MAP_PRIVATE, 3, 0) = 0x7f769313a000
close(3)                                = 0
openat(AT_FDCWD, "/usr/lib/libmetadata.so", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\21\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=16888, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7693138000
mmap(NULL, 16552, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7693133000
mmap(0x7f7693134000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7f7693134000
mmap(0x7f7693135000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f7693135000
mmap(0x7f7693136000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f7693136000
close(3)                                = 0
munmap(0x7f769313a000, 24)              = 0
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=40491, ...}) = 0
mmap(NULL, 40491, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7693129000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@k\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=155296, ...}) = 0
mmap(NULL, 2259632, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7692f01000
mprotect(0x7f7692f26000, 2093056, PROT_NONE) = 0
mmap(0x7f7693125000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x24000) = 0x7f7693125000
mmap(0x7f7693127000, 6832, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7693127000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260A\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1824496, ...}) = 0
mmap(NULL, 1837056, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7692d40000
mprotect(0x7f7692d62000, 1658880, PROT_NONE) = 0
mmap(0x7f7692d62000, 1343488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x22000) = 0x7f7692d62000
mmap(0x7f7692eaa000, 311296, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16a000) = 0x7f7692eaa000
mmap(0x7f7692ef7000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b6000) = 0x7f7692ef7000
mmap(0x7f7692efd000, 14336, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7692efd000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000\21\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=14592, ...}) = 0
mmap(NULL, 16656, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7692d3b000
mmap(0x7f7692d3c000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7f7692d3c000
mmap(0x7f7692d3d000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f7692d3d000
mmap(0x7f7692d3e000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f7692d3e000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpcre.so.3", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340!\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=468944, ...}) = 0
mmap(NULL, 471304, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7692cc7000
mmap(0x7f7692cc9000, 335872, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f7692cc9000
mmap(0x7f7692d1b000, 122880, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x54000) = 0x7f7692d1b000
mmap(0x7f7692d39000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x71000) = 0x7f7692d39000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@l\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=146968, ...}) = 0
mmap(NULL, 132288, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7692ca6000
mmap(0x7f7692cac000, 61440, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7f7692cac000
mmap(0x7f7692cbb000, 24576, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x15000) = 0x7f7692cbb000
mmap(0x7f7692cc1000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1a000) = 0x7f7692cc1000
mmap(0x7f7692cc3000, 13504, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7692cc3000
close(3)                                = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7692ca4000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7692ca2000
arch_prctl(ARCH_SET_FS, 0x7f7692ca5680) = 0
mprotect(0x7f7692ef7000, 16384, PROT_READ) = 0
mprotect(0x7f7692cc1000, 4096, PROT_READ) = 0
mprotect(0x7f7692d39000, 4096, PROT_READ) = 0
mprotect(0x7f7692d3e000, 4096, PROT_READ) = 0
mprotect(0x7f7693125000, 4096, PROT_READ) = 0
mprotect(0x7f7693136000, 4096, PROT_READ) = 0
mprotect(0x55d529c53000, 4096, PROT_READ) = 0
mprotect(0x7f7693162000, 4096, PROT_READ) = 0
munmap(0x7f7693129000, 40491)           = 0
set_tid_address(0x7f7692ca5950)         = 2659065
set_robust_list(0x7f7692ca5960, 24)     = 0
rt_sigaction(SIGRTMIN, {sa_handler=0x7f7692cac6b0, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7f7692cb87         30}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {sa_handler=0x7f7692cac740, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART|SA_SIGINFO, sa_restorer=0x         7f7692cb8730}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
statfs("/sys/fs/selinux", 0x7ffc2931b140) = -1 ENOENT (No such file or directory)
statfs("/selinux", 0x7ffc2931b140)      = -1 ENOENT (No such file or directory)
brk(NULL)                               = 0x55d52b8e2000
brk(0x55d52b903000)                     = 0x55d52b903000
openat(AT_FDCWD, "/proc/filesystems", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
read(3, "nodev\tsysfs\nnodev\ttmpfs\nnodev\tbd"..., 1024) = 415
read(3, "", 1024)                       = 0
close(3)                                = 0
access("/etc/selinux/config", F_OK)     = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=3031632, ...}) = 0
mmap(NULL, 3031632, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f76929bd000
close(3)                                = 0
access("/etc/passwd", F_OK)             = 0
openat(AT_FDCWD, "/etc/.pwd.lock", O_WRONLY|O_CREAT|O_CLOEXEC, 0600) = 3
rt_sigaction(SIGALRM, {sa_handler=0x7f7692e3f100, sa_mask=~[], sa_flags=SA_RESTORER, sa_restorer=0x7f7692d77840}, {sa_ha         ndler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [ALRM], [], 8) = 0
alarm(15)                               = 0
 
Stoiko Ivanov said:
could you try to strace the vipw invocation - check which syscall creates the error
Click to expand...
Sorry could not paste the entire output in 1 message...

Code: 
fcntl(3, F_SETLKW, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=0}) = 0

alarm(0)                                = 15

rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0

rt_sigaction(SIGALRM, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f7692d77840}, NULL, 8) = 0

getpid()                                = 2659065

openat(AT_FDCWD, "/etc/passwd.2659065", O_WRONLY|O_CREAT|O_EXCL, 0600) = 4

getpid()                                = 2659065

write(4, "2659065\0", 8)                = 8

close(4)                                = 0

link("/etc/passwd.2659065", "/etc/passwd.lock") = -1 EEXIST (File exists)

openat(AT_FDCWD, "/etc/passwd.lock", O_RDWR) = 4

read(4, "2657352\0", 31)                = 8

close(4)                                = 0

kill(2657352, 0)                        = -1 ESRCH (No such process)

unlink("/etc/passwd.lock")              = 0

link("/etc/passwd.2659065", "/etc/passwd.lock") = 0

stat("/etc/passwd.2659065", {st_mode=S_IFREG|0600, st_size=8, ...}) = 0

unlink("/etc/passwd.2659065")           = 0

stat("/etc/passwd", {st_mode=S_IFREG|0644, st_size=1661, ...}) = 0

openat(AT_FDCWD, "/etc/passwd", O_RDONLY) = 4

umask(077)                              = 022

openat(AT_FDCWD, "/etc/passwd.edit", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 5

umask(022)                              = 077

fstat(4, {st_mode=S_IFREG|0644, st_size=1661, ...}) = 0

lseek(4, 0, SEEK_SET)                   = 0

read(4, "root:x:0:0:root:/root:/bin/bash\n"..., 2048) = 1661

fstat(5, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

read(4, "", 2048)                       = 0

write(5, "root:x:0:0:root:/root:/bin/bash\n"..., 1661) = 1661

fsync(5)                                = 0

close(5)                                = 0

utime("/etc/passwd.edit", {actime=1597753021 /* 2020-08-18T14:17:01+0200 */, modtime=1597753021 /* 2020-08-18T14:17:01+0         200 */}) = 0

chmod("/etc/passwd.edit", 0100644)      = 0

chown("/etc/passwd.edit", 0, 0)         = 0

close(4)                                = 0

clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f7692ca5950) = 2659066

[{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WSTOPPED, NULL) = 2659066

--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2659066, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---

stat("/etc/passwd.edit", {st_mode=S_IFREG|0644, st_size=1661, ...}) = 0

unlink("/etc/passwd-")                  = 0

link("/etc/passwd", "/etc/passwd-")     = -1 EPERM (Operation not permitted)

rename("/etc/passwd.edit", "/etc/passwd") = -1 EPERM (Operation not permitted)

openat(AT_FDCWD, "/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 4

fstat(4, {st_mode=S_IFREG|0644, st_size=2995, ...}) = 0

read(4, "# Locale name alias data base.\n#"..., 3072) = 2995

read(4, "", 3072)                       = 0

close(4)                                = 0

openat(AT_FDCWD, "/usr/share/locale/en_US.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

openat(AT_FDCWD, "/usr/share/locale/en_US.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

openat(AT_FDCWD, "/usr/share/locale/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

openat(AT_FDCWD, "/usr/share/locale/en.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

openat(AT_FDCWD, "/usr/share/locale/en.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

openat(AT_FDCWD, "/usr/share/locale/en_US.UTF-8/LC_MESSAGES/shadow.mo", O_RDONLY) = -1 ENOENT (No such file or directory         )

openat(AT_FDCWD, "/usr/share/locale/en_US.utf8/LC_MESSAGES/shadow.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

openat(AT_FDCWD, "/usr/share/locale/en_US/LC_MESSAGES/shadow.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

openat(AT_FDCWD, "/usr/share/locale/en.UTF-8/LC_MESSAGES/shadow.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

openat(AT_FDCWD, "/usr/share/locale/en.utf8/LC_MESSAGES/shadow.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/shadow.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

write(2, "vipw: can't restore /etc/passwd:"..., 96vipw: can't restore /etc/passwd: Operation not permitted (your changes          are in /etc/passwd.edit)

) = 96

unlink("/etc/passwd.lock")              = 0

close(3)                                = 0

fstat(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0x1), ...}) = 0

write(1, "vipw: /etc/passwd is unchanged\n", 31vipw: /etc/passwd is unchanged

) = 31

exit_group(1)                           = ?

+++ exited with 1 +++
 
This is not getting any less odd:
* What's the permissions of '/etc'? `ls -ld /etc`
* What's the permissions of '/etc/passwd-'? `ls -la /etc/passwd-`
* What's the content of '/etc/passwd-'? `diff -s /etc/passwd /etc/passwd-`
* can you create a hardlink in '/etc/' ? `ln /etc/passwd /etc/passwd.tst`
 
Stoiko Ivanov said:
This is not getting any less odd:
Click to expand...
I'm telling you! D:

Stoiko Ivanov said:
* What's the permissions of '/etc'? `ls -ld /etc`
Click to expand...
root@c5:~# ls -ld /etc
drwxr-xr-x 92 root root 190 Aug 18 15:17 /etc

Stoiko Ivanov said:
* What's the permissions of '/etc/passwd-'? `ls -la /etc/passwd-`
Click to expand...
-rw-r--r-- 1 root root 1661 Aug 18 15:17 /etc/passwd-

Stoiko Ivanov said:
* What's the content of '/etc/passwd-'? `diff -s /etc/passwd /etc/passwd-`
Click to expand...
root@c5:~# diff -s /etc/passwd /etc/passwd-
Files /etc/passwd and /etc/passwd- are identical

Stoiko Ivanov said:
* can you create a hardlink in '/etc/' ? `ln /etc/passwd /etc/passwd.tst`
Click to expand...
root@c5:~# ln /etc/passwd /etc/passwd.tst
ln: failed to create hard link '/etc/passwd.tst' => '/etc/passwd': Operation not permitted
 
* can you create other hardlinks in '/etc/'?
`ln /etc/motd /etc/motd.tst`
* `lsattr /etc/passwd`

else - check `man 2 link` for the conditions where the call from the strace output yields a EPERM
 
Stoiko Ivanov said:
* can you create other hardlinks in '/etc/'?
`ln /etc/motd /etc/motd.tst`
* `lsattr /etc/passwd`

else - check `man 2 link` for the conditions where the call from the strace output yields a EPERM
Click to expand...
The ln /etc/motd /etc/motd.tst was successfull.

* `lsattr /etc/passwd` returned ----i-------------- /etc/passwd

I'm not sure what else I can check?
 
infinityM said:
returned ----i-------------- /etc/passwd
Click to expand...
the file is marked as immutable - see `man lsattr`, `man chattr`.

Unless you know how that happened I would suggest to take a good and close look at the system (and especially at the user accounts in /etc/passwd) - this is something I've seen happening on hacked systems (though it can also be a cautios local admin, who wishes to prevent such things)

In any case I guess this explains the problems of `usermod` initially causing the problems

I hope this helps!
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom