Can't install Ceph even on clean Proxmox installs.

[Webster I]

We have been using Proxmox and Ceph for quite a while now; unfortunately, when we attempted to setup a new cluster a few days ago, the Ceph install failed. We are doing all of this as the root user. The error occurs when Ceph attempts to add the ssh keys to localhost error: PermissionError: [Errno 1] Operation not permitted. We tried to chmod 770 /root/.ssh/authorized_keys and got chmod: changing permissions of '/root/.ssh/authorized_keys': Operation not permitted.

This is the same exact way we setup the last cluster without issues. Please help.

 

[BobhWasatch]

Please post the output of:

id
ls -al /root

 

[Webster I]

Please post the output of:

id
ls -al /root

uid=0(root) gid=0(root) groups=0(root)
total 32
drwx------ 3 root root 4096 Aug 5 12:46 .
drwxr-xr-x 18 root root 4096 Aug 5 12:43 ..
-rw------- 1 root root 1353 Aug 5 13:28 .bash_history
-rw-r--r-- 1 root root 571 Apr 10 2021 .bashrc
-rw-r--r-- 1 root root 31 Aug 5 12:46 .forward
-rw-r--r-- 1 root root 161 Jul 9 2019 .profile
-rw------- 1 root root 1024 Aug 5 12:27 .rnd
drwx------ 2 root root 4096 Aug 5 12:27 .ssh

 

[BobhWasatch]

Ok, so you really are root and the permissions look ok at the top level. Next thing to check is maybe the rootfs is mounted read-only due to errors or if the permissions of /root/.ssh/authorized_keys (with an o not an e as you spelled it) are messed up. Post the following in code tags:

touch /root/.ssh/myfile
ls -al /root/.ssh
lsattr /root/.ssh
mount

 

[Webster I]

Ok, so you really are root and the permissions look ok at the top level. Next thing to check is maybe the rootfs is mounted read-only due to errors or if the permissions of /root/.ssh/authorized_keys (with an o not an e as you spelled it) are messed up. Post the following in code tags:

touch /root/.ssh/myfile
ls -al /root/.ssh
lsattr /root/.ssh
mount

ls -al /root/.ssh
total 20
drwx------ 2 root root 4096 Aug  5 15:13 .
drwx------ 3 root root 4096 Aug  5 12:46 ..
lrwxrwxrwx 1 root root   29 Aug  5 12:27 authorized_keys -> /etc/pve/priv/authorized_keys
-rw-r----- 1 root root  117 Aug  5 12:27 config
-rw------- 1 root root 1811 Aug  5 12:27 id_rsa
-rw-r--r-- 1 root root  392 Aug  5 12:27 id_rsa.pub
-rw-r--r-- 1 root root    0 Aug  5 15:13 myfile

This is the other error I ran into. I thought the file had the immutable or append attribute, but i can't even read the attributes.

lsattr /root/.ssh
--------------e------- /root/.ssh/config
lsattr: Operation not supported While reading flags on /root/.ssh/authorized_keys
--------------e------- /root/.ssh/id_rsa.pub
--------------e------- /root/.ssh/myfile
--------------e------- /root/.ssh/id_rsa

mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=32879388k,nr_inodes=8219847,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=6582652k,mode=755,inode64)
/dev/mapper/pve-root on / type ext4 (rw,relatime,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=33014)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
/dev/sda2 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
/dev/mapper/pve-root on /var/lib/containers/storage/overlay type ext4 (rw,relatime,errors=remount-ro)
overlay on /var/lib/containers/storage/overlay/be036b600a34f721d0ad97f37fb5c8c73eb1b3b2307787bb7b18b7deedd78d3f/merged type overlay (rw,relatime,lowerdir=/var/lib/containers/storage/overlay/l/H5JEC2YMVN3I5X76DEXN2XT7CB:/var/lib/containers/storage/overlay/l/YG4LFEN3GFB6NYHH2J6CV4T55M,upperdir=/var/lib/containers/storage/overlay/be036b600a34f721d0ad97f37fb5c8c73eb1b3b2307787bb7b18b7deedd78d3f/diff,workdir=/var/lib/containers/storage/overlay/be036b600a34f721d0ad97f37fb5c8c73eb1b3b2307787bb7b18b7deedd78d3f/work,uuid=on,volatile,nouserxattr)
overlay on /var/lib/containers/storage/overlay/7ab7797704fa2cd6e8f06b394b8e6a9a3ccb9a421506b6b27582e5e4cbe6b766/merged type overlay (rw,relatime,lowerdir=/var/lib/containers/storage/overlay/l/H5JEC2YMVN3I5X76DEXN2XT7CB:/var/lib/containers/storage/overlay/l/YG4LFEN3GFB6NYHH2J6CV4T55M,upperdir=/var/lib/containers/storage/overlay/7ab7797704fa2cd6e8f06b394b8e6a9a3ccb9a421506b6b27582e5e4cbe6b766/diff,workdir=/var/lib/containers/storage/overlay/7ab7797704fa2cd6e8f06b394b8e6a9a3ccb9a421506b6b27582e5e4cbe6b766/work,uuid=on,volatile,nouserxattr)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=6582648k,nr_inodes=1645662,mode=700,inode64)

 

[BobhWasatch]

I got the email notification of your reply but it hasn't shown up in the forum yet. Weird.

Anyhow, I forgot that /root/.ssh/authorized_keys is a symlink to /etc/pve/priv/authorized_keys. The reason lsattr fails is because /etc/pve is a FUSE filesystem that doesn't support attributes.

Point is, you should check the permissions of /etc/pve/priv/authorized_keys. If /etc/pve doesn't exist then you have a different problem.

 

[Webster I]

I got the email notification of your reply but it hasn't shown up in the forum yet. Weird.

Anyhow, I forgot that /root/.ssh/authorized_keys is a symlink to /etc/pve/priv/authorized_keys. The reason lsattr fails is because /etc/pve is a FUSE filesystem that doesn't support attributes.

Point is, you should check the permissions of /etc/pve/priv/authorized_keys. If /etc/pve doesn't exist then you have a different problem.

All my posts are now awaiting mod approval. "This message is awaiting moderator approval, and is invisible to normal visitors."

ls -al /etc/pve/priv/authorized_keys
-rw------- 1 root www-data 393 Aug  5 14:44 /etc/pve/priv/authorized_keys

I can't chmod that either.

chmod: changing permissions of '/etc/pve/priv/authorized_keys': Operation not permitted

 

[BobhWasatch]

That is what my /etc/pve/priv looks like (except file sizes). That /etc/pve directory isn't really on disk, it is generated from a database by one of the pve daemons. You can't change permissions but the ones that are there look like mine at least.

https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_pmxcfs

Only thing I can think of is that you did some customization after install to cause this but I don't know what that might be. Did you run any "tteck" scripts or similar? Have you already joined this node to a cluster?

Out of ideas for the moment.

 

[Webster I]

That is what my /etc/pve/priv looks like (except file sizes). That /etc/pve directory isn't really on disk, it is generated from a database by one of the pve daemons. You can't change permissions but the ones that are there look like mine at least.

https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_pmxcfs

Only thing I can think of is that you did some customization after install to cause this but I don't know what that might be. Did you run any "tteck" scripts or similar? Have you already joined this node to a cluster?

Out of ideas for the moment.

The only thing I did to this node was run

apt update && apt upgrade -y

and reboot.

It is not in a cluster. I have attempted this on multiple fresh installs and it fails every time.

Are you running Ceph?

 

[UdoB]

apt update && apt upgrade -y

It is highly recommended to never use "upgrade", always use apt update && apt full-upgrade. This resolves new dependencies in a better way...

 

[BobhWasatch]

No, I don't run Ceph, but this seemed like an installation problem, not specifically a Ceph problem, so thought I could help.

I don't know if @UdoB suggestion is the cause but I suppose it could be. It has been the cause of many strange happenings. It is worth a shot to try I think.