Configuration Management System (Saltstack) Permission Errors but working for files in /etc/pve/nodes/<mynode> ???

[silverstone]

I am bit surprised (and I think it's only happening since a few Weeks/Months) by the Error but it seems to be working somehow.

I use certbot in a Podman/Docker Container to generates ALL Certificates for my Infrastructure. No, I do NOT use the ACME "plugin" of Proxmox VE, since I have wildcard certificates and managing them in a centralized Place is much much easier.

Then I use saltstack / salt (and in some cases normal ssh) to distribute these Certificates & Keys to my Servers.

What I noticed is that for Proxmox VE Hosts (as said, I think it's quite a recent thing), I am getting lots of Errors:

Logs for Before and After disabling setting ownership using chown attached (too long for the Forum to accept the Code within the Post).


Originally I thought that maybe salt didn't like the symlink /etc/pve/local -> /etc/pve/nodes/<node>. Therefore that was the first thing I attempted to fix.

After that it was however still failing.

Logging in via SSH on the Proxmox VE Host shows that I can however manually rename, move, remove etc files in /etc/pve/nodes/<node>.

Both operations are performed as root (salt running as root, ssh logged in as root).

In the end, the files are correctly overwritten (looking at the timestamp shows that they have indeed been updated) and a (remotely issued) salt "PVE" cmd.run "nohup systemctl --no-block restart pveproxy &" allows the Certificate to be correctly loaded by pveproxy. And when I visit the website, I correctly have no more "Self-Signed SSL Certificate" Error.

I don't understand however why this is happening. Any ideas what is special about using a Configuration Management System (such as salt / saltstack) vs raw ssh that could explain these permission errors, yet the thing still works as intended ?

 

[gfngfn256]

salt running as root

I don't have any experience or knowledge at all with saltstack/salt, & I may be barking up the wrong tree, but when you say "salt as root" wouldn't that be root of its OWN environment, so "root of the Podman/Docker Container", but not necessarily as "root on the PVE host". With SSH however I imagine you are actually logging in as root on the host. As I say I don't have any idea on any of the workings of saltstack/salt - I'm just using my head.

 

[silverstone]

I don't have any experience or knowledge at all with saltstack/salt, & I may be barking up the wrong tree, but when you say "salt as root" wouldn't that be root of its OWN environment, so "root of the Podman/Docker Container", but not necessarily as "root on the PVE host". With SSH however I imagine you are actually logging in as root on the host. As I say I don't have any idea on any of the workings of saltstack/salt - I'm just using my head.

salt-minion is definitively running as REAL root, otherwise it won't be able to do anything on the minion:

root@PVE:~# ps aux | grep salt
root        6257  0.2  0.0 131264 26368 ?        Ss   12:14   0:00 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion
root        6266  1.6  0.0 734016 69328 ?        Sl   12:14   0:00 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager
root        7074  1.3  0.0      0     0 ?        Z    12:15   0:00 [/opt/saltstack/] <defunct>
root        7081  0.0  0.0   6332  1536 pts/0    S+   12:15   0:00 grep salt

Nothing in dmesg about apparmor potentially blocking things.

I am starting to think maybe it's the combination of FUSE and Python (e.g. os.chown) Modules that cause Issues ?

 

[fabian]

the permissions and ownership in /etc/pve is hard-coded, you can't change it

 

[silverstone]

the permissions and ownership in /etc/pve is hard-coded, you can't change it

But also if I removed the chown command I still get one error. See attached files in the original Post.

It works, but it's weird to have failure logs in that case ...

 

[fabian]

well, it doesn't contain the information which operation it tried, but my guess is something like chmod/chattr which also doesn't work on pmxcfs