[SOLVED] Custom files in /etc/pve

[whataboutpereira]

I realized it's possible to store custom files in /etc/pve so they will be replicated (unfortunately no executables (hook scripts)).

Is that a bad idea for some reason or is it fine to have a few custom files there?

Any danger of them being wiped when updating or similar?

 

[t.lamprecht]

I realized it's possible to store custom files in /etc/pve so they will be replicated (unfortunately no executables (hook scripts)).

Yes, file permissions are quite limited by design:
https://pve.proxmox.com/pve-docs/chapter-pmxcfs.html#_posix_compatibility

Is that a bad idea for some reason or is it fine to have a few custom files there?

It really depends on:

• how big each file is
• how many you plan to add
• how often those get updated

With 6.4 you still have 512 KiB per-file, 30 MiB total and 10k inodes.
With Proxmox VE 7.0 the limits got a bit increased, 1 MiB per-file, 128 MiB total and 256k inodes, see:
https://git.proxmox.com/?p=pve-cluster.git;a=commit;h=a8df0863b5851dacb4f76ae6364ac1a02fbd56db

Also remember that due to the replication any write will also replicate to all other nodes and thus increase cluster traffic, so frequent updates may cause issues depending on the size of the cluster and the network you're primary using for corosync traffic.

Any danger of them being wiped when updating or similar?

No, we do not wipe anything not under the explicit control of the API (VM configs, API auth keys, ...) and that naturally only if there's an API call that destroys the related resource.

So it can be fine for some use cases (ideal a combination of infrequent modifications & relatively small file size), and it really should be avoided, or is impossible, for others.

 

[whataboutpereira]

Thanks, that's good news.

Currently I'm only storing a Fedora CoreOS ignition file there, so I can have the same reference to it should I migrate it to another node.

I also tried to put my MySQL container backup hook scripts there, but it seems that's a no-go, because I can't use executable permissions and hook scripts didn't want to work with "/usr/bin/bash /etc/pve/custom/vzdump-hook-script.sh" either.

 

[t.lamprecht]

I also tried to put my MySQL container backup hook scripts there, but it seems that's a no-go, because I can't use executable permissions and hook scripts didn't want to work with "/usr/bin/bash /etc/pve/custom/vzdump-hook-script.sh" either.

While the executable permission cannot be added it still works just fine to execute a script there explicitly:

root@prod1:~$ cat /etc/pve/test.sh
#!/bin/bash

echo "executing $0 works"
root@prod1:~$ bash /etc/pve/test.sh
executing /etc/pve/test.sh works

 

[whataboutpereira]

While the executable permission cannot be added it still works just fine to execute a script there explicitly:

root@prod1:~$ cat /etc/pve/test.sh
#!/bin/bash

echo "executing $0 works"
root@prod1:~$ bash /etc/pve/test.sh
executing /etc/pve/test.sh works

From command like it's fine indeed, but I couldn't get it working in a vzdump backup hook. I might have another look sometime, but it didn't seem to matter for vzdump if I used "/etc/pve/test.sh" or "bash /etc/pve/test.sh" as the script.

 

[t.lamprecht]

From command like it's fine indeed, but I couldn't get it working in a vzdump backup hook. I might have another look sometime, but it didn't seem to matter for vzdump if I used "/etc/pve/test.sh" or "bash /etc/pve/test.sh" as the script.

Ah yes, those are executed directly and thus need the executable permission. What you could do as workaround is setting up an intermediate local script that just executes the one at /etc/pve.

Example local "jump" script that is copied to all nodes at the same path:

#!/bin/bash
bash /etc/pve/custom/vzdump-hook-script.sh

 

[whataboutpereira]

Ah yes, those are executed directly and thus need the executable permission. What you could do as workaround is setting up an intermediate local script that just executes the one at /etc/pve.

Example local "jump" script that is copied to all nodes at the same path:

#!/bin/bash
bash /etc/pve/custom/vzdump-hook-script.sh

That would work, but I was hoping to get away with having PVE sync the scripts. It would have been (too) convenient!

 

[whataboutpereira]

I ended up installing the backup hook scripts on each node with ansible.

Also realized I can test with 'pct exec 120 pidof mysqld' if there's a running mysql or redis server in any container and flush/save, so I don't have to hardcode container ids in the scripts.

Just to mention another thing for posterity, if anyone is having trouble with extremely slow backup speed of a MySQL container with datasets using primarycache=metadata, I solved it by setting primarycache=all on the container datasets in backup pre-hook and back to metadata in post-hook.

 

[arubenstein]

Thanks for asking this question - I was going to ask the same. For me, it's snmpd.conf which is the same for all nodes in the cluster - saves time. Thanks again!