Ansible 'Operation not permitted' copying ssl keys directly

rml

Member
Apr 24, 2019
31
0
11
45
I've got as far as working out that /etc/pve is a fuse mount and that I should use the gui to replace the certs (although it would be nice to automate this)

So, more for my leaning, is there a better way to copy these into place through ansible that I'm missing please?
Or what am I missing here about why it's not copying please?

Some interesting behaviour...

1) it doesn't seem to be an immutable switch but interesting that I'm not getting back information on the files.
Code:
lsattr: Function not implemented While reading flags on ./pve-ssl.pem
lsattr: Function not implemented While reading flags on ./pve-ssl.key

2) trying to delete- Ansible looks like it's deleting the previous files ok, should it?
I only ask as I thought it report they were deleted but it wasn't doing it until I manuall deleted them- but I may be misremembering

Code:
- name:
    file:
      path: "{{item}}"
      state: absent
    loop:
      - /etc/pve/nodes/{{node}}/pveproxy-ssl.pem
      - /etc/pve/nodes/{{node}}/pveproxy-ssl.key
Code:
TASK [file] ******************************************************************************
changed: [rmlprox] => (item=/etc/pve/nodes/rmlprox/pveproxy-ssl.pem)
changed: [proxrml] => (item=/etc/pve/nodes/pve/pveproxy-ssl.pem)
changed: [rmlprox] => (item=/etc/pve/nodes/rmlprox/pveproxy-ssl.key)
changed: [proxrml] => (item=/etc/pve/nodes/pve/pveproxy-ssl.key)

3) Copying them directly with ansible.builtin.copy with doesn't work, either directly, or by copying them locally and them using
copy:
remote_src: yes

Code:
- name: move the files
    ansible.builtin.copy:
      src: "{{item}}"
      dest: "/etc/pve/nodes/{{node}}/"
      remote_src: yes
      owner: root
      group: www-data
      mode: 0640
    become: yes
    loop:
      - /root/certs/pveproxy-ssl.pem
      - /root/certs/pveproxy-ssl.key

Code:
failed: [rmlprox] (item=/root/certs/pveproxy-ssl.pem) => {"ansible_loop_var": "item", "changed": false, "item": "/root/certs/pveproxy-ssl.pem", "msg": "failed to copy: /root/certs/pveproxy-ssl.pem to /etc/pve/nodes/rmlprox/pveproxy-ssl.pem", "traceback": "Traceback (most recent call last):\n  File \"/tmp/ansible_ansible.legacy.copy_payload_j8q7eurp/ansible_ansible.legacy.copy_payload.zip/ansible/modules/copy.py\", line 684, in main\n  File \"/tmp/ansible_ansible.legacy.copy_payload_j8q7eurp/ansible_ansible.legacy.copy_payload.zip/ansible/module_utils/basic.py\", line 2468, in atomic_move\n    os.chmod(b_dest, DEFAULT_PERM & ~umask)\nPermissionError: [Errno 1] Operation not permitted: b'/etc/pve/nodes/rmlprox/pveproxy-ssl.pem'\n"}


4) but you can run a command

- name: sudo move files ansible.builtin.command: cmd: "cp {{item}} /etc/pve/nodes/{{node}}/" loop: - /root/certs/pveproxy-ssl.pem - /root/certs/pveproxy-ssl.key become: yes
 
can you try leaving out the mode, owner, and group from your move command? pmxcfs handles that itself, and the error indicates that it chokes on setting the permissions...
 
Thanks but doesn't work for me I'm getting two different errors (some info obsfucated)
Don't want to waste anyone's time as it works the shell way works - but interested to understand this

In both cases I'm running these as sudo ansible-playbook
(not to self: is there a better way to get privilege escalation for the control machine to access the files?
Perhaps import_playbook with delegate_to: localhost?)

Copying straight in place (r

YAML:
- name: move the files as directly (sudo ansible-playbook)
    ansible.builtin.copy:
      src: "{{item}}"
      dest: "/etc/pve/nodes/{{node}}/"
      # remote_src: yes
      # owner: root
      # group: www-data
      # mode: 0640
    become: yes
    become_user: root
    ansible_ssh_user: root
    loop:
      - "/etc/letsencrypt/live/{{inventory_hostname}}.{{web_domain}}.{{web_suffix}}/fullchain.pem"
      - "/etc/letsencrypt/live/{{inventory_hostname}}.{{web_domain}}.{{web_suffix}}/privkey.pem"
    ignore_errors: yes

YAML:
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: PermissionError: [Errno 1] Operation not permitted: b'/etc/pve/nodes/rmlprox/.ansible_tmpn50wguabfullchain.pem'
failed: [rmlprox] (item=/etc/letsencrypt/live/rmlprox.X.X.X/fullchain.pem) => {"ansible_loop_var": "item", "changed": false, "checksum": "f750aa0d1bedefae4d5e35e54f48f63c3b73dd1d", "item": "/etc/letsencrypt/live/rmlprox.X.X.X/fullchain.pem", "msg": "Failed to replace file: b'/tmp/.ansible-root/tmp/ansible-tmp-1619611155.5270648-115854-61259630597801/source' to /etc/pve/nodes/rmlprox/fullchain.pem: [Errno 1] Operation not permitted: b'/etc/pve/nodes/rmlprox/.ansible_tmpn50wguabfullchain.pem'"}

YAML:
- name: move the files with remote_src: yes
    ansible.builtin.copy:
      src: "{{item}}"
      dest: "/etc/pve/nodes/{{node}}/"
      remote_src: yes
      # owner: root
      # group: www-data
      # mode: 0640
    become: yes
    become_user: root
    ansible_ssh_user: root
    loop:
      - /root/certs/pveproxy-ssl.pem
      - /root/certs/pveproxy-ssl.key

YAML:
WARNING]: File '/etc/pve/nodes/rmlprox/pveproxy-ssl.pem' created with default permissions '666'. The previous default was '666'. Specify 'mode' to avoid this warning.
failed: [rmlprox] (item=/root/certs/pveproxy-ssl.pem) => {"ansible_loop_var": "item", "changed": false, "item": "/root/certs/pveproxy-ssl.pem", "msg": "failed to copy: /root/certs/pveproxy-ssl.pem to /etc/pve/nodes/rmlprox/pveproxy-ssl.pem", "traceback": "Traceback (most recent call last):\n  File \"/tmp/ansible_ansible.legacy.copy_payload_hhfr78kt/ansible_ansible.legacy.copy_payload.zip/ansible/modules/copy.py\", line 684, in main\n  File \"/tmp/ansible_ansible.legacy.copy_payload_hhfr78kt/ansible_ansible.legacy.copy_payload.zip/ansible/module_utils/basic.py\", line 2468, in atomic_move\n    os.chmod(b_dest, DEFAULT_PERM & ~umask)\nPermissionError: [Errno 1] Operation not permitted: b'/etc/pve/nodes/rmlprox/pveproxy-ssl.pem'\n"}

Though I understand a bit better what's going on now....

For the next guy
* /etc/pve is actually a fuse mounted sqlite database
** "you can’t change file permissions (permissions are based on path)"
** see https://pve.proxmox.com/wiki/Proxmox_Cluster_File_System_(pmxcfs)
 
yeah, not sure what ansible is doing under the hood there ;) simply copying the file should work, guess they do something more "magic".
 
Cool, don't let it bother you, thanks for spending the time looking.
 
Thanks, that's what I did in the end but, just to check, how did you so that copy from tmp to nodes? With ansible.builtin.copy? or with a shell sudo command?
 
Cool, thanks, that was in the original question but appreciate it was long. Thanks
 
Touch the files before copying, if the file already exists ansible won't try to set mode or owner.

Code:
- name: "Create empty cert files"
  shell: "touch /etc/pve/local/pveproxy-ssl.pem /etc/pve/local/pveproxy-ssl.key"
  creates: "/etc/pve/local/pveproxy-ssl.pem"

- name: "Copy certificate files"
  copy:
    dest: "/etc/pve/local/pveproxy-ssl.pem"
    content: |
             {{ lookup('file', 'resources/cert.pem') }}
             {{ lookup('file', 'resources/cert_chain.pem') }}

- name: "Copy keys files"
  copy:
    dest: "/etc/pve/local/pveproxy-ssl.key"
    content: "{{ vault_key_file }}"
 
This thread was very helpful in figuring out I just needed to touch the files to solve for it. Thusly dont need to worry about perms when performing the template

Code:
- name: Touch the files to set permissions
  ansible.builtin.file:
    path: "{{ files }}"
    state: touch
  loop:
    - /etc/pve/local/pveproxy-ssl.pem
    - /etc/pve/local/pveproxy-ssl.key
  loop_control:
    loop_var: files

- name: Template LE certificate to Proxmox host
  ansible.builtin.template:
    src: "{{ cert.src }}"
    dest: "{{ cert.dest }}"
    follow: true
  loop:
    - { src: 'fullchain.pem.j2', dest: '/etc/pve/local/pveproxy-ssl.pem' }
    - { src: 'privkey.pem.j2', dest: '/etc/pve/local/pveproxy-ssl.key' }
  loop_control:
    loop_var: cert
  notify:
    - Restart pveproxy
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!