Using Resource Mappings on LXCs

[danthonywalker]

Hello,

I have successfully setup an USB Device in Resource Mappings. While there is documentation on attaching Resource Mapping devices to VMs, I cannot verify if it's possible to use them for LXCs and there is no way add the device via Device Passthrough in the GUI. I took a look around at /dev and while I can see the device itself, I do not see any /dev entry that matches the name I specified in Resource Mappings.

If there is no such thing, it would be nice to be able to use Resource Mappings with LXCs. It would allow the LXC to migrate between nodes with the device and a common path; very similar to the usecase of VMs. I see there is a /dev/mapper which seems like an excellent place to put the device, in the form of /dev/mapper/<type>/<name>.

 

[danthonywalker]

Should I add udev rules as an alternative to having a consistent (across hosts) user-friendly way to add the device to the LXC?

 

[danthonywalker]

I want to share some Ansible tasks I have written to accomplish synchronizing Resource Mappings to /dev entries:

- name: Get PCI resource mappings
  ansible.builtin.command: pvesh get /cluster/mapping/pci --output-format json
  register: pci_resource_mappings
  changed_when: false

- name: Get USB resource mappings
  ansible.builtin.command: pvesh get /cluster/mapping/usb --output-format json
  register: usb_resource_mappings
  changed_when: false

- name: Add udev rules
  vars:
    usb_devices: "{{ usb_resource_mappings.stdout | from_json }}"
    pci_devices: "{{ pci_resource_mappings.stdout | from_json }}"
  ansible.builtin.template:
    src: 99-resource-mappings.rules.j2
    dest: /etc/udev/rules.d/99-resource-mappings.rules
    mode: "0644"
  notify:
    - Reboot

And the accompanying template:

# This file was automatically generated using Ansible.
#
# Adds stable /dev paths for devices used in Resource Mappings.

# PCI
{% for pci_device in pci_devices -%}
  {% for raw_mapping in pci_device.map -%}
    {% set mapping = raw_mapping.split(',') | map('split', '=') | community.general.dict -%}
    {% if mapping.node == inventory_hostname -%}
      {% set name = pci_device.id -%}
      {% set ids = mapping.id.split(':') -%}
      {% if "gpu".casefold() in name.casefold() -%}
## GPU
{# TODO Use mapping iommugroup, path, and/or subsystem-id #}
SUBSYSTEM=="drm", KERNEL=="card*", ATTRS{vendor}=="0x{{ ids[0] }}", ATTRS{device}=="0x{{ ids[1] }}", SYMLINK+="{{ name }}-card"
SUBSYSTEM=="drm", KERNEL=="render*", ATTRS{vendor}=="0x{{ ids[0] }}", ATTRS{device}=="0x{{ ids[1] }}", SYMLINK+="{{ name }}-render"
      {% else -%}
## Other
# TODO "{{ pci_device }}"
      {%- endif %}
    {%- endif %}
  {%- endfor %}
{%- endfor %}

# USB
{% for usb_device in usb_devices -%}
  {% for raw_mapping in usb_device.map -%}
    {% set mapping = raw_mapping.split(',') | map('split', '=') | community.general.dict -%}
    {% if mapping.node == inventory_hostname -%}
      {% set name = usb_device.id -%}
      {% set ids = mapping.id.split(':') -%}
SUBSYSTEM=="usb", ATTRS{idVendor}=="{{ ids[0] }}", ATTRS{idProduct}=="{{ ids[1] }}", SYMLINK+="{{ name }}"
    {%- endif %}
  {%- endfor %}
{%- endfor %}

This seems to work as far as creating /dev entries is concerned. I have not yet tested if these can be passed through to an LXC container yet. There are also TODOs that may be relevant to your setup to address, for example, if using IOMMU or SR-IOV. However, for me, udevadm info /dev/<name> --attribute-walk output seems promising.

 

[danthonywalker]

As it turns out you cannot use the /dev/<device> in the LXC.

I was thinking as an alternative to maybe use lxc.hook in some way to modify the devX entry in config in some way (and base it off the Resource Mappings). However, if I'm understanding hooks correctly, these would run too late and wouldn't work. So I'm stumped on this.

It would be helpful if Proxmox could do this automatically. When selecting "Device Passthrough" for an LXC, you can select a Resource Mapping, and it'll automatically add the devX entry to the LXC config that points to the real /dev device. Then when the LXC migrates from one node to another, Proxmox automatically changes that config when it moves nodes to the appropriate /dev for that node as stated by the Resource Mapping.

 

[evanb]

I would like an simple solution for this too. Most times when my node restarts the use device passthrough changes and then the lxc wont boot.