Can't mount NVME Drive Using vfio-pci Driver

C

cbridgeman

New Member
Nov 25, 2024
8
0
1
I installed a new NVME drive, but I cannot access it with Proxmox. I cannot see it when doing a lsblk command.

After much troubleshooting, I think that it is because it is using the vfio-pci driver. I can see it listed in a VM machine's hardware section under "PCI Devices." I do not have this drive currently in use via passthrough with any VM.

I am using GPU passthrough and I also pass the PCI USB controller through to my main Windows VM.

I have tried to issue the command "echo -n "0000:0c:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind" which changes the drive attribute listed below from lspci -v (below) from "Kernel driver in use: vfio-pci Kernel modules: nvme" to "Kernel modules: nvme" When I issue the command "echo -n "0000:0c:00.0" > /sys/bus/pci/drivers/nvme/bind" I get an error "-bash: echo: write error: Device or resource busy"

When I reboot the PVE the listing from lspci -v (below) returns to its original output.

0c:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a (DRAM-less) (prog-if 02 [NVM Express])
Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a (DRAM-less)
Flags: fast devsel, IRQ 24, IOMMU group 20
Memory at a0200000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/16 Maskable- 64bit+
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [b0] MSI-X: Enable- Count=17 Masked-
Capabilities: [100] Advanced Error Reporting
Capabilities: [168] Secondary PCI Express
Capabilities: [188] Physical Layer 16.0 GT/s <?>
Capabilities: [1ac] Lane Margining at the Receiver <?>
Capabilities: [1c4] Extended Capability ID 0x2a
Capabilities: [1e8] Latency Tolerance Reporting
Capabilities: [1f0] L1 PM Substates
Capabilities: [374] Data Link Feature <?>
Kernel driver in use: vfio-pci
Kernel modules: nvme


Any help would be much appreciated.
 
Last edited:
Additional info. The drive in question is nvme2:

root@proxmox2:/sys/bus# dmesg -H | grep -i nvme
[ +0.003784] nvme 0000:02:00.0: platform quirk: setting simple suspend
[ +0.000001] nvme 0000:05:00.0: platform quirk: setting simple suspend
[ +0.000025] nvme 0000:0c:00.0: platform quirk: setting simple suspend
[ +0.000033] nvme nvme0: pci function 0000:05:00.0
[ +0.000004] nvme nvme1: pci function 0000:02:00.0
[ +0.000018] nvme nvme2: pci function 0000:0c:00.0
[ +0.000249] nvme nvme0: missing or invalid SUBNQN field.
[ +0.009699] nvme nvme2: Shutdown timeout set to 10 seconds
[ +0.011413] nvme nvme2: allocated 64 MiB host memory buffer.
[ +0.005560] nvme nvme0: allocated 32 MiB host memory buffer.
[ +0.022199] nvme nvme1: 32/0/0 default/read/poll queues
[ +0.002394] nvme nvme2: 16/0/0 default/read/poll queues
[ +0.002277] nvme1n1: p1 p2 p3
[ +0.003765] nvme2n1: p1
[ +0.001748] nvme nvme0: 8/0/0 default/read/poll queues
 
Additional info:

If I set amd_iommu=off in Grub, I am able to see the disk and add it as a disk to various VMs. When i re-enable iommu it disappears again and has the same issues I listed in the original post.
 
I do not believe it is, but here is the relevant output of lspci -nnk. My device is 0c:00.0.

0b:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port [1022:43f5] (rev 01)
Subsystem: ASMedia Technology Inc. 600 Series Chipset PCIe Switch Downstream Port [1b21:3328]
Kernel driver in use: pcieport
0c:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a (DRAM-less) [144d:a80d]
Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller PM9C1a (DRAM-less) [144d:a801]
Kernel driver in use: vfio-pci
Kernel modules: nvme
0d:00.0 Network controller [0280]: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter [14c3:0616]
Subsystem: MEDIATEK Corp. MT7922 802.11ax PCI Express Wireless Network Adapter [14c3:0616]

I think that i need to exclude the NVME from loading the vfio-pci driver, but I cannot get that accomplished. I tried to do an unbind from the vfio-pci driver, but it will not let it bind to the nvme driver with this command:
root@proxmox2:~# echo -n "0000:0c:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
root@proxmox2:~# echo -n "0000:0c:00.0" > /sys/bus/pci/drivers/nvme/bind
-bash: echo: write error: Device or resource busy
 
It does indeed look like the device is early bound to vfio-pci or passed through to a VM. What are the outputs of grep -i a80d /proc/cmdline /etc/modprobe.d/*.conf and grep -i 0c:00 /etc/pve/qemu-server/*.conf ? Or did you use driverctl or a script from the internet maybe?
 
BobhWasatch said:
The fact that it works with IOMMU disabled kind of says otherwise. Plus the fact that the kernel is binding vfio to the device says it thinks the device is being passed thru.

What does "lspci -vmm" say?
Click to expand...
root@proxmox2:~# lspci -vmm
Slot: 00:00.0
Class: Host bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Raphael/Granite Ridge Root Complex
SVendor: Micro-Star International Co., Ltd. [MSI]
SDevice: Raphael/Granite Ridge Root Complex
ProgIf: 00

Slot: 00:00.2
Class: IOMMU
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Raphael/Granite Ridge IOMMU
SVendor: Micro-Star International Co., Ltd. [MSI]
SDevice: Raphael/Granite Ridge IOMMU
ProgIf: 00

Slot: 00:01.0
Class: Host bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Raphael/Granite Ridge Dummy Host Bridge
ProgIf: 00
IOMMUGroup: 0

Slot: 00:01.1
Class: PCI bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Raphael/Granite Ridge GPP Bridge
SVendor: Micro-Star International Co., Ltd. [MSI]
SDevice: Raphael/Granite Ridge GPP Bridge
ProgIf: 00
IOMMUGroup: 1

Slot: 00:01.2
Class: PCI bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Raphael/Granite Ridge GPP Bridge
SVendor: Micro-Star International Co., Ltd. [MSI]
SDevice: Raphael/Granite Ridge GPP Bridge
ProgIf: 00
IOMMUGroup: 2

Slot: 00:02.0
Class: Host bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Raphael/Granite Ridge Dummy Host Bridge
ProgIf: 00
IOMMUGroup: 3

Slot: 00:02.1
Class: PCI bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Raphael/Granite Ridge GPP Bridge
SVendor: Micro-Star International Co., Ltd. [MSI]
SDevice: Raphael/Granite Ridge GPP Bridge
ProgIf: 00
IOMMUGroup: 4

Slot: 00:03.0
Class: Host bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Raphael/Granite Ridge Dummy Host Bridge
ProgIf: 00
IOMMUGroup: 5

Slot: 00:04.0
Class: Host bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Raphael/Granite Ridge Dummy Host Bridge
ProgIf: 00
IOMMUGroup: 6

Slot: 00:08.0
Class: Host bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Raphael/Granite Ridge Dummy Host Bridge
ProgIf: 00
IOMMUGroup: 7

Slot: 00:08.1
Class: PCI bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Raphael/Granite Ridge Internal GPP Bridge to Bus [C:A]
SVendor: Unknown vendor 7e12
SDevice: Raphael/Granite Ridge Internal GPP Bridge to Bus [C:A]
ProgIf: 00
IOMMUGroup: 8

Slot: 00:08.3
Class: PCI bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Raphael/Granite Ridge Internal GPP Bridge to Bus [C:A]
SVendor: Unknown vendor 7e12
SDevice: Raphael/Granite Ridge Internal GPP Bridge to Bus [C:A]
ProgIf: 00
IOMMUGroup: 9

Slot: 00:14.0
Class: SMBus
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: FCH SMBus Controller
SVendor: Micro-Star International Co., Ltd. [MSI]
SDevice: FCH SMBus Controller
Rev: 71
ProgIf: 00
IOMMUGroup: 10

Slot: 00:14.3
Class: ISA bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: FCH LPC Bridge
SVendor: Micro-Star International Co., Ltd. [MSI]
SDevice: FCH LPC Bridge
Rev: 51
ProgIf: 00
IOMMUGroup: 10

Slot: 00:18.0
Class: Host bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Raphael/Granite Ridge Data Fabric; Function 0
ProgIf: 00
IOMMUGroup: 11

Slot: 00:18.1
Class: Host bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Raphael/Granite Ridge Data Fabric; Function 1
ProgIf: 00
IOMMUGroup: 11

Slot: 00:18.2
Class: Host bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Raphael/Granite Ridge Data Fabric; Function 2
ProgIf: 00
IOMMUGroup: 11

Slot: 00:18.3
Class: Host bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Raphael/Granite Ridge Data Fabric; Function 3
ProgIf: 00
IOMMUGroup: 11

Slot: 00:18.4
Class: Host bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Raphael/Granite Ridge Data Fabric; Function 4
ProgIf: 00
IOMMUGroup: 11

Slot: 00:18.5
Class: Host bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Raphael/Granite Ridge Data Fabric; Function 5
ProgIf: 00
IOMMUGroup: 11

Slot: 00:18.6
Class: Host bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Raphael/Granite Ridge Data Fabric; Function 6
ProgIf: 00
IOMMUGroup: 11

Slot: 00:18.7
Class: Host bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Raphael/Granite Ridge Data Fabric; Function 7
ProgIf: 00
IOMMUGroup: 11

Slot: 01:00.0
Class: VGA compatible controller
Vendor: NVIDIA Corporation
Device: AD103 [GeForce RTX 4070 Ti SUPER]
SVendor: PNY
SDevice: AD103 [GeForce RTX 4070 Ti SUPER]
Rev: a1
ProgIf: 00
IOMMUGroup: 12

Slot: 01:00.1
Class: Audio device
Vendor: NVIDIA Corporation
Device: Device 22bb
SVendor: PNY
SDevice: Device 141f
Rev: a1
ProgIf: 00
IOMMUGroup: 12

Slot: 02:00.0
Class: Non-Volatile memory controller
Vendor: Micron/Crucial Technology
Device: T705 NVMe PCIe SSD
SVendor: Micron/Crucial Technology
SDevice: T705 NVMe PCIe SSD
ProgIf: 02
IOMMUGroup: 13

Slot: 03:00.0
Class: PCI bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: 600 Series Chipset PCIe Switch Upstream Port
SVendor: ASMedia Technology Inc.
SDevice: 600 Series Chipset PCIe Switch Upstream Port
Rev: 01
ProgIf: 00
IOMMUGroup: 14

Slot: 04:00.0
Class: PCI bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: 600 Series Chipset PCIe Switch Downstream Port
SVendor: ASMedia Technology Inc.
SDevice: 600 Series Chipset PCIe Switch Downstream Port
Rev: 01
ProgIf: 00
IOMMUGroup: 15

Slot: 04:04.0
Class: PCI bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: 600 Series Chipset PCIe Switch Downstream Port
SVendor: ASMedia Technology Inc.
SDevice: 600 Series Chipset PCIe Switch Downstream Port
Rev: 01
ProgIf: 00
IOMMUGroup: 16

Slot: 04:05.0
Class: PCI bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: 600 Series Chipset PCIe Switch Downstream Port
SVendor: ASMedia Technology Inc.
SDevice: 600 Series Chipset PCIe Switch Downstream Port
Rev: 01
ProgIf: 00
IOMMUGroup: 17

Slot: 04:06.0
Class: PCI bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: 600 Series Chipset PCIe Switch Downstream Port
SVendor: ASMedia Technology Inc.
SDevice: 600 Series Chipset PCIe Switch Downstream Port
Rev: 01
ProgIf: 00
IOMMUGroup: 18

Slot: 04:07.0
Class: PCI bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: 600 Series Chipset PCIe Switch Downstream Port
SVendor: ASMedia Technology Inc.
SDevice: 600 Series Chipset PCIe Switch Downstream Port
Rev: 01
ProgIf: 00
IOMMUGroup: 19

Slot: 04:08.0
Class: PCI bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: 600 Series Chipset PCIe Switch Downstream Port
SVendor: ASMedia Technology Inc.
SDevice: 600 Series Chipset PCIe Switch Downstream Port
Rev: 01
ProgIf: 00
IOMMUGroup: 20

Slot: 04:0c.0
Class: PCI bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: 600 Series Chipset PCIe Switch Downstream Port
SVendor: ASMedia Technology Inc.
SDevice: 600 Series Chipset PCIe Switch Downstream Port
Rev: 01
ProgIf: 00
IOMMUGroup: 21

Slot: 04:0d.0
Class: PCI bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: 600 Series Chipset PCIe Switch Downstream Port
SVendor: ASMedia Technology Inc.
SDevice: 600 Series Chipset PCIe Switch Downstream Port
Rev: 01
ProgIf: 00
IOMMUGroup: 22

Slot: 05:00.0
Class: Non-Volatile memory controller
Vendor: Micron/Crucial Technology
Device: P3 Plus NVMe PCIe SSD (DRAM-less)
SVendor: Micron/Crucial Technology
SDevice: P3 Plus NVMe PCIe SSD (DRAM-less)
Rev: 01
ProgIf: 02
IOMMUGroup: 15

Slot: 06:00.0
Class: Ethernet controller
Vendor: Realtek Semiconductor Co., Ltd.
Device: RTL8125 2.5GbE Controller
SVendor: Micro-Star International Co., Ltd. [MSI]
SDevice: RTL8125 2.5GbE Controller
Rev: 05
ProgIf: 00
IOMMUGroup: 16

Slot: 0a:00.0
Class: PCI bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: 600 Series Chipset PCIe Switch Upstream Port
SVendor: ASMedia Technology Inc.
SDevice: 600 Series Chipset PCIe Switch Upstream Port
Rev: 01
ProgIf: 00
IOMMUGroup: 20

Slot: 0b:00.0
Class: PCI bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: 600 Series Chipset PCIe Switch Downstream Port
SVendor: ASMedia Technology Inc.
SDevice: 600 Series Chipset PCIe Switch Downstream Port
Rev: 01
ProgIf: 00
IOMMUGroup: 20

Slot: 0b:04.0
Class: PCI bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: 600 Series Chipset PCIe Switch Downstream Port
SVendor: ASMedia Technology Inc.
SDevice: 600 Series Chipset PCIe Switch Downstream Port
Rev: 01
ProgIf: 00
IOMMUGroup: 20

Slot: 0b:05.0
Class: PCI bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: 600 Series Chipset PCIe Switch Downstream Port
SVendor: ASMedia Technology Inc.
SDevice: 600 Series Chipset PCIe Switch Downstream Port
Rev: 01
ProgIf: 00
IOMMUGroup: 20

Slot: 0b:06.0
Class: PCI bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: 600 Series Chipset PCIe Switch Downstream Port
SVendor: ASMedia Technology Inc.
SDevice: 600 Series Chipset PCIe Switch Downstream Port
Rev: 01
ProgIf: 00
IOMMUGroup: 20

Slot: 0b:07.0
Class: PCI bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: 600 Series Chipset PCIe Switch Downstream Port
SVendor: ASMedia Technology Inc.
SDevice: 600 Series Chipset PCIe Switch Downstream Port
Rev: 01
ProgIf: 00
IOMMUGroup: 20

Slot: 0b:08.0
Class: PCI bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: 600 Series Chipset PCIe Switch Downstream Port
SVendor: ASMedia Technology Inc.
SDevice: 600 Series Chipset PCIe Switch Downstream Port
Rev: 01
ProgIf: 00
IOMMUGroup: 20

Slot: 0b:0c.0
Class: PCI bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: 600 Series Chipset PCIe Switch Downstream Port
SVendor: ASMedia Technology Inc.
SDevice: 600 Series Chipset PCIe Switch Downstream Port
Rev: 01
ProgIf: 00
IOMMUGroup: 20

Slot: 0b:0d.0
Class: PCI bridge
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: 600 Series Chipset PCIe Switch Downstream Port
SVendor: ASMedia Technology Inc.
SDevice: 600 Series Chipset PCIe Switch Downstream Port
Rev: 01
ProgIf: 00
IOMMUGroup: 20

Slot: 0c:00.0
Class: Non-Volatile memory controller
Vendor: Samsung Electronics Co Ltd
Device: NVMe SSD Controller PM9C1a (DRAM-less)
SVendor: Samsung Electronics Co Ltd
SDevice: NVMe SSD Controller PM9C1a (DRAM-less)
ProgIf: 02
IOMMUGroup: 20

Slot: 0d:00.0
Class: Network controller
Vendor: MEDIATEK Corp.
Device: MT7922 802.11ax PCI Express Wireless Network Adapter
SVendor: MEDIATEK Corp.
SDevice: MT7922 802.11ax PCI Express Wireless Network Adapter
ProgIf: 00
IOMMUGroup: 20

Slot: 12:00.0
Class: USB controller
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: 600 Series Chipset USB 3.2 Controller
SVendor: ASMedia Technology Inc.
SDevice: 600 Series Chipset USB 3.2 Controller
Rev: 01
ProgIf: 30
IOMMUGroup: 20

Slot: 13:00.0
Class: SATA controller
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: 600 Series Chipset SATA Controller
SVendor: ASMedia Technology Inc.
SDevice: 600 Series Chipset SATA Controller
Rev: 01
ProgIf: 01
IOMMUGroup: 20

Slot: 14:00.0
Class: USB controller
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: 600 Series Chipset USB 3.2 Controller
SVendor: ASMedia Technology Inc.
SDevice: 600 Series Chipset USB 3.2 Controller
Rev: 01
ProgIf: 30
IOMMUGroup: 21

Slot: 15:00.0
Class: SATA controller
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: 600 Series Chipset SATA Controller
SVendor: ASMedia Technology Inc.
SDevice: 600 Series Chipset SATA Controller
Rev: 01
ProgIf: 01
IOMMUGroup: 22

Slot: 16:00.0
Class: VGA compatible controller
Vendor: Advanced Micro Devices, Inc. [AMD/ATI]
Device: Raphael
SVendor: Micro-Star International Co., Ltd. [MSI]
SDevice: Raphael
Rev: c9
ProgIf: 00
IOMMUGroup: 23

Slot: 16:00.1
Class: Audio device
Vendor: Advanced Micro Devices, Inc. [AMD/ATI]
Device: Rembrandt Radeon High Definition Audio Controller
SVendor: Micro-Star International Co., Ltd. [MSI]
SDevice: Rembrandt Radeon High Definition Audio Controller
ProgIf: 00
IOMMUGroup: 24

Slot: 16:00.2
Class: Encryption controller
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: VanGogh PSP/CCP
SVendor: Micro-Star International Co., Ltd. [MSI]
SDevice: Family 19h PSP/CCP
ProgIf: 00
IOMMUGroup: 25

Slot: 16:00.3
Class: USB controller
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Raphael/Granite Ridge USB 3.1 xHCI
SVendor: Micro-Star International Co., Ltd. [MSI]
SDevice: Raphael/Granite Ridge USB 3.1 xHCI
ProgIf: 30
IOMMUGroup: 26

Slot: 16:00.4
Class: USB controller
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Raphael/Granite Ridge USB 3.1 xHCI
SVendor: Micro-Star International Co., Ltd. [MSI]
SDevice: Raphael/Granite Ridge USB 3.1 xHCI
ProgIf: 30
IOMMUGroup: 27

Slot: 16:00.6
Class: Audio device
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Family 17h/19h HD Audio Controller
SVendor: Micro-Star International Co., Ltd. [MSI]
SDevice: Family 17h/19h/1ah HD Audio Controller
ProgIf: 00
IOMMUGroup: 28

Slot: 17:00.0
Class: USB controller
Vendor: Advanced Micro Devices, Inc. [AMD]
Device: Raphael/Granite Ridge USB 2.0 xHCI
SVendor: Micro-Star International Co., Ltd. [MSI]
SDevice: Raphael/Granite Ridge USB 2.0 xHCI
ProgIf: 30
IOMMUGroup: 29
 
leesteken said:
It does indeed look like the device is early bound to vfio-pci or passed through to a VM. What are the outputs of grep -i a80d /proc/cmdline /etc/modprobe.d/*.conf and grep -i 0c:00 /etc/pve/qemu-server/*.conf ? Or did you use driverctl or a script from the internet maybe?
Click to expand...
root@proxmox2:~# grep -i a80d /proc/cmdline /etc/modprobe.d/*.conf
root@proxmox2:~# grep -i 0c:00 /etc/pve/qemu-server/*.conf
root@proxmox2:~#
 
Ok, I solved it. I added pcie_acs_override=downstream,multifunction to the GRUB_CMDLINE_LINUX_DEFAULT= line in my Grub config. This put the devices into their own iommu groups. I know it has some security implications, but this is a home server that I alone use. Big thanks to BobhWasatch for pointing me in the right direction!
 
You must log in or register to reply here.
Top Bottom