[TUTORIAL] Configuring Fusion-Io (SanDisk) ioDrive, ioDrive2, ioScale and ioScale2 cards with Proxmox

[mrwizardno2]

Heads up: Had some trouble building DKMS with the PVE 8.1 release and new kernel. Ended up manually building and loading the module with insmod to get my lab back up.

 

[macgro1]

it's been my case... as well, rectified:
git clone https://github.com/RemixVSL/iomemory-vsl
cd iomemory-vsl/
sudo make dkms

 

[MJFlanigan]

How did you get it to run on proxmox 8?
What kernel are you using?

I am currently on proxmox 7.4-17/513c62be (running kernel: 6.1.15-1-pve)

but i am afraid to upgrade ...
Installing them was a nightmare... and i don't want to start all that again...

I would like to know the answer to this as well. I just bought a card from dell and I would like to use it in my proxmox server but I have no idea based on these instructions how to install this thing at all. I read though the repo and it's still unclear how to do this on proxmox. Is there a new guide for 8 out yet?

 

[Jan Panoch]

hello,

here is a updated guide for Proxmox v8.x (successfully build on proxmox 8.1.4 with kernel 6.5.11-7-pve):

apt update && apt --assume-yes install git zip unzip pve-headers proxmox-headers-`uname -r` && apt --assume-yes upgrade && apt --assume-yes autoremove && \
mkdir /home/temp  && cd /home/temp  && \
git clone https://github.com/RemixVSL/iomemory-vsl && \
wget -O fio-common_3.2.16.1731-1.0_amd64.deb https://www.dropbox.com/s/pd2ohfaufhwqc34/fio-common_3.2.16.1731-1.0_amd64.deb?dl=1 && \
wget -O fio-firmware-fusion_3.2.16.20180821-1_all.deb https://www.dropbox.com/s/kcn5agi6lyikicf/fio-firmware-fusion_3.2.16.20180821-1_all.deb?dl=1 && \
wget -O fio-sysvinit_3.2.16.1731-1.0_all.deb https://www.dropbox.com/s/g39l6lg9of6eqze/fio-sysvinit_3.2.16.1731-1.0_all.deb?dl=1 && \
wget -O fio-util_3.2.16.1731-1.0_amd64.deb https://www.dropbox.com/s/57huby17mteg6wp/fio-util_3.2.16.1731-1.0_amd64.deb?dl=1 && \
apt update && apt --assume-yes install gcc fakeroot build-essential debhelper rsync dkms && \
cp -r /home/temp/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16 /usr/src/ && \
mkdir -p /var/lib/dkms/iomemory-vsl/3.2.16/build && \
ln -s /usr/src/iomemory-vsl-3.2.16 /var/lib/dkms/iomemory-vsl/3.2.16/source && \
dkms build -m iomemory-vsl -v 3.2.16 && \
dkms install -m iomemory-vsl -v 3.2.16 && \
modprobe iomemory-vsl && \
cd /home/temp && \
dpkg -i fio-firmware-fusion_3.2.16.20180821-1_all.deb fio-util_3.2.16.1731-1.0_amd64.deb fio-sysvinit_3.2.16.1731-1.0_all.deb fio-common_3.2.16.1731-1.0_amd64.deb

btw. if you stayed - like me - on older kernel 5.15 with proxmox 8, because iomemory-vsl cannot be compiled with newer one, you have to install proper headers and compile driver a bit different way:

apt install proxmox-headers-6.5.11-7-pve
dkms build -m iomemory-vsl -v 3.2.16 -k 6.5.11-7-pve
dkms install -m iomemory-vsl -v 3.2.16 -k 6.5.11-7-pve

There are just minor changes from 7.x guide, it looks like the real work was done by RemixVSL who updated driver sources to support newer kernels..

Jan

 

[Jan Panoch]

I would like to know the answer to this as well. I just bought a card from dell and I would like to use it in my proxmox server but I have no idea based on these instructions how to install this thing at all. I read though the repo and it's still unclear how to do this on proxmox. Is there a new guide for 8 out yet?

I just posted updated guide for Proxmox 8.x

 

[Jan Panoch]

hmm, proxmox kernel proxmox-kernel-6.8.4-2-pve-signed has some changes, need to update driver sources from github repo.
i prepare small script to update:

#!/bin/sh
cd /home/temp
rm -Rf iomemory-vsl
git clone https://github.com/RemixVSL/iomemory-vsl
cp -r /home/temp/iomemory-vsl/root/usr/src/iomemory-vsl-3.2.16 /usr/src/

 

[Jan Panoch]

EDIT: fixed - i forgot to add iommu=pt to boot parameters.

There is a problem with kernel 6.8.x

snippet from dmesg

[    8.227469] <6>fioinf Copyright (c) 2006-2014 Fusion-io, Inc. (acquired by SanDisk Corp. 2014)
[   13.058621] <6>fioinf ioDrive 0000:08:00.0: Found device fct0 (Fusion-io ioScale 3.20TB 0000:08:00.0) on pipeline 0
[   13.059061] <3>fioerr Fusion-io ioScale 3.20TB 0000:08:00.0: failed to map append request
[   13.059063] <3>fioerr Fusion-io ioScale 3.20TB 0000:08:00.0: request page program 000000007bca9ea9 failed -22
[   18.862619] <6>fioinf ioDrive 0000:08:00.0: Found device fct0 (Fusion-io ioScale 3.20TB 0000:08:00.0) on pipeline 0
[   18.862976] <6>fioinf Fusion-io ioScale 3.20TB 0000:08:00.0: probed fct0
[   18.862979] <6>fioinf Fusion-io ioScale 3.20TB 0000:08:00.0: Attaching explicitly disabled
[   18.862987] <3>fioerr Fusion-io ioScale 3.20TB 0000:08:00.0: auto attach failed with error EINVAL: Invalid argument
[   35.664271] <6>fioinf Fusion-io ioScale 3.20TB 0000:08:00.0: IOCTL 0xc030684d unsupported in minimal mode
[   35.664419] <6>fioinf Fusion-io ioScale 3.20TB 0000:08:00.0: IOCTL 0x80286828 unsupported in minimal mode

fio-status -a

Found 1 ioMemory device in this system
Driver version: 3.2.16 build 1731

Adapter: Single Controller Adapter
    Fusion-io ioScale 3.20TB, Product Number:F11-002-3T20-CS-0001, SN:1439D178B, FIO SN:1439D178B
    ioDrive2 Adapter Controller, PN:PA005064001
    External Power: NOT connected
    PCIe Power limit threshold: 24.75W
    PCIe slot available power: unavailable
    PCIe negotiated link: 4 lanes at 5.0 Gt/sec each, 2000.00 MBytes/sec total
    Connected ioMemory modules:
      fct0:    Product Number:F11-002-3T20-CS-0001, SN:1439D178B

fct0    Status unknown: Driver is in MINIMAL MODE:
        Device has a hardware failure
    ioDrive2 Adapter Controller, Product Number:F11-002-3T20-CS-0001, SN:1439D178B
!! --->    There are active errors or warnings on this device!  Read below for details.
    ioDrive2 Adapter Controller, PN:PA005064001
    SMP(AVR) Versions: App Version: 1.0.21.0, Boot Version: 1.0.6.1
    Located in slot 0 Center of ioDrive2 Adapter Controller SN:1439D178B
    Powerloss protection: not available
    PCI:08:00.0, Slot Number:1
    Vendor:1aed, Device:2001, Sub vendor:1aed, Sub device:2001
    Firmware v7.1.17, rev 116786 Public
    Geometry and capacity information not available.
    Format: not low-level formatted
    PCIe slot available power: unavailable
    PCIe negotiated link: 4 lanes at 5.0 Gt/sec each, 2000.00 MBytes/sec total
    Internal temperature: 46.76 degC, max 46.76 degC
    Internal voltage: avg 1.02V, max 1.02V
    Aux voltage: avg 2.48V, max 2.48V
    Rated PBW: 20.00 PB
    Lifetime data volumes:
       Physical bytes written: 0
       Physical bytes read   : 0
    RAM usage:
       Current: 0 bytes
       Peak   : 0 bytes

    ACTIVE WARNINGS:
        The ioMemory is currently running in a minimal state.

kernel 6.5.x is working normally (tested on 6.5.13-5-pve)

i submitted a issue on RemixVSL github https://github.com/RemixVSL/iomemory-vsl/issues/129

 

[charmety]

Hi,
Firstly thank you for all of your time spending on this tutorial, it really helps me alot in setting up promox, I usually use esxi but couldn't get vgpu working on esxi. I've managed to install vgpu and iodrive2 to promox.

Here's the result of fio-status -a
----------------------------------------------------------------------------------------------
Found 1 ioMemory device in this system
Driver version: 3.2.16 build 1731

Adapter: Single Controller Adapter
Fusion-io ioDrive2 1.205TB, Product Number:F00-001-1T20-CS-0001, SN:1308D02F0, FIO SN:1308D02F0
ioDrive2 Adapter Controller, PNA004137045
External Power: NOT connected
PCIe Bus voltage: avg 11.94V
PCIe Bus current: avg 0.75A
PCIe Bus power: avg 8.81W
PCIe Power limit threshold: 24.75W
PCIe slot available power: unavailable
PCIe negotiated link: 4 lanes at 5.0 Gt/sec each, 2000.00 MBytes/sec total
Connected ioMemory modules:
fct0: Product Number:F00-001-1T20-CS-0001, SN:1308D02F0

fct0 Attached
ioDrive2 Adapter Controller, Product Number:F00-001-1T20-CS-0001, SN:1308D02F0
ioDrive2 Adapter Controller, PNA004137045
SMP(AVR) Versions: App Version: 1.0.23.0, Boot Version: 0.0.8.1
Located in slot 0 Center of ioDrive2 Adapter Controller SN:1308D02F0
Powerloss protection: protected
PCI:04:00.0
Vendor:1aed, Device:2001, Sub vendor:1aed, Sub device:2001
Firmware v7.1.13, rev 109322 Public
1205.00 GBytes device size
Format: v500, 2353515625 sectors of 512 bytes
PCIe slot available power: unavailable
PCIe negotiated link: 4 lanes at 5.0 Gt/sec each, 2000.00 MBytes/sec total
Internal temperature: 44.79 degC, max 47.25 degC
Internal voltage: avg 1.02V, max 1.02V
Aux voltage: avg 2.48V, max 2.49V
Reserve space status: Healthy; Reserves: 100.00%, warn at 10.00%
Active media: 100.00%
Rated PBW: 17.00 PB, 99.65% remaining
Lifetime data volumes:
Physical bytes written: 59,096,468,533,320
Physical bytes read : 91,554,925,116,096
RAM usage:
Current: 389,441,280 bytes
Peak : 526,804,480 bytes
Contained VSUs:
fioa: ID:0, UUID:a3deba39-14f7-403f-8df9-864e632f562f

fioa State: Online, Type: block device
ID:0, UUID:a3deba39-14f7-403f-8df9-864e632f562f
1205.00 GBytes device size
Format: 2353515625 sectors of 512 bytes

----------------------------------------------------------------------------------------------

I already fio-format and use sgdisk --zap-all /dev/fioa
I also can see the iodrive with lsblk

root@pve:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 136.7G 0 disk
├─sda1 8:1 0 1007K 0 part
├─sda2 8:2 0 1G 0 part
└─sda3 8:3 0 135.7G 0 part
├─pve-swap 252:0 0 8G 0 lvm [SWAP]
├─pve-root 252:1 0 43.9G 0 lvm /
├─pve-data_tmeta 252:2 0 1G 0 lvm
│ └─pve-data-tpool 252:4 0 65.8G 0 lvm
│ └─pve-data 252:5 0 65.8G 1 lvm
└─pve-data_tdata 252:3 0 65.8G 0 lvm
└─pve-data-tpool 252:4 0 65.8G 0 lvm
└─pve-data 252:5 0 65.8G 1 lvm
sdb 8:16 0 931.5G 0 disk
├─sdb1 8:17 0 931.5G 0 part
└─sdb9 8:25 0 8M 0 part
sr0 11:0 1 1024M 0 rom
zd0 230:0 0 4G 0 disk
fioa 251:0 0 1.1T 0 disk <<< THIS ONE

----------------------------------------------------------------------------------------------
But I still can't see and use the iodrive in promox.



Am I missing or skipping steps in adding the drive?

Thank you

 

[kogare]

Hi,
I just skimmed previous page, I won’t show on GUI page but you can still can use the drive with the console.

 

[Orion Pax]

That is right, on my server they never showed up on the GIU.

You can however, make a zfs pool on them using the consiole and i think that will show up in the GUI.
I have a different server now so i can't really check for you anymore.