Hit and miss GIM module runtime configuration

cjones · Aug 24, 2022

I have configured the GIM module following this. I am able to pass VFs through to VMs and it works great. However, I have been trying to modify the runtime options as indicated here. Some seem to work, and some don't.
The default 256M per VF is too low for what I want to use it for, so I am trying to modify the vf_fb option alongside the vf_num option. With each modification of /etc/gim_config, I am reloading the gim module. vf_num takes effect, but vf_fb won't. The VF is staying at 256M.
Not sure if I'm missing something, any suggestions?

dcsapak · Aug 24, 2022

how many vf do you configure? did you try to set the vf_fb option to 0 (so it gets divided automatically) ?
AFAIR this did work for me. also i think i remember you could set those via the module parameters, maybe that works?

can you post the dmesg output after loading the module (and the content of the /etc/gim_config would also be good)

cjones · Aug 25, 2022

dcsapak said:
how many vf do you configure? did you try to set the vf_fb option to 0 (so it gets divided automatically) ?
AFAIR this did work for me. also i think i remember you could set those via the module parameters, maybe that works?

can you post the dmesg output after loading the module (and the content of the /etc/gim_config would also be good)

I am trying to configure 2 VFs and I've tried both an explicit value of 4096 and 0 for vf_fb, same outcome regardless.

Here's dmesg:

Code:

[857541.201982] gim info:(gim_exit:187) Exit AMD open source GIM!
[857541.202337] gim info:(idle_all_adapters:1061) idle adapter[1]
[857541.202696] gim info:(pause_scheduler:124) Stop the Scheduler
[857541.203008] gim warning:(stop_current_vf:2110) runlist is empty, ignore stop
[857541.203630] gim info:(ih_irq_source_disable:629) disabled irq sourceID 0x89
[857541.204282] gim info:(ih_irq_source_disable:629) disabled irq sourceID 0x88
[857541.204886] gim info:(free_iv_ring:246) unmap the iv ring
[857541.205654] gim info:(disable_sriov:322) Disable SRIOV
[857541.206417] pci 0000:01:02.0: Removing from iommu group 48
[857541.207298] pci 0000:01:02.1: Removing from iommu group 49
[857547.749999] gim info:(gim_init:149) Start AMD open source GIM initialization
[857547.750387] gim info:(gim_init:151) GPU IOV MODULE - version 1.1.4
[857547.750756] gim info:(gim_init:154) Copyright (c) 2014-2017 Advanced Micro Devices, Inc. All rights reserved.
[857547.751600] gim info:(parse_config_file:217) AMD GIM fb_option = 0
[857547.752342] gim info:(parse_config_file:217) AMD GIM sched_option = 0
[857547.753014] gim info:(parse_config_file:217) AMD GIM vf_num = 2
[857547.753639] gim info:(parse_config_file:217) AMD GIM pf_fb = 0
[857547.754294] gim info:(parse_config_file:217) AMD GIM vf_fb = 0
[857547.754914] gim info:(parse_config_file:217) AMD GIM sched_interval = 0
[857547.755544] gim info:(parse_config_file:217) AMD GIM sched_interval_us = 0
[857547.756113] gim info:(parse_config_file:217) AMD GIM fb_clear = 0
[857547.756710] gim info:(init_config:341) INIT CONFIG
[857547.760796] gim info:(set_new_adapter:572) curr allocated at 000000007e329256
[857547.761448] gim info:(set_new_adapter:579) SRIOV is supported
[857547.762074] gim info:(set_new_adapter:587) found PCI bridge device
[857547.762668] gim info:(set_new_adapter:588) found: 00:1.0
[857547.763326] gim info:(set_new_adapter:608) mmio_base = 00000000c096b150
[857547.763345] gim info:(set_new_adapter:610) doorbell = 00000000d308e604
[857547.766579] gim info:(set_new_adapter:612) pf.fb_va = 00000000799324ae
[857547.767255] gim info:(sriov_is_ari_enabled:164) PCI_SRIOV_CAP = 0x00000002
[857547.768483] gim info:(sriov_is_ari_enabled:174) PCI_SRIOV_CTRL = 0x00000010
[857547.769078] gim info:(sriov_is_ari_enabled:177) PCI_SRIOV_CTRL_ARI is set --> ARI is supported
[857547.769716] gim info:(program_ari_mode:441) Read bif_strap8 = 0x00000004
[857547.770371] gim info:(program_ari_mode:446) program_ari_mode - Set ARI_Mode = PF_BUS
[857547.771018] gim info:(program_ari_mode:456) Write bif_strap8 = 0x00000004
[857547.771639] gim info:(gim_read_rom_from_reg:181) Reading VBios from ROM
[857547.771820] gim info:(gim_read_vbios:243) VBIOS starts:  0x55, 0xaa
[857547.773119] gim info:(gim_read_vbios:246) VBios size is 0x10000
[857547.773772] gim info:(gim_read_vbios:249) vbios allocated at 00000000455937bb
[857547.774437] gim info:(gim_read_rom_from_reg:181) Reading VBios from ROM
[857547.957341] gim info:(gim_read_vbios:255) BIOS Version Major 0xF Minor 0x31
[857547.958220] gim info:(gim_read_vbios:270) Valid video BIOS image,
[857547.958222] gim info:(gim_read_vbios:271) size = 0x10000, check sum is 0x53b100
[857547.959130] gim info:(set_new_adapter:661) Scheduler Time interval is per-vf from XL
[857547.959131] gim info:(set_new_adapter:662) config file
[857547.959992] gim info:(enable_sriov:298) Enable SRIOV
[857547.960461] gim info:(enable_sriov:299) Enable SRIOV vfs count = 2
[857548.066054] pci 0000:01:02.0: [1002:692f] type 00 class 0x030000
[857548.066533] pci 0000:01:02.0: enabling Extended Tags
[857548.067090] pci 0000:01:02.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[857548.067558] pci 0000:01:02.0: Adding to iommu group 48
[857548.068266] pci 0000:01:02.1: [1002:692f] type 00 class 0x030000
[857548.068711] pci 0000:01:02.1: enabling Extended Tags
[857548.069248] pci 0000:01:02.1: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[857548.069726] pci 0000:01:02.1: Adding to iommu group 49
[857548.070383] gim info:(enumerate_vfs:122) vf found: 01:2.0
[857548.070840] gim info:(enumerate_vfs:122) vf found: 01:2.1
[857548.071346] gim info:(pci_disable_error_reporting:731) Disable error reporting for device: 01:2.0
[857548.071799] gim info:(pci_disable_error_reporting:738) Mask before -> corr = 0x00000000, uncorr = 0x00000000
[857548.072253] gim info:(pci_disable_error_reporting:749) Mask after -> corr = 0x00000000, uncorr = 0x00000000
[857548.072800] gim info:(pci_disable_error_reporting:731) Disable error reporting for device: 01:2.1
[857548.073242] gim info:(pci_disable_error_reporting:738) Mask before -> corr = 0x00000000, uncorr = 0x00000000
[857548.073665] gim info:(pci_disable_error_reporting:749) Mask after -> corr = 0x00000000, uncorr = 0x00000000
[857548.074144] gim info:(pci_gpu_iov_init:87) total_fb_available = 8190
                
[857548.074145] gim info:(pci_gpu_iov_init:88) AMD GIM pci_gpu_iov_init pos = 400
[857548.075595] gim info:(pci_gpu_iov_init:89) AMD GIM pci_gpu_iov_init total_fb_available = 1ffe
[857548.076083] gim info:(init_frame_buffer_partition:218) PCI defined PF FB size = 256 MB
[857548.076618] gim info:(init_frame_buffer_partition:222) PCI defined VF FB size = 256 MB
[857548.077162] gim info:(init_frame_buffer_partition:226) Total FB Available = 8190 MB, CSA = 8 MB
[857548.077603] gim info:(init_frame_buffer_partition:228) Max Remaining FB Size = 8160
[857548.078106] gim info:(init_frame_buffer_partition:240) PF FB size after checking limits from config file = 256MB
[857548.078629] gim info:(init_frame_buffer_partition:243) PF rounded down to nearest 16MB boundary = 256
[857548.079150] gim info:(init_pf_fb:59) total framebuffer available = 1ffe
[857548.079613] gim info:(init_pf_fb:61) pf framebuffer = 100
[857548.080095] gim info:(init_pf_fb:62) total framebuffer consumed = 1efe
[857548.080582] gim info:(init_frame_buffer_partition:251) CSA starts at offset 256MB
[857548.081110] gim info:(init_context_save_area:41) AMD GIM init_context_save_area: base =100 size=1.
[857548.081639] gim info:(init_frame_buffer_partition:257) VF FB base = 272MB (256 + 16)
[857548.082168] gim info:(init_frame_buffer_partition:261) VF FB Size = 7904MB (8160 - 256)
[857548.082719] gim info:(init_fb_static:145) AMD GIM init_fb_static: num_vf = 2, base= 272, total_size=7904, min_size=256
[857548.083285] gim info:(init_fb_static:166) AMD GIM init_fb_static: vf_fb_size = 3952, base= 272
[857548.083774] gim info:(init_fb_static:176) AMD GIM init_fb_static: partition 0 base =272,size= 3952
[857548.084303] gim info:(init_fb_static:176) AMD GIM init_fb_static: partition 1 base =4224,size= 3952
[857548.086109] gim info:(set_new_adapter:707) enable MSI
[857548.086247] gim info:(ih_iv_ring_disable:383) disable iv ring successfully
[857548.088366] gim info:(alloc_iv_ring:99) ih->ivr_num_entries = 256
[857548.089406] gim info:(alloc_iv_ring:102) ih->ivr_size_in_bytes = 4096
[857548.090502] gim info:(alloc_iv_ring:107) ih->ivr_alloc_size_in_bytes = 4100
[857548.091500] gim info:(alloc_iv_ring:110) iv ring page_cnt = 2
[857548.092532] gim info:(alloc_iv_ring:141) ih->ivr_va = 00000000b032a676
[857548.093541] gim info:(alloc_iv_ring:147) ih->ivr_ma.quad_part = 0xffffb000
[857548.094519] gim info:(alloc_iv_ring:151) ih->ivr_wptr_wb = 00000000c47df127
[857548.095501] gim info:(alloc_iv_ring:157) ih->ivr_wptr_wa.quad_part = 0xffffa000
[857548.096516] gim info:(alloc_iv_ring:163) update rptr via doorbell
[857548.097443] gim info:(ih_iv_ring_init:291) ih->rptr_doorbell = 000000006041e279
[857548.097445] gim info:(ih_iv_ring_init:292) ih->rptr_doorbell_offset = 0x1e8
[857548.098456] gim info:(ih_iv_ring_hw_init:184) the physical address of ring buffer: 0xffffb0
[857548.099467] gim info:(ih_iv_ring_setup_rptr:450) write mmBIF_DOORBELL_APER_EN: 0x1
[857548.101544] gim info:(ih_iv_ring_enable:350) ih->ivr_wptr_reg = 0x0
[857548.102491] gim info:(ih_iv_ring_enable:352) ih->ivr_wptr = 0
[857548.103452] gim info:(ih_iv_ring_enable:354) ih->ivr_rptr_reg = 0x0
[857548.104459] gim info:(ih_iv_ring_enable:356) ih->ivr_rptr = 0
[857548.105411] gim info:(ih_iv_ring_enable:358) *(ih->rptr_doorbell) = 0x0
[857548.106336] gim info:(ih_iv_ring_init:299) init iv ring successfully
[857548.106342] gim info:(set_new_adapter:720) init work
[857548.107208] gim info:(set_new_adapter:726) register interrupt
[857548.108192] gim info:(ih_irq_source_enable:583) IH: read 0x00000000 from mask_reg 0x14d1
[857548.109201] gim info:(ih_irq_source_enable:589) IH: write 0x00000001 to mask_reg 0x14d1
[857548.110106] gim info:(ih_irq_source_enable:592) irq sourceID 0x89 get enabled
[857548.111823] gim info:(ih_irq_source_enable:583) IH: read 0x00000001 from mask_reg 0x14d1
[857548.111833] gim info:(ih_irq_source_enable:589) IH: write 0x00000003 to mask_reg 0x14d1
[857548.112762] gim info:(ih_irq_source_enable:592) irq sourceID 0x88 get enabled
[857548.115158] gim info:(gim_clear_all_errors:357) PCIE cap pos 58
[857548.116002] gim info:(gim_clear_all_errors:362) AER ext cap pos 150
[857548.116892] gim info:(resume_scheduler:136) No functions on the runlist.
[857548.116897] gim info:(resume_scheduler:137) Don't need to restart the scheduler
[857548.118630] gim info:(gim_probe:91) AMD GIM probe: pf_count = 1

Here's my latest /etc/gim_config:

Code:

fb_option=0
sched_option=0
vf_num=2
pf_fb=0
vf_fb=0
sched_interval=0
sched_interval_us=0
fb_clear=0

I will experiment with the module parameters.

dcsapak · Aug 25, 2022

Code:

[857548.077162] gim info:(init_frame_buffer_partition:226) Total FB Available = 8190 MB, CSA = 8 MB
[857548.077603] gim info:(init_frame_buffer_partition:228) Max Remaining FB Size = 8160
[857548.078106] gim info:(init_frame_buffer_partition:240) PF FB size after checking limits from config file = 256MB
[857548.078629] gim info:(init_frame_buffer_partition:243) PF rounded down to nearest 16MB boundary = 256
[857548.079150] gim info:(init_pf_fb:59) total framebuffer available = 1ffe
[857548.079613] gim info:(init_pf_fb:61) pf framebuffer = 100
[857548.080095] gim info:(init_pf_fb:62) total framebuffer consumed = 1efe
[857548.080582] gim info:(init_frame_buffer_partition:251) CSA starts at offset 256MB
[857548.081110] gim info:(init_context_save_area:41) AMD GIM init_context_save_area: base =100 size=1.
[857548.081639] gim info:(init_frame_buffer_partition:257) VF FB base = 272MB (256 + 16)
[857548.082168] gim info:(init_frame_buffer_partition:261) VF FB Size = 7904MB (8160 - 256)
[857548.082719] gim info:(init_fb_static:145) AMD GIM init_fb_static: num_vf = 2, base= 272, total_size=7904, min_size=256
[857548.083285] gim info:(init_fb_static:166) AMD GIM init_fb_static: vf_fb_size = 3952, base= 272

seems to work though?

it says (summed) VF FB size = 7904MB and then 'vf_fb_size = 3952' (7902/2 = 3952)

cjones · Aug 26, 2022

dcsapak said:

Code:

[857548.077162] gim info:(init_frame_buffer_partition:226) Total FB Available = 8190 MB, CSA = 8 MB
[857548.077603] gim info:(init_frame_buffer_partition:228) Max Remaining FB Size = 8160
[857548.078106] gim info:(init_frame_buffer_partition:240) PF FB size after checking limits from config file = 256MB
[857548.078629] gim info:(init_frame_buffer_partition:243) PF rounded down to nearest 16MB boundary = 256
[857548.079150] gim info:(init_pf_fb:59) total framebuffer available = 1ffe
[857548.079613] gim info:(init_pf_fb:61) pf framebuffer = 100
[857548.080095] gim info:(init_pf_fb:62) total framebuffer consumed = 1efe
[857548.080582] gim info:(init_frame_buffer_partition:251) CSA starts at offset 256MB
[857548.081110] gim info:(init_context_save_area:41) AMD GIM init_context_save_area: base =100 size=1.
[857548.081639] gim info:(init_frame_buffer_partition:257) VF FB base = 272MB (256 + 16)
[857548.082168] gim info:(init_frame_buffer_partition:261) VF FB Size = 7904MB (8160 - 256)
[857548.082719] gim info:(init_fb_static:145) AMD GIM init_fb_static: num_vf = 2, base= 272, total_size=7904, min_size=256
[857548.083285] gim info:(init_fb_static:166) AMD GIM init_fb_static: vf_fb_size = 3952, base= 272

seems to work though?

it says (summed) VF FB size = 7904MB and then 'vf_fb_size = 3952' (7902/2 = 3952)

That's what I would've thought, but listing the PCI details on the VFs reveals that the memory size is still at 256M. This is the same within the guest that a VF is passed-through to.

The PF:

Code:

lspci -s 01:00 -vv
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XT GL [FirePro S7150] (prog-if 00 [VGA controller])
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XT GL [FirePro S7150]
        Physical Slot: 4
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 91
        NUMA node: 0
        IOMMU group: 40
        Region 0: Memory at 383fe0000000 (64-bit, prefetchable) [size=256M]
        Region 2: Memory at 383ff4000000 (64-bit, prefetchable) [size=2M]
        Region 4: I/O ports at e000 [size=256]
        Region 5: Memory at fb400000 (32-bit, non-prefetchable) [size=256K]
        Expansion ROM at fb440000 [disabled] [size=128K]
        Capabilities: [48] Vendor Specific Information: Len=08 <?>
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
                        RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s (ok), Width x8 (downgraded)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
                         EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [a0] MSI: Enable+ Count=1/4 Maskable+ 64bit+
                Address: 00000000fee00ab8  Data: 0000
                Masking: 0000000e  Pending: 00000000
        Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP+ SDES- TLP+ FCP- CmpltTO+ CmpltAbrt- UnxCmplt+ RxOF- MalfTLP+ ECRC+ UnsupReq+ ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP+ BadDLLP+ Rollover+ Timeout+ AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [200 v1] Physical Resizable BAR
                BAR 0: current size: 256MB, supported: 256MB 512MB 1GB 2GB 4GB 8GB
        Capabilities: [270 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Capabilities: [2b0 v1] Address Translation Service (ATS)
                ATSCap: Invalidate Queue Depth: 00
                ATSCtl: Enable+, Smallest Translation Unit: 00
        Capabilities: [2c0 v1] Page Request Interface (PRI)
                PRICtl: Enable- Reset-
                PRISta: RF- UPRGI- Stopped+
                Page Request Capacity: 00000020, Page Request Allocation: 00000000
        Capabilities: [2d0 v1] Process Address Space ID (PASID)
                PASIDCap: Exec+ Priv+, Max PASID Width: 10
                PASIDCtl: Enable- Exec- Priv-
        Capabilities: [328 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [330 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration-, Interrupt Message Number: 000
                IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy+
                IOVSta: Migration-
                Initial VFs: 16, Total VFs: 16, Number of VFs: 2, Function Dependency Link: 00
                VF offset: 16, stride: 1, Device ID: 692f
                Supported Page Size: 00000553, System Page Size: 00000001
                Region 0: Memory at 0000383ee0000000 (64-bit, prefetchable)
                Region 2: Memory at 0000383ff0000000 (64-bit, prefetchable)
                Region 5: Memory at f7400000 (32-bit, non-prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [400 v1] Vendor Specific Information: ID=0002 Rev=1 Len=070 <?>
        Kernel modules: amdgpu

One of the VFs:

Code:

lspci -s 01:02 -vv
01:02.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] (prog-if 00 [VGA controller])
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        NUMA node: 0
        IOMMU group: 48
        Region 0: Memory at 383ee0000000 (64-bit, prefetchable) [virtual] [size=256M]
        Region 2: Memory at 383ff0000000 (64-bit, prefetchable) [virtual] [size=4M]
        Region 5: Memory at f7400000 (32-bit, non-prefetchable) [virtual] [size=4M]
        Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed unknown (downgraded), Width x0 (downgraded)
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [a0] MSI: Enable- Count=1/4 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: ffffffff ffffffff ffffffff ffffffff
        Capabilities: [2b0 v1] Address Translation Service (ATS)
                ATSCap: Invalidate Queue Depth: 00
                ATSCtl: Enable+, Smallest Translation Unit: 00
        Capabilities: [328 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu

dcsapak · Aug 26, 2022

is that on the host? did you try to check what is available if you pass it through on a guest?

cjones said:
[857548.076618] gim info:(init_frame_buffer_partition:222) PCI defined VF FB size = 256 MB

i think its just the pci defined size

cjones · Aug 27, 2022

dcsapak said:
is that on the host? did you try to check what is available if you pass it through on a guest?

i think its just the pci defined size

The guest was showing the same thing, but I ended up running glxinfo and it shows the right amount of video memory. I guess I misunderstand the lspci output.

Thank you for your help.

Search

Search

Hit and miss GIM module runtime configuration

cjones

Member

dcsapak

Proxmox Staff Member

cjones

Member

dcsapak

Proxmox Staff Member

cjones

Member

dcsapak

Proxmox Staff Member

cjones

Member