Hit and miss GIM module runtime configuration

C

cjones

Active Member
Proxmox Subscriber
Jun 28, 2019
27
1
43
I have configured the GIM module following this. I am able to pass VFs through to VMs and it works great. However, I have been trying to modify the runtime options as indicated here. Some seem to work, and some don't.
The default 256M per VF is too low for what I want to use it for, so I am trying to modify the vf_fb option alongside the vf_num option. With each modification of /etc/gim_config, I am reloading the gim module. vf_num takes effect, but vf_fb won't. The VF is staying at 256M.
Not sure if I'm missing something, any suggestions?
 
how many vf do you configure? did you try to set the vf_fb option to 0 (so it gets divided automatically) ?
AFAIR this did work for me. also i think i remember you could set those via the module parameters, maybe that works?

can you post the dmesg output after loading the module (and the content of the /etc/gim_config would also be good)
 
dcsapak said:
how many vf do you configure? did you try to set the vf_fb option to 0 (so it gets divided automatically) ?
AFAIR this did work for me. also i think i remember you could set those via the module parameters, maybe that works?

can you post the dmesg output after loading the module (and the content of the /etc/gim_config would also be good)
Click to expand...
I am trying to configure 2 VFs and I've tried both an explicit value of 4096 and 0 for vf_fb, same outcome regardless.

Here's dmesg:

Code: 
[857541.201982] gim info:(gim_exit:187) Exit AMD open source GIM!
[857541.202337] gim info:(idle_all_adapters:1061) idle adapter[1]
[857541.202696] gim info:(pause_scheduler:124) Stop the Scheduler
[857541.203008] gim warning:(stop_current_vf:2110) runlist is empty, ignore stop
[857541.203630] gim info:(ih_irq_source_disable:629) disabled irq sourceID 0x89
[857541.204282] gim info:(ih_irq_source_disable:629) disabled irq sourceID 0x88
[857541.204886] gim info:(free_iv_ring:246) unmap the iv ring
[857541.205654] gim info:(disable_sriov:322) Disable SRIOV
[857541.206417] pci 0000:01:02.0: Removing from iommu group 48
[857541.207298] pci 0000:01:02.1: Removing from iommu group 49
[857547.749999] gim info:(gim_init:149) Start AMD open source GIM initialization
[857547.750387] gim info:(gim_init:151) GPU IOV MODULE - version 1.1.4
[857547.750756] gim info:(gim_init:154) Copyright (c) 2014-2017 Advanced Micro Devices, Inc. All rights reserved.
[857547.751600] gim info:(parse_config_file:217) AMD GIM fb_option = 0
[857547.752342] gim info:(parse_config_file:217) AMD GIM sched_option = 0
[857547.753014] gim info:(parse_config_file:217) AMD GIM vf_num = 2
[857547.753639] gim info:(parse_config_file:217) AMD GIM pf_fb = 0
[857547.754294] gim info:(parse_config_file:217) AMD GIM vf_fb = 0
[857547.754914] gim info:(parse_config_file:217) AMD GIM sched_interval = 0
[857547.755544] gim info:(parse_config_file:217) AMD GIM sched_interval_us = 0
[857547.756113] gim info:(parse_config_file:217) AMD GIM fb_clear = 0
[857547.756710] gim info:(init_config:341) INIT CONFIG
[857547.760796] gim info:(set_new_adapter:572) curr allocated at 000000007e329256
[857547.761448] gim info:(set_new_adapter:579) SRIOV is supported
[857547.762074] gim info:(set_new_adapter:587) found PCI bridge device
[857547.762668] gim info:(set_new_adapter:588) found: 00:1.0
[857547.763326] gim info:(set_new_adapter:608) mmio_base = 00000000c096b150
[857547.763345] gim info:(set_new_adapter:610) doorbell = 00000000d308e604
[857547.766579] gim info:(set_new_adapter:612) pf.fb_va = 00000000799324ae
[857547.767255] gim info:(sriov_is_ari_enabled:164) PCI_SRIOV_CAP = 0x00000002
[857547.768483] gim info:(sriov_is_ari_enabled:174) PCI_SRIOV_CTRL = 0x00000010
[857547.769078] gim info:(sriov_is_ari_enabled:177) PCI_SRIOV_CTRL_ARI is set --> ARI is supported
[857547.769716] gim info:(program_ari_mode:441) Read bif_strap8 = 0x00000004
[857547.770371] gim info:(program_ari_mode:446) program_ari_mode - Set ARI_Mode = PF_BUS
[857547.771018] gim info:(program_ari_mode:456) Write bif_strap8 = 0x00000004
[857547.771639] gim info:(gim_read_rom_from_reg:181) Reading VBios from ROM
[857547.771820] gim info:(gim_read_vbios:243) VBIOS starts:  0x55, 0xaa
[857547.773119] gim info:(gim_read_vbios:246) VBios size is 0x10000
[857547.773772] gim info:(gim_read_vbios:249) vbios allocated at 00000000455937bb
[857547.774437] gim info:(gim_read_rom_from_reg:181) Reading VBios from ROM
[857547.957341] gim info:(gim_read_vbios:255) BIOS Version Major 0xF Minor 0x31
[857547.958220] gim info:(gim_read_vbios:270) Valid video BIOS image,
[857547.958222] gim info:(gim_read_vbios:271) size = 0x10000, check sum is 0x53b100
[857547.959130] gim info:(set_new_adapter:661) Scheduler Time interval is per-vf from XL
[857547.959131] gim info:(set_new_adapter:662) config file
[857547.959992] gim info:(enable_sriov:298) Enable SRIOV
[857547.960461] gim info:(enable_sriov:299) Enable SRIOV vfs count = 2
[857548.066054] pci 0000:01:02.0: [1002:692f] type 00 class 0x030000
[857548.066533] pci 0000:01:02.0: enabling Extended Tags
[857548.067090] pci 0000:01:02.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[857548.067558] pci 0000:01:02.0: Adding to iommu group 48
[857548.068266] pci 0000:01:02.1: [1002:692f] type 00 class 0x030000
[857548.068711] pci 0000:01:02.1: enabling Extended Tags
[857548.069248] pci 0000:01:02.1: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[857548.069726] pci 0000:01:02.1: Adding to iommu group 49
[857548.070383] gim info:(enumerate_vfs:122) vf found: 01:2.0
[857548.070840] gim info:(enumerate_vfs:122) vf found: 01:2.1
[857548.071346] gim info:(pci_disable_error_reporting:731) Disable error reporting for device: 01:2.0
[857548.071799] gim info:(pci_disable_error_reporting:738) Mask before -> corr = 0x00000000, uncorr = 0x00000000
[857548.072253] gim info:(pci_disable_error_reporting:749) Mask after -> corr = 0x00000000, uncorr = 0x00000000
[857548.072800] gim info:(pci_disable_error_reporting:731) Disable error reporting for device: 01:2.1
[857548.073242] gim info:(pci_disable_error_reporting:738) Mask before -> corr = 0x00000000, uncorr = 0x00000000
[857548.073665] gim info:(pci_disable_error_reporting:749) Mask after -> corr = 0x00000000, uncorr = 0x00000000
[857548.074144] gim info:(pci_gpu_iov_init:87) total_fb_available = 8190
                
[857548.074145] gim info:(pci_gpu_iov_init:88) AMD GIM pci_gpu_iov_init pos = 400
[857548.075595] gim info:(pci_gpu_iov_init:89) AMD GIM pci_gpu_iov_init total_fb_available = 1ffe
[857548.076083] gim info:(init_frame_buffer_partition:218) PCI defined PF FB size = 256 MB
[857548.076618] gim info:(init_frame_buffer_partition:222) PCI defined VF FB size = 256 MB
[857548.077162] gim info:(init_frame_buffer_partition:226) Total FB Available = 8190 MB, CSA = 8 MB
[857548.077603] gim info:(init_frame_buffer_partition:228) Max Remaining FB Size = 8160
[857548.078106] gim info:(init_frame_buffer_partition:240) PF FB size after checking limits from config file = 256MB
[857548.078629] gim info:(init_frame_buffer_partition:243) PF rounded down to nearest 16MB boundary = 256
[857548.079150] gim info:(init_pf_fb:59) total framebuffer available = 1ffe
[857548.079613] gim info:(init_pf_fb:61) pf framebuffer = 100
[857548.080095] gim info:(init_pf_fb:62) total framebuffer consumed = 1efe
[857548.080582] gim info:(init_frame_buffer_partition:251) CSA starts at offset 256MB
[857548.081110] gim info:(init_context_save_area:41) AMD GIM init_context_save_area: base =100 size=1.
[857548.081639] gim info:(init_frame_buffer_partition:257) VF FB base = 272MB (256 + 16)
[857548.082168] gim info:(init_frame_buffer_partition:261) VF FB Size = 7904MB (8160 - 256)
[857548.082719] gim info:(init_fb_static:145) AMD GIM init_fb_static: num_vf = 2, base= 272, total_size=7904, min_size=256
[857548.083285] gim info:(init_fb_static:166) AMD GIM init_fb_static: vf_fb_size = 3952, base= 272
[857548.083774] gim info:(init_fb_static:176) AMD GIM init_fb_static: partition 0 base =272,size= 3952
[857548.084303] gim info:(init_fb_static:176) AMD GIM init_fb_static: partition 1 base =4224,size= 3952
[857548.086109] gim info:(set_new_adapter:707) enable MSI
[857548.086247] gim info:(ih_iv_ring_disable:383) disable iv ring successfully
[857548.088366] gim info:(alloc_iv_ring:99) ih->ivr_num_entries = 256
[857548.089406] gim info:(alloc_iv_ring:102) ih->ivr_size_in_bytes = 4096
[857548.090502] gim info:(alloc_iv_ring:107) ih->ivr_alloc_size_in_bytes = 4100
[857548.091500] gim info:(alloc_iv_ring:110) iv ring page_cnt = 2
[857548.092532] gim info:(alloc_iv_ring:141) ih->ivr_va = 00000000b032a676
[857548.093541] gim info:(alloc_iv_ring:147) ih->ivr_ma.quad_part = 0xffffb000
[857548.094519] gim info:(alloc_iv_ring:151) ih->ivr_wptr_wb = 00000000c47df127
[857548.095501] gim info:(alloc_iv_ring:157) ih->ivr_wptr_wa.quad_part = 0xffffa000
[857548.096516] gim info:(alloc_iv_ring:163) update rptr via doorbell
[857548.097443] gim info:(ih_iv_ring_init:291) ih->rptr_doorbell = 000000006041e279
[857548.097445] gim info:(ih_iv_ring_init:292) ih->rptr_doorbell_offset = 0x1e8
[857548.098456] gim info:(ih_iv_ring_hw_init:184) the physical address of ring buffer: 0xffffb0
[857548.099467] gim info:(ih_iv_ring_setup_rptr:450) write mmBIF_DOORBELL_APER_EN: 0x1
[857548.101544] gim info:(ih_iv_ring_enable:350) ih->ivr_wptr_reg = 0x0
[857548.102491] gim info:(ih_iv_ring_enable:352) ih->ivr_wptr = 0
[857548.103452] gim info:(ih_iv_ring_enable:354) ih->ivr_rptr_reg = 0x0
[857548.104459] gim info:(ih_iv_ring_enable:356) ih->ivr_rptr = 0
[857548.105411] gim info:(ih_iv_ring_enable:358) *(ih->rptr_doorbell) = 0x0
[857548.106336] gim info:(ih_iv_ring_init:299) init iv ring successfully
[857548.106342] gim info:(set_new_adapter:720) init work
[857548.107208] gim info:(set_new_adapter:726) register interrupt
[857548.108192] gim info:(ih_irq_source_enable:583) IH: read 0x00000000 from mask_reg 0x14d1
[857548.109201] gim info:(ih_irq_source_enable:589) IH: write 0x00000001 to mask_reg 0x14d1
[857548.110106] gim info:(ih_irq_source_enable:592) irq sourceID 0x89 get enabled
[857548.111823] gim info:(ih_irq_source_enable:583) IH: read 0x00000001 from mask_reg 0x14d1
[857548.111833] gim info:(ih_irq_source_enable:589) IH: write 0x00000003 to mask_reg 0x14d1
[857548.112762] gim info:(ih_irq_source_enable:592) irq sourceID 0x88 get enabled
[857548.115158] gim info:(gim_clear_all_errors:357) PCIE cap pos 58
[857548.116002] gim info:(gim_clear_all_errors:362) AER ext cap pos 150
[857548.116892] gim info:(resume_scheduler:136) No functions on the runlist.
[857548.116897] gim info:(resume_scheduler:137) Don't need to restart the scheduler
[857548.118630] gim info:(gim_probe:91) AMD GIM probe: pf_count = 1

Here's my latest /etc/gim_config:
Code: 
fb_option=0
sched_option=0
vf_num=2
pf_fb=0
vf_fb=0
sched_interval=0
sched_interval_us=0
fb_clear=0

I will experiment with the module parameters.
 
Code: 
[857548.077162] gim info:(init_frame_buffer_partition:226) Total FB Available = 8190 MB, CSA = 8 MB
[857548.077603] gim info:(init_frame_buffer_partition:228) Max Remaining FB Size = 8160
[857548.078106] gim info:(init_frame_buffer_partition:240) PF FB size after checking limits from config file = 256MB
[857548.078629] gim info:(init_frame_buffer_partition:243) PF rounded down to nearest 16MB boundary = 256
[857548.079150] gim info:(init_pf_fb:59) total framebuffer available = 1ffe
[857548.079613] gim info:(init_pf_fb:61) pf framebuffer = 100
[857548.080095] gim info:(init_pf_fb:62) total framebuffer consumed = 1efe
[857548.080582] gim info:(init_frame_buffer_partition:251) CSA starts at offset 256MB
[857548.081110] gim info:(init_context_save_area:41) AMD GIM init_context_save_area: base =100 size=1.
[857548.081639] gim info:(init_frame_buffer_partition:257) VF FB base = 272MB (256 + 16)
[857548.082168] gim info:(init_frame_buffer_partition:261) VF FB Size = 7904MB (8160 - 256)
[857548.082719] gim info:(init_fb_static:145) AMD GIM init_fb_static: num_vf = 2, base= 272, total_size=7904, min_size=256
[857548.083285] gim info:(init_fb_static:166) AMD GIM init_fb_static: vf_fb_size = 3952, base= 272
seems to work though?

it says (summed) VF FB size = 7904MB and then 'vf_fb_size = 3952' (7902/2 = 3952)
 
dcsapak said:
Code: 
[857548.077162] gim info:(init_frame_buffer_partition:226) Total FB Available = 8190 MB, CSA = 8 MB
[857548.077603] gim info:(init_frame_buffer_partition:228) Max Remaining FB Size = 8160
[857548.078106] gim info:(init_frame_buffer_partition:240) PF FB size after checking limits from config file = 256MB
[857548.078629] gim info:(init_frame_buffer_partition:243) PF rounded down to nearest 16MB boundary = 256
[857548.079150] gim info:(init_pf_fb:59) total framebuffer available = 1ffe
[857548.079613] gim info:(init_pf_fb:61) pf framebuffer = 100
[857548.080095] gim info:(init_pf_fb:62) total framebuffer consumed = 1efe
[857548.080582] gim info:(init_frame_buffer_partition:251) CSA starts at offset 256MB
[857548.081110] gim info:(init_context_save_area:41) AMD GIM init_context_save_area: base =100 size=1.
[857548.081639] gim info:(init_frame_buffer_partition:257) VF FB base = 272MB (256 + 16)
[857548.082168] gim info:(init_frame_buffer_partition:261) VF FB Size = 7904MB (8160 - 256)
[857548.082719] gim info:(init_fb_static:145) AMD GIM init_fb_static: num_vf = 2, base= 272, total_size=7904, min_size=256
[857548.083285] gim info:(init_fb_static:166) AMD GIM init_fb_static: vf_fb_size = 3952, base= 272
seems to work though?

it says (summed) VF FB size = 7904MB and then 'vf_fb_size = 3952' (7902/2 = 3952)
Click to expand...
That's what I would've thought, but listing the PCI details on the VFs reveals that the memory size is still at 256M. This is the same within the guest that a VF is passed-through to.

The PF:
Code: 
lspci -s 01:00 -vv
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XT GL [FirePro S7150] (prog-if 00 [VGA controller])
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XT GL [FirePro S7150]
        Physical Slot: 4
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 91
        NUMA node: 0
        IOMMU group: 40
        Region 0: Memory at 383fe0000000 (64-bit, prefetchable) [size=256M]
        Region 2: Memory at 383ff4000000 (64-bit, prefetchable) [size=2M]
        Region 4: I/O ports at e000 [size=256]
        Region 5: Memory at fb400000 (32-bit, non-prefetchable) [size=256K]
        Expansion ROM at fb440000 [disabled] [size=128K]
        Capabilities: [48] Vendor Specific Information: Len=08 <?>
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
                        RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s (ok), Width x8 (downgraded)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
                         EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [a0] MSI: Enable+ Count=1/4 Maskable+ 64bit+
                Address: 00000000fee00ab8  Data: 0000
                Masking: 0000000e  Pending: 00000000
        Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP+ SDES- TLP+ FCP- CmpltTO+ CmpltAbrt- UnxCmplt+ RxOF- MalfTLP+ ECRC+ UnsupReq+ ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP+ BadDLLP+ Rollover+ Timeout+ AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [200 v1] Physical Resizable BAR
                BAR 0: current size: 256MB, supported: 256MB 512MB 1GB 2GB 4GB 8GB
        Capabilities: [270 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Capabilities: [2b0 v1] Address Translation Service (ATS)
                ATSCap: Invalidate Queue Depth: 00
                ATSCtl: Enable+, Smallest Translation Unit: 00
        Capabilities: [2c0 v1] Page Request Interface (PRI)
                PRICtl: Enable- Reset-
                PRISta: RF- UPRGI- Stopped+
                Page Request Capacity: 00000020, Page Request Allocation: 00000000
        Capabilities: [2d0 v1] Process Address Space ID (PASID)
                PASIDCap: Exec+ Priv+, Max PASID Width: 10
                PASIDCtl: Enable- Exec- Priv-
        Capabilities: [328 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [330 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration-, Interrupt Message Number: 000
                IOVCtl: Enable+ Migration- Interrupt- MSE+ ARIHierarchy+
                IOVSta: Migration-
                Initial VFs: 16, Total VFs: 16, Number of VFs: 2, Function Dependency Link: 00
                VF offset: 16, stride: 1, Device ID: 692f
                Supported Page Size: 00000553, System Page Size: 00000001
                Region 0: Memory at 0000383ee0000000 (64-bit, prefetchable)
                Region 2: Memory at 0000383ff0000000 (64-bit, prefetchable)
                Region 5: Memory at f7400000 (32-bit, non-prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [400 v1] Vendor Specific Information: ID=0002 Rev=1 Len=070 <?>
        Kernel modules: amdgpu

One of the VFs:
Code: 
lspci -s 01:02 -vv
01:02.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] (prog-if 00 [VGA controller])
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        NUMA node: 0
        IOMMU group: 48
        Region 0: Memory at 383ee0000000 (64-bit, prefetchable) [virtual] [size=256M]
        Region 2: Memory at 383ff0000000 (64-bit, prefetchable) [virtual] [size=4M]
        Region 5: Memory at f7400000 (32-bit, non-prefetchable) [virtual] [size=4M]
        Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed unknown (downgraded), Width x0 (downgraded)
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis- NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [a0] MSI: Enable- Count=1/4 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: ffffffff ffffffff ffffffff ffffffff
        Capabilities: [2b0 v1] Address Translation Service (ATS)
                ATSCap: Invalidate Queue Depth: 00
                ATSCtl: Enable+, Smallest Translation Unit: 00
        Capabilities: [328 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu
 
is that on the host? did you try to check what is available if you pass it through on a guest?

cjones said:
[857548.076618] gim info:(init_frame_buffer_partition:222) PCI defined VF FB size = 256 MB
Click to expand...
i think its just the pci defined size
 
dcsapak said:
is that on the host? did you try to check what is available if you pass it through on a guest?


i think its just the pci defined size
Click to expand...
The guest was showing the same thing, but I ended up running glxinfo and it shows the right amount of video memory. I guess I misunderstand the lspci output.

Thank you for your help.
 
You must log in or register to reply here.
Top Bottom