Skip to contentSkip to conversation

MSI-X capability display of pciutils

DISCUSSION David Epping 2 participants 7 messages May 31, 2013 – Jun 1, 2013 1 message not in the archive PCI

lore
Share & tools
lore
#0 lore raw

Hello,

I am inspecting the MSI-X setup of some PCIe devices and have questions about the output of lspci.

It looks something like this:
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-

    Vector table: BAR=0 offset=00002000
    PBA: BAR=0 offset=00003000

As far as I understand the PCI 3.0 specification MSI-X capabilities define a 32 bit register for the upper address portion of the MSI target addresses (at capability offset 4) and another 32 bit register pointing to the BAR and inter-BAR offset for the table of lower address portions of the MSI target addresses (at capability offset 8). Why are there two lines in the lspci output indicating some BAR and offset? What do they mean?

Looking at the source code of pciutils (ls-caps.c) indicates that the PBA line is the information pointing to the table of MSI-X entries, while the vector table line is actually the upper 32bit of each MSI target address. This should however not be split into BAR and offset in that case, I believe.

What did I miss?
Thanks for clarifying,
David

maintainer
#1 lore raw
Quoted 10 lines · David Epping

Hello,

I am inspecting the MSI-X setup of some PCIe devices and have questions about the output of lspci.

It looks something like this:
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-

    Vector table: BAR=0 offset=00002000
    PBA: BAR=0 offset=00003000
Quoted 7 lines

As far as I understand the PCI 3.0 specification MSI-X capabilities define a 32 bit register for the upper address portion of the MSI target addresses (at capability offset 4) and another 32 bit register pointing to the BAR and inter-BAR offset for the table of lower address portions of the MSI target addresses (at capability offset 8).
Why are there two lines in the lspci output indicating some BAR and offset? What do they mean?

My understanding is that there are three data structures here:
1) The MSI-X Capability (Figure 6-10) in config space
2) The "MSI-X Table" (Figure 6-11) in PCI memory space
3) The "MSI-X PBA" (Pending Bit Array, Figure 6-12) in PCI memory space

The "Vector table" line tells you how to find the "MSI-X Table," and the "PBA" line tells you how to find the "MSI-X PBA" table.

In MSI (not MSI-X), there is only one MSI target address, and it is in the MSI capability.

In MSI-X, each MSI-X vector can have its own target address, and those addresses are not in the capability itself; they are stored in the memory decoded by a BAR.

In your case, you would read the bus address from BAR 0, and the vector table should be at offsets 0x2000-0x21ff (32 16-byte entries) from that address. The PBA should be at offsets 0x3000-0x30ff (32 8-byte entries).

lspci will show you the BAR values, but it doesn't dump PCI memory space, so you'd have to use /dev/mem or a tool like
http://cmp.felk.cvut.cz/~pisa/linux/rdwrmem.c to actually look at the tables.

Bjorn

#2 lore raw
Quoted 44 lines · Bjorn Helgaas
Quoted 10 lines · David Epping

Hello,

I am inspecting the MSI-X setup of some PCIe devices and have questions about the output of lspci.

It looks something like this:
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-

     Vector table: BAR=0 offset=00002000
     PBA: BAR=0 offset=00003000
Quoted 7 lines

As far as I understand the PCI 3.0 specification MSI-X capabilities define a 32 bit register for the upper address portion of the MSI target addresses (at capability offset 4) and another 32 bit register pointing to the BAR and inter-BAR offset for the table of lower address portions of the MSI target addresses (at capability offset 8).
Why are there two lines in the lspci output indicating some BAR and offset? What do they mean?

My understanding is that there are three data structures here:
1) The MSI-X Capability (Figure 6-10) in config space
2) The "MSI-X Table" (Figure 6-11) in PCI memory space
3) The "MSI-X PBA" (Pending Bit Array, Figure 6-12) in PCI memory space

The "Vector table" line tells you how to find the "MSI-X Table," and the "PBA" line tells you how to find the "MSI-X PBA" table.

In MSI (not MSI-X), there is only one MSI target address, and it is in the MSI capability.

In MSI-X, each MSI-X vector can have its own target address, and those addresses are not in the capability itself; they are stored in the memory decoded by a BAR.

In your case, you would read the bus address from BAR 0, and the vector table should be at offsets 0x2000-0x21ff (32 16-byte entries) from that address. The PBA should be at offsets 0x3000-0x30ff (32 8-byte entries).

lspci will show you the BAR values, but it doesn't dump PCI memory space, so you'd have to use /dev/mem or a tool like
http://cmp.felk.cvut.cz/~pisa/linux/rdwrmem.c to actually look at the tables.

Quoted 5 lines

Bjorn
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Bjorn,

thank you for your quick reply.

So far the concepts of MSI and MSI-X are clear. I did not have the meaning of the PBA acronym, but now I know what it stands for.
I do however not see that reflected in the spec. As your figure references are off by one and I do not have the PBA figure, we should clarify the document used. I'm looking at PCI Local Bus Specification Revision 3.0 from August 2002.

According to this document (and in case I'm reading it correctly) the pending bit is part of each MSI-X Vector Table entry (lowest bit in the address portion - section 6.8.2.6). The lspci source code and its output (and your explanation) however suggest that the pending bits are stored in a separate table. In addition this specification document has the common upper address portion of each MSI target address at capability offset 4 and the BAR and intra-BAR offset at capability offset 8. The pciutils source code defines the later to be at 4 and uses offset 8 to derive the PBA.

I wonder where our disconnect is. What is the title and revision of the document you are referring to? Thanks again,
David

#3 lore raw

Bjorn,

I found a ECN and presentation by PCI-SIG. Both documents describe exactly the scheme you pointed out. Seems I have been using an old or draft version. I will request the current version right now.

Thank you for your kind help and sorry for any confusion caused. Best regards,
David

Quoted 70 lines · David Epping
Quoted 40 lines · Bjorn Helgaas
Quoted 10 lines · David Epping

Hello,

I am inspecting the MSI-X setup of some PCIe devices and have questions about the output of lspci.

It looks something like this:
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-

     Vector table: BAR=0 offset=00002000
     PBA: BAR=0 offset=00003000
Quoted 7 lines

As far as I understand the PCI 3.0 specification MSI-X capabilities define a 32 bit register for the upper address portion of the MSI target addresses (at capability offset 4) and another 32 bit register pointing to the BAR and inter-BAR offset for the table of lower address portions of the MSI target addresses (at capability offset 8).
Why are there two lines in the lspci output indicating some BAR and offset? What do they mean?

My understanding is that there are three data structures here:
1) The MSI-X Capability (Figure 6-10) in config space
2) The "MSI-X Table" (Figure 6-11) in PCI memory space
3) The "MSI-X PBA" (Pending Bit Array, Figure 6-12) in PCI memory
space

The "Vector table" line tells you how to find the "MSI-X Table," and the "PBA" line tells you how to find the "MSI-X PBA" table.

In MSI (not MSI-X), there is only one MSI target address, and it is in the MSI capability.

In MSI-X, each MSI-X vector can have its own target address, and those addresses are not in the capability itself; they are stored in the memory decoded by a BAR.

In your case, you would read the bus address from BAR 0, and the vector table should be at offsets 0x2000-0x21ff (32 16-byte entries) from that address. The PBA should be at offsets 0x3000-0x30ff (32 8-byte entries).

Quoted 10 lines

lspci will show you the BAR values, but it doesn't dump PCI memory space, so you'd have to use /dev/mem or a tool like
http://cmp.felk.cvut.cz/~pisa/linux/rdwrmem.c to actually look at the tables.

Bjorn

Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Bjorn,

thank you for your quick reply.

So far the concepts of MSI and MSI-X are clear. I did not have the meaning of the PBA acronym, but now I know what it stands for.
I do however not see that reflected in the spec. As your figure references are off by one and I do not have the PBA figure, we should clarify the document used. I'm looking at PCI Local Bus Specification Revision 3.0 from August 2002.

According to this document (and in case I'm reading it correctly) the pending bit is part of each MSI-X Vector Table entry (lowest bit in the address portion - section 6.8.2.6). The lspci source code and its output (and your explanation) however suggest that the pending bits are stored in a separate table. In addition this specification document has the common upper address portion of each MSI target address at capability offset 4 and the BAR and intra-BAR offset at capability offset 8. The pciutils source code defines the later to be at 4 and uses offset 8 to derive the PBA.

I wonder where our disconnect is. What is the title and revision of the document you are referring to? Thanks again,
David

Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 11 lines
David Epping
[email protected]
Skype: david.epping
Mobile US: +1 (408) 4757-574
Mobile DE: +49 (179) 9656-223

Missing Link Electronics
http://www.missinglinkelectronics.com
Office DE: +49 (731) 802-3930
Office US: +1  (408) 457-0700
Skype: missing.link.electronics
#5 lore raw
In reply to a message not in the archive
Quoted 133 lines

David/All,
Not sure if I can use this thread to ask but wonderingwhere do hobbyists can get to see the PCIe spec?

Quoted 128 lines

Date: Sat, 1 Jun 2013 13:30:28 +0200
From: [email protected]
To: [email protected]
CC: [email protected]
Subject: Re: MSI-X capability display of pciutils

Bjorn,

I found a ECN and presentation by PCI-SIG. Both documents describe exactly the scheme you pointed out.
Seems I have been using an old or draft version. I will request the current version right now.

Thank you for your kind help and sorry for any confusion caused. Best regards,
David

Quoted 6 lines · David Epping
Quoted 5 lines · Bjorn Helgaas
Quoted 3 lines · David Epping

Hello,

I am inspecting the MSI-X setup of some PCIe devices and have

questions

Quoted 25 lines
Quoted 25 lines
Quoted 7 lines

about the output of lspci.

It looks something like this:
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-

     Vector table: BAR=0 offset=00002000
     PBA: BAR=0 offset=00003000
Quoted 12 lines

As far as I understand the PCI 3.0 specification MSI-X capabilities define a
32 bit register for the upper address portion of the MSI target addresses
(at capability offset 4) and another 32 bit register pointing to the BAR and
inter-BAR offset for the table of lower address portions of the MSI target
addresses (at capability offset 8).
Why are there two lines in the lspci output indicating some BAR and offset?
What do they mean?

My understanding is that there are three data structures here:
1) The MSI-X Capability (Figure 6-10) in config space
2) The "MSI-X Table" (Figure 6-11) in PCI memory space
3) The "MSI-X PBA" (Pending Bit Array, Figure 6-12) in PCI memory
space

Quoted 4 lines
Quoted 4 lines

The "Vector table" line tells you how to find the "MSI-X Table," and the "PBA" line tells you how to find the "MSI-X PBA" table.

In MSI (not MSI-X), there is only one MSI target address, and it is

in

Quoted 3 lines
Quoted 3 lines

the MSI capability.

In MSI-X, each MSI-X vector can have its own target address, and

those

Quoted 8 lines
Quoted 8 lines

addresses are not in the capability itself; they are stored in the memory decoded by a BAR.

In your case, you would read the bus address from BAR 0, and the vector table should be at offsets 0x2000-0x21ff (32 16-byte entries) from that address. The PBA should be at offsets 0x3000-0x30ff (32 8-byte entries).

Quoted 8 lines
Quoted 8 lines

lspci will show you the BAR values, but it doesn't dump PCI memory space, so you'd have to use /dev/mem or a tool like
http://cmp.felk.cvut.cz/~pisa/linux/rdwrmem.c to actually look at the tables.

Bjorn
--
To unsubscribe from this list: send the line "unsubscribe linux-pci"

in

Quoted 24 lines
Quoted 2 lines

the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Bjorn,

thank you for your quick reply.

So far the concepts of MSI and MSI-X are clear. I did not have the meaning of the PBA acronym, but now I know what it stands for. I do however not see that reflected in the spec. As your figure references are off by one and I do not have the PBA figure, we should clarify the document used. I'm looking at PCI Local Bus Specification Revision 3.0 from August 2002.

According to this document (and in case I'm reading it correctly) the pending bit is part of each MSI-X Vector Table entry (lowest bit in the address portion - section 6.8.2.6). The lspci source code and its output (and your explanation) however suggest that the pending bits are stored in a separate table.
In addition this specification document has the common upper address portion of each MSI target address at capability offset 4 and the BAR and intra-BAR offset at capability offset 8. The pciutils source code defines the later to be at 4 and uses offset 8 to derive the PBA.

Quoted 7 lines

I wonder where our disconnect is. What is the title and revision of the document you are referring to?
Thanks again,
David

--
To unsubscribe from this list: send the line "unsubscribe linux-pci"

in

Quoted 2 lines

the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
David Epping
[email protected]
Skype: david.epping
Mobile US: +1 (408) 4757-574
Mobile DE: +49 (179) 9656-223

Missing Link Electronics
http://www.missinglinkelectronics.com
Office DE: +49 (731) 802-3930
Office US: +1 (408) 457-0700
Skype: missing.link.electronics

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

AFAIK there is no public (full or reduced) version of the relevant specifications. There are some documents available for download for non-members, but for the real spec your employer is probably your best chance.

maintainer
#7 lore raw
In reply to David Epping

Interesting. I'm using the "PCI Local Bus Specification, Rev. 3.0," but dated Feb 3, 2004. Sounds like they screwed up assigning revision numbers, and released two different things as "rev 3.0." Sigh.

Quoted 120 lines · David Epping

Bjorn,

I found a ECN and presentation by PCI-SIG. Both documents describe exactly the scheme you pointed out.
Seems I have been using an old or draft version. I will request the current version right now.

Thank you for your kind help and sorry for any confusion caused. Best regards,
David

Quoted 88 lines · David Epping
Quoted 53 lines · Bjorn Helgaas
Quoted 11 lines · David Epping

Hello,

I am inspecting the MSI-X setup of some PCIe devices and have questions about the output of lspci.

It looks something like this:
Capabilities: [50] MSI-X: Enable+ Count=32 Masked-

     Vector table: BAR=0 offset=00002000
     PBA: BAR=0 offset=00003000
Quoted 12 lines

As far as I understand the PCI 3.0 specification MSI-X capabilities define a
32 bit register for the upper address portion of the MSI target addresses
(at capability offset 4) and another 32 bit register pointing to the BAR and
inter-BAR offset for the table of lower address portions of the MSI target
addresses (at capability offset 8).
Why are there two lines in the lspci output indicating some BAR and offset?
What do they mean?

My understanding is that there are three data structures here:
1) The MSI-X Capability (Figure 6-10) in config space
2) The "MSI-X Table" (Figure 6-11) in PCI memory space
3) The "MSI-X PBA" (Pending Bit Array, Figure 6-12) in PCI memory
space

The "Vector table" line tells you how to find the "MSI-X Table," and the "PBA" line tells you how to find the "MSI-X PBA" table.

In MSI (not MSI-X), there is only one MSI target address, and it is in the MSI capability.

In MSI-X, each MSI-X vector can have its own target address, and those addresses are not in the capability itself; they are stored in the memory decoded by a BAR.

In your case, you would read the bus address from BAR 0, and the vector table should be at offsets 0x2000-0x21ff (32 16-byte entries) from that address. The PBA should be at offsets 0x3000-0x30ff (32 8-byte entries).

lspci will show you the BAR values, but it doesn't dump PCI memory space, so you'd have to use /dev/mem or a tool like
http://cmp.felk.cvut.cz/~pisa/linux/rdwrmem.c to actually look at the tables.

Quoted 5 lines

Bjorn
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Bjorn,

thank you for your quick reply.

So far the concepts of MSI and MSI-X are clear. I did not have the meaning of the PBA acronym, but now I know what it stands for.
I do however not see that reflected in the spec. As your figure references are off by one and I do not have the PBA figure, we should clarify the document used. I'm looking at PCI Local Bus Specification Revision 3.0 from August 2002.

According to this document (and in case I'm reading it correctly) the pending bit is part of each MSI-X Vector Table entry (lowest bit in the address portion - section 6.8.2.6). The lspci source code and its output (and your explanation) however suggest that the pending bits are stored in a separate table.
In addition this specification document has the common upper address portion of each MSI target address at capability offset 4 and the BAR and intra-BAR offset at capability offset 8. The pciutils source code defines the later to be at 4 and uses offset 8 to derive the PBA.

I wonder where our disconnect is. What is the title and revision of the document you are referring to?
Thanks again,
David

Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
David Epping
[email protected]
Skype: david.epping
Mobile US: +1 (408) 4757-574
Mobile DE: +49 (179) 9656-223

Missing Link Electronics
http://www.missinglinkelectronics.com
Office DE: +49 (731) 802-3930
Office US: +1 (408) 457-0700
Skype: missing.link.electronics

maintainer
#6 lore raw
In reply to David Epping
Quoted 8 lines · David Epping
Quoted 3 lines

David/All,
Not sure if I can use this thread to ask but wonderingwhere do hobbyists can get to see the PCIe spec?

AFAIK there is no public (full or reduced) version of the relevant specifications. There are some documents available for download for non-members, but for the real spec your employer is probably your best chance.

The MindShare books are the only real public docs I know of. But in most cases, those books are pretty good.

Bjorn