Does my understanding correct?
All,
I draw a picture about the physical layout on a part of the pci system.
Each block represent a bridge or a device.
Am I correct?
+-----------------------------+
| |
| |
| Bus#2 |
| ------------------ |
| |
| |
| |
| +-------------+
| |DownStream |02:00.0 pci_dev
+---------------+-------+-----+
| Bus#3
---------------+------------------------------------------+---
| |
| |
+---------------+-------+-----+ +---------------+----+--+-----+
| |UpStream |03:00.0 | |UpStream |03:01.0
| +-------------+ | +-------------+ a normal device
| | | |
| | | |
| Bus#4 | | |
| ------------------- | | |
| +----------+ | |
| |DownStream| | |
+-----------------------+-----+ +-----------------------------+
|
| Bus#5
------------+------------ Signature · 2 lines
Richard Yang Help you, Help me
For PCI, there's no internal bus Bus#4 in the left bottom device.
Quoted 37 lines · Richard Yang
All,
I draw a picture about the physical layout on a part of the pci system.
Each block represent a bridge or a device.
Am I correct?+-----------------------------+ | | | | | Bus#2 | | ------------------ | | | | | | | | +-------------+ | |DownStream |02:00.0 pci_dev +---------------+-------+-----+ | Bus#3 ---------------+------------------------------------------+--- | | | | +---------------+-------+-----+ +---------------+----+--+-----+ | |UpStream |03:00.0 | |UpStream |03:01.0 | +-------------+ | +-------------+ a normal device | | | | | | | | | Bus#4 | | | | ------------------- | | | | +----------+ | | | |DownStream| | | +-----------------------+-----+ +-----------------------------+ | | Bus#5 ------------+------------
Bjorn Helgaas
maintainerT24gRnJpLCBBcHIgMjcsIDIwMTIgYXQgODoxMCBBTSwgSmlhbmcgTGl1IDxsaXVqOTdAZ21haWwu Y29tPiB3cm90ZToKPiBGb3IgUENJLCB0aGVyZSdzIG5vIGludGVybmFsIGJ1cyBCdXMjNCBpbiB0 aGUgbGVmdCBib3R0b20gZGV2aWNlLgoKWWVzLCB0aGVyZSBpcy4gIFRoZSBsZWZ0IGJvdHRvbSBi b3ggaXMgYSBQQ0llIHN3aXRjaCB3aXRoIHVwc3RyZWFtCnBvcnQgMDM6MDAuMC4gIFRoZSB1cHN0 cmVhbSBwb3J0J3Mgc2Vjb25kYXJ5IGJ1cyBpcyAwNCwgd2hpY2ggaXMgdGhlCnN3aXRjaCdzIGlu dGVybmFsIGJ1cy4gIFRoZSBzd2l0Y2gncyBkb3duc3RyZWFtIHBvcnRzIHdvdWxkIGJlLCBmb3IK ZXhhbXBsZSwgMDQ6MDAuMCwgMDQ6MDEuMCwgMDQ6MDIuMCwgZXRjIChvdGhlciBudW1iZXJpbmdz IGFyZQpwb3NzaWJsZSkuICBJZiB0aGUgYm94IGxhYmVsbGVkICJEb3duc3RyZWFtIiBpcyAwNDow MC4wLCBpdHMgc2Vjb25kYXJ5CmJ1cyB3b3VsZCBiZSAwNS4KCkFzIEkgbWVudGlvbmVkIGJlZm9y ZSwgSSB0aGluayB0aGUgYnVzIDA1IGxpbmUgaXMgbWlzbGVhZGluZywgYmVjYXVzZQppdCdzIGRy YXduIGFzIGEgYnVzIHdoZXJlIG11bHRpcGxlIGRldmljZXMgY291bGQgYmUgYXR0YWNoZWQuICBJ dApzaG91bGQgaW5zdGVhZCBiZSBhIHBvaW50LXRvLXBvaW50IGxpbmsgdG8gYSBzaW5nbGUgZGV2 aWNlLiAgKEV4Y2VwdAppbiB0aGlzIFN0cmF0dXMgc3lzdGVtLCB3aGljaCBzZWVtcyBicm9rZW4g YmVjYXVzZSBpdCAqZG9lcyogaGF2ZSB0d28KZGV2aWNlcyBhdHRhY2hlZCB0byB0aGUgc2FtZSBk b3duc3RyZWFtIHBvcnQuKQoKPiBPbiAwNC8yNy8yMDEyIDA1OjI3IFBNLCBSaWNoYXJkIFlhbmcg d3JvdGU6Cj4+IEFsbCwKPj4KPj4gSSBkcmF3IGEgcGljdHVyZSBhYm91dCB0aGUgcGh5c2ljYWwg bGF5b3V0IG9uIGEgcGFydCBvZiB0aGUgcGNpIHN5c3RlbS4KPj4KPj4gRWFjaCBibG9jayByZXBy ZXNlbnQgYSBicmlkZ2Ugb3IgYSBkZXZpY2UuCj4+IEFtIEkgY29ycmVjdD8KPj4KPj4goCCgICst LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSsKPj4goCCgIHwgoCCgIKAgoCCgIKAgoCCgIKAg oCCgIKAgoCCgIHwKPj4goCCgIHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIHwKPj4goCCg IHwgoCCgIKBCdXMjMiCgIKAgoCCgIKAgoCCgIKAgoHwKPj4goCCgIHwgoCAtLS0tLS0tLS0tLS0t LS0tLS0goCCgIKAgoHwKPj4goCCgIHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIHwKPj4g oCCgIHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIHwKPj4goCCgIHwgoCCgIKAgoCCgIKAg oCCgIKAgoCCgIKAgoCCgIHwKPj4goCCgIHwgoCCgIKAgoCCgIKAgoCArLS0tLS0tLS0tLS0tLSsK Pj4goCCgIHwgoCCgIKAgoCCgIKAgoCB8RG93blN0cmVhbSCgIHwwMjowMC4wIHBjaV9kZXYKPj4g oCCgICstLS0tLS0tLS0tLS0tLS0rLS0tLS0tLSstLS0tLSsKPj4goCCgIKAgoCCgIKAgoCCgIKAg oCCgIKAgoCCgIHwgoCBCdXMjMwo+PiCgIKAgoCCgIKAgoCCgLS0tLS0tLS0tLS0tLS0tKy0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSstLS0KPj4goCCgIKAgoCCgIKAg oCCgIKAgoCCgIKAgoCCgIHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCg IKB8Cj4+IKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCB8IKAgoCCgIKAgoCCgIKAgoCCgIKAg oCCgIKAgoCCgIKAgoCCgIKAgoCCgfAo+PiCgIKAgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tKy0t LS0tKyCgIKAgoCCgIKAgoCCgICstLS0tLS0tLS0tLS0tLS0rLS0tLSstLSstLS0tLSsKPj4goCCg IHwgoCCgIKAgoCCgIKAgoCB8VXBTdHJlYW0goCCgIHwwMzowMC4wIKAgoCCgIKB8IKAgoCCgIKAg oCCgIKAgfFVwU3RyZWFtIKAgoCB8MDM6MDEuMAo+PiCgIKAgfCCgIKAgoCCgIKAgoCCgICstLS0t LS0tLS0tLS0tKyCgIKAgoCCgIKAgoCCgIHwgoCCgIKAgoCCgIKAgoCArLS0tLS0tLS0tLS0tLSsg YSBub3JtYWwgZGV2aWNlCj4+IKAgoCB8IKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCB8IKAg oCCgIKAgoCCgIKAgfCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgfAo+PiCgIKAgfCCgIKAg oCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgfCCgIKAgoCCgIKAgoCCgIHwgoCCgIKAgoCCgIKAgoCCg IKAgoCCgIKAgoCCgIHwKPj4goCCgIHwgoCCgIKAgoCCgIKAgQnVzIzQgoCCgIKAgoCCgIHwgoCCg IKAgoCCgIKAgoCB8IKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCB8Cj4+IKAgoCB8IKAgoCAt LS0tLS0tLS0tLS0tLS0tLS0tIKAgoCB8IKAgoCCgIKAgoCCgIKAgfCCgIKAgoCCgIKAgoCCgIKAg oCCgIKAgoCCgIKAgfAo+PiCgIKAgfCCgIKAgoCCgIKAgoCCgIKAgoCstLS0tLS0tLS0tKyCgIKAg oCCgIKAgoCCgIHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIHwKPj4goCCgIHwgoCCgIKAg oCCgIKAgoCCgIKB8RG93blN0cmVhbXwgoCCgIKAgoCCgIKAgoCB8IKAgoCCgIKAgoCCgIKAgoCCg IKAgoCCgIKAgoCB8Cj4+IKAgoCArLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0rIKAgoCCg IKAgoCCgIKAgKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tKwo+PiCgIKAgoCCgIKAgoCCg IKAgoCCgIKAgoCCgIKAgfAo+PiCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgfCBCdXMjNQo+ PiCgIKAgoCCgIKAgoCCgIKAgLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLQo+Pgo+Pgo+Cg==
Bjorn Helgaas
maintainerT24gRnJpLCBBcHIgMjcsIDIwMTIgYXQgMzoyNyBBTSwgUmljaGFyZCBZYW5nCjx3ZWl5YW5nQGxp bnV4LnZuZXQuaWJtLmNvbT4gd3JvdGU6Cj4gQWxsLAo+Cj4gSSBkcmF3IGEgcGljdHVyZSBhYm91 dCB0aGUgcGh5c2ljYWwgbGF5b3V0IG9uIGEgcGFydCBvZiB0aGUgcGNpIHN5c3RlbS4KPgo+IEVh Y2ggYmxvY2sgcmVwcmVzZW50IGEgYnJpZGdlIG9yIGEgZGV2aWNlLgo+IEFtIEkgY29ycmVjdD8K Pgo+IKAgoCstLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLSsKPiCgIKB8IKAgoCCgIKAgoCCg IKAgoCCgIKAgoCCgIKAgoCB8Cj4goCCgfCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgfAo+ IKAgoHwgoCCgIKBCdXMjMiCgIKAgoCCgIKAgoCCgIKAgoHwKPiCgIKB8IKAgLS0tLS0tLS0tLS0t LS0tLS0tIKAgoCCgIKB8Cj4goCCgfCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgfAo+IKAg oHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIHwKPiCgIKB8IKAgoCCgIKAgoCCgIKAgoCCg IKAgoCCgIKAgoCB8Cj4goCCgfCCgIKAgoCCgIKAgoCCgICstLS0tLS0tLS0tLS0tKwo+IKAgoHwg oCCgIKAgoCCgIKAgoCB8RG93blN0cmVhbSCgIHwwMjowMC4wIHBjaV9kZXYKPiCgIKArLS0tLS0t LS0tLS0tLS0tKy0tLS0tLS0rLS0tLS0rCj4goCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgfCCg IEJ1cyMzCj4goCCgIKAgoCCgIKAgLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLSstLS0KPiCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKB8 IKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgfAo+IKAgoCCgIKAgoCCg IKAgoCCgIKAgoCCgIKAgoHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCg IKB8Cj4goCCgKy0tLS0tLS0tLS0tLS0tLSstLS0tLS0tKy0tLS0tKyCgIKAgoCCgIKAgoCCgICst LS0tLS0tLS0tLS0tLS0rLS0tLSstLSstLS0tLSsKPiCgIKB8IKAgoCCgIKAgoCCgIKAgfFVwU3Ry ZWFtIKAgoCB8MDM6MDAuMCCgIKAgoCCgfCCgIKAgoCCgIKAgoCCgIHxVcFN0cmVhbSCgIKAgfDAz OjAxLjAKPiCgIKB8IKAgoCCgIKAgoCCgIKAgKy0tLS0tLS0tLS0tLS0rIKAgoCCgIKAgoCCgIKAg fCCgIKAgoCCgIKAgoCCgICstLS0tLS0tLS0tLS0tKyBhIG5vcm1hbCBkZXZpY2UKPiCgIKB8IKAg oCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCB8IKAgoCCgIKAgoCCgIKAgfCCgIKAgoCCgIKAgoCCg IKAgoCCgIKAgoCCgIKAgfAo+IKAgoHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIHwgoCCg IKAgoCCgIKAgoCB8IKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCB8Cj4goCCgfCCgIKAgoCCg IKAgoCBCdXMjNCCgIKAgoCCgIKAgfCCgIKAgoCCgIKAgoCCgIHwgoCCgIKAgoCCgIKAgoCCgIKAg oCCgIKAgoCCgIHwKPiCgIKB8IKAgoCAtLS0tLS0tLS0tLS0tLS0tLS0tIKAgoCB8IKAgoCCgIKAg oCCgIKAgfCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgfAo+IKAgoHwgoCCgIKAgoCCgIKAg oCCgIKArLS0tLS0tLS0tLSsgoCCgIKAgoCCgIKAgoCB8IKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCg IKAgoCB8Cj4goCCgfCCgIKAgoCCgIKAgoCCgIKAgoHxEb3duU3RyZWFtfCCgIKAgoCCgIKAgoCCg IHwgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIHwKPiCgIKArLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0rLS0tLS0rIKAgoCCgIKAgoCCgIKAgKy0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t Kwo+IKAgoCCgIKAgoCCgIKAgoCCgIKAgoCCgIKAgoHwKPiCgIKAgoCCgIKAgoCCgIKAgoCCgIKAg oCCgIKB8IEJ1cyM1Cj4goCCgIKAgoCCgIKAgoCCgLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0tLQoK SSBhc3N1bWUgeW91ciBxdWVzdGlvbiByZWxhdGVzIHRvIHRoZSBTdHJhdHVzIGZ0U2VydmVyIHRv cG9sb2d5LiAgSWYKc28sIHRoZSBsc3BjaSBkZXRhaWxzIG1pZ2h0IGNsYXJpZnkgdGhpbmdzLgoK SW4gdGhhdCBzeXN0ZW0sIG15IHVuZGVyc3RhbmRpbmcgaXMgdGhhdCAwMzowMS4wIGlzIGEgZG93 bnN0cmVhbSBwb3J0LApub3QgYW4gdXBzdHJlYW0gcG9ydC4KCkkgdGhpbmsgeW91ciBwaWN0dXJl IGlzIHNsaWdodGx5IG1pc2xlYWRpbmcgYmVjYXVzZSBQQ0llIGxpbmtzIGFyZSBub3QKYnVzZXM7 IHRoZXkncmUgcG9pbnQtdG8tcG9pbnQgbGlua3MgYmV0d2VlbiB0d28gZGV2aWNlcy4gIFlvdSd2 ZSBkcmF3bgojMyBhbmQgIzUgYXMgYnVzZXMgdGhhdCBjYW4gaGF2ZSBzZXZlcmFsIGRldmljZXMg b24gdGhlbSwgd2hpY2ggaXMgbm90CnJlYWxseSB0aGUgY2FzZS4gIFRoZSBsaW5rIGZyb20gYSBk b3duc3RyZWFtIHBvcnQgc2hvdWxkIGxlYWQgdG8KZXhhY3RseSBvbmUgZGV2aWNlLgoKVGhhdCdz IG9uZSB0aGluZyB0aGF0J3Mgc3RyYW5nZSBpbiB0aGUgZnRTZXJ2ZXIgdG9wb2xvZ3k6IGFwcGFy ZW50bHkKdGhlcmUgYXJlICp0d28qIGRldmljZXMgb24gYnVzIDAzOiB0aGUgMDM6MDAuMCB1cHN0 cmVhbSBwb3J0IGFuZCB0aGUKMDM6MDEuMCBkb3duc3RyZWFtIHBvcnQuICBJIHRoaW5rIDAzOjAw LjAgaXMgdGhlIHVwc3RyZWFtIHBvcnQgb2YgYQpQQ0llIHN3aXRjaCwgd2hpY2ggaXMgcGVyZmVj dGx5IG5vcm1hbC4gIE15IHVuZGVyc3RhbmRpbmcgaXMgdGhhdAowMzowMS4wIGlzIGFub3RoZXIg KmRvd25zdHJlYW0qIHBvcnQgdGhhdCBsZWFkcyB0byBzZXZlcmFsIG1vcmUKZGV2aWNlcyAoVVNC LCBOSUMsIGV0YykuCgpCam9ybgo=
Quoted 6 lines · Bjorn Helgaas
On Fri, Apr 27, 2012 at 3:27 AM, Richard Yang
<[email protected]> wrote:I assume your question relates to the Stratus ftServer topology. If so, the lspci details might clarify things.
Yes, my picture is a little bit related to your previous mail. While my intention is to find out how the physical world is represented in the kernel.
Below is a typical topology in PCIe spec r3.0.
+------------------+
| |
| RC |
| Bus#0 |
| ------------- |
| |
+-+-----+--------+-+
00:0.0 | | | 00:02.0
+---------+---------+ | | | +------------+-------------+
| +------+ | +-------| PCIe 2 PCI Bridge |
| PCIe Endpoint | | | |
+-------------------+ | | Bus#2 |
| | -------------- |
| +-------+---------------+--+
| | |
| 00:01.0 |02:00.0 |02:01.0
+------------+-------------+ +-------+------+ +---+-------+
| | |PCI dev | |PCI dev |
| Switch | | | | |
| Bus#1 | | | | |
| --------------- | +--------------+ +-----------+
| |
+------------------------+-+
| |
| |
| 01:00.0 | 01:01.0
+---------+-------+ +--------+----------------+
| | | |
| PCI Endpoint | | PCIe Endpoint |
| | | |
| | | |
+-----------------+ +-------------------------+ Do you think the current assignment of bus number and pci_dev is correct?
Quoted 17 lines
In that system, my understanding is that 03:01.0 is a downstream port, not an upstream port.
I think your picture is slightly misleading because PCIe links are not buses; they're point-to-point links between two devices. You've drawn #3 and #5 as buses that can have several devices on them, which is not really the case. The link from a downstream port should lead to exactly one device.
That's one thing that's strange in the ftServer topology: apparently there are *two* devices on bus 03: the 03:00.0 upstream port and the 03:01.0 downstream port. I think 03:00.0 is the upstream port of a PCIe switch, which is perfectly normal. My understanding is that 03:01.0 is another *downstream* port that leads to several more devices (USB, NIC, etc).
Bjorn
Signature · 2 lines
Richard Yang Help you, Help me
Quoted 46 lines · Richard Yang
Quoted 6 lines · Bjorn Helgaas
On Fri, Apr 27, 2012 at 3:27 AM, Richard Yang
<[email protected]> wrote:I assume your question relates to the Stratus ftServer topology. If so, the lspci details might clarify things.
Yes, my picture is a little bit related to your previous mail. While my intention is to find out how the physical world is represented in the kernel.
Below is a typical topology in PCIe spec r3.0.
+------------------+ | | | RC | | Bus#0 | | ------------- | | | +-+-----+--------+-+ 00:0.0 | | | 00:02.0 +---------+---------+ | | | +------------+-------------+ | +------+ | +-------| PCIe 2 PCI Bridge | | PCIe Endpoint | | | | +-------------------+ | | Bus#2 | | | -------------- | | +-------+---------------+--+ | | | | 00:01.0 |02:00.0 |02:01.0 +------------+-------------+ +-------+------+ +---+-------+ | | |PCI dev | |PCI dev | | Switch | | | | | | Bus#1 | | | | | | --------------- | +--------------+ +-----------+ | | +------------------------+-+ | | | | | 01:00.0 | 01:01.0 +---------+-------+ +--------+----------------+ | | | | | PCI Endpoint | | PCIe Endpoint | | | | | | | | | +-----------------+ +-------------------------+
Quoted 3 lines
Do you think the current assignment of bus number and
pci_devis correct?
BTW, if the chart is correct, 01:00.0 reprents Switch DownStream Port or the PCI Endpoint?
Quoted 22 lines
Quoted 18 lines
In that system, my understanding is that 03:01.0 is a downstream port, not an upstream port.
I think your picture is slightly misleading because PCIe links are not buses; they're point-to-point links between two devices. You've drawn #3 and #5 as buses that can have several devices on them, which is not really the case. The link from a downstream port should lead to exactly one device.
That's one thing that's strange in the ftServer topology: apparently there are *two* devices on bus 03: the 03:00.0 upstream port and the 03:01.0 downstream port. I think 03:00.0 is the upstream port of a PCIe switch, which is perfectly normal. My understanding is that 03:01.0 is another *downstream* port that leads to several more devices (USB, NIC, etc).
Bjorn
Signature · 2 lines
Richard Yang Help you, Help me
Signature · 2 lines
Richard Yang Help you, Help me
Bjorn Helgaas
maintainerQuoted 46 lines · Richard Yang
Quoted 6 lines · Bjorn Helgaas
On Fri, Apr 27, 2012 at 3:27 AM, Richard Yang
<[email protected]> wrote:I assume your question relates to the Stratus ftServer topology. If so, the lspci details might clarify things.
Yes, my picture is a little bit related to your previous mail. While my intention is to find out how the physical world is represented in the kernel.
Below is a typical topology in PCIe spec r3.0.
+------------------+ | | | RC | | Bus#0 | | ------------- | | | +-+-----+--------+-+ 00:0.0 | | | 00:02.0 +---------+---------+ | | | +------------+-------------+ | +------+ | +-------| PCIe 2 PCI Bridge | | PCIe Endpoint | | | | +-------------------+ | | Bus#2 | | | -------------- | | +-------+---------------+--+ | | | | 00:01.0 |02:00.0 |02:01.0 +------------+-------------+ +-------+------+ +---+-------+ | | |PCI dev | |PCI dev | | Switch | | | | | | Bus#1 | | | | | | --------------- | +--------------+ +-----------+ | | +------------------------+-+ | | | | | 01:00.0 | 01:01.0 +---------+-------+ +--------+----------------+ | | | | | PCI Endpoint | | PCIe Endpoint | | | | | | | | | +-----------------+ +-------------------------+
Quoted 2 lines
Do you think the current assignment of bus number and
pci_devis correct?
I think assignments shown for the PCIe-to-PCI bridge are OK, although I would draw it like this because the bridge originates a single bus 02 that may have multiple devices attached to it (this side is PCI, not PCIe, so it really is a shared bus):
^
|
+--------+--------+
| 00:02.0 |
| PCIe-PCI bridge |
| |
+--------+--------+
|
|
+---------------------+ Bus 02
| |
| |
| |
+----v----+ +----v----+
| 02:00.0 | | 02:01.0 |
+---------+ +---------+I think the PCIe switch part is incorrect. Here's Figure 1-3 from sec 1.3.3 of the PCIe r3 spec:
^
|
+-------------------------------|------------------------------+
| | |
| +----+----+ |
| | virtual | |
| | PCI-PCI | |
| | bridge | |
| +----+----+ |
| | |
| | |
| | |
| +----------------------------------------+ |
| | | | |
| | | | |
| | | | |
| +----+----+ +----+----+ +----++---+ |
| | virtual | | virtual | | virtual | |
| | PCI-PCI | | PCI-PCI | | PCI-PCI | |
| | bridge | | bridge | | bridge | |
| +----+----+ +----+----+ +----+----+ |
| | | | |
| | | | |
+----------|--------------------|-------------------|----------+
| | |
v v v
A PCIe switch appears as two or more PCI-PCI bridges. One is associated with the upstream port; the others with the downstream ports.
A bridge always has a primary side and a secondary side. In your diagram, the bridge associated with the upstream port would be 00:01.0 (primary bus 00) and could have a secondary bus of 03 (since 02 is already consumed by the PCIe-PCI bridge).
The bridges associated with the downstream ports are all logically on bus 03. Their primary bus number would be 03; they might be 03:00.0, 03:01.0, 03:02.0, etc. Each would have its own secondary bus number, for example 04, 05, 06. That secondary bus number is for the downstream link from the corresponding downstream port.
The endpoints below the PCIe switch could then be 04:00.0 and 05:00.0 (or these could be the upstream ports of more PCIe switches).
Bjorn
Quoted 3 lines · Bjorn Helgaas
On Fri, Apr 27, 2012 at 11:01 PM, Richard Yang
<[email protected]> wrote:
Thanks for your nice chart.
Quoted 5 lines
I think assignments shown for the PCIe-to-PCI bridge are OK, although I would draw it like this because the bridge originates a single bus 02 that may have multiple devices attached to it (this side is PCI, not PCIe, so it really is a shared bus):
Quoted 17 lines
^
|
+--------+--------+
| 00:02.0 |
| PCIe-PCI bridge |
| |
+--------+--------+
|
|
+---------------------+ Bus 02
| |
| |
| |
+----v----+ +----v----+
| 02:00.0 | | 02:01.0 |
+---------+ +---------+So for this case, there is not internal bus, while this is really a physical shared bus, not a logical one.
Quoted 2 lines
I think the PCIe switch part is incorrect. Here's Figure 1-3 from sec 1.3.3 of the PCIe r3 spec:
^
|
+-------------------------------|------------------------------+
| | |
| +----+----+ |
| | virtual | |
| | PCI-PCI | |
| | bridge | |
| +----+----+ |
| | |
| |Bus#3 |
| | |
| +----------------------------------------+ |
| | | | |
| | | | |
| |03:00.0 |03:01.0 |03:02.0 |
| +----+----+ +----+----+ +----++---+ |
| | virtual | | virtual | | virtual | |
| | PCI-PCI | | PCI-PCI | | PCI-PCI | |
| | bridge | | bridge | | bridge | |
| +----+----+ +----+----+ +----+----+ |
| |Bus#4? | | |
| -----+------- | | |
+----------|--------------------|-------------------|----------+
| | |
v v v
Quoted 8 lines
A PCIe switch appears as two or more PCI-PCI bridges. One is associated with the upstream port; the others with the downstream ports.
A bridge always has a primary side and a secondary side. In your diagram, the bridge associated with the upstream port would be 00:01.0 (primary bus 00) and could have a secondary bus of 03 (since 02 is already consumed by the PCIe-PCI bridge).
Hmm... I am confused why is 03. 02 is used but 01 is not used. Switch should be configured after PCIe2PCI bridge?
Quoted 6 lines
The bridges associated with the downstream ports are all logically on bus 03. Their primary bus number would be 03; they might be 03:00.0, 03:01.0, 03:02.0, etc. Each would have its own secondary bus number, for example 04, 05, 06. That secondary bus number is for the downstream link from the corresponding downstream port.
Hmm, as you mentioned in previous letter, PCIe is an point-to-point protocol, then the secondary bus should reside in the Switch? Do you think my Bus#4 is correct?
Quoted 3 lines
The endpoints below the PCIe switch could then be 04:00.0 and 05:00.0 (or these could be the upstream ports of more PCIe switches).
So below the PCIe downstream port, there is only on PCIe device. The whole bus is occupied by this device?
That is why the device could have upto 256 functions?
Quoted 2 lines
Bjorn
Signature · 2 lines
Richard Yang Help you, Help me
Bjorn Helgaas
maintainerQuoted 27 lines · Richard Yang
Quoted 3 lines · Bjorn Helgaas
On Fri, Apr 27, 2012 at 11:01 PM, Richard Yang
<[email protected]> wrote:Thanks for your nice chart.
Quoted 7 lines
I think assignments shown for the PCIe-to-PCI bridge are OK, although I would draw it like this because the bridge originates a single bus 02 that may have multiple devices attached to it (this side is PCI, not PCIe, so it really is a shared bus):
^
|Quoted 15 lines
+--------+--------+ | 00:02.0 | | PCIe-PCI bridge | | | +--------+--------+ | | +---------------------+ Bus 02 | | | | | | +----v----+ +----v----+ | 02:00.0 | | 02:01.0 | +---------+ +---------+
Quoted 2 lines
So for this case, there is not internal bus, while this is really a physical shared bus, not a logical one.
Yes. The downstream side of the PCIe-PCI bridge is PCI.
Quoted 35 lines
Quoted 2 lines
I think the PCIe switch part is incorrect. Here's Figure 1-3 from sec 1.3.3 of the PCIe r3 spec:
^
|+-------------------------------|------------------------------+ | | | | +----+----+ | | | virtual | | | | PCI-PCI | | | | bridge | | | +----+----+ | | | | | |Bus#3 | | | | | +----------------------------------------+ | | | | | | | | | | | | |03:00.0 |03:01.0 |03:02.0 | | +----+----+ +----+----+ +----++---+ | | | virtual | | virtual | | virtual | | | | PCI-PCI | | PCI-PCI | | PCI-PCI | | | | bridge | | bridge | | bridge | | | +----+----+ +----+----+ +----+----+ | | |Bus#4? | | | | -----+------- | | | +----------|--------------------|-------------------|----------+ | | |v v v
Quoted 4 lines
A PCIe switch appears as two or more PCI-PCI bridges. One is associated with the upstream port; the others with the downstream ports.
Quoted 6 lines
Quoted 4 lines
A bridge always has a primary side and a secondary side. In your diagram, the bridge associated with the upstream port would be 00:01.0 (primary bus 00) and could have a secondary bus of 03 (since 02 is already consumed by the PCIe-PCI bridge).
Hmm... I am confused why is 03. 02 is used but 01 is not used. Switch should be configured after PCIe2PCI bridge?
It's likely that the PCIe switch would be configured first, since its device number is lower, but that is not a requirement. The requirement is that the bus number ranges consumed by bridges be non-overlapping. In this case (using your original topology plus my PCIe switch diagram), we'd have:
1. an endpoint at 00:00.0 -- consumes no additional buses
2. a PCIe switch at 00:01.0 -- consumes at least [bus 03-06]
3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 02] (its secondary bus)
Bus 03 is the internal PCIe switch bus that connects the upstream port to the downstream ports. Buses 04, 05, and 06 are the links originating from the downstream ports.
Quoted 8 lines
Quoted 5 lines
The bridges associated with the downstream ports are all logically on bus 03. Their primary bus number would be 03; they might be 03:00.0, 03:01.0, 03:02.0, etc. Each would have its own secondary bus number, for example 04, 05, 06. That secondary bus number is for the downstream link from the corresponding downstream port.
Hmm, as you mentioned in previous letter, PCIe is an point-to-point protocol, then the secondary bus should reside in the Switch? Do you think my Bus#4 is correct?
No. Bus 04 is a PCIe link that connects the downstream port (03:00.0) to a single PCIe device. That device (04:00) could be a PCIe endpoint, or it could be the upstream port of another PCIe switch. (If it is a switch, more bus numbers would be required.)
Quoted 5 lines
Quoted 2 lines
The endpoints below the PCIe switch could then be 04:00.0 and 05:00.0 (or these could be the upstream ports of more PCIe switches).
So below the PCIe downstream port, there is only on PCIe device. The whole bus is occupied by this device?
That is why the device could have upto 256 functions?
Yes, if it supports ARI.
Bjorn
Quoted 104 lines · Bjorn Helgaas
Quoted 27 lines · Richard Yang
Quoted 3 lines · Bjorn Helgaas
On Fri, Apr 27, 2012 at 11:01 PM, Richard Yang
<[email protected]> wrote:Thanks for your nice chart.
Quoted 5 lines
I think assignments shown for the PCIe-to-PCI bridge are OK, although I would draw it like this because the bridge originates a single bus 02 that may have multiple devices attached to it (this side is PCI, not PCIe, so it really is a shared bus):
Quoted 17 lines
^ | +--------+--------+ | 00:02.0 | | PCIe-PCI bridge | | | +--------+--------+ | | +---------------------+ Bus 02 | | | | | | +----v----+ +----v----+ | 02:00.0 | | 02:01.0 | +---------+ +---------+Quoted 2 lines
So for this case, there is not internal bus, while this is really a physical shared bus, not a logical one.
Yes. The downstream side of the PCIe-PCI bridge is PCI.
Quoted 35 lines
Quoted 2 lines
I think the PCIe switch part is incorrect. Here's Figure 1-3 from sec 1.3.3 of the PCIe r3 spec:
^
|
+-------------------------------|------------------------------+
| | |
| +----+----+ |
| | virtual | |
| | PCI-PCI | |
| | bridge | |
| +----+----+ |
| | |
| |Bus#3 |
| | |
| +----------------------------------------+ |
| | | | |
| | | | |
| |03:00.0 |03:01.0 |03:02.0 |
| +----+----+ +----+----+ +----++---+ |
| | virtual | | virtual | | virtual | |
| | PCI-PCI | | PCI-PCI | | PCI-PCI | |
| | bridge | | bridge | | bridge | |
| +----+----+ +----+----+ +----+----+ |
| |Bus#4? | | |
| -----+------- | | |
+----------|--------------------|-------------------|----------+
| | |
v v vQuoted 4 lines
A PCIe switch appears as two or more PCI-PCI bridges. One is associated with the upstream port; the others with the downstream ports.
Quoted 6 lines
Quoted 4 lines
A bridge always has a primary side and a secondary side. In your diagram, the bridge associated with the upstream port would be 00:01.0 (primary bus 00) and could have a secondary bus of 03 (since 02 is already consumed by the PCIe-PCI bridge).
Hmm... I am confused why is 03. 02 is used but 01 is not used. Switch should be configured after PCIe2PCI bridge?
It's likely that the PCIe switch would be configured first, since its device number is lower, but that is not a requirement. The requirement is that the bus number ranges consumed by bridges be non-overlapping. In this case (using your original topology plus my PCIe switch diagram), we'd have:
1. an endpoint at 00:00.0 -- consumes no additional buses
2. a PCIe switch at 00:01.0 -- consumes at least [bus 03-06]
3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 02] (its secondary bus)Bus 03 is the internal PCIe switch bus that connects the upstream port to the downstream ports. Buses 04, 05, and 06 are the links originating from the downstream ports.
Quoted 8 lines
Quoted 5 lines
The bridges associated with the downstream ports are all logically on bus 03. Their primary bus number would be 03; they might be 03:00.0, 03:01.0, 03:02.0, etc. Each would have its own secondary bus number, for example 04, 05, 06. That secondary bus number is for the downstream link from the corresponding downstream port.
Hmm, as you mentioned in previous letter, PCIe is an point-to-point protocol, then the secondary bus should reside in the Switch? Do you think my Bus#4 is correct?
No. Bus 04 is a PCIe link that connects the downstream port (03:00.0) to a single PCIe device. That device (04:00) could be a PCIe endpoint, or it could be the upstream port of another PCIe switch. (If it is a switch, more bus numbers would be required.)
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
The endpoints below the PCIe switch could then be 04:00.0 and 05:00.0 (or these could be the upstream ports of more PCIe switches).
So below the PCIe downstream port, there is only on PCIe device. The whole bus is occupied by this device?
That is why the device could have upto 256 functions?Yes, if it supports ARI.
minor tweak: if the device *and* the downstream PCIe bridge support ARI... both must exist. I know of one such module, e.g., non-ARI PCIe switch w/PCIe endpoints) and the endpoints have SRIOV/VF support... not a very smart hw combo. .. nuf said.
Quoted 5 lines
Bjorn
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 54 lines · Bjorn Helgaas
Quoted 27 lines · Richard Yang
Quoted 3 lines · Bjorn Helgaas
On Fri, Apr 27, 2012 at 11:01 PM, Richard Yang
<[email protected]> wrote:Thanks for your nice chart.
Quoted 7 lines
I think assignments shown for the PCIe-to-PCI bridge are OK, although I would draw it like this because the bridge originates a single bus 02 that may have multiple devices attached to it (this side is PCI, not PCIe, so it really is a shared bus):
^
|Quoted 15 lines
+--------+--------+ | 00:02.0 | | PCIe-PCI bridge | | | +--------+--------+ | | +---------------------+ Bus 02 | | | | | | +----v----+ +----v----+ | 02:00.0 | | 02:01.0 | +---------+ +---------+Quoted 2 lines
So for this case, there is not internal bus, while this is really a physical shared bus, not a logical one.
Yes. The downstream side of the PCIe-PCI bridge is PCI.
Quoted 7 lines
Quoted 7 lines
I think the PCIe switch part is incorrect. Here's Figure 1-3 from sec 1.3.3 of the PCIe r3 spec:
A PCIe switch appears as two or more PCI-PCI bridges. One is associated with the upstream port; the others with the downstream ports.
Quoted 6 lines
Quoted 4 lines
A bridge always has a primary side and a secondary side. In your diagram, the bridge associated with the upstream port would be 00:01.0 (primary bus 00) and could have a secondary bus of 03 (since 02 is already consumed by the PCIe-PCI bridge).
Hmm... I am confused why is 03. 02 is used but 01 is not used. Switch should be configured after PCIe2PCI bridge?
It's likely that the PCIe switch would be configured first, since its device number is lower, but that is not a requirement. The requirement is that the bus number ranges consumed by bridges be non-overlapping. In this case (using your original topology plus my PCIe switch diagram), we'd have:
Quoted 3 lines
1. an endpoint at 00:00.0 -- consumes no additional buses
2. a PCIe switch at 00:01.0 -- consumes at least [bus 03-06]
3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 02] (its secondary bus)
So this is a sequence issue. Below is also a valid configuration.
2. a PCIe switch at 00:01.0 -- consumes at least [bus 02-05]
3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 06] (its secondary bus)
Quoted 10 lines
Bus 03 is the internal PCIe switch bus that connects the upstream port to the downstream ports. Buses 04, 05, and 06 are the links originating from the downstream ports.
Quoted 5 lines
Quoted 5 lines
The bridges associated with the downstream ports are all logically on bus 03. Their primary bus number would be 03; they might be 03:00.0, 03:01.0, 03:02.0, etc. Each would have its own secondary bus number, for example 04, 05, 06. That secondary bus number is for the downstream link from the corresponding downstream port.
Quoted 8 lines
Quoted 3 lines
Hmm, as you mentioned in previous letter, PCIe is an point-to-point protocol, then the secondary bus should reside in the Switch? Do you think my Bus#4 is correct?
No. Bus 04 is a PCIe link that connects the downstream port (03:00.0) to a single PCIe device. That device (04:00) could be a PCIe endpoint, or it could be the upstream port of another PCIe switch. (If it is a switch, more bus numbers would be required.)
^
|
+-------------------------------|------------------------------+
| | |
| +----+----+ |
| | virtual | |
| | PCI-PCI | |
| | bridge | |
| +----+----+ |
| | |
| |Bus#3 |
| | |
| +----------------------------------------+ |
| | | | |
| | | | |
| |03:00.0 |03:01.0 |03:02.0 |
| +----+----+ +----+----+ +----++---+ |
| | virtual | | virtual | | virtual | |
| | PCI-PCI | | PCI-PCI | | PCI-PCI | |
| | bridge | | bridge | | bridge | |
| +----+----+ +----+----+ +----+----+ |
| | | | |
| | | | |
+----------|--------------------|-------------------|----------+
| Bus#4? | |
v v v
So the link itself is Bus#4?
If there is no device under this link, will Bus#4 appear in kernel? And then the PCIe switch is represented by 4 pci_dev structure and each is bridge type?
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
The endpoints below the PCIe switch could then be 04:00.0 and 05:00.0 (or these could be the upstream ports of more PCIe switches).
So below the PCIe downstream port, there is only on PCIe device. The whole bus is occupied by this device?
That is why the device could have upto 256 functions?Yes, if it supports ARI.
Bjorn
Signature · 2 lines
Richard Yang Help you, Help me
Bjorn Helgaas
maintainerQuoted 55 lines · Richard Yang
Quoted 54 lines · Bjorn Helgaas
Quoted 27 lines · Richard Yang
Quoted 3 lines · Bjorn Helgaas
On Fri, Apr 27, 2012 at 11:01 PM, Richard Yang
<[email protected]> wrote:Thanks for your nice chart.
Quoted 7 lines
I think assignments shown for the PCIe-to-PCI bridge are OK, although I would draw it like this because the bridge originates a single bus 02 that may have multiple devices attached to it (this side is PCI, not PCIe, so it really is a shared bus):
^
|Quoted 15 lines
+--------+--------+ | 00:02.0 | | PCIe-PCI bridge | | | +--------+--------+ | | +---------------------+ Bus 02 | | | | | | +----v----+ +----v----+ | 02:00.0 | | 02:01.0 | +---------+ +---------+Quoted 2 lines
So for this case, there is not internal bus, while this is really a physical shared bus, not a logical one.
Yes. The downstream side of the PCIe-PCI bridge is PCI.
Quoted 7 lines
Quoted 7 lines
I think the PCIe switch part is incorrect. Here's Figure 1-3 from sec 1.3.3 of the PCIe r3 spec:
A PCIe switch appears as two or more PCI-PCI bridges. One is associated with the upstream port; the others with the downstream ports.
Quoted 6 lines
Quoted 4 lines
A bridge always has a primary side and a secondary side. In your diagram, the bridge associated with the upstream port would be 00:01.0 (primary bus 00) and could have a secondary bus of 03 (since 02 is already consumed by the PCIe-PCI bridge).
Hmm... I am confused why is 03. 02 is used but 01 is not used. Switch should be configured after PCIe2PCI bridge?
It's likely that the PCIe switch would be configured first, since its device number is lower, but that is not a requirement. The requirement is that the bus number ranges consumed by bridges be non-overlapping. In this case (using your original topology plus my PCIe switch diagram), we'd have:
Quoted 7 lines
Quoted 3 lines
1. an endpoint at 00:00.0 -- consumes no additional buses
2. a PCIe switch at 00:01.0 -- consumes at least [bus 03-06]
3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 02] (its secondary bus)So this is a sequence issue. Below is also a valid configuration.
2. a PCIe switch at 00:01.0 -- consumes at least [bus 02-05]
3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 06] (its secondary bus)
Yes, this would also be valid.
Quoted 44 lines
Quoted 9 lines
Bus 03 is the internal PCIe switch bus that connects the upstream port to the downstream ports. Buses 04, 05, and 06 are the links originating from the downstream ports.
Quoted 5 lines
Quoted 5 lines
The bridges associated with the downstream ports are all logically on bus 03. Their primary bus number would be 03; they might be 03:00.0, 03:01.0, 03:02.0, etc. Each would have its own secondary bus number, for example 04, 05, 06. That secondary bus number is for the downstream link from the corresponding downstream port.
Quoted 8 lines
Quoted 3 lines
Hmm, as you mentioned in previous letter, PCIe is an point-to-point protocol, then the secondary bus should reside in the Switch? Do you think my Bus#4 is correct?
No. Bus 04 is a PCIe link that connects the downstream port (03:00.0) to a single PCIe device. That device (04:00) could be a PCIe endpoint, or it could be the upstream port of another PCIe switch. (If it is a switch, more bus numbers would be required.)
^
|+-------------------------------|------------------------------+ | | | | +----+----+ | | | virtual | | | | PCI-PCI | | | | bridge | | | +----+----+ | | | | | |Bus#3 | | | | | +----------------------------------------+ | | | | | | | | | | | | |03:00.0 |03:01.0 |03:02.0 | | +----+----+ +----+----+ +----++---+ | | | virtual | | virtual | | virtual | | | | PCI-PCI | | PCI-PCI | | PCI-PCI | | | | bridge | | bridge | | bridge | | | +----+----+ +----+----+ +----+----+ | | | | | | | | | | | +----------|--------------------|-------------------|----------+ | Bus#4? | |
Quoted 3 lines
v v v
So the link itself is Bus#4?
Exactly.
Quoted 1 line
If there is no device under this link, will Bus#4 appear in kernel?
Yes, I'm pretty sure we build the bus 04 pci_bus structure, then scan bus 04 for devices. Even if we find none, the pci_bus struct for bus 04 remains.
Quoted 2 lines
And then the PCIe switch is represented by 4
pci_devstructure and each is bridge type?
Yes. And each has an associated pci_bus struct for its secondary bus.
Bjorn
Quoted 119 lines · Bjorn Helgaas
Quoted 55 lines · Richard Yang
Quoted 54 lines · Bjorn Helgaas
Quoted 27 lines · Richard Yang
Quoted 3 lines · Bjorn Helgaas
On Fri, Apr 27, 2012 at 11:01 PM, Richard Yang
<[email protected]> wrote:Thanks for your nice chart.
Quoted 7 lines
I think assignments shown for the PCIe-to-PCI bridge are OK, although I would draw it like this because the bridge originates a single bus 02 that may have multiple devices attached to it (this side is PCI, not PCIe, so it really is a shared bus):
^
|Quoted 15 lines
+--------+--------+ | 00:02.0 | | PCIe-PCI bridge | | | +--------+--------+ | | +---------------------+ Bus 02 | | | | | | +----v----+ +----v----+ | 02:00.0 | | 02:01.0 | +---------+ +---------+Quoted 2 lines
So for this case, there is not internal bus, while this is really a physical shared bus, not a logical one.
Yes. The downstream side of the PCIe-PCI bridge is PCI.
Quoted 7 lines
Quoted 7 lines
I think the PCIe switch part is incorrect. Here's Figure 1-3 from sec 1.3.3 of the PCIe r3 spec:
A PCIe switch appears as two or more PCI-PCI bridges. One is associated with the upstream port; the others with the downstream ports.
Quoted 6 lines
Quoted 4 lines
A bridge always has a primary side and a secondary side. In your diagram, the bridge associated with the upstream port would be 00:01.0 (primary bus 00) and could have a secondary bus of 03 (since 02 is already consumed by the PCIe-PCI bridge).
Hmm... I am confused why is 03. 02 is used but 01 is not used. Switch should be configured after PCIe2PCI bridge?
It's likely that the PCIe switch would be configured first, since its device number is lower, but that is not a requirement. The requirement is that the bus number ranges consumed by bridges be non-overlapping. In this case (using your original topology plus my PCIe switch diagram), we'd have:
Quoted 7 lines
Quoted 3 lines
1. an endpoint at 00:00.0 -- consumes no additional buses
2. a PCIe switch at 00:01.0 -- consumes at least [bus 03-06]
3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 02] (its secondary bus)So this is a sequence issue. Below is also a valid configuration.
2. a PCIe switch at 00:01.0 -- consumes at least [bus 02-05]
3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 06] (its secondary bus)Yes, this would also be valid.
Quoted 44 lines
Quoted 9 lines
Bus 03 is the internal PCIe switch bus that connects the upstream port to the downstream ports. Buses 04, 05, and 06 are the links originating from the downstream ports.
Quoted 5 lines
Quoted 5 lines
The bridges associated with the downstream ports are all logically on bus 03. Their primary bus number would be 03; they might be 03:00.0, 03:01.0, 03:02.0, etc. Each would have its own secondary bus number, for example 04, 05, 06. That secondary bus number is for the downstream link from the corresponding downstream port.
Quoted 8 lines
Quoted 3 lines
Hmm, as you mentioned in previous letter, PCIe is an point-to-point protocol, then the secondary bus should reside in the Switch? Do you think my Bus#4 is correct?
No. Bus 04 is a PCIe link that connects the downstream port (03:00.0) to a single PCIe device. That device (04:00) could be a PCIe endpoint, or it could be the upstream port of another PCIe switch. (If it is a switch, more bus numbers would be required.)
^
|+-------------------------------|------------------------------+ | | | | +----+----+ | | | virtual | | | | PCI-PCI | | | | bridge | | | +----+----+ | | | | | |Bus#3 | | | | | +----------------------------------------+ | | | | | | | | | | | | |03:00.0 |03:01.0 |03:02.0 | | +----+----+ +----+----+ +----++---+ | | | virtual | | virtual | | virtual | | | | PCI-PCI | | PCI-PCI | | PCI-PCI | | | | bridge | | bridge | | bridge | | | +----+----+ +----+----+ +----+----+ | | | | | | | | | | | +----------|--------------------|-------------------|----------+ | Bus#4? | |Quoted 3 lines
v v v
So the link itself is Bus#4?
Exactly.
Quoted 1 line
If there is no device under this link, will Bus#4 appear in kernel?
Quoted 8 lines
Yes, I'm pretty sure we build the bus 04
pci_busstructure, then scan bus 04 for devices. Even if we find none, thepci_busstruct for bus 04 remains.Quoted 2 lines
And then the PCIe switch is represented by 4
pci_devstructure and each is bridge type?Yes. And each has an associated
pci_busstruct for its secondary bus.
Bjorn
Thanks a lot :)
Your patient reply answer my question clearly.
Those questions confused me a lot.
Now I think I get a more clear view about how pci system is represented in the kernel.
Thanks again.
Quoted 2 lines
Bjorn
Signature · 2 lines
Richard Yang Help you, Help me
Quoted 61 lines · Bjorn Helgaas
Quoted 58 lines · Richard Yang
Quoted 57 lines · Bjorn Helgaas
Quoted 27 lines · Richard Yang
Quoted 3 lines · Bjorn Helgaas
On Fri, Apr 27, 2012 at 11:01 PM, Richard Yang
<[email protected]> wrote:Thanks for your nice chart.
Quoted 7 lines
I think assignments shown for the PCIe-to-PCI bridge are OK, although I would draw it like this because the bridge originates a single bus 02 that may have multiple devices attached to it (this side is PCI, not PCIe, so it really is a shared bus):
^
|Quoted 15 lines
+--------+--------+ | 00:02.0 | | PCIe-PCI bridge | | | +--------+--------+ | | +---------------------+ Bus 02 | | | | | | +----v----+ +----v----+ | 02:00.0 | | 02:01.0 | +---------+ +---------+Quoted 2 lines
So for this case, there is not internal bus, while this is really a physical shared bus, not a logical one.
Yes. The downstream side of the PCIe-PCI bridge is PCI.
Quoted 7 lines
Quoted 7 lines
I think the PCIe switch part is incorrect. Here's Figure 1-3 from sec 1.3.3 of the PCIe r3 spec:
A PCIe switch appears as two or more PCI-PCI bridges. One is associated with the upstream port; the others with the downstream ports.
Quoted 6 lines
Quoted 4 lines
A bridge always has a primary side and a secondary side. In your diagram, the bridge associated with the upstream port would be 00:01.0 (primary bus 00) and could have a secondary bus of 03 (since 02 is already consumed by the PCIe-PCI bridge).
Hmm... I am confused why is 03. 02 is used but 01 is not used. Switch should be configured after PCIe2PCI bridge?
It's likely that the PCIe switch would be configured first, since its device number is lower, but that is not a requirement. The requirement is that the bus number ranges consumed by bridges be non-overlapping. In this case (using your original topology plus my PCIe switch diagram), we'd have:
1. an endpoint at 00:00.0 -- consumes no additional buses
2. a PCIe switch at 00:01.0 -- consumes at least [bus 03-06]
3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 02] (its secondary bus)
Quoted 53 lines
Quoted 3 lines
So this is a sequence issue. Below is also a valid configuration.
2. a PCIe switch at 00:01.0 -- consumes at least [bus 02-05]
3. a PCIe-PCI bridge at 00:02.0 -- consumes least [bus 06] (its secondary bus)Yes, this would also be valid.
Quoted 47 lines
Quoted 9 lines
Bus 03 is the internal PCIe switch bus that connects the upstream port to the downstream ports. Buses 04, 05, and 06 are the links originating from the downstream ports.
Quoted 5 lines
Quoted 5 lines
The bridges associated with the downstream ports are all logically on bus 03. Their primary bus number would be 03; they might be 03:00.0, 03:01.0, 03:02.0, etc. Each would have its own secondary bus number, for example 04, 05, 06. That secondary bus number is for the downstream link from the corresponding downstream port.
Quoted 8 lines
Quoted 3 lines
Hmm, as you mentioned in previous letter, PCIe is an point-to-point protocol, then the secondary bus should reside in the Switch? Do you think my Bus#4 is correct?
No. Bus 04 is a PCIe link that connects the downstream port (03:00.0) to a single PCIe device. That device (04:00) could be a PCIe endpoint, or it could be the upstream port of another PCIe switch. (If it is a switch, more bus numbers would be required.)
^
|+-------------------------------|------------------------------+ | | | | +----+----+ | | | virtual | | | | PCI-PCI | | | | bridge | | | +----+----+ | | | | | |Bus#3 | | | | | +----------------------------------------+ | | | | | | | | | | | | |03:00.0 |03:01.0 |03:02.0 | | +----+----+ +----+----+ +----++---+ | | | virtual | | virtual | | virtual | | | | PCI-PCI | | PCI-PCI | | PCI-PCI | | | | bridge | | bridge | | bridge | | | +----+----+ +----+----+ +----+----+ | | | | | | | | | | | +----------|--------------------|-------------------|----------+ | Bus#4? | |v v v
So the link itself is Bus#4?
Bjorn,
Sorry to disturb you again.
If under this link, one SRIOV device is attached. And VF is enabled, more than 256 VFs.
How the Bus#4 handle those PFs and VFs? How new Buses is organized?
Or this is not standarded?
Quoted 15 lines
Exactly.
Quoted 1 line
If there is no device under this link, will Bus#4 appear in kernel?
Yes, I'm pretty sure we build the bus 04
pci_busstructure, then scan bus 04 for devices. Even if we find none, thepci_busstruct for bus 04 remains.Quoted 2 lines
And then the PCIe switch is represented by 4
pci_devstructure and each is bridge type?Yes. And each has an associated
pci_busstruct for its secondary bus.Bjorn
Signature · 2 lines
Richard Yang Help you, Help me
Quoted 21 lines · Bjorn Helgaas
Quoted 21 lines
+-------------------------------|------------------------------+ | | | | +----+----+ | | | virtual | | | | PCI-PCI | | | | bridge | | | +----+----+ | | | | |Bus#3 | | | | +----------------------------------------+ | | | | | | | | | | | | |03:00.0 |03:01.0 |03:02.0 | | +----+----+ +----+----+ +----++---+ | | | virtual | | virtual | | virtual | | | | PCI-PCI | | PCI-PCI | | PCI-PCI | |
Quoted 8 lines
Quoted 8 lines
| | bridge | | bridge | | bridge | | | +----+----+ +----+----+ +----+----+ | | | | | | | | | | | +----------|--------------------|-------------------|----------+ | Bus#4? | |v v v
I read the code of pci_scan_child_bus(), in this function it will call pci_scan_slot() like this.
for (devfn = 0; devfn < 0x100; devfn += 8) pci_scan_slot(bus, devfn);
I think this scheme is based on the PCI LB specification.
If the bus->self is ari enabled, one call of pci_scan_slot(bus, 0) will scan all the functions under this bus.
Then next 2^5-1 times of pci_scan_slot() will configure the function which has already been configured, or even empty function.
My idea is
if the bus is ari enabled, just one call of pci_scan_slot() will be called.
Do you think this works? Or may I miss some real case?
Quoted 16 lines
Quoted 1 line
So the link itself is Bus#4?
Exactly.
Quoted 1 line
If there is no device under this link, will Bus#4 appear in kernel?
Yes, I'm pretty sure we build the bus 04
pci_busstructure, then scan bus 04 for devices. Even if we find none, thepci_busstruct for bus 04 remains.Quoted 2 lines
And then the PCIe switch is represented by 4
pci_devstructure and each is bridge type?Yes. And each has an associated
pci_busstruct for its secondary bus.Bjorn
Signature · 2 lines
Richard Yang Help you, Help me
Quoted 11 lines · Bjorn Helgaas
I assume your question relates to the Stratus ftServer topology. If so, the lspci details might clarify things.
In that system, my understanding is that 03:01.0 is a downstream port, not an upstream port.
I think your picture is slightly misleading because PCIe links are not buses; they're point-to-point links between two devices. You've drawn #3 and #5 as buses that can have several devices on them, which is not really the case. The link from a downstream port should lead to exactly one device.
Hmm... if this is a PCI bridge, then it will have both internal bus and externel bus? or no internal pci bus, just one external bus?
Quoted 9 lines
That's one thing that's strange in the ftServer topology: apparently there are *two* devices on bus 03: the 03:00.0 upstream port and the 03:01.0 downstream port. I think 03:00.0 is the upstream port of a PCIe switch, which is perfectly normal. My understanding is that 03:01.0 is another *downstream* port that leads to several more devices (USB, NIC, etc).
Bjorn
Signature · 2 lines
Richard Yang Help you, Help me