Skip to contentSkip to conversation

Proposed PCI ECR for description of Advanced Peer to Peer Capabilities

RFC Jonathan Cameron 5 participants 10 messages Dec 10, 2018 – Dec 13, 2018 PCI

lore
Share & tools
lore
#0 lore raw

Hi All,

This is perhaps a somewhat unusual RFC as it's to ask for feedback on an ECR that is currently under discussion in the PCI SIG. That ECR, at least partly, came out of the issues highlighted during the discussion of P2PDMA. In that patch set, p2pdma is only allowed if there is a switch between the devices involved ensuring the path does not pass through a root port. Put simply there is no way to know if a root complex will allow p2p or not.

So we are asking for feedback at a stage where modifications to the proposal can be made. Eric has agreed to this being shared with mailing list to hopefully get such feedback. Note this is under active discussion within the SIG so inputs in that forum also welcome.

I would recommend reading the ECR and presentation for more details. Anyone who can't get to Google Drive, email me and I'll send a copy directly.

ECR: https://drive.google.com/open?id=1X08pfyjkNwSfpTVBuiut90ajxBZd7BXg
Slides: https://drive.google.com/open?id=11d3egA8x91d99-QqfYGZ-MxLGrHpJHCK

My summary probably won't be great, but I'll give it a go to persuade people to look at the real docs!

Problem Statement:

Linux does not generally enable Peer to Peer because (amongst other reasons) it does not know which Root Complexes support Peer to Peer.

* Some Root Complexes do not support Peer to Peer
* Some Root Complexes only support Posted Peer to Peer
* Some Root Complexes only support Peer to Peer within one PCIe Segment
* Some Root Complexes only support Address Decode Peer to Peer (no ID routing)
* Some Root Complexes may not support Peer to Peer from Integrated End Points

Related other issue
* MPS must be programmed the same across all devices supporting P2P
* Hot plugged devices might require a smaller MPS than was programmed.

Solution:
* New extended capability and control (though only one bit in the control)
* Advertise what a Peer to Peer Root Port and Switch Down Stream ports can do.
* Define packet splitting scheme for MPS mismatches.
- Rules on how this must be done if supported.

All feedback welcome. Let's make sure we get this right.

Thanks,

Jonathan Cameron
Huawei

#1 lore raw

Hi Jonathan,

Thanks for taking this on.

Quoted 5 lines · Jonathan Cameron

I would recommend reading the ECR and presentation for more details. Anyone who can't get to Google Drive, email me and I'll send a copy directly.

ECR: https://drive.google.com/open?id=1X08pfyjkNwSfpTVBuiut90ajxBZd7BXg
Slides: https://drive.google.com/open?id=11d3egA8x91d99-QqfYGZ-MxLGrHpJHCK

I've looked through the ECR now a couple times and haven't really spotted any major issues with it.

The minimum supported bandwidth requirement is pretty vague and my biggest concern is that hardware vendors must report consistently otherwise this is useless. So I'd maybe suggest expanding on what's required before hardware is allowed to indicate support.

Also, one thought is, it might be better to add another bit indicating performance. So the hardware can say: these TLPs will be supported but they will not be supported at high bandwidth.

In terms of defining high bandwidth: what I care about is whether the bandwidth through the P2P link will be significantly slower than the bandwidth of going to/from system RAM. If that's the case it probably means a subset of P2P applications would rather just transfer to RAM instead of going P2P.

Logan

#2 lore raw

Hi Logan,

Thank you for reviewing this.

Although I mentioned the minimum bandwidth requirement in the introduction of the ECR, I did not include any specification changes that would require this. I recognized that for p2p to be useful, there must be a useful level of bandwidth, however, I could see no easy way to adequately advertise this. Ultimately, bandwidth levels could be seen as part of the platform architecture of each OEM. Performance differences between OEM's is often what makes a buyer choose one OEM's system over another in a competitive environment. So requiring a particular bandwidth level is like requiring an OEM to include a minimum number of cores in a system or a minimum number of PCIe links. So ultimately, I excluded any discussion of bandwidth in the ECR spec changes.

However, if you see a particular way that this issue could be approached, I am still willing to consider a change.

Eric Wehage
Huawei Principal Engineer, PCIe

-----Original Message-----
From: Logan Gunthorpe [mailto:[email protected]]
Sent: Monday, December 10, 2018 9:20 AM
To: Jonathan Cameron <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Eric Wehage <[email protected]>; Linuxarm <[email protected]>; Stephen Bates <[email protected]>
Subject: Re: [RFC] Proposed PCI ECR for description of Advanced Peer to Peer Capabilities

Hi Jonathan,

Thanks for taking this on.

Quoted 7 lines · Jonathan Cameron

I would recommend reading the ECR and presentation for more details. Anyone who can't get to Google Drive, email me and I'll send a copy directly.

ECR:
https://drive.google.com/open?id=1X08pfyjkNwSfpTVBuiut90ajxBZd7BXg
Slides:
https://drive.google.com/open?id=11d3egA8x91d99-QqfYGZ-MxLGrHpJHCK

I've looked through the ECR now a couple times and haven't really spotted any major issues with it.

The minimum supported bandwidth requirement is pretty vague and my biggest concern is that hardware vendors must report consistently otherwise this is useless. So I'd maybe suggest expanding on what's required before hardware is allowed to indicate support.

Also, one thought is, it might be better to add another bit indicating performance. So the hardware can say: these TLPs will be supported but they will not be supported at high bandwidth.

In terms of defining high bandwidth: what I care about is whether the bandwidth through the P2P link will be significantly slower than the bandwidth of going to/from system RAM. If that's the case it probably means a subset of P2P applications would rather just transfer to RAM instead of going P2P.

Logan

#3 lore raw
Quoted 7 lines · Eric Wehage

Hi Logan,

Thank you for reviewing this.

Although I mentioned the minimum bandwidth requirement in the introduction of the ECR, I did not include any specification changes that would require this. I recognized that for p2p to be useful, there must be a useful level of bandwidth, however, I could see no easy way to adequately advertise this. Ultimately, bandwidth levels could be seen as part of the platform architecture of each OEM. Performance differences between OEM's is often what makes a buyer choose one OEM's system over another in a competitive environment. So requiring a particular bandwidth level is like requiring an OEM to include a minimum number of cores in a system or a minimum number of PCIe links. So ultimately, I excluded any discussion of bandwidth in the ECR spec changes.

However, if you see a particular way that this issue could be approached, I am still willing to consider a change.

Yeah, per my last email, I don't want to advertise bandwidth. And I gave a rough rule as to what might qualify as "sufficient" bandwidth (ie. a boolean indicating whether it is roughly expected to be as fast or faster than RAM).

I also was trying to get the point across that even if a particular piece of hardware does not have "sufficient" bandwidth it would still be valuable to publish whether the link is expected to work for the different TLP types. Some applications don't care about bandwidth and just want to know if they can ring doorbells, etc.

Logan

#4 lore raw

Hi Eric/Jonathan

I'd also like to thank you for taking this work on.

Quoted 2 lines

However, if you see a particular way that this issue could be approached, I am still willing to consider a change.

I would echo Logan's sentiments that a simple Boolean to indicate the p2p performance between RPs is comparable (or better) than RP<->DRAM performance or not would be useful and sufficient. It might be useful to at least add such a field to the ECR so it can be discussed within PCI-SIG.

Cheers

Stephen

#9 lore raw
In reply to Jonathan Cameron
Quoted 44 lines · Jonathan Cameron

Hi All,

This is perhaps a somewhat unusual RFC as it's to ask for feedback on an ECR that is currently under discussion in the PCI SIG. That ECR, at least partly, came out of the issues highlighted during the discussion of P2PDMA. In that patch set, p2pdma is only allowed if there is a switch between the devices involved ensuring the path does not pass through a root port. Put simply there is no way to know if a root complex will allow p2p or not.

So we are asking for feedback at a stage where modifications to the proposal can be made. Eric has agreed to this being shared with mailing list to hopefully get such feedback. Note this is under active discussion within the SIG so inputs in that forum also welcome.

I would recommend reading the ECR and presentation for more details. Anyone who can't get to Google Drive, email me and I'll send a copy directly.

ECR: https://drive.google.com/open?id=1X08pfyjkNwSfpTVBuiut90ajxBZd7BXg
Slides: https://drive.google.com/open?id=11d3egA8x91d99-QqfYGZ-MxLGrHpJHCK

My summary probably won't be great, but I'll give it a go to persuade people to look at the real docs!

Problem Statement:

Linux does not generally enable Peer to Peer because (amongst other reasons) it does not know which Root Complexes support Peer to Peer.

* Some Root Complexes do not support Peer to Peer
* Some Root Complexes only support Posted Peer to Peer
* Some Root Complexes only support Peer to Peer within one PCIe Segment
* Some Root Complexes only support Address Decode Peer to Peer (no ID routing)
* Some Root Complexes may not support Peer to Peer from Integrated End Points

Related other issue
* MPS must be programmed the same across all devices supporting P2P
* Hot plugged devices might require a smaller MPS than was programmed.

Solution:
* New extended capability and control (though only one bit in the control)
* Advertise what a Peer to Peer Root Port and Switch Down Stream ports can do.
* Define packet splitting scheme for MPS mismatches.
- Rules on how this must be done if supported.

Quoted 6 lines

All feedback welcome. Let's make sure we get this right.

Thanks,

Jonathan Cameron
Huawei

Looking this over some of this seems redundant with the ACS P2P Request Redirect and Completion Redirect functionality.

I was curious if there was a reason why you couldn't use something like that to at least get some early enabling going for peer-to-peer root port pass through?

Also, I have not seen called out anywhere in your documents how this might interact with something like Address Translation Services (ATS). I'm assuming there would be scenarios where the translation agent would need to be a part of the p2p topology.

Thanks.

- Alex

#5 lore raw
In reply to Logan Gunthorpe

There are 5 TLP types described in each of the three P2P Routes. I'm not sure that anyone would care about Message Bandwidth, so I could simply add 3 bits to each Route register indicating whether the P2P route bandwidth supports the same or more bandwidth than if it targeted DRAM.

The three RO/HWInit bits for each route register would be:
- P2P Memory Read Bandwidth is equal to or greater than to Main Memory
- P2P Memory Write Bandwidth is equal to or greater than to Main Memory
- P2P Memory AtomicOp Bandwidth is equal to or greater than to Main Memory

Eric Wehage
Huawei Principal Engineer, PCIe

#6 lore raw
Quoted 6 lines · Eric Wehage

There are 5 TLP types described in each of the three P2P Routes. I'm not sure that anyone would care about Message Bandwidth, so I could simply add 3 bits to each Route register indicating whether the P2P route bandwidth supports the same or more bandwidth than if it targeted DRAM.

Quoted 7 lines

The three RO/HWInit bits for each route register would be:
- P2P Memory Read Bandwidth is equal to or greater than to Main Memory
- P2P Memory Write Bandwidth is equal to or greater than to Main Memory
- P2P Memory AtomicOp Bandwidth is equal to or greater than to Main Memory

Yeah! That sounds great. Thanks.

Logan

#7 lore raw
Quoted 15 lines · On Mon, 10 Dec 2018 12:39:36 -0700 Logan Gunthorpe <[email protected]> wrote:
Quoted 6 lines · Eric Wehage

There are 5 TLP types described in each of the three P2P Routes. I'm not sure that anyone would care about Message Bandwidth, so I could simply add 3 bits to each Route register indicating whether the P2P route bandwidth supports the same or more bandwidth than if it targeted DRAM.

Quoted 7 lines

The three RO/HWInit bits for each route register would be:
- P2P Memory Read Bandwidth is equal to or greater than to Main Memory
- P2P Memory Write Bandwidth is equal to or greater than to Main Memory
- P2P Memory AtomicOp Bandwidth is equal to or greater than to Main Memory

Quoted 3 lines

Yeah! That sounds great. Thanks.

Logan

Updated ECR and presentation at:
Presentation: https://drive.google.com/open?id=1fgwz8w32K2ju9aIySP8RXwQwUAcT9icF
ECR: https://drive.google.com/open?id=1cHn4OHgTJpa0KkiVab_CyK1NMjPap3MM

A few small editorial bits and pieces.

The main change is making the Memory Read, Memory Write and Atomic capabilities in addition to indicating they support peer to peer at all, can now give an indication of supporting 'same bandwidth as to system memory'.

Note that from a PCI viewpoint system memory really means anything beyond the host so caches etc are all rolled into that term.

Thanks,

Eric and Jonathan

#8 lore raw
Quoted 4 lines · Jonathan Cameron

Updated ECR and presentation at:
Presentation: https://drive.google.com/open?id=1fgwz8w32K2ju9aIySP8RXwQwUAcT9icF
ECR: https://drive.google.com/open?id=1cHn4OHgTJpa0KkiVab_CyK1NMjPap3MM

Quoted 8 lines

A few small editorial bits and pieces.

The main change is making the Memory Read, Memory Write and Atomic capabilities in addition to indicating they support peer to peer at all, can now give an indication of supporting 'same bandwidth as to system memory'.

Note that from a PCI viewpoint system memory really means anything beyond the host so caches etc are all rolled into that term.

Looks good to me. Thanks,

Logan