Proposed PCI ECR for description of Advanced Peer to Peer Capabilities
Hi All,
This is perhaps a somewhat unusual RFC as it's to ask for feedback on an ECR that is currently under discussion in the PCI SIG. That ECR, at least partly, came out of the issues highlighted during the discussion of P2PDMA. In that patch set, p2pdma is only allowed if there is a switch between the devices involved ensuring the path does not pass through a root port. Put simply there is no way to know if a root complex will allow p2p or not.
So we are asking for feedback at a stage where modifications to the proposal can be made. Eric has agreed to this being shared with mailing list to hopefully get such feedback. Note this is under active discussion within the SIG so inputs in that forum also welcome.
I would recommend reading the ECR and presentation for more details. Anyone who can't get to Google Drive, email me and I'll send a copy directly.
ECR: https://drive.google.com/open?id=1X08pfyjkNwSfpTVBuiut90ajxBZd7BXg
Slides: https://drive.google.com/open?id=11d3egA8x91d99-QqfYGZ-MxLGrHpJHCK
My summary probably won't be great, but I'll give it a go to persuade people to look at the real docs!
Problem Statement:
Linux does not generally enable Peer to Peer because (amongst other reasons) it does not know which Root Complexes support Peer to Peer.
* Some Root Complexes do not support Peer to Peer
* Some Root Complexes only support Posted Peer to Peer
* Some Root Complexes only support Peer to Peer within one PCIe Segment
* Some Root Complexes only support Address Decode Peer to Peer (no ID routing)
* Some Root Complexes may not support Peer to Peer from Integrated End Points
Related other issue
* MPS must be programmed the same across all devices supporting P2P
* Hot plugged devices might require a smaller MPS than was programmed.
Solution:
* New extended capability and control (though only one bit in the control)
* Advertise what a Peer to Peer Root Port and Switch Down Stream ports can do.
* Define packet splitting scheme for MPS mismatches.
- Rules on how this must be done if supported.
All feedback welcome. Let's make sure we get this right.
Thanks,
Jonathan Cameron
Huawei
Hi Jonathan,
Thanks for taking this on.
Quoted 5 lines · Jonathan Cameron
I would recommend reading the ECR and presentation for more details. Anyone who can't get to Google Drive, email me and I'll send a copy directly.
ECR: https://drive.google.com/open?id=1X08pfyjkNwSfpTVBuiut90ajxBZd7BXg
Slides: https://drive.google.com/open?id=11d3egA8x91d99-QqfYGZ-MxLGrHpJHCK
I've looked through the ECR now a couple times and haven't really spotted any major issues with it.
The minimum supported bandwidth requirement is pretty vague and my biggest concern is that hardware vendors must report consistently otherwise this is useless. So I'd maybe suggest expanding on what's required before hardware is allowed to indicate support.
Also, one thought is, it might be better to add another bit indicating performance. So the hardware can say: these TLPs will be supported but they will not be supported at high bandwidth.
In terms of defining high bandwidth: what I care about is whether the bandwidth through the P2P link will be significantly slower than the bandwidth of going to/from system RAM. If that's the case it probably means a subset of P2P applications would rather just transfer to RAM instead of going P2P.
Logan
Hi Logan,
Thank you for reviewing this.
Although I mentioned the minimum bandwidth requirement in the introduction of the ECR, I did not include any specification changes that would require this. I recognized that for p2p to be useful, there must be a useful level of bandwidth, however, I could see no easy way to adequately advertise this. Ultimately, bandwidth levels could be seen as part of the platform architecture of each OEM. Performance differences between OEM's is often what makes a buyer choose one OEM's system over another in a competitive environment. So requiring a particular bandwidth level is like requiring an OEM to include a minimum number of cores in a system or a minimum number of PCIe links. So ultimately, I excluded any discussion of bandwidth in the ECR spec changes.
However, if you see a particular way that this issue could be approached, I am still willing to consider a change.
Eric Wehage
Huawei Principal Engineer, PCIe
-----Original Message-----
From: Logan Gunthorpe [mailto:[email protected]]
Sent: Monday, December 10, 2018 9:20 AM
To: Jonathan Cameron <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; Eric Wehage <[email protected]>; Linuxarm <[email protected]>; Stephen Bates <[email protected]>
Subject: Re: [RFC] Proposed PCI ECR for description of Advanced Peer to Peer Capabilities
Hi Jonathan,
Thanks for taking this on.
Quoted 7 lines · Jonathan Cameron
I would recommend reading the ECR and presentation for more details. Anyone who can't get to Google Drive, email me and I'll send a copy directly.
ECR:
https://drive.google.com/open?id=1X08pfyjkNwSfpTVBuiut90ajxBZd7BXg
Slides:
https://drive.google.com/open?id=11d3egA8x91d99-QqfYGZ-MxLGrHpJHCK
I've looked through the ECR now a couple times and haven't really spotted any major issues with it.
The minimum supported bandwidth requirement is pretty vague and my biggest concern is that hardware vendors must report consistently otherwise this is useless. So I'd maybe suggest expanding on what's required before hardware is allowed to indicate support.
Also, one thought is, it might be better to add another bit indicating performance. So the hardware can say: these TLPs will be supported but they will not be supported at high bandwidth.
In terms of defining high bandwidth: what I care about is whether the bandwidth through the P2P link will be significantly slower than the bandwidth of going to/from system RAM. If that's the case it probably means a subset of P2P applications would rather just transfer to RAM instead of going P2P.
Logan
Quoted 7 lines · Eric Wehage
Hi Logan,
Thank you for reviewing this.
Although I mentioned the minimum bandwidth requirement in the introduction of the ECR, I did not include any specification changes that would require this. I recognized that for p2p to be useful, there must be a useful level of bandwidth, however, I could see no easy way to adequately advertise this. Ultimately, bandwidth levels could be seen as part of the platform architecture of each OEM. Performance differences between OEM's is often what makes a buyer choose one OEM's system over another in a competitive environment. So requiring a particular bandwidth level is like requiring an OEM to include a minimum number of cores in a system or a minimum number of PCIe links. So ultimately, I excluded any discussion of bandwidth in the ECR spec changes.
However, if you see a particular way that this issue could be approached, I am still willing to consider a change.
Yeah, per my last email, I don't want to advertise bandwidth. And I gave a rough rule as to what might qualify as "sufficient" bandwidth (ie. a boolean indicating whether it is roughly expected to be as fast or faster than RAM).
I also was trying to get the point across that even if a particular piece of hardware does not have "sufficient" bandwidth it would still be valuable to publish whether the link is expected to work for the different TLP types. Some applications don't care about bandwidth and just want to know if they can ring doorbells, etc.
Logan
Hi Eric/Jonathan
I'd also like to thank you for taking this work on.
Quoted 2 lines
However, if you see a particular way that this issue could be approached, I am still willing to consider a change.
I would echo Logan's sentiments that a simple Boolean to indicate the p2p performance between RPs is comparable (or better) than RP<->DRAM performance or not would be useful and sufficient. It might be useful to at least add such a field to the ECR so it can be discussed within PCI-SIG.
Cheers
Stephen
There are 5 TLP types described in each of the three P2P Routes. I'm not sure that anyone would care about Message Bandwidth, so I could simply add 3 bits to each Route register indicating whether the P2P route bandwidth supports the same or more bandwidth than if it targeted DRAM.
The three RO/HWInit bits for each route register would be:
- P2P Memory Read Bandwidth is equal to or greater than to Main Memory
- P2P Memory Write Bandwidth is equal to or greater than to Main Memory
- P2P Memory AtomicOp Bandwidth is equal to or greater than to Main Memory
Eric Wehage
Huawei Principal Engineer, PCIe
Quoted 6 lines · Eric Wehage
There are 5 TLP types described in each of the three P2P Routes. I'm not sure that anyone would care about Message Bandwidth, so I could simply add 3 bits to each Route register indicating whether the P2P route bandwidth supports the same or more bandwidth than if it targeted DRAM.
Quoted 7 lines
The three RO/HWInit bits for each route register would be:
- P2P Memory Read Bandwidth is equal to or greater than to Main Memory
- P2P Memory Write Bandwidth is equal to or greater than to Main Memory
- P2P Memory AtomicOp Bandwidth is equal to or greater than to Main Memory
Yeah! That sounds great. Thanks.
Logan
Quoted 15 lines · On Mon, 10 Dec 2018 12:39:36 -0700 Logan Gunthorpe <[email protected]> wrote:
Quoted 6 lines · Eric Wehage
There are 5 TLP types described in each of the three P2P Routes. I'm not sure that anyone would care about Message Bandwidth, so I could simply add 3 bits to each Route register indicating whether the P2P route bandwidth supports the same or more bandwidth than if it targeted DRAM.
Quoted 7 lines
The three RO/HWInit bits for each route register would be:
- P2P Memory Read Bandwidth is equal to or greater than to Main Memory
- P2P Memory Write Bandwidth is equal to or greater than to Main Memory
- P2P Memory AtomicOp Bandwidth is equal to or greater than to Main Memory
Quoted 3 lines
Yeah! That sounds great. Thanks.
Logan
Updated ECR and presentation at:
Presentation: https://drive.google.com/open?id=1fgwz8w32K2ju9aIySP8RXwQwUAcT9icF
ECR: https://drive.google.com/open?id=1cHn4OHgTJpa0KkiVab_CyK1NMjPap3MM
A few small editorial bits and pieces.
The main change is making the Memory Read, Memory Write and Atomic capabilities in addition to indicating they support peer to peer at all, can now give an indication of supporting 'same bandwidth as to system memory'.
Note that from a PCI viewpoint system memory really means anything beyond the host so caches etc are all rolled into that term.
Thanks,
Eric and Jonathan
Quoted 4 lines · Jonathan Cameron
Updated ECR and presentation at:
Presentation: https://drive.google.com/open?id=1fgwz8w32K2ju9aIySP8RXwQwUAcT9icF
ECR: https://drive.google.com/open?id=1cHn4OHgTJpa0KkiVab_CyK1NMjPap3MM
Quoted 8 lines
A few small editorial bits and pieces.
The main change is making the Memory Read, Memory Write and Atomic capabilities in addition to indicating they support peer to peer at all, can now give an indication of supporting 'same bandwidth as to system memory'.
Note that from a PCI viewpoint system memory really means anything beyond the host so caches etc are all rolled into that term.
Looks good to me. Thanks,
Logan
Quoted 44 lines · Jonathan Cameron
Hi All,
This is perhaps a somewhat unusual RFC as it's to ask for feedback on an ECR that is currently under discussion in the PCI SIG. That ECR, at least partly, came out of the issues highlighted during the discussion of P2PDMA. In that patch set, p2pdma is only allowed if there is a switch between the devices involved ensuring the path does not pass through a root port. Put simply there is no way to know if a root complex will allow p2p or not.
So we are asking for feedback at a stage where modifications to the proposal can be made. Eric has agreed to this being shared with mailing list to hopefully get such feedback. Note this is under active discussion within the SIG so inputs in that forum also welcome.
I would recommend reading the ECR and presentation for more details. Anyone who can't get to Google Drive, email me and I'll send a copy directly.
ECR: https://drive.google.com/open?id=1X08pfyjkNwSfpTVBuiut90ajxBZd7BXg
Slides: https://drive.google.com/open?id=11d3egA8x91d99-QqfYGZ-MxLGrHpJHCKMy summary probably won't be great, but I'll give it a go to persuade people to look at the real docs!
Problem Statement:
Linux does not generally enable Peer to Peer because (amongst other reasons) it does not know which Root Complexes support Peer to Peer.
* Some Root Complexes do not support Peer to Peer
* Some Root Complexes only support Posted Peer to Peer
* Some Root Complexes only support Peer to Peer within one PCIe Segment
* Some Root Complexes only support Address Decode Peer to Peer (no ID routing)
* Some Root Complexes may not support Peer to Peer from Integrated End PointsRelated other issue
* MPS must be programmed the same across all devices supporting P2P
* Hot plugged devices might require a smaller MPS than was programmed.Solution:
* New extended capability and control (though only one bit in the control)
* Advertise what a Peer to Peer Root Port and Switch Down Stream ports can do.
* Define packet splitting scheme for MPS mismatches.
- Rules on how this must be done if supported.
Quoted 6 lines
All feedback welcome. Let's make sure we get this right.
Thanks,
Jonathan Cameron
Huawei
Looking this over some of this seems redundant with the ACS P2P Request Redirect and Completion Redirect functionality.
I was curious if there was a reason why you couldn't use something like that to at least get some early enabling going for peer-to-peer root port pass through?
Also, I have not seen called out anywhere in your documents how this might interact with something like Address Translation Services (ATS). I'm assuming there would be scenarios where the translation agent would need to be a part of the p2p topology.
Thanks.
- Alex