PAT support
Hi all,
quite a while back, I sent out email about adding Page Attribute Table support to the kernel (http://www.ker
neltraffic.org/kernel-traffic/kt20030616_219.html#3).
At the time, the concern was being able to mark remapped i/o pages Write Combined in the case that we ran out of MTRRs to do so. This was mainly for the agp aperture and framebuffer. With PCI Express systems coming out, the need changes slightly. PCI Express does not have a centralized aperture like agp that can be marked WC. Instead, individual system pages of memory need to be marked WC via PAT in the page tables. This significantly increases the need for PAT support under linux to maintain high performance levels on PCI Express systems.
I thought the best approach would be to handle some of the original feedback in the code before I came backto ping lkml. I discussed things a little offline with Andi Kleen. He suggested I focus on the simpler i/o regions first, then come back to handling main memory once I had that done and had gotten feedback on it.
I've worked on a mechanism (cachemap) to track what type of caching a region of memory is currently mapped as. when a new region of memory is mapped, cachemap is queried to make sure the new region's caching type matches the old type (or is compatible with the old type). if the cachemap query succeeds, it's safe to map the new i/o region, otherwise it's not safe.
for the first pass, I focused on testing ioremap. so the cachemap queries are only made from ioremap. I also added code to have the mtrr code call the cachemap code (in this case, it's a report rather than a query to indicate the mapping's already made). I've made a few test runs on systems here, and it seems to work fairly well.
this current patch includes the original PAT support and the new cachemap mechanism. note that the cachemap mechanism does not actually change any caching attributes, it only keeps track of the attributes and tests regions. I think the end idea would be that drivers would use the normal ioremap/change_page_attr/remap_page_range mechanisms like they already do, and these mechanisms would in turn use cachemap to make sure there's no conflicts. I'm completely open to how any specific details should work, and any changes needed to be made.
Thanks,
Terence
--On 12 April 2004 17:29 -0500 Terence Ripperda <[email protected]>
Quoted 4 lines · wrote:
quite a while back, I sent out email about adding Page Attribute Table support to the kernel (http://www.ker
neltraffic.org/kernel-traffic/kt20030616_219.html#3).
Quoted 8 lines
this current patch includes the original PAT support and the new cachemap mechanism. note that the cachemap mechanism does not actually change any caching attributes, it only keeps track of the attributes and tests regions. I think the end idea would be that drivers would use the normal ioremap/change_page_attr/remap_page_range mechanisms like they already do, and these mechanisms would in turn use cachemap to make sure there's no conflicts. I'm completely open to how any specific details should work, and any changes needed to be made.
Yeah, I've experemented recently with PAT too. Not had a chance to look at the code in any specifics. But I did notice there appear to be no notes or warnings on the issues of using PAT based mappings. Cirtainly there are some _very_ onerus restrictions which I have personally tested and found to be true :(. Perhaps we could get some big fat warning style comments.
Cirtainly the documentation is _very_ specific about this limitation, I think I saw this being implmented in the code.
+ * According to the INTEL documentation it is the systems responsibility
+ * to ensure that the PAT registers are kept in agreement on all processors
+ * in a system. Changing these registers must occu in a manner which
+ * maintains the consistency of the processor caches and translation
+ * lookaside buggers (TLB).
Also, I have confirmed that if you have any Intel processors which do not have the SS (Self Snoop) capability, you cannot have two independent mappings to a page with different cache attributes. I have been hit by this and you get stale data returned from the cache.
-apw
Quoted 4 lines · On Tue, Apr 13, 2004 at 01:36:13AM -0700, [email protected] wrote:
But I did notice there appear to be no notes or
warnings on the issues of using PAT based mappings. Cirtainly there are some _very_ onerus restrictions which I have personally tested and found to be true :(. Perhaps we could get some big fat warning style comments.
where would you like to see such warnings? arguably, all of the dangerous conditions should be handled by this core code to avoid problems (such as only using the first 4 pat entries, flushing the correct caches when updating the pat entries or pte bits). these problems really aren't all that different than any other cache aliasing/pte flushing issues that always exist, right?
Quoted 5 lines
+ * According to the INTEL documentation it is the systems responsibility
+ * to ensure that the PAT registers are kept in agreement on all processors
+ * in a system. Changing these registers must occu in a manner which
+ * maintains the consistency of the processor caches and translation
+ * lookaside buggers (TLB).
absolutely. I tried to handle this by initializing the pat entries as each cpu comes online at boot time, with cache flushes. I think Andi mentioned flushing the TLBs as well, I'll check up on that to make sure I'm doing that as well.
Quoted 4 lines
Also, I have confirmed that if you have any Intel processors which do not have the SS (Self Snoop) capability, you cannot have two independent mappings to a page with different cache attributes. I have been hit by this and you get stale data returned from the cache.
certainly, that is more or less what this mechanism is intended to prevent. cmap_compare_cachings is an arch-dependent function, which allows architectures to allow/disallow different cache attributes. I certainly consider the current implementation to be a sample that needs some tweaking. for example, I forgot that I had allowed CACHED/NOCACHED overlaps, due to MTRRs that legally overlap like this. but that's probably not a situation we want for any other case, so I need to fix that.
Thanks,
Terence