pat support in the kernel
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 3 lines · Terence Ripperda
Hello all,
I've discussed adding Page Attribute Table (PAT) support to the kernel w/ a few developers offline. They were very supportive and suggested I bring the discussion to lkml so others could get involved.
change_page_attr() will already do it for the kernel mappings. Just define a PAGE_KERNEL_WC. Drawback is that it will convert the mapping to 4K pages (from 2/4MB), but there is probably no alternative unless all your mappings are 2MB aligned.
But the tricky part of it is that you need to make sure all mappings to that memory have the same caching attribute, otherwise you invoke undefined x86 behaviour and risk cache corruptions on some CPUs.
For the special case of AGP it's quite simple - when an user process maps the aperture it can just set the correct bits in its own mmap method like it already does for uncachable mappings. But for other mappings it is more difficult.
For normal memory you would need to find a way to synchronize the attributes in all mappers (e.g. setting a flag in struct page or similar). For frame buffer you also need to handle it in all mmap'ers (like fbcon or /dev/mem). I think handling these generic cases will need a few VM changes.
[actually even the agp aperture can be accessed using /dev/mem, but thats probably unlikely to happen because there is a better interface and could be ignored]
-Andi
Quoted 2 lines · On Tue, May 20, 2003 at 09:10:18PM +0200, [email protected] wrote:
change_page_attr()will already do it for the kernel mappings. Just define aPAGE_KERNEL_WC.
correct. that was the intent.
Quoted 3 lines
But the tricky part of it is that you need to make sure all mappings to that memory have the same caching attribute, otherwise you invoke undefined x86 behaviour and risk cache corruptions on some CPUs.
yes. implementing the basic PAT support is pretty trivial. it's dealing with these cache attribute issues that is the hard part.
Quoted 3 lines
For normal memory you would need to find a way to synchronize the attributes in all mappers (e.g. setting a flag in struct page or similar).
are you refering to generic memory that might have shared mappings between multiple processes? I had really only thought of memory explicitly allocated by a driver and mapped to a process, in which this wouldn't be an issue (or is an issue isolated to the specific driver).
it seems it would be easy enough to add a flag to struct page indicating that any future mappings of this memory must be marked with the given attribute. But you'd need to also worry about previous mappings of that page. wouldn't that require a fairly exhaustive scan of who has the physical memory mapped?
would it make sense to limit this functionality to memory mmapped with MAP_PRIVATE rather than MAP_SHARED?
what if process 1 mapped a region WC, forcing process 2 to later map it the same way even though process 2 doesn't care. then process 1 exits and process 3 decides to map the memory. does the caching attribute remain sticky with process 2 (causing process 3 to also need the memory WC), or revert to cached/whatever when the requestor's mapping is removed?
what if 2 processes ask for conflicting mappings? process 1 wants the framebuffer mapped WC, but process 2 asks for it cacheable. or process 1 maps 1/2 of the framebuffer WC and process 2 asks for the full framebuffer uncached.
a lot of these are corner cases that are unlikely to be desirable, but probably should be protected against.
Quoted 3 lines
For frame buffer you also need to handle it in all mmap'ers (like fbcon or
/dev/mem). I think handling these generic cases will need a few VM changes.
yes, this was the case I was more worried about, but it looks like the case above will have the same issues.
I don't think there's any way currently to determine if anyone already has a mapping to a given address range. And it seems that scanning for pre-existing mappings would be pretty ugly. are there any other suggestions for how to handle this?
Thanks,
Terence
Quoted 5 lines · Terence Ripperda
Quoted 3 lines
For normal memory you would need to find a way to synchronize the attributes in all mappers (e.g. setting a flag in struct page or similar).
are you refering to generic memory that might have shared mappings between multiple processes? I had really only thought of memory explicitly allocated by a driver and mapped to a process, in which this wouldn't be an issue (or is an issue isolated to the specific driver).
Yes.
Quoted 2 lines
it seems it would be easy enough to add a flag to struct page indicating that any future mappings of this memory must be marked with the given attribute. But you'd need to also worry about previous mappings of that page. wouldn't that require a fairly exhaustive scan of who has the physical memory mapped?
Not in 2.5 - it has the new RMAP vm with backlinks from struct page to ptes. I already used that in a new machine check handler that has similar requirements.
Quoted 2 lines
would it make sense to limit this functionality to memory mmapped with
MAP_PRIVATErather thanMAP_SHARED?
That would be a bit ugly, but possible if there is no better way.
Quoted 2 lines
what if process 1 mapped a region WC, forcing process 2 to later map it the same way even though process 2 doesn't care. then process 1 exits and process 3 decides to map the memory. does the caching attribute remain sticky with process 2 (causing process 3 to also need the memory WC), or revert to cached/whatever when the requestor's mapping is removed?
You have to walk the rmap chains and change the ptes, or return -EINVAL.
(e.g. you could define the API that only transition from writeback to other mappings is allowed or reversal)
The locking of this is a bit tricky however.
Quoted 2 lines
what if 2 processes ask for conflicting mappings? process 1 wants the framebuffer mapped WC, but process 2 asks for it cacheable. or process 1 maps 1/2 of the framebuffer WC and process 2 asks for the full framebuffer uncached.
One of them has to lose. Or use the EINVAL method above.
Quoted 2 lines
a lot of these are corner cases that are unlikely to be desirable, but probably should be protected against.
Yes, definitely.
Quoted 5 lines
Quoted 3 lines
For frame buffer you also need to handle it in all mmap'ers (like fbcon or
/dev/mem). I think handling these generic cases will need a few VM changes.yes, this was the case I was more worried about, but it looks like the case above will have the same issues.
The problem is that the frame buffer and the agp aperture normally have no struct page, so you need to find a different way to store the shared state for them (e.g. a new rbtree)
Quoted 2 lines
I don't think there's any way currently to determine if anyone already has a mapping to a given address range. And it seems that scanning for pre-existing mappings would be pretty ugly. are there any other suggestions for how to handle this?
In 2.4 there isn't, unless you run a RMAP kernel.
In 2.5 it's easy.
-Andi
Thanks for the tips Andi,
The rmap lookups sound like a good route to go. I'll work on that and post another patch when I have something working.
And I agree that a "first come, first serve" approach that fails any conflicting mapping attempts is a good route to go.
Terence