Skip to contentSkip to conversation

Mapping PCI BAR through /sys/devices/pci* sets cache-disable and write-through

DISCUSSION Keith Packard 3 participants 5 messages Apr 16, 2008 – Apr 17, 2008

lore
Share & tools
lore
#0 lore raw

The X server recently (within the last year) switched from mapping the frame buffer from /dev/mem to using /sys/devices/pci*. This caused a significant memory bandwidth reduction for writes, similar to the effect caused when the associated MTRR is set to UC instead of WC.

Looking at the code path, we find, in i386:pci_mmap_page_range:

        prot = pgprot_val(vma->vm_page_prot);
        if (boot_cpu_data.x86 > 3)
                prot |= _PAGE_PCD | _PAGE_PWT;
        vma->vm_page_prot = __pgprot(prot);

Which is to say, on any CPU which supports it, force the cache-disable and write-through bits on.

Is there some reason this is there? Surely applications should be expected to set these attributes correctly, currently using MTRR and in the future, using PAT directly.

Signature · 1 line
[email protected]
#1 lore raw

RE: Mapping PCI BAR through /sys/devices/pci* sets cache-disable andwrite-through

Quoted 23 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Keith Packard
Sent: Wednesday, April 16, 2008 10:40 AM
To: linux-kernel
Subject: Mapping PCI BAR through /sys/devices/pci* sets cache-disable andwrite-through

The X server recently (within the last year) switched from mapping the frame buffer from /dev/mem to using /sys/devices/pci*. This caused a significant memory bandwidth reduction for writes, similar to the effect
caused when the associated MTRR is set to UC instead of WC.

Looking at the code path, we find, in i386:pci_mmap_page_range:

       prot = pgprot_val(vma->vm_page_prot);
       if (boot_cpu_data.x86 > 3)
               prot |= _PAGE_PCD | _PAGE_PWT;
       vma->vm_page_prot = __pgprot(prot);
Quoted 7 lines

Which is to say, on any CPU which supports it, force the cache-disable and write-through bits on.

Is there some reason this is there? Surely applications should be expected to set these attributes correctly, currently using MTRR and in the future, using PAT directly.

Setting flags "_PAGE_PCD | _PAGE_PWT" maps to PAT_3 which is "UC". So, the mapping X server will get is UC and if X sets MTRR to WC, it will not have any effect on this mapping, which will be still UC. There may be apps which depend on this /sys/ interface giving UC mapping always...

With PAT patches, we now have /sys/devices/pci*resource_wc (in addition to resource) for any PREFETCHABLE region. So, X has to change to use that new interface to get WC mapping.

Thanks,
Venki

#2 lore raw

RE: Mapping PCI BAR through /sys/devices/pci* sets cache-disable andwrite-through

Quoted 4 lines · Venkatesh

Setting flags "_PAGE_PCD | _PAGE_PWT" maps to PAT_3 which is "UC". So, the mapping X server will get is UC and if X sets MTRR to WC, it will not have any effect on this mapping, which will be still UC. There may be apps which depend on this /sys/ interface giving UC mapping always...

Sure, but X wants WC in the biggest way (our measurements show WC being more than 6 times faster for writes, the operation we do the most of).

Jesse Barnes suggested using the fact that the BAR was prefetchable to guess that this region should not be mapped UC. I don't know of any other information available at this API that would help make a better choice; there are no ioctls on /sys files that we could use to manipulate the mapping.

Quoted 3 lines

With PAT patches, we now have /sys/devices/pci*resource_wc (in addition to resource) for any PREFETCHABLE region. So, X has to change to use that new interface to get WC mapping.

Yeah, we'll do that when it becomes available. Would it make sense in the pre-PAT world to use Jesse's guess?

In any case, we'll continue to use the fact that mprotect is also broken to get our WC mapping working (using mprotect PROT_NONE followed by mprotect PROT_READ|PROT_WRITE causes the CD and WT bits to get cleared). We're fortunate in this case that we've found a bug to exploit that gives us the desired behaviour.

Signature · 1 line
[email protected]
#3 lore raw

RE: Mapping PCI BAR through /sys/devices/pci* sets cache-disableandwrite-through

Quoted 20 lines

-----Original Message-----
From: Keith Packard [mailto:[email protected]]
Sent: Wednesday, April 16, 2008 10:21 PM
To: Pallipadi, Venkatesh
Cc: linux-kernel; Siddha, Suresh B
Subject: RE: Mapping PCI BAR through /sys/devices/pci* sets cache-disableandwrite-through

Quoted 1 line · Venkatesh

Setting flags "_PAGE_PCD | _PAGE_PWT" maps to PAT_3 which is

"UC". So,

Quoted 2 lines

the mapping X server will get is UC and if X sets MTRR to WC, it will not have any effect on this mapping, which will be still UC.

There may

Quoted 1 line

be apps which depend on this /sys/ interface giving UC

mapping always...

Quoted 8 lines

Sure, but X wants WC in the biggest way (our measurements show WC being more than 6 times faster for writes, the operation we do the most of).

Jesse Barnes suggested using the fact that the BAR was prefetchable to guess that this region should not be mapped UC. I don't know of any other information available at this API that would help make a better choice; there are no ioctls on /sys files that we could use to manipulate the mapping.

Yes. But, the API being there for some time may mean that there is some user who always wants UC behavior with or without PREFETCHABLE flag. With PAT changes we thought of ioctl and then settled on new interface.

Quoted 8 lines
Quoted 1 line

With PAT patches, we now have /sys/devices/pci*resource_wc

(in addition

Quoted 2 lines

to resource) for any PREFETCHABLE region. So, X has to change to use that new interface to get WC mapping.

Yeah, we'll do that when it becomes available. Would it make sense in the pre-PAT world to use Jesse's guess?

Yes. Easiest way to do that will be to use UC_MINUS (Just set PCD bit and not PWT bit) until PAT changes.
Ioremap() uses UC_MINUS, it should not have any side effects to other users who expect this mapping to be UC without any MTRR setting.

Quoted 8 lines

In any case, we'll continue to use the fact that mprotect is also broken
to get our WC mapping working (using mprotect PROT_NONE followed by mprotect PROT_READ|PROT_WRITE causes the CD and WT bits to get cleared).
We're fortunate in this case that we've found a bug to exploit that gives us the desired behaviour.

Noo.. Now we have one more thing in PAT to do list :-(. To go and plug those mprotect APIs to prevent users from doing things like this.

Thanks,
Venki

#4 lore raw

Re: Mapping PCI BAR through /sys/devices/pci* sets cache-disableandwrite-through

Quoted 9 lines · Venkatesh
Quoted 5 lines

Jesse Barnes suggested using the fact that the BAR was prefetchable to guess that this region should not be mapped UC. I don't know of any other information available at this API that would help make a better choice; there are no ioctls on /sys files that we could use to manipulate the mapping.

Yes. But, the API being there for some time may mean that there is some user who always wants UC behavior with or without PREFETCHABLE flag. With PAT changes we thought of ioctl and then settled on new interface.

Yeah, we can't really know. There *probably* aren't any users here that would break, but you never know, and it's best not to change the behavior of an existing API...

Quoted 7 lines
Quoted 2 lines

Yeah, we'll do that when it becomes available. Would it make sense in the pre-PAT world to use Jesse's guess?

Yes. Easiest way to do that will be to use UC_MINUS (Just set PCD bit and not PWT bit) until PAT changes.
Ioremap() uses UC_MINUS, it should not have any side effects to other users who expect this mapping to be UC without any MTRR setting.

We could probably do that unconditionally though, rather than dependent on the prefetchable bit like my patch does.

Quoted 10 lines
Quoted 7 lines

In any case, we'll continue to use the fact that mprotect is also broken
to get our WC mapping working (using mprotect PROT_NONE followed by mprotect PROT_READ|PROT_WRITE causes the CD and WT bits to get cleared).
We're fortunate in this case that we've found a bug to exploit that gives us the desired behaviour.

Noo.. Now we have one more thing in PAT to do list :-(. To go and plug those mprotect APIs to prevent users from doing things like this.

I think it's just a long standing bug... mprotect really shouldn't be clobbering PTE bits regardless of PAT (though with PAT the bug becomes more serious).

Jesse