x86/ioremap: fix boundary calculation and boundary judgment issues for ioremap()
v3 cover x86/ioremap: fix boundary calculation and boundary judgment issues for ioremap()
ioremap_xxx() functions should fail if the memory address range contains normal RAM. But due to some boundary calculation and boundary judgment issues, the RAM check may be omitted for the very start or the very end page in the memory range. As a consequence, ioremap_xxx() can be applied to normal RAM pages by mistake. This raises the risk of misusing ioremap_xxx() functions on normal RAM ranges, and may incur terrible performance issues.
For example, suppose [phys_addr ~ phys_addr + PAGE_SIZE - 1] is a normal RAM page. Calling ioremap(phys_addr, PAGE_SIZE-1) will succeed (but it should not). This will set the cache flag of the phys_addr's directing mapping pte to be PCD. What's worse, iounmap() won't revert this cache flag in the directing mapping. So the pte in the directing mapping keeps polluted until workarounds are applied (by invoking ioremap_cache() on phys_addr) to fix the cache bit. If there is important data/code in the polluted page, which is accessed frequently, then the performance of the machine will drop terribly.
These two patches aim to address this issue.
Yahui Wang (2):
x86/ioremap: fix the pfn calculation mistake in __ioremap_check_ram() kernel/resource: fix boundary judgment issues in find_next_iomem_res()
and __walk_iomem_res_desc()
arch/x86/mm/ioremap.c | 16 ++++++++-------- kernel/resource.c | 4 ++-- 2 files changed, 10 insertions(+), 10 deletions(-)
base-commit: 13311e74253fe64329390df80bed3f07314ddd61
Signature · 1 line
2.25.1
PATCH v3 1/2 x86/ioremap: fix the pfn calculation mistake in __ioremap_check_ram()
In __ioremap_check_ram(), the pfn wrapping calculation supposes res->start to be page-aligned and res->end to be PAGE_SIZE - 1 aligned. But res->start and res->end may not follow such alignment, which may make the RAM checking be omitted for the very start page or the very end page of the memory range. This can cause ioremap_xxx() to succeed on normal RAM by mistake.
For example, suppose memory range [phys_addr ~ phys_addr + PAGE_SIZE - 1] is a normal RAM page. ioremap(phys_addr, PAGE_SIZE - 1) will succeed (but it should not) because the pfn wrapping prevents this page to be checked whether it touches non-ioremappable resources.
The new pfn wrapping calculation makes sure the resulting pfn range covers [res->start, res->end] completely.
--- arch/x86/mm/ioremap.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
arch/x86/mm/ioremap.c +8 -8
@@ -68,19 +68,19 @@ int ioremap_change_attr(unsigned long vaddr, unsigned long size,/* Does the range (or a subset of) contain normal RAM? */ static unsigned int __ioremap_check_ram (struct resource *res) { -unsigned long start_pfn, stop_pfn; +unsigned long start_pfn, stop_pfn, npages;unsigned long i;if ((res->flags &IORESOURCE_SYSTEM_RAM ) !=IORESOURCE_SYSTEM_RAM )return 0 ; - start_pfn = (res->start +PAGE_SIZE -1 ) >>PAGE_SHIFT ; - stop_pfn = (res->end +1 ) >>PAGE_SHIFT ; -if (stop_pfn > start_pfn) { -for (i =0 ; i < (stop_pfn - start_pfn); ++i) -if (pfn_valid (start_pfn + i) && - !PageReserved (pfn_to_page (start_pfn + i))) -return IORES_MAP_SYSTEM_RAM ; + start_pfn =PFN_DOWN (res->start); + stop_pfn =PFN_DOWN (res->end); + npages = stop_pfn - start_pfn +1 ; +for (i =0 ; i < npages; ++i) { +if (pfn_valid (start_pfn + i) && + !PageReserved (pfn_to_page (start_pfn + i))) +return IORES_MAP_SYSTEM_RAM ; }return 0 ;
Signature · 1 line
2.25.1
Re: [PATCH v3 1/2] x86/ioremap: fix the pfn calculation mistake in __ioremap_check_ram()
Quoted 4 lines · Yaohui Wang
For example, suppose memory range [
phys_addr~phys_addr+PAGE_SIZE- 1] is a normal RAM page.ioremap(phys_addr, PAGE_SIZE - 1)will succeed (but it should not) because the pfn wrapping prevents this page to be checked whether it touches non-ioremappable resources.
I would have expected such a scenario to get caught by this check:
if ((res->flags & IORESOURCE_SYSTEM_RAM) != IORESOURCE_SYSTEM_RAM) return false;
Was this issue found by inspection, or is it causing an actual problem in practice?
Also, it would be really nice to include the original authors when you send Fixes: for patches. BTW, scripts/get_maintainer.pl would have done this for you. I've added Tom and Brijesh. Please cc them in the future.
Re: [PATCH v3 1/2] x86/ioremap: fix the pfn calculation mistake in __ioremap_check_ram()
Quoted 2 lines · Dave Hansen
Was this issue found by inspection, or is it causing an actual problem in practice?
This issus truly caused terrible perforamnce downgrade in the practice. When developing an out of tree module in our testing environment, invoking ioremap() on normal RAM causes apparent CLI lag. The Unixbench score also decreases a lot (5x slowdown in the worst case).
Debugging such performance issue is extremely difficult, especially when the code of the faulty module itself is already very complex. I tested the system in many aspects before finally located this problem.
Quoted 3 lines
Also, it would be really nice to include the original authors when you send Fixes: for patches. BTW,
scripts/get_maintainer.plwould have done this for you. I've added Tom and Brijesh. Please cc them in the future.
I'll pay attention to this in the future. Thank you for your time and patience!
Re: [PATCH v3 1/2] x86/ioremap: fix the pfn calculation mistake in __ioremap_check_ram()
Quoted 4 lines · Yaohui Wang
Quoted 2 lines · Dave Hansen
Was this issue found by inspection, or is it causing an actual problem in practice?
Quoted 8 lines
This issus truly caused terrible perforamnce downgrade in the practice. When developing an out of tree module in our testing environment, invoking
ioremap()on normal RAM causes apparent CLI lag. The Unixbench score also decreases a lot (5x slowdown in the worst case).Debugging such performance issue is extremely difficult, especially when the code of the faulty module itself is already very complex. I tested the system in many aspects before finally located this problem.
Do you know why this check:
Quoted 2 lines
if ((res->flags & IORESOURCE_SYSTEM_RAM) != IORESOURCE_SYSTEM_RAM) return false;
did not catch your out-of-tree driver's errant ioremap()?
Re: [PATCH v3 1/2] x86/ioremap: fix the pfn calculation mistake in __ioremap_check_ram()
Quoted 7 lines · Dave Hansen
Do you know why this check:
Quoted 2 lines
if ((res->flags & IORESOURCE_SYSTEM_RAM) != IORESOURCE_SYSTEM_RAM) return false;did not catch your out-of-tree driver's errant
ioremap()?
If ioremap() is invoked on an area that contains normal memory, then:
(res->flags & IORESOURCE_SYSTEM_RAM) == IORESOURCE_SYSTEM_RAM)
is true, so the original check is false. The code following the check will continue to scan whether this area contains any page that is not PageReserved (i.e. that is truly normal RAM).
Your idea should be:
if ((res->flags & IORESOURCE_SYSTEM_RAM) == IORESOURCE_SYSTEM_RAM) return IORES_MAP_SYSTEM_RAM;
But this check is too strict as IORESOURCE_SYSTEM_RAM area may contain PageReserved pages, and PageReserved pages should be ioremap-able. So the checking logic of the original __ioremap_check_ram() function is reasonable.
PATCH v3 2/2 kernel/resource: fix boundary judgment issues in find_next_iomem_res() and __walk_iomem_res_desc()
In reply to Yaohui WangMemory resources are described with the start address and the inclusive end address, which means for a resource with 1 byte length the start address is the same as the end address.
find_next_iomem_res() and __walk_iomem_res_desc() ignore resources with 1 byte length, which prevents that ioremap_xxx(phys_addr, 1) is checked whether it touches non-ioremappable resources.
--- kernel/resource.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/resource.c +2 -2
@@ -344,7 +344,7 @@ static int find_next_iomem_res(resource_size_t start, resource_size_t end,if (!res)return -EINVAL ; -if (start >= end) +if (start > end)return -EINVAL ;read_lock (&resource_lock);
@@ -392,7 +392,7 @@ static int __walk_iomem_res_desc(resource_size_t start, resource_size_t end,struct resource res;int ret = -EINVAL ; -while (start < end && +while (start <= end && !find_next_iomem_res (start, end, flags, desc, &res)) { ret = (*func)(&res, arg);if (ret)
Signature · 1 line
2.25.1
Re: [PATCH v3 2/2] kernel/resource: fix boundary judgment issues in find_next_iomem_res() and __walk_iomem_res_desc()
Quoted 3 lines · Yaohui Wang
Memory resources are described with the start address and the inclusive end address, which means for a resource with 1 byte length the start address is the same as the end address.
Is it just me or does this little "feature" of the resource code continue to bite us over and over?
It might be some time for some unit tests for kernel/resource.c.
Hi, maintainers,
It's been 10 days since I sent the patches at Jun 21st. Would you please help to review them? Wish to hear your feedbacks, Thanks!
Yaohui
Quoted 32 lines · Yaohui Wang
ioremap_xxx()functions should fail if the memory address range contains normal RAM. But due to some boundary calculation and boundary judgment issues, the RAM check may be omitted for the very start or the very end page in the memory range. As a consequence,ioremap_xxx()can be applied to normal RAM pages by mistake. This raises the risk of misusingioremap_xxx()functions on normal RAM ranges, and may incur terrible performance issues.For example, suppose [
phys_addr~phys_addr+PAGE_SIZE- 1] is a normal RAM page. Callingioremap(phys_addr, PAGE_SIZE-1)will succeed (but it should not). This will set the cache flag of the phys_addr's directing mapping pte to be PCD. What's worse,iounmap()won't revert this cache flag in the directing mapping. So the pte in the directing mapping keeps polluted until workarounds are applied (by invokingioremap_cache()onphys_addr) to fix the cache bit. If there is important data/code in the polluted page, which is accessed frequently, then the performance of the machine will drop terribly.These two patches aim to address this issue.
Yahui Wang (2):
x86/ioremap: fix the pfn calculation mistake in__ioremap_check_ram()kernel/resource: fix boundary judgment issues infind_next_iomem_res()and __walk_iomem_res_desc()arch/x86/mm/ioremap.c | 16 ++++++++-------- kernel/resource.c | 4 ++-- 2 files changed, 10 insertions(+), 10 deletions(-)base-commit:
13311e74253fe64329390df80bed3f07314ddd61