Skip to contentSkip to conversation

RW and executable hole in page tables on x86_64

DISCUSSION Kees Cook 3 participants 5 messages Oct 25, 2013 – Nov 14, 2014

lore
Share & tools
lore

Kees Cook

author
#0 lore raw

Hi,

I've noticed there's a chunk of kernel memory still marked RW and x. See 0xffffffff82956000 below...

---[ High Kernel Mapping ]---

0xffffffff80000000-0xffffffff81000000          16M                           pmd
0xffffffff81000000-0xffffffff81a00000          10M     ro         PSE GLB x  pmd
0xffffffff81a00000-0xffffffff81e00000           4M     ro         PSE GLB NX pmd
0xffffffff81e00000-0xffffffff82200000           4M     RW             GLB NX pte
0xffffffff82200000-0xffffffff82800000           6M     RW         PSE GLB NX pmd
0xffffffff82800000-0xffffffff82956000        1368K     RW             GLB NX pte
0xffffffff82956000-0xffffffff82a00000         680K     RW             GLB x  pte

0xffffffff82a00000-0xffffffffa0000000 470M pmd

HPA looked at it for a bit, but it wasn't obvious what was going on. It's after the end of bss. I do note that the two adjacent regions add up to 2MiB. Is this some kind of leftover mapping? What is this region? Is there a sensible place to clean it up?

Thanks,

-Kees

Signature · 1 line
Kees Cook                                            @outflux.net
#1 lore raw
Quoted 13 lines · Kees Cook

Hi,

I've noticed there's a chunk of kernel memory still marked RW and x. See 0xffffffff82956000 below...

---[ High Kernel Mapping ]---

0xffffffff80000000-0xffffffff81000000          16M                           pmd
0xffffffff81000000-0xffffffff81a00000          10M     ro         PSE GLB x  pmd
0xffffffff81a00000-0xffffffff81e00000           4M     ro         PSE GLB NX pmd
0xffffffff81e00000-0xffffffff82200000           4M     RW             GLB NX pte
0xffffffff82200000-0xffffffff82800000           6M     RW         PSE GLB NX pmd
0xffffffff82800000-0xffffffff82956000        1368K     RW             GLB NX pte
0xffffffff82956000-0xffffffff82a00000         680K     RW             GLB x  pte
Quoted 7 lines

0xffffffff82a00000-0xffffffffa0000000 470M pmd

HPA looked at it for a bit, but it wasn't obvious what was going on. It's after the end of bss. I do note that the two adjacent regions add up to 2MiB. Is this some kind of leftover mapping? What is this region? Is there a sensible place to clean it up?

It looks to be what is left after the 2 MB page for bss is broken up. It doesn't mean it isn't broken, though.

-hpa

author
#2 lore raw

Continuing a thread from a year ago...

Quoted 16 lines · H. Peter Anvin
Quoted 15 lines · Kees Cook

Hi,

I've noticed there's a chunk of kernel memory still marked RW and x. See 0xffffffff82956000 below...

---[ High Kernel Mapping ]---

0xffffffff80000000-0xffffffff81000000          16M                           pmd
0xffffffff81000000-0xffffffff81a00000          10M     ro         PSE GLB x  pmd
0xffffffff81a00000-0xffffffff81e00000           4M     ro         PSE GLB NX pmd
0xffffffff81e00000-0xffffffff82200000           4M     RW             GLB NX pte
0xffffffff82200000-0xffffffff82800000           6M     RW         PSE GLB NX pmd
0xffffffff82800000-0xffffffff82956000        1368K     RW             GLB NX pte
0xffffffff82956000-0xffffffff82a00000         680K     RW             GLB x  pte

0xffffffff82a00000-0xffffffffa0000000 470M pmd

Quoted 8 lines
Quoted 5 lines

HPA looked at it for a bit, but it wasn't obvious what was going on. It's after the end of bss. I do note that the two adjacent regions add up to 2MiB. Is this some kind of leftover mapping? What is this region? Is there a sensible place to clean it up?

It looks to be what is left after the 2 MB page for bss is broken up. It doesn't mean it isn't broken, though.

It looks like the problem still exists:

---[ High Kernel Mapping ]---

0xffffffff80000000-0xffffffff9ca00000     458M                   pmd
0xffffffff9ca00000-0xffffffff9d200000       8M   ro   PSE GLB x  pmd
0xffffffff9d200000-0xffffffff9d3f3000    1996K   ro       GLB x  pte
0xffffffff9d3f3000-0xffffffff9d400000      52K   ro           x  pte
0xffffffff9d400000-0xffffffff9d600000       2M   ro   PSE GLB NX pmd
0xffffffff9d600000-0xffffffff9d7e8000    1952K   ro       GLB NX pte
0xffffffff9d7e8000-0xffffffff9d800000      96K   ro           NX pte
0xffffffff9d800000-0xffffffff9d8ff000    1020K   RW       GLB NX pte
0xffffffff9d8ff000-0xffffffff9da2d000    1208K   RW           NX pte
0xffffffff9da2d000-0xffffffff9dc00000    1868K   RW       GLB NX pte
0xffffffff9dc00000-0xffffffff9e600000      10M   RW   PSE GLB NX pmd
0xffffffff9e600000-0xffffffff9e7f5000    2004K   RW       GLB NX pte
0xffffffff9e7f5000-0xffffffff9e800000      44K   RW       GLB x  pte

0xffffffff9e800000-0xffffffffc0000000 536M pmd

Still seems to be the bss getting broken up. What is this left-over memory used for? Any pointers to where it happens? I'd really like to kill this area.

-Kees

Signature · 1 line
Kees Cook                                            @outflux.net
#3 lore raw
Quoted 44 lines · Kees Cook

Continuing a thread from a year ago...

Quoted 16 lines · H. Peter Anvin
Quoted 15 lines · Kees Cook

Hi,

I've noticed there's a chunk of kernel memory still marked RW and x. See 0xffffffff82956000 below...

---[ High Kernel Mapping ]---

0xffffffff80000000-0xffffffff81000000          16M                           pmd
0xffffffff81000000-0xffffffff81a00000          10M     ro         PSE GLB x  pmd
0xffffffff81a00000-0xffffffff81e00000           4M     ro         PSE GLB NX pmd
0xffffffff81e00000-0xffffffff82200000           4M     RW             GLB NX pte
0xffffffff82200000-0xffffffff82800000           6M     RW         PSE GLB NX pmd
0xffffffff82800000-0xffffffff82956000        1368K     RW             GLB NX pte
0xffffffff82956000-0xffffffff82a00000         680K     RW             GLB x  pte

0xffffffff82a00000-0xffffffffa0000000 470M pmd

Quoted 8 lines
Quoted 5 lines

HPA looked at it for a bit, but it wasn't obvious what was going on. It's after the end of bss. I do note that the two adjacent regions add up to 2MiB. Is this some kind of leftover mapping? What is this region? Is there a sensible place to clean it up?

It looks to be what is left after the 2 MB page for bss is broken up. It doesn't mean it isn't broken, though.

It looks like the problem still exists:

---[ High Kernel Mapping ]---

0xffffffff80000000-0xffffffff9ca00000     458M                   pmd
0xffffffff9ca00000-0xffffffff9d200000       8M   ro   PSE GLB x  pmd
0xffffffff9d200000-0xffffffff9d3f3000    1996K   ro       GLB x  pte
0xffffffff9d3f3000-0xffffffff9d400000      52K   ro           x  pte
0xffffffff9d400000-0xffffffff9d600000       2M   ro   PSE GLB NX pmd
0xffffffff9d600000-0xffffffff9d7e8000    1952K   ro       GLB NX pte
0xffffffff9d7e8000-0xffffffff9d800000      96K   ro           NX pte
0xffffffff9d800000-0xffffffff9d8ff000    1020K   RW       GLB NX pte
0xffffffff9d8ff000-0xffffffff9da2d000    1208K   RW           NX pte
0xffffffff9da2d000-0xffffffff9dc00000    1868K   RW       GLB NX pte
0xffffffff9dc00000-0xffffffff9e600000      10M   RW   PSE GLB NX pmd
0xffffffff9e600000-0xffffffff9e7f5000    2004K   RW       GLB NX pte
0xffffffff9e7f5000-0xffffffff9e800000      44K   RW       GLB x  pte
Quoted 5 lines

0xffffffff9e800000-0xffffffffc0000000 536M pmd

Still seems to be the bss getting broken up. What is this left-over memory used for? Any pointers to where it happens? I'd really like to kill this area.

mark_rodata_ro()

	set_memory_nx(rodata_start, (all_end - rodata_start) >> PAGE_SHIFT);

all_end is _end which is the end of the the __brk section.

That looks indeed like a 2MB page which is split by set_memory_nx().

That reminder of the 2MB is a leftover from cleanup_highmap()

* We limit the mappings to the region from _text to _brk_end. _brk_end
* is rounded up to the 2MB boundary.

So what you see is the reminder between _brk_end and the 2MB boundary.

Now the simple solution is to round up all_end in mark_rodata_ro() to the 2MB boundary, but we should probably get rid of the mapping completely.

Something like:

   end = roundup((unsigned long)all_end, PMD_SIZE) - 1;	  

free_init_pages(all_end + 1, end)

should do the trick.

Thanks,

tglx

author
#4 lore raw

Hi Thomas,

Quoted 61 lines · Thomas Gleixner
Quoted 44 lines · Kees Cook

Continuing a thread from a year ago...

Quoted 16 lines · H. Peter Anvin
Quoted 15 lines · Kees Cook

Hi,

I've noticed there's a chunk of kernel memory still marked RW and x. See 0xffffffff82956000 below...

---[ High Kernel Mapping ]---

0xffffffff80000000-0xffffffff81000000          16M                           pmd
0xffffffff81000000-0xffffffff81a00000          10M     ro         PSE GLB x  pmd
0xffffffff81a00000-0xffffffff81e00000           4M     ro         PSE GLB NX pmd
0xffffffff81e00000-0xffffffff82200000           4M     RW             GLB NX pte
0xffffffff82200000-0xffffffff82800000           6M     RW         PSE GLB NX pmd
0xffffffff82800000-0xffffffff82956000        1368K     RW             GLB NX pte
0xffffffff82956000-0xffffffff82a00000         680K     RW             GLB x  pte

0xffffffff82a00000-0xffffffffa0000000 470M pmd

Quoted 8 lines
Quoted 5 lines

HPA looked at it for a bit, but it wasn't obvious what was going on. It's after the end of bss. I do note that the two adjacent regions add up to 2MiB. Is this some kind of leftover mapping? What is this region? Is there a sensible place to clean it up?

It looks to be what is left after the 2 MB page for bss is broken up. It doesn't mean it isn't broken, though.

It looks like the problem still exists:

---[ High Kernel Mapping ]---

0xffffffff80000000-0xffffffff9ca00000     458M                   pmd
0xffffffff9ca00000-0xffffffff9d200000       8M   ro   PSE GLB x  pmd
0xffffffff9d200000-0xffffffff9d3f3000    1996K   ro       GLB x  pte
0xffffffff9d3f3000-0xffffffff9d400000      52K   ro           x  pte
0xffffffff9d400000-0xffffffff9d600000       2M   ro   PSE GLB NX pmd
0xffffffff9d600000-0xffffffff9d7e8000    1952K   ro       GLB NX pte
0xffffffff9d7e8000-0xffffffff9d800000      96K   ro           NX pte
0xffffffff9d800000-0xffffffff9d8ff000    1020K   RW       GLB NX pte
0xffffffff9d8ff000-0xffffffff9da2d000    1208K   RW           NX pte
0xffffffff9da2d000-0xffffffff9dc00000    1868K   RW       GLB NX pte
0xffffffff9dc00000-0xffffffff9e600000      10M   RW   PSE GLB NX pmd
0xffffffff9e600000-0xffffffff9e7f5000    2004K   RW       GLB NX pte
0xffffffff9e7f5000-0xffffffff9e800000      44K   RW       GLB x  pte
Quoted 5 lines

0xffffffff9e800000-0xffffffffc0000000 536M pmd

Still seems to be the bss getting broken up. What is this left-over memory used for? Any pointers to where it happens? I'd really like to kill this area.

mark_rodata_ro()

	set_memory_nx(rodata_start, (all_end - rodata_start) >> PAGE_SHIFT);

all_end is _end which is the end of the the __brk section.

That looks indeed like a 2MB page which is split by set_memory_nx().

That reminder of the 2MB is a leftover from cleanup_highmap()

Quoted 7 lines

* We limit the mappings to the region from _text to _brk_end. _brk_end
* is rounded up to the 2MB boundary.

So what you see is the reminder between _brk_end and the 2MB boundary.

Now the simple solution is to round up all_end in mark_rodata_ro() to the 2MB boundary,

Thanks! I just managed to work these things out from Peter's earlier hints, just before you sent this. Good to have my guess confirmed. :)

Quoted 10 lines

but we should probably get rid of the mapping
completely.

Something like:

   end = roundup((unsigned long)all_end, PMD_SIZE) - 1;	  

free_init_pages(all_end + 1, end)

should do the trick.

Ah-ha! Okay, I'll send a new patch that does this. Thanks!

-Kees

Signature · 1 line
Kees Cook                                            @outflux.net