makedumpfile issues many readpage_elf: Attempt to read non-existent page

makedumpfile -c -d 31 /proc/vmcore /var/crash/201609221517/dump-incomplete

[    7.513337] kdump-tools[715]: readpage_elf: Attempt to read non-existent page at 0x134dfff78000.
[    7.524186] kdump-tools[715]: readmem: type_addr: 0, addr:ffff9b4dfff78000, size:16
[    7.528440] kdump-tools[715]: section_mem_map_addr: Can't get a struct mem_section(ffff9b4dfff78000).
[    7.536562] kdump-tools[715]: readpage_elf: Attempt to read non-existent page at 0x134dfff78000.
[    7.544356] kdump-tools[715]: readmem: type_addr: 0, addr:ffff9b4dfff78010, size:16
[    7.552422] kdump-tools[715]: section_mem_map_addr: Can't get a struct mem_section(ffff9b4dfff78010).
[    7.560317] kdump-tools[715]: readpage_elf: Attempt to read non-existent page at 0x134dfff78000.
[    7.568422] kdump-tools[715]: readmem: type_addr: 0, addr:ffff9b4dfff78020, size:16
[    7.572296] kdump-tools[715]: section_mem_map_addr: Can't get a struct mem_section(ffff9b4dfff78020).

Hello,

I am investigating an issue with makedumpfile and kernel 4.8 where makedumpfile (1.6.0) exits on error with the following message :

get_mem_map: Can't distinguish the memory type.

I found commit 2c21d4656e8d3c2af2b1e14809d076941ae69e96 in the upstream development branch that is supposed to fix this :

[PATCH v2] Support _count -> _refcount rename in struct page

_count member was renamed to _refcount in linux commit 0139aa7b7fa12 ("mm: rename _count, field of the struct page, to _refcount") and this broke makedumpfile. The reason for making the change was to find all users accessing it directly and not through the recommended API. I tried suggesting to revert the change but failed, I see no other choice than to start supporting both _count and _refcount in makedumpfile.

Though, when I apply the patch and test on either Ubuntu's 4.8.0-11 kernel, or kernel.org's mainline 4.8.0-040800rc7 kernel, I get the following repeated multiple times :

Quoted 10 lines

makedumpfile -c -d 31 /proc/vmcore /var/crash/201609221517/dump-incomplete

[    7.513337] kdump-tools[715]: readpage_elf: Attempt to read non-existent page at 0x134dfff78000.
[    7.524186] kdump-tools[715]: readmem: type_addr: 0, addr:ffff9b4dfff78000, size:16
[    7.528440] kdump-tools[715]: section_mem_map_addr: Can't get a struct mem_section(ffff9b4dfff78000).
[    7.536562] kdump-tools[715]: readpage_elf: Attempt to read non-existent page at 0x134dfff78000.
[    7.544356] kdump-tools[715]: readmem: type_addr: 0, addr:ffff9b4dfff78010, size:16
[    7.552422] kdump-tools[715]: section_mem_map_addr: Can't get a struct mem_section(ffff9b4dfff78010).
[    7.560317] kdump-tools[715]: readpage_elf: Attempt to read non-existent page at 0x134dfff78000.
[    7.568422] kdump-tools[715]: readmem: type_addr: 0, addr:ffff9b4dfff78020, size:16
[    7.572296] kdump-tools[715]: section_mem_map_addr: Can't get a struct mem_section(ffff9b4dfff78020).

I also tested with all the commits in the upstream/development branch with no luck, I still get the same behavior.

Does someone have an idea of where this could come from ?

TIA,

Louis
--
Louis Bouchard
Software engineer, Cloud & Sustaining eng. Canonical Ltd
Ubuntu developer Debian Maintainer
GPG : 429D 7A3B DD05 B6F8 AF63 B9C4 8B3D 867C 823E 7A61

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

Quoted 1 line

Does someone have an idea of where this could come from ?

Thanks for your report, but I'm afraid that I'm going on vacation till Oct 2.
I hope someone helps you until I'm back.

Regards,
Atsushi Kumagai

Hello,

Quoted 10 lines · Atsushi Kumagai

Quoted 1 line

Does someone have an idea of where this could come from ?

Thanks for your report, but I'm afraid that I'm going on vacation till Oct 2.
I hope someone helps you until I'm back.

Regards,
Atsushi Kumagai

As a follow up to my question, I've been investigating the issue. The source of the problem seems to come from readpage_elf() which tries to read to an inexistant page :

readpage_elf: Attempt to read non-existent page at 0x1a7b7fff5000.

The error happens when readpage_elf() is called from section_mem_map_addr with pgaddr, which is derived from paddr. This one comes from :

paddr = vaddr_to_paddr(addr)

When tracing vaddr_to_paddr, I see a difference in the calculation. on a 4.4 kernel :

info->phys_base = 0

On 4.8 :
info->phys_base = 0xffffffffe3800000

So the paddr values used to define pgaddr are :

for 4.4 kernels : 0x3fff4000
for 4.8 kernels : 0x1a7b7fff5000

Running makedumpfile --dump-dmesg in debug mode leads to :

4.4 kernel :

sadump: does not have partition header
sadump: read dump device as unknown format
sadump: unknown format
LOAD (0)
phys_start : 1000000
phys_end : 2224000
virt_start : ffffffff81000000
virt_end : ffffffff82224000
LOAD (1)
phys_start : 1000
phys_end : 9fc00
virt_start : ffff880000001000
virt_end : ffff88000009fc00
LOAD (2)
phys_start : 100000
phys_end : 2b000000
virt_start : ffff880000100000
virt_end : ffff88002b000000
LOAD (3)
phys_start : 33000000
phys_end : 3fffe000
virt_start : ffff880033000000
virt_end : ffff88003fffe000
Linux kdump
page_size : 4096

max_mapnr : 3fffe

Buffer size for the cyclic mode: 65535

num of NODEs : 1

Memory type : SPARSEMEM_EX

mem_map (0)

  mem_map    : ffffea0000000000
  pfn_start  : 0
  pfn_end    : 8000
mem_map (1)
  mem_map    : ffffea0000200000
  pfn_start  : 8000
  pfn_end    : 10000
mem_map (2)
  mem_map    : ffffea0000400000
  pfn_start  : 10000
  pfn_end    : 18000
mem_map (3)
  mem_map    : ffffea0000600000
  pfn_start  : 18000
  pfn_end    : 20000
mem_map (4)
  mem_map    : ffffea0000800000
  pfn_start  : 20000
  pfn_end    : 28000
mem_map (5)
  mem_map    : ffffea0000a00000
  pfn_start  : 28000
  pfn_end    : 30000
mem_map (6)
  mem_map    : ffffea0000c00000
  pfn_start  : 30000
  pfn_end    : 38000
mem_map (7)
  mem_map    : ffffea0000e00000
  pfn_start  : 38000
  pfn_end    : 3fffe

mmap() is available on the kernel.

log_buf : ffffffff8210521c
log_end : 0
log_buf_len : 262144
log_first_idx : 0
log_next_idx : 141176

The dmesg log is saved to /tmp/titi.

makedumpfile Completed.

For 4.8 kernels :
sadump: does not have partition header
sadump: read dump device as unknown format
sadump: unknown format
LOAD (0)
phys_start : 7200000
phys_end : 847d000
virt_start : ffffffffa3a00000
virt_end : ffffffffa4c7d000
LOAD (1)
phys_start : 1000
phys_end : 9fc00
virt_start : ffff880000001000
virt_end : ffff88000009fc00
LOAD (2)
phys_start : 100000
phys_end : 2b000000
virt_start : ffff880000100000
virt_end : ffff88002b000000
LOAD (3)
phys_start : 33000000
phys_end : 3fffe000
virt_start : ffff880033000000
virt_end : ffff88003fffe000
Linux kdump
page_size : 4096

max_mapnr : 3fffe

Buffer size for the cyclic mode: 65535
The kernel version is not supported.
The makedumpfile operation may be incomplete.

num of NODEs : 1

Memory type : SPARSEMEM_EX

mem_map (0)

  mem_map    : 0
  pfn_start  : 0
  pfn_end    : 8000
mem_map (1)
  mem_map    : 0
  pfn_start  : 8000
  pfn_end    : 10000
mem_map (2)
  mem_map    : 0
  pfn_start  : 10000
  pfn_end    : 18000
mem_map (3)
  mem_map    : 0
  pfn_start  : 18000
  pfn_end    : 20000
mem_map (4)
  mem_map    : 0
  pfn_start  : 20000
  pfn_end    : 28000
mem_map (5)
  mem_map    : 0
  pfn_start  : 28000
  pfn_end    : 30000
mem_map (6)
  mem_map    : 0
  pfn_start  : 30000
  pfn_end    : 38000
mem_map (7)
  mem_map    : 0
  pfn_start  : 38000
  pfn_end    : 3fffe

mmap() is available on the kernel.

log_buf : ffffffffa4b3c19c
log_end : 0
log_buf_len : 262144
log_first_idx : 0
log_next_idx : 43160

The dmesg log is saved to /tmp/toto.

makedumpfile Completed.

I don't know if it is to be expected with kernels 4.8 to have all mem_map addresses set to 0, unlike with k4.4 :

mem_map (0)
mem_map : 0
pfn_start : 0
pfn_end : 8000

Any comment appreciated while I continue to investigate.

Kind regards,

...Louis

commit 021182e52fe01c1f7b126f97fd6ba048dc4234fd

Author: Thomas Garnier <[email protected]>
Date:   Tue Jun 21 17:47:03 2016 -0700

x86/mm: Enable KASLR for physical mapping memory regions

Add the physical mapping in the list of randomized memory regions.

The physical memory mapping holds most allocations from boot and heap allocators. Knowing the base address and physical memory size, an attacker can deduce the PDE virtual address for the vDSO memory page. This attack was demonstrated at CanSecWest 2016, in the following presentation:

"Getting Physical: Extreme Abuse of Intel Based Paged Systems":
https://github.com/n3k/CansecWest2016_Getting_Physical_Extreme_Abuse_of_Intel_Based_Paging_Systems/blob/master/Presentation/CanSec2016_Presentation.pdf

(See second part of the presentation).

The exploits used against Linux worked successfully against 4.6+ but fail with KASLR memory enabled:

https://github.com/n3k/CansecWest2016_Getting_Physical_Extreme_Abuse_of_Intel_Based_Paging_Systems/tree/master/Demos/Linux/exploits

Similar research was done at Google leading to this patch proposal.

Variants exists to overwrite /proc or /sys objects ACLs leading to elevation of privileges. These variants were tested against 4.6+.

The page offset used by the compressed kernel retains the static value since it is not yet randomized during this boot stage.

Signed-off-by Thomas Garnier <[email protected]>

Signed-off-by Kees Cook <[email protected]>

Cc Alexander Kuleshov <[email protected]>

Hello,

Quoted 213 lines · Louis Bouchard

Hello,

Quoted 10 lines · Atsushi Kumagai

Quoted 1 line

Does someone have an idea of where this could come from ?

Thanks for your report, but I'm afraid that I'm going on vacation till Oct 2.
I hope someone helps you until I'm back.

Regards,
Atsushi Kumagai

As a follow up to my question, I've been investigating the issue. The source of the problem seems to come from readpage_elf() which tries to read to an inexistant page :

readpage_elf: Attempt to read non-existent page at 0x1a7b7fff5000.

The error happens when readpage_elf() is called from section_mem_map_addr with pgaddr, which is derived from paddr. This one comes from :

paddr = vaddr_to_paddr(addr)

When tracing vaddr_to_paddr, I see a difference in the calculation. on a 4.4 kernel :

info->phys_base = 0

On 4.8 :
info->phys_base = 0xffffffffe3800000

So the paddr values used to define pgaddr are :

for 4.4 kernels : 0x3fff4000
for 4.8 kernels : 0x1a7b7fff5000

Running makedumpfile --dump-dmesg in debug mode leads to :

4.4 kernel :

sadump: does not have partition header
sadump: read dump device as unknown format
sadump: unknown format
LOAD (0)
phys_start : 1000000
phys_end : 2224000
virt_start : ffffffff81000000
virt_end : ffffffff82224000
LOAD (1)
phys_start : 1000
phys_end : 9fc00
virt_start : ffff880000001000
virt_end : ffff88000009fc00
LOAD (2)
phys_start : 100000
phys_end : 2b000000
virt_start : ffff880000100000
virt_end : ffff88002b000000
LOAD (3)
phys_start : 33000000
phys_end : 3fffe000
virt_start : ffff880033000000
virt_end : ffff88003fffe000
Linux kdump
page_size : 4096

max_mapnr : 3fffe

Buffer size for the cyclic mode: 65535

num of NODEs : 1

Memory type : SPARSEMEM_EX

mem_map (0)

  mem_map    : ffffea0000000000
  pfn_start  : 0
  pfn_end    : 8000
mem_map (1)
  mem_map    : ffffea0000200000
  pfn_start  : 8000
  pfn_end    : 10000
mem_map (2)
  mem_map    : ffffea0000400000
  pfn_start  : 10000
  pfn_end    : 18000
mem_map (3)
  mem_map    : ffffea0000600000
  pfn_start  : 18000
  pfn_end    : 20000
mem_map (4)
  mem_map    : ffffea0000800000
  pfn_start  : 20000
  pfn_end    : 28000
mem_map (5)
  mem_map    : ffffea0000a00000
  pfn_start  : 28000
  pfn_end    : 30000
mem_map (6)
  mem_map    : ffffea0000c00000
  pfn_start  : 30000
  pfn_end    : 38000
mem_map (7)
  mem_map    : ffffea0000e00000
  pfn_start  : 38000
  pfn_end    : 3fffe

mmap() is available on the kernel.

log_buf : ffffffff8210521c
log_end : 0
log_buf_len : 262144
log_first_idx : 0
log_next_idx : 141176

The dmesg log is saved to /tmp/titi.

makedumpfile Completed.

For 4.8 kernels :
sadump: does not have partition header
sadump: read dump device as unknown format
sadump: unknown format
LOAD (0)
phys_start : 7200000
phys_end : 847d000
virt_start : ffffffffa3a00000
virt_end : ffffffffa4c7d000
LOAD (1)
phys_start : 1000
phys_end : 9fc00
virt_start : ffff880000001000
virt_end : ffff88000009fc00
LOAD (2)
phys_start : 100000
phys_end : 2b000000
virt_start : ffff880000100000
virt_end : ffff88002b000000
LOAD (3)
phys_start : 33000000
phys_end : 3fffe000
virt_start : ffff880033000000
virt_end : ffff88003fffe000
Linux kdump
page_size : 4096

max_mapnr : 3fffe

Buffer size for the cyclic mode: 65535
The kernel version is not supported.
The makedumpfile operation may be incomplete.

num of NODEs : 1

Memory type : SPARSEMEM_EX

mem_map (0)

  mem_map    : 0
  pfn_start  : 0
  pfn_end    : 8000
mem_map (1)
  mem_map    : 0
  pfn_start  : 8000
  pfn_end    : 10000
mem_map (2)
  mem_map    : 0
  pfn_start  : 10000
  pfn_end    : 18000
mem_map (3)
  mem_map    : 0
  pfn_start  : 18000
  pfn_end    : 20000
mem_map (4)
  mem_map    : 0
  pfn_start  : 20000
  pfn_end    : 28000
mem_map (5)
  mem_map    : 0
  pfn_start  : 28000
  pfn_end    : 30000
mem_map (6)
  mem_map    : 0
  pfn_start  : 30000
  pfn_end    : 38000
mem_map (7)
  mem_map    : 0
  pfn_start  : 38000
  pfn_end    : 3fffe

mmap() is available on the kernel.

log_buf : ffffffffa4b3c19c
log_end : 0
log_buf_len : 262144
log_first_idx : 0
log_next_idx : 43160

The dmesg log is saved to /tmp/toto.

makedumpfile Completed.

I don't know if it is to be expected with kernels 4.8 to have all mem_map addresses set to 0, unlike with k4.4 :

mem_map (0)
mem_map : 0
pfn_start : 0
pfn_end : 8000

Quoted 6 lines

Any comment appreciated while I continue to investigate.

Kind regards,

...Louis

I have now completed the kernel bisection between 4.7.8 and 4.8-rc1 and identified the kernel modification that triggers the errors cited above :

Quoted 28 lines

commit 021182e52fe01c1f7b126f97fd6ba048dc4234fd

Author: Thomas Garnier <[email protected]>
Date:   Tue Jun 21 17:47:03 2016 -0700

x86/mm: Enable KASLR for physical mapping memory regions

Add the physical mapping in the list of randomized memory regions.

The physical memory mapping holds most allocations from boot and heap allocators. Knowing the base address and physical memory size, an attacker can deduce the PDE virtual address for the vDSO memory page. This attack was demonstrated at CanSecWest 2016, in the following presentation:

"Getting Physical: Extreme Abuse of Intel Based Paged Systems":
https://github.com/n3k/CansecWest2016_Getting_Physical_Extreme_Abuse_of_Intel_Based_Paging_Systems/blob/master/Presentation/CanSec2016_Presentation.pdf

(See second part of the presentation).

The exploits used against Linux worked successfully against 4.6+ but fail with KASLR memory enabled:

https://github.com/n3k/CansecWest2016_Getting_Physical_Extreme_Abuse_of_Intel_Based_Paging_Systems/tree/master/Demos/Linux/exploits

Similar research was done at Google leading to this patch proposal.

Variants exists to overwrite /proc or /sys objects ACLs leading to elevation of privileges. These variants were tested against 4.6+.

Quoted 6 lines

The page offset used by the compressed kernel retains the static value since it is not yet randomized during this boot stage.

Signed-off-by Thomas Garnier <[email protected]>

Signed-off-by Kees Cook <[email protected]>

Cc Alexander Kuleshov <[email protected]>

<truncated>

The interesting change seems to be :

Review

Quoted 7 lines

(quoted diff) +6 -1

-#define __PAGE_OFFSET           _AC(0xffff880000000000, UL)
+#define __PAGE_OFFSET_BASE      _AC(0xffff880000000000, UL)
+#ifdef CONFIG_RANDOMIZE_MEMORY
+#define __PAGE_OFFSET           page_offset_base
+#else
+#define __PAGE_OFFSET           __PAGE_OFFSET_BASE
+#endif /* CONFIG_RANDOMIZE_MEMORY */

I'll try to see if I can fix that.

Kind regards,

...Louis

Quoted 18 lines

I have now completed the kernel bisection between 4.7.8 and 4.8-rc1 and identified the kernel modification that triggers the errors cited above :

Quoted 15 lines

commit 021182e52fe01c1f7b126f97fd6ba048dc4234fd

Author: Thomas Garnier <[email protected]>
Date:   Tue Jun 21 17:47:03 2016 -0700

x86/mm: Enable KASLR for physical mapping memory regions

Add the physical mapping in the list of randomized memory regions.

The physical memory mapping holds most allocations from boot and heap allocators. Knowing the base address and physical memory size, an attacker can deduce the PDE virtual address for the vDSO memory page. This attack was demonstrated at CanSecWest 2016, in the following presentation:

"Getting Physical: Extreme Abuse of Intel Based Paged Systems":

https://github.com/n3k/CansecWest2016_Getting_Physical_Extreme_Abuse_of_Intel_Based_Paging_Systems/blob/master/Prese
ntation/CanSec2016_Presentation.pdf

Quoted 7 lines

Quoted 7 lines

(See second part of the presentation).

The exploits used against Linux worked successfully against 4.6+ but fail with KASLR memory enabled:

https://github.com/n3k/CansecWest2016_Getting_Physical_Extreme_Abuse_of_Intel_Based_Paging_Systems/tree/master/Demos
/Linux/exploits

Quoted 26 lines

Quoted 6 lines

Similar research was done at Google leading to this patch proposal.

Variants exists to overwrite /proc or /sys objects ACLs leading to elevation of privileges. These variants were tested against 4.6+.

Quoted 6 lines

The page offset used by the compressed kernel retains the static value since it is not yet randomized during this boot stage.

Signed-off-by Thomas Garnier <[email protected]>

Signed-off-by Kees Cook <[email protected]>

Cc Alexander Kuleshov <[email protected]>

<truncated>

The interesting change seems to be :

Review

Quoted 7 lines

(quoted diff) +6 -1

-#define __PAGE_OFFSET           _AC(0xffff880000000000, UL)
+#define __PAGE_OFFSET_BASE      _AC(0xffff880000000000, UL)
+#ifdef CONFIG_RANDOMIZE_MEMORY
+#define __PAGE_OFFSET           page_offset_base
+#else
+#define __PAGE_OFFSET           __PAGE_OFFSET_BASE
+#endif /* CONFIG_RANDOMIZE_MEMORY */

I'll try to see if I can fix that.

Quoted 7 lines

Kind regards,

...Louis

Some more *important* information in this mostly monologue thread : Pratyush Anand has pushed a patch to the list earlier today that apparently fixes this issue :

[PATCH Makedumpfile 1/4] x86_64: Calculate page_offset from pt_load[1]

HTH,

Kind regards,

...Louis

[1] https://www.mail-archive.com/[email protected]/msg16628.html
--
Louis Bouchard
Software engineer, Cloud & Sustaining eng.
Canonical Ltd
Ubuntu developer Debian Maintainer
GPG : 429D 7A3B DD05 B6F8 AF63 B9C4 8B3D 867C 823E 7A61