bad pmd ffff810000207238(9090909090909090).

DISCUSSION Applied Fede 6 participants 9 messages May 28, 2008

Subsystems (3)

Memory management x86 x86 MM

lore

Share & tools

lore

Fede

author

2008-05-28 02:06 UTC #0 lore raw

Hi,

Today I tried to start a firewalling script and failed due to an unrelated issue, but when I checked the log I saw this:

May 27 20:38:15 kaoz ip_tables: (C) 2000-2006 Netfilter Core Team
May 27 20:38:28 kaoz Netfilter messages via NETLINK v0.30.
May 27 20:38:28 kaoz nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
May 27 20:38:28 kaoz ctnetlink v0.93: registering with nfnetlink.
May 27 20:38:28 kaoz ClusterIP Version 0.8 loaded successfully
May 27 20:38:28 kaoz mm/memory.c:127: bad pmd ffff810000207238(9090909090909090).

I also found another post with a very similar issue. The other post had almost the same message (*mm*/*memory*.*c*:*127*: *bad* *pmd* ffff810000207808(9090909090909090).)

Does anyone know what is it?

Arjan van de Ven

2008-05-28 04:10 UTC #1 lore raw

Quoted 5 lines · On Tue, 27 May 2008 23:01:01 -0300 Fede <[email protected]> wrote:

Hi,
Today I tried to start a firewalling script and failed due to an unrelated issue, but when I checked the log I saw this:

Quoted 7 lines

May 27 20:38:15 kaoz ip_tables: (C) 2000-2006 Netfilter Core Team
May 27 20:38:28 kaoz Netfilter messages via NETLINK v0.30.
May 27 20:38:28 kaoz nf_conntrack version 0.5.0 (16384 buckets, 65536
max) May 27 20:38:28 kaoz ctnetlink v0.93: registering with nfnetlink.
May 27 20:38:28 kaoz ClusterIP Version 0.8 loaded successfully
May 27 20:38:28 kaoz mm/memory.c:127: bad pmd

ffff810000207238(9090909090909090).

909090 is asm for "nop nop nop", a pattern often used in rootkits to deal with slightly unknown memory locations

I would REALLY REALLY strongly suggest that you check for rootkits and the like on your system...

Signature · 3 lines

If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Hugh Dickins

2008-05-28 18:36 UTC #2 lore raw

In reply to Fede

Quoted 10 lines · Fede

Today I tried to start a firewalling script and failed due to an unrelated issue, but when I checked the log I saw this:

May 27 20:38:15 kaoz ip_tables: (C) 2000-2006 Netfilter Core Team
May 27 20:38:28 kaoz Netfilter messages via NETLINK v0.30.
May 27 20:38:28 kaoz nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
May 27 20:38:28 kaoz ctnetlink v0.93: registering with nfnetlink.
May 27 20:38:28 kaoz ClusterIP Version 0.8 loaded successfully
May 27 20:38:28 kaoz mm/memory.c:127: bad pmd

Quoted 7 lines

ffff810000207238(9090909090909090).
I also found another post with a very similar issue. The other post had almost the same message (*mm*/*memory*.*c*:*127*: *bad* *pmd* ffff810000207808(9090909090909090).)
Does anyone know what is it?

Thanks a lot for re-reporting this: it was fun to work it out. It's not a rootkit, it's harmless, but we ought to fix the noise. Simple patch below, but let me explain more verbosely first.

What was really interesting in your report was that the address is so close to that in OGAWA-San's report. I had a look at that page on my x86_64 boxes, and they have lots of 0x90s there too. It's just some page alignment filler that x86_64 kernel startup has missed cleaning up - patch below fixes that. There's no security aspect to it: the entries were already not-present, they just generate this noise by triggering the pmd_bad test.

But why do you occasionally see those messages (I never have)? I was puzzled awhile because those tests are usually for user address space, but this is up in kernel address space: in the modules area. Ah, vmalloc.c uses them: it's almost coincidence, but those user space tests do work on the vmalloc and modules areas, though they'd fail on many other parts of the kernel address space (because of pse and global and nx bits).

Those messages come just occasionally from unloading a module. There's a years-old anomaly in vmalloc.c, that the allocation routines add an empty guard page slot to the size, then the freeing routines include that empty slot. (I've always thought the guard page should be private to vmalloc.c, consistently left out of the public size; but that would need wider changes.) When it would occupy the first slot of a new page table, allocation won't have assigned one, and freeing will then hit this "bad" pmd entry.

Hugh

[PATCH] x86: fix bad pmd ffff810000207xxx(9090909090909090)

OGAWA Hirofumi and Fede have reported rare pmd_ERROR messages:
mm/memory.c:127: bad pmd ffff810000207xxx(9090909090909090).

Initialization's cleanup_highmap was leaving alignment filler behind in the pmd for MODULES_VADDR: when vmalloc's guard page would occupy a new page table, it's not allocated, and then module unload's vfree hits the bad 9090 pmd entry left over.

Signed-off-by Hugh Dickins <[email protected]>

---

 arch/x86/mm/init_64.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

arch/x86/mm/init_64.c +1 -1

@@ -206,7 +206,7 @@ void __init cleanup_highmap(void)
 	pmd_t *last_pmd = pmd + PTRS_PER_PMD;
 
 	for (; pmd < last_pmd; pmd++, vaddr += PMD_SIZE) {
-		if (!pmd_present(*pmd))
+		if (pmd_none(*pmd))
 			continue;
 		if (vaddr < (unsigned long) _text || vaddr > end)
 			set_pmd(pmd, __pmd(0));

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Ingo Molnar

Applied

2008-05-28 18:40 UTC #3 lore raw

Quoted 9 lines · Hugh Dickins

[PATCH] x86: fix bad pmd ffff810000207xxx(9090909090909090)
OGAWA Hirofumi and Fede have reported rare pmd_ERROR messages:
mm/memory.c:127: bad pmd ffff810000207xxx(9090909090909090).
Initialization's cleanup_highmap was leaving alignment filler behind in the pmd for MODULES_VADDR: when vmalloc's guard page would occupy a new page table, it's not allocated, and then module unload's vfree hits the bad 9090 pmd entry left over.

applied, nice catch!

Ingo

Willy Tarreau

2008-05-28 19:56 UTC #4 lore raw

In reply to Hugh Dickins

Quoted 23 lines · Hugh Dickins

Quoted 10 lines · Fede
Today I tried to start a firewalling script and failed due to an unrelated issue, but when I checked the log I saw this:
May 27 20:38:15 kaoz ip_tables: (C) 2000-2006 Netfilter Core Team
May 27 20:38:28 kaoz Netfilter messages via NETLINK v0.30.
May 27 20:38:28 kaoz nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
May 27 20:38:28 kaoz ctnetlink v0.93: registering with nfnetlink.
May 27 20:38:28 kaoz ClusterIP Version 0.8 loaded successfully
May 27 20:38:28 kaoz mm/memory.c:127: bad pmd
Quoted 7 lines
ffff810000207238(9090909090909090).
I also found another post with a very similar issue. The other post had almost the same message (*mm*/*memory*.*c*:*127*: *bad* *pmd* ffff810000207808(9090909090909090).)
Does anyone know what is it?
Thanks a lot for re-reporting this: it was fun to work it out. It's not a rootkit, it's harmless, but we ought to fix the noise. Simple patch below, but let me explain more verbosely first.

Quoted 7 lines

What was really interesting in your report was that the address is so close to that in OGAWA-San's report. I had a look at that page on my x86_64 boxes, and they have lots of 0x90s there too. It's just some page alignment filler that x86_64 kernel startup has missed cleaning up - patch below fixes that. There's no security aspect to it: the entries were already not-present, they just generate this noise by triggering the pmd_bad test.

Is there a particular reason we use 0x90 as an alignment filler ? If we can put anything else, at least next time it will not get confused with NOPs. We could use 0xAF (Alignment Filler) for instance.

Well done BTW ;-)

Reagrds,
Willy

Jan Engelhardt

2008-05-28 20:14 UTC #5 lore raw

Quoted 32 lines · Willy Tarreau

Quoted 23 lines · Hugh Dickins
Quoted 10 lines · Fede
Today I tried to start a firewalling script and failed due to an unrelated issue, but when I checked the log I saw this:
May 27 20:38:15 kaoz ip_tables: (C) 2000-2006 Netfilter Core Team
May 27 20:38:28 kaoz Netfilter messages via NETLINK v0.30.
May 27 20:38:28 kaoz nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
May 27 20:38:28 kaoz ctnetlink v0.93: registering with nfnetlink.
May 27 20:38:28 kaoz ClusterIP Version 0.8 loaded successfully
May 27 20:38:28 kaoz mm/memory.c:127: bad pmd
Quoted 7 lines
ffff810000207238(9090909090909090).
I also found another post with a very similar issue. The other post had almost the same message (*mm*/*memory*.*c*:*127*: *bad* *pmd* ffff810000207808(9090909090909090).)
Does anyone know what is it?
Thanks a lot for re-reporting this: it was fun to work it out. It's not a rootkit, it's harmless, but we ought to fix the noise. Simple patch below, but let me explain more verbosely first.
Quoted 7 lines
What was really interesting in your report was that the address is so close to that in OGAWA-San's report. I had a look at that page on my x86_64 boxes, and they have lots of 0x90s there too. It's just some page alignment filler that x86_64 kernel startup has missed cleaning up - patch below fixes that. There's no security aspect to it: the entries were already not-present, they just generate this noise by triggering the pmd_bad test.

Quoted 1 line

Is there a particular reason we use 0x90 as an alignment filler ?

Alignment within functions. You could use a JMP to jump over the alignment, but that would be costly. So in order to "run through the wall", you need an opcode that does not do anything, something like 0x90.
0xAF would map to scasd on x86, and I'd hardly call that a no-op.

Willy Tarreau

2008-05-28 20:43 UTC #6 lore raw

Quoted 34 lines · Jan Engelhardt

Quoted 32 lines · Willy Tarreau
Quoted 30 lines · Hugh Dickins
Quoted 10 lines · Fede
Today I tried to start a firewalling script and failed due to an unrelated issue, but when I checked the log I saw this:
May 27 20:38:15 kaoz ip_tables: (C) 2000-2006 Netfilter Core Team
May 27 20:38:28 kaoz Netfilter messages via NETLINK v0.30.
May 27 20:38:28 kaoz nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
May 27 20:38:28 kaoz ctnetlink v0.93: registering with nfnetlink.
May 27 20:38:28 kaoz ClusterIP Version 0.8 loaded successfully
May 27 20:38:28 kaoz mm/memory.c:127: bad pmd
Quoted 7 lines
ffff810000207238(9090909090909090).
I also found another post with a very similar issue. The other post had almost the same message (*mm*/*memory*.*c*:*127*: *bad* *pmd* ffff810000207808(9090909090909090).)
Does anyone know what is it?
Thanks a lot for re-reporting this: it was fun to work it out. It's not a rootkit, it's harmless, but we ought to fix the noise. Simple patch below, but let me explain more verbosely first.
What was really interesting in your report was that the address is so close to that in OGAWA-San's report. I had a look at that page on my x86_64 boxes, and they have lots of 0x90s there too. It's just some page alignment filler that x86_64 kernel startup has missed cleaning up - patch below fixes that. There's no security aspect to it: the entries were already not-present, they just generate this noise by triggering the pmd_bad test.

Quoted 8 lines

Quoted 1 line
Is there a particular reason we use 0x90 as an alignment filler ?
Alignment within functions. You could use a JMP to jump over the alignment, but that would be costly. So in order to "run through the wall", you need an opcode that does not do anything, something like 0x90.
0xAF would map to scasd on x86, and I'd hardly call that a no-op.

OK, I did not understand from Hugh's explanation that it was all about alignment within functions. Of course, 0x90 is fine there (though there are multi-byte NOPs available).

Cheers,
Willy

Jan Engelhardt

2008-05-28 21:24 UTC #7 lore raw

Quoted 10 lines · Willy Tarreau

Quoted 9 lines
Quoted 2 lines
Is there a particular reason we use 0x90 as an alignment filler ?
Alignment within functions. You could use a JMP to jump over the alignment, but that would be costly. So in order to "run through the wall", you need an opcode that does not do anything, something like 0x90.
0xAF would map to scasd on x86, and I'd hardly call that a no-op.

Quoted 3 lines

OK, I did not understand from Hugh's explanation that it was all about alignment within functions. Of course, 0x90 is fine there (though there are multi-byte NOPs available).

"All about alignment within functions" -- I am not sure about that, you just happened to ask about 0x90 :)
And if you have a 1-byte NOP (which fits perfectly everywhere), which is also a real NOP (and not just a filler byte that could possibly be an opcode doing something very different), you've got an ideal candidate for padding, no?
There is probably nothing wrong with padding .data sections with 0xAF or even 0xDB and ud2 to catch execute-readonly-data cases. To that end, I think something like that should be proposed to binutils.

Hugh Dickins

2008-05-28 22:20 UTC #8 lore raw

In reply to Willy Tarreau

Quoted 10 lines · Willy Tarreau

Quoted 9 lines · Jan Engelhardt
Quoted 7 lines · Willy Tarreau
Quoted 4 lines · Hugh Dickins
page on my x86_64 boxes, and they have lots of 0x90s there too. It's just some page alignment filler that x86_64 kernel startup has missed cleaning up - patch below fixes that. There's no
Is there a particular reason we use 0x90 as an alignment filler ?

Quoted 8 lines

Quoted 4 lines
Alignment within functions. You could use a JMP to jump over the alignment, but that would be costly. So in order to "run through the wall", you need an opcode that does not do anything, something like 0x90.
OK, I did not understand from Hugh's explanation that it was all about alignment within functions. Of course, 0x90 is fine there (though there are multi-byte NOPs available).

I'm hardly the right person to answer on this, but I believe that because the 0x90 NOP is particularly appropriate in .text (and prevents stalls even where it cannot be reached?), it gets used
as the default alignment filler all over. (It's even specified
for the arch-independent default _ALIGN in linux/linkage.h.)

Hugh