Skip to contentSkip to conversation

Reading /proc/kcore causes a BUG()

DISCUSSION Nick Craig-Wood 4 participants 10 messages Sep 3, 2009 – Sep 7, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Is a fix for this going to make 2.6.31?

To replicate

cat /proc/kcore >/dev/null

See also

http://bugzilla.kernel.org/show_bug.cgi?id=13850

To get

BUG: unable to handle kernel paging request at eda08000
IP: [<c01a0e98>] read_kcore+0x27f/0x341
*pdpt = 0000000071809027
Oops: 0000 [#1] SMP

last sysfs file: /sys/class/net/lo/operstate

Pid: 2361, comm: cat Not tainted (2.6.31.git-x1 #1)
EIP: 0061:[<c01a0e98>] EFLAGS: 00010286 CPU: 1
EIP is at read_kcore+0x27f/0x341
EAX: c04404fc EBX: 00001000 ECX: 00000400 EDX: eda09000
ESI: eda08000 EDI: ec129000 EBP: ec60bf48 ESP: ec60bf10
DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
Process cat (pid: 2361, ti=ec60a000 task=ecf800b0 task.ti=ec60a000)

Stack:
 00001000 08051000 00000000 00001000 00000003 eda08000 ec129000 c04404fc
<0> eda09000 00000000 00001000 ecd39380 fffffffb c01a0c19 ec60bf6c c019a907
<0> ec60bf98 00001000 08051000 eccbab00 eccbab00 c019a8af 00001000 ec60bf8c
Call Trace:
 [<c01a0c19>] ? read_kcore+0x0/0x341
 [<c019a907>] ? proc_reg_read+0x58/0x6c
 [<c019a8af>] ? proc_reg_read+0x0/0x6c
 [<c016905e>] ? vfs_read+0x87/0x110
 [<c0169180>] ? sys_read+0x3b/0x60
 [<c0106a4d>] ? syscall_call+0x7/0xb
Code: d3 29 f3 89 f8 29 f0 39 d7 0f 46 d8 8d 0c 33 89 4d e8 29 5d ec 8b 45 e4 f6 40 0c 01 75 18 89 d9 c1 e9 02 89 f7 2b 7d dc 03 7d e0 <f3> a5 89 d9 83 e1 03 74 02 f3 a4 8b 55 e4 8b 12 89 55 e4 83 7d
EIP: [<c01a0e98>] read_kcore+0x27f/0x341 SS:ESP 0069:ec60bf10
CR2: 00000000eda08000
---[ end trace 4387f828fd1590eb ]---

I tried this on the latest git checkout (as of 2009-09-03 11:00 GMT) under Xen as a domU. The bugzilla report states it happens on non xen machines also.

I know reading /proc/kcore isn't such a good idea, but badly written backup scripts are triggering this on our customer's servers :-(

Signature · 1 line
Nick Craig-Wood <[email protected]> -- http://www.craig-wood.com/nick
#1 lore raw
Quoted 41 lines · Nick Craig-Wood

Is a fix for this going to make 2.6.31?

To replicate

 cat /proc/kcore >/dev/null

See also

 http://bugzilla.kernel.org/show_bug.cgi?id=13850

To get

BUG: unable to handle kernel paging request at eda08000
IP: [<c01a0e98>] read_kcore+0x27f/0x341
*pdpt = 0000000071809027
Oops: 0000 [#1] SMP

last sysfs file: /sys/class/net/lo/operstate

Pid: 2361, comm: cat Not tainted (2.6.31.git-x1 #1)
EIP: 0061:[<c01a0e98>] EFLAGS: 00010286 CPU: 1
EIP is at read_kcore+0x27f/0x341
EAX: c04404fc EBX: 00001000 ECX: 00000400 EDX: eda09000
ESI: eda08000 EDI: ec129000 EBP: ec60bf48 ESP: ec60bf10
 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
Process cat (pid: 2361, ti=ec60a000 task=ecf800b0 task.ti=ec60a000)

Stack:
 00001000 08051000 00000000 00001000 00000003 eda08000 ec129000 c04404fc
<0> eda09000 00000000 00001000 ecd39380 fffffffb c01a0c19 ec60bf6c c019a907
<0> ec60bf98 00001000 08051000 eccbab00 eccbab00 c019a8af 00001000 ec60bf8c
Call Trace:
 [<c01a0c19>] ? read_kcore+0x0/0x341
 [<c019a907>] ? proc_reg_read+0x58/0x6c
 [<c019a8af>] ? proc_reg_read+0x0/0x6c
 [<c016905e>] ? vfs_read+0x87/0x110
 [<c0169180>] ? sys_read+0x3b/0x60
 [<c0106a4d>] ? syscall_call+0x7/0xb
Code: d3 29 f3 89 f8 29 f0 39 d7 0f 46 d8 8d 0c 33 89 4d e8 29 5d ec 8b 45 e4 f6 40 0c 01 75 18 89 d9 c1 e9 02 89 f7 2b 7d dc 03 7d e0 <f3> a5 89 d9 83 e1 03 74 02 f3 a4 8b 55 e4 8b 12 89 55 e4 83 7d
EIP: [<c01a0e98>] read_kcore+0x27f/0x341 SS:ESP 0069:ec60bf10
CR2: 00000000eda08000
---[ end trace 4387f828fd1590eb ]---
Quoted 6 lines

I tried this on the latest git checkout (as of 2009-09-03 11:00 GMT) under Xen as a domU.  The bugzilla report states it happens on non xen machines also.

I know reading /proc/kcore isn't such a good idea, but badly written backup scripts are triggering this on our customer's servers :-(

AFAICT the bug was fixed but I can't seem to find the patches in Linus' git either. Lets CC Andrew and Hiroyuki-san.

Pekka

#2 lore raw
Quoted 46 lines · On Thu, 3 Sep 2009 15:14:55 +0300 Pekka Enberg <[email protected]> wrote:
Quoted 45 lines · Nick Craig-Wood

Is a fix for this going to make 2.6.31?

To replicate

 cat /proc/kcore >/dev/null

See also

 http://bugzilla.kernel.org/show_bug.cgi?id=13850

To get

BUG: unable to handle kernel paging request at eda08000
IP: [<c01a0e98>] read_kcore+0x27f/0x341
*pdpt = 0000000071809027
Oops: 0000 [#1] SMP

last sysfs file: /sys/class/net/lo/operstate

Pid: 2361, comm: cat Not tainted (2.6.31.git-x1 #1)
EIP: 0061:[<c01a0e98>] EFLAGS: 00010286 CPU: 1
EIP is at read_kcore+0x27f/0x341
EAX: c04404fc EBX: 00001000 ECX: 00000400 EDX: eda09000
ESI: eda08000 EDI: ec129000 EBP: ec60bf48 ESP: ec60bf10
 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
Process cat (pid: 2361, ti=ec60a000 task=ecf800b0 task.ti=ec60a000)

Stack:
 00001000 08051000 00000000 00001000 00000003 eda08000 ec129000 c04404fc
<0> eda09000 00000000 00001000 ecd39380 fffffffb c01a0c19 ec60bf6c c019a907
<0> ec60bf98 00001000 08051000 eccbab00 eccbab00 c019a8af 00001000 ec60bf8c
Call Trace:
 [<c01a0c19>] ? read_kcore+0x0/0x341
 [<c019a907>] ? proc_reg_read+0x58/0x6c
 [<c019a8af>] ? proc_reg_read+0x0/0x6c
 [<c016905e>] ? vfs_read+0x87/0x110
 [<c0169180>] ? sys_read+0x3b/0x60
 [<c0106a4d>] ? syscall_call+0x7/0xb
Code: d3 29 f3 89 f8 29 f0 39 d7 0f 46 d8 8d 0c 33 89 4d e8 29 5d ec 8b 45 e4 f6 40 0c 01 75 18 89 d9 c1 e9 02 89 f7 2b 7d dc 03 7d e0 <f3> a5 89 d9 83 e1 03 74 02 f3 a4 8b 55 e4 8b 12 89 55 e4 83 7d
EIP: [<c01a0e98>] read_kcore+0x27f/0x341 SS:ESP 0069:ec60bf10
CR2: 00000000eda08000
---[ end trace 4387f828fd1590eb ]---

I tried this on the latest git checkout (as of 2009-09-03 11:00 GMT) under Xen as a domU.  The bugzilla report states it happens on non xen machines also.

Quoted 6 lines
Quoted 2 lines

I know reading /proc/kcore isn't such a good idea, but badly written backup scripts are triggering this on our customer's servers :-(

AFAICT the bug was fixed but I can't seem to find the patches in Linus' git either. Lets CC Andrew and Hiroyuki-san.

Ah, it's now tested under mmotm. please wait.

Thanks,
-Kame

#3 lore raw
Quoted 12 lines · KAMEZAWA Hiroyuki
Quoted 10 lines · On Thu, 3 Sep 2009 15:14:55 +0300 Pekka Enberg <[email protected]> wrote:
Quoted 9 lines · Nick Craig-Wood

Is a fix for this going to make 2.6.31?

To replicate

 cat /proc/kcore >/dev/null

See also

 http://bugzilla.kernel.org/show_bug.cgi?id=13850

[snip]

Quoted 11 lines
Quoted 9 lines
Quoted 6 lines

I tried this on the latest git checkout (as of 2009-09-03 11:00 GMT) under Xen as a domU.  The bugzilla report states it happens on non xen machines also.

I know reading /proc/kcore isn't such a good idea, but badly written backup scripts are triggering this on our customer's servers :-(

AFAICT the bug was fixed but I can't seem to find the patches in Linus' git either. Lets CC Andrew and Hiroyuki-san.

Ah, it's now tested under mmotm. please wait.

I tried mmotm but I couldn't get it to boot under Xen :-(

If you send me a patch against latest git I'm willing to test it (I tried to extract the relevant patch from mmotm but failed dismally)

Thanks

Nick

Signature · 1 line
Nick Craig-Wood <[email protected]> -- http://www.craig-wood.com/nick
#4 lore raw
Quoted 24 lines · On Fri, 4 Sep 2009 10:00:39 +0100 Nick Craig-Wood <[email protected]> wrote:
Quoted 12 lines · KAMEZAWA Hiroyuki
Quoted 10 lines · On Thu, 3 Sep 2009 15:14:55 +0300 Pekka Enberg <[email protected]> wrote:
Quoted 9 lines · Nick Craig-Wood

Is a fix for this going to make 2.6.31?

To replicate

 cat /proc/kcore >/dev/null

See also

 http://bugzilla.kernel.org/show_bug.cgi?id=13850

[snip]

Quoted 10 lines
Quoted 9 lines
Quoted 6 lines

I tried this on the latest git checkout (as of 2009-09-03 11:00 GMT) under Xen as a domU.  The bugzilla report states it happens on non xen machines also.

I know reading /proc/kcore isn't such a good idea, but badly written backup scripts are triggering this on our customer's servers :-(

AFAICT the bug was fixed but I can't seem to find the patches in Linus' git either. Lets CC Andrew and Hiroyuki-san.

Quoted 9 lines
Quoted 1 line

Ah, it's now tested under mmotm. please wait.

I tried mmotm but I couldn't get it to boot under Xen :-(

If you send me a patch against latest git I'm willing to test it (I tried to extract the relevant patch from mmotm but failed dismally)

Thanks

If you already downloaded mmotm, use these patches.

vmalloc-unmap-vmalloc-area-after-hiding-it.patch kcore-fix-vread-vwrite-to-be-aware-of-holes.patch kcore-fix-vread-vwrite-to-be-aware-of-holes-update.patch kcore-proc-kcore-should-use-vread.patch

All I tested was x86-32/x86-64. then more tests are welcomed.

-Kame

Quoted 4 lines

Nick

Signature · 1 line
Nick Craig-Wood <[email protected]> -- http://www.craig-wood.com/nick
#5 lore raw
Quoted 34 lines · KAMEZAWA Hiroyuki
Quoted 24 lines · On Fri, 4 Sep 2009 10:00:39 +0100 Nick Craig-Wood <[email protected]> wrote:
Quoted 12 lines · KAMEZAWA Hiroyuki
Quoted 10 lines · On Thu, 3 Sep 2009 15:14:55 +0300 Pekka Enberg <[email protected]> wrote:
Quoted 9 lines · Nick Craig-Wood

Is a fix for this going to make 2.6.31?

To replicate

 cat /proc/kcore >/dev/null

See also

 http://bugzilla.kernel.org/show_bug.cgi?id=13850

[snip]

Quoted 10 lines
Quoted 9 lines
Quoted 6 lines

I tried this on the latest git checkout (as of 2009-09-03 11:00 GMT) under Xen as a domU.  The bugzilla report states it happens on non xen machines also.

I know reading /proc/kcore isn't such a good idea, but badly written backup scripts are triggering this on our customer's servers :-(

AFAICT the bug was fixed but I can't seem to find the patches in Linus' git either. Lets CC Andrew and Hiroyuki-san.

Quoted 6 lines
Quoted 1 line

Ah, it's now tested under mmotm. please wait.

I tried mmotm but I couldn't get it to boot under Xen :-(

If you send me a patch against latest git I'm willing to test it (I tried to extract the relevant patch from mmotm but failed dismally)

Quoted 8 lines

If you already downloaded mmotm, use these patches.

vmalloc-unmap-vmalloc-area-after-hiding-it.patch kcore-fix-vread-vwrite-to-be-aware-of-holes.patch kcore-fix-vread-vwrite-to-be-aware-of-holes-update.patch kcore-proc-kcore-should-use-vread.patch

All I tested was x86-32/x86-64. then more tests are welcomed.

I can confirm that these patches fix the problem on x86-32 running under Xen DomU - thank you very much.

Signature · 1 line
Nick Craig-Wood <[email protected]> -- http://www.craig-wood.com/nick
#6 lore raw
In reply to KAMEZAWA Hiroyuki
Quoted 13 lines · On Fri, 4 Sep 2009 18:16:45 +0900 KAMEZAWA Hiroyuki…
Quoted 10 lines · On Fri, 4 Sep 2009 10:00:39 +0100 Nick Craig-Wood <[email protected]> wrote:
Quoted 9 lines · KAMEZAWA Hiroyuki
Quoted 7 lines · On Thu, 3 Sep 2009 15:14:55 +0300 Pekka Enberg <[email protected]> wrote:
Quoted 6 lines · Nick Craig-Wood

Is a fix for this going to make 2.6.31?

To replicate

__cat /proc/kcore >/dev/null

Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

See also

__http://bugzilla.kernel.org/show_bug.cgi?id=13850

[snip]

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

I tried this on the latest git checkout (as of 2009-09-03 11:00 GMT) under Xen as a domU. __The bugzilla report states it happens on non xen machines also.

Is this a regression? I've lost track..

Quoted 16 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

I know reading /proc/kcore isn't such a good idea, but badly written backup scripts are triggering this on our customer's servers :-(

AFAICT the bug was fixed but I can't seem to find the patches in Linus' git either. Lets CC Andrew and Hiroyuki-san.

Quoted 9 lines
Quoted 1 line

Ah, it's now tested under mmotm. please wait.

I tried mmotm but I couldn't get it to boot under Xen :-(

If you send me a patch against latest git I'm willing to test it (I tried to extract the relevant patch from mmotm but failed dismally)

Thanks

Quoted 8 lines

If you already downloaded mmotm, use these patches.

vmalloc-unmap-vmalloc-area-after-hiding-it.patch kcore-fix-vread-vwrite-to-be-aware-of-holes.patch kcore-fix-vread-vwrite-to-be-aware-of-holes-update.patch kcore-proc-kcore-should-use-vread.patch

All I tested was x86-32/x86-64. then more tests are welcomed.

That's a lot of stuff for 2.6.31. Is there some simple quickfix we can do?

#7 lore raw
Quoted 17 lines · On Fri, 4 Sep 2009 10:31:56 -0700 Andrew Morton <[email protected]>…
Quoted 15 lines · On Fri, 4 Sep 2009 18:16:45 +0900 KAMEZAWA Hiroyuki…
Quoted 12 lines · On Fri, 4 Sep 2009 10:00:39 +0100 Nick Craig-Wood <[email protected]> wrote:
Quoted 11 lines · KAMEZAWA Hiroyuki
Quoted 9 lines · On Thu, 3 Sep 2009 15:14:55 +0300 Pekka Enberg <[email protected]> wrote:
Quoted 8 lines · Nick Craig-Wood

Is a fix for this going to make 2.6.31?

To replicate

__cat /proc/kcore >/dev/null

See also

Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

__http://bugzilla.kernel.org/show_bug.cgi?id=13850

[snip]

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

I tried this on the latest git checkout (as of 2009-09-03 11:00 GMT) under Xen as a domU. __The bugzilla report states it happens on non xen machines also.

Is this a regression? I've lost track..

regression. (this comes from new per-cpu area implemantation, which uses vmalloc area with memory holes.)

Quoted 18 lines
Quoted 18 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

I know reading /proc/kcore isn't such a good idea, but badly written backup scripts are triggering this on our customer's servers :-(

AFAICT the bug was fixed but I can't seem to find the patches in Linus' git either. Lets CC Andrew and Hiroyuki-san.

Quoted 9 lines
Quoted 1 line

Ah, it's now tested under mmotm. please wait.

I tried mmotm but I couldn't get it to boot under Xen :-(

If you send me a patch against latest git I'm willing to test it (I tried to extract the relevant patch from mmotm but failed dismally)

Thanks

If you already downloaded mmotm, use these patches.

Quoted 9 lines
Quoted 6 lines

vmalloc-unmap-vmalloc-area-after-hiding-it.patch kcore-fix-vread-vwrite-to-be-aware-of-holes.patch kcore-fix-vread-vwrite-to-be-aware-of-holes-update.patch kcore-proc-kcore-should-use-vread.patch

All I tested was x86-32/x86-64. then more tests are welcomed.

That's a lot of stuff for 2.6.31. Is there some simple quickfix we can do?

I don't like this patch. But here, my 1st version which wasn't sent out. -Kame
==
/proc/kcore may access in memory holes in vmalloc area. skip it by using copy_from_user().

---
 fs/proc/kcore.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
fs/proc/kcore.c +3 -1
Index: linux-2.6.31-rc8/fs/proc/kcore.c
===================================================================
@@ -361,7 +361,9 @@ read_kcore(struct file *file, char __use
 				/* don't dump ioremap'd stuff! (TA) */
 				if (m->flags & VM_IOREMAP)
 					continue;
-				memcpy(elf_buf + (vmstart - start),
+				/* we may access memory holes */
+				__copy_from_user_inatomic(
+					elf_buf + (vmstart - start),
 					(char *)vmstart, vmsize);
 			}
 			read_unlock(&vmlist_lock);
#8 lore raw
Quoted 4 lines · On Mon, 7 Sep 2009 09:38:56 +0900 KAMEZAWA Hiroyuki…
Quoted 2 lines

That's a lot of stuff for 2.6.31. Is there some simple quickfix we can do?

I don't like this patch. But here, my 1st version which wasn't sent out. -Kame

Fixed warnings. (I confirmed this patch works well on x86-64.) ==
To access vmalloc area which may have memory holes, copy_from_user is useful. By this,

 # cat /proc/kcore > /dev/null
will not panic.
---
 fs/proc/kcore.c |    8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
fs/proc/kcore.c +7 -1
Index: linux-2.6.31-rc8/fs/proc/kcore.c
===================================================================
@@ -361,7 +361,13 @@ read_kcore(struct file *file, char __use
 				/* don't dump ioremap'd stuff! (TA) */
 				if (m->flags & VM_IOREMAP)
 					continue;
-				memcpy(elf_buf + (vmstart - start),
+				/*
+				 * we may access memory holes, then use
+				 * ex_table. checking return value just for
+				 * avoid warnings.
+				 */
+				vmsize = __copy_from_user_inatomic(
+					elf_buf + (vmstart - start),
 					(char *)vmstart, vmsize);
 			}
 			read_unlock(&vmlist_lock);
#9 lore raw
Quoted 28 lines · KAMEZAWA Hiroyuki
Quoted 4 lines · On Mon, 7 Sep 2009 09:38:56 +0900 KAMEZAWA Hiroyuki…
Quoted 2 lines

That's a lot of stuff for 2.6.31. Is there some simple quickfix we can do?

I don't like this patch. But here, my 1st version which wasn't sent out. -Kame

Fixed warnings. (I confirmed this patch works well on x86-64.) ==
To access vmalloc area which may have memory holes, copy_from_user is useful. By this,

 # cat /proc/kcore > /dev/null
will not panic.
---
 fs/proc/kcore.c |    8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

Index: linux-2.6.31-rc8/fs/proc/kcore.c
===================================================================

fs/proc/kcore.c +0 -1
@@ -361,7 +361,13 @@ read_kcore(struct file *file, char __use
 				/* don't dump ioremap'd stuff! (TA) */
 				if (m->flags & VM_IOREMAP)
 					continue;
-				memcpy(elf_buf + (vmstart - start),

fs/proc/kcore.c read_kcore(struct file *file, char __use · line 373

Quoted 10 lines
fs/proc/kcore.c +7 -0
+				/*
+				 * we may access memory holes, then use
+				 * ex_table. checking return value just for
+				 * avoid warnings.
+				 */
+				vmsize = __copy_from_user_inatomic(
+					elf_buf + (vmstart - start),
 					(char *)vmstart, vmsize);
 			}
 			read_unlock(&vmlist_lock);

I gave this patch a thrashing and it works fine for me too (Xen domU x86)

It would be great if we could have this quick fix for 2.6.31.

Perhaps it should also go out to 2.6.30-stable also?

I can confirm that this patch applies to 2.6.30.5 and fixes the problem there too.

Signature · 1 line
Nick Craig-Wood <[email protected]> -- http://www.craig-wood.com/nick