BUG: using rootfstype=ext4 causes oops
Using rootfstype=ext4 with today's linux-2.6.git causes a panic on my two amd64 machines (haven't tested it on any others). The steps to reproduce go something like:
- add rootfstype=ext4 kernel param
- boot
- do some I/O
My root fs is actually ext3 (if that's relevant). Booting without rootfstype=ext4 seems stable. Let me know if you need more info or patch testing. This trace was obtained over netconsole:
[ 281.714039] ------------[ cut here ]------------ [ 281.714207] kernel BUG at mm/slub.c:2802! [ 281.714366] invalid opcode: 0000 [#1] PREEMPT SMP [ 281.714709] last sysfs file: /sys/class/net/lo/operstate [ 281.714865] CPU 0 [ 281.715017] Modules linked in: [ 281.715017] Pid: 0, comm: swapper Not tainted 2.6.30-rc2-diogenes7 #1 System Product Name [ 281.715017] RIP: 0010:[<ffffffff802b3d4e>] [<ffffffff802b3d4e>] kfree+0x12e/0x140 [ 281.715017] RSP: 0018:ffff88000100ddf8 EFLAGS: 00010246 [ 281.715017] RAX: 0100000000080000 RBX: ffff88003e828908 RCX: 0000000000000000 [ 281.715017] RDX: ffffe20000000000 RSI: ffffe20000dac8c0 RDI: ffff88003e828908 [ 281.715017] RBP: ffffffff8041f885 R08: 0000000002202000 R09: 0000000000000001 [ 281.715017] R10: 0000000000000016 R11: ffffffff80224b60 R12: ffff88003e828908 [ 281.715017] R13: 00000000000000e7 R14: 0000000000000000 R15: ffff88003e828908 [ 281.715017] FS: 00007ffe72c136f0(0000) GS:ffff88000100a000(0000) knlGS:0000000000000000 [ 281.715017] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b [ 281.715017] CR2: 00007ffe725c48f0 CR3: 000000003f2ac000 CR4: 00000000000006e0 [ 281.715017] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 281.715017] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 281.715017] Process swapper (pid: 0, threadinfo ffffffff806e2000, task ffffffff80688360) [ 281.715017] Stack: [ 281.715017] ffff88003e828908 ffff88003fb8a800 0000000000000000 ffffffff8041f885 [ 281.715017] 0000000000000050 ffff88003fb8a800 ffff88003e804d88 0000000000000000 [ 281.715017] 0000000000000050 ffff88003fb8a800 ffff88003e804d88 ffffffff80423801 [ 281.715017] Call Trace: [ 281.715017] <IRQ> <0> [<ffffffff8041f885>] ? ide_complete_cmd+0x75/0x100 [ 281.715017] [<ffffffff80423801>] ? ide_finish_cmd+0x51/0xa0 [ 281.715017] [<ffffffff80424101>] ? task_no_data_intr+0xf1/0x170 [ 281.715017] [<ffffffff80424010>] ? task_no_data_intr+0x0/0x170 [ 281.715017] [<ffffffff8041f4ec>] ? ide_intr+0x1ec/0x250 [ 281.715017] [<ffffffff8026c3dd>] ? handle_IRQ_event+0xad/0x220 [ 281.715017] [<ffffffff8026e081>] ? handle_edge_irq+0xc1/0x160 [ 281.715017] [<ffffffff8020e167>] ? handle_irq+0x17/0x20 [ 281.715017] [<ffffffff8020d8d5>] ? do_IRQ+0x65/0xf0 [ 281.715017] [<ffffffff8020bd53>] ? ret_from_intr+0x0/0xa [ 281.715017] <EOI> <0> [<ffffffff8021321b>] ? default_idle+0x9b/0x150 [ 281.715017] [<ffffffff802590f7>] ? notifier_call_chain+0x37/0x70 [ 281.715017] [<ffffffff8020a64a>] ? cpu_idle+0x5a/0xc0 [ 281.715017] [<ffffffff806ebc55>] ? start_kernel+0x342/0x408 [ 281.715017] [<ffffffff806eb378>] ? x86_64_start_kernel+0xe1/0xf2 [ 281.715017] Code: 14 49 8b 00 49 89 04 d4 4d 89 20 eb cc e8 3b 1f 2c 00 e9 57 ff ff ff 66 a9 00 c0 66 90 74 48 89 f7 fd ff <0f> 0b eb fe 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 48 81 ef [ 281.715017] RIP [<ffffffff802b3d4e>] kfree+0x12e/0x140 [ 281.715017] RSP <ffff88000100ddf8> [ 281.715017] ---[ end trace 0e7df544696dff6e ]--- [ 281.715017] Kernel panic - not syncing: Fatal exception in interrupt [ 281.715017] Pid: 0, comm: swapper Tainted: G D 2.6.30-rc2-diogenes7 #1 [ 281.715017] Call Trace: [ 281.715017] <IRQ> [<ffffffff80574c4a>] ? panic+0x95/0x152 [ 281.715017] [<ffffffff8020e371>] ? show_registers+0x91/0x2f0 [ 281.715017] [<ffffffff802596f9>] ? __atomic_notifier_call_chain+0x19/0x50 [ 281.715017] [<ffffffff80258f16>] ? up+0x16/0x50 [ 281.715017] [<ffffffff8023e195>] ? release_console_sem+0x1a5/0x1f0 [ 281.715017] [<ffffffff8020f77d>] ? oops_end+0x8d/0xa0 [ 281.715017] [<ffffffff8020d1c4>] ? do_invalid_op+0x84/0xa0 [ 281.715017] [<ffffffff802b3d4e>] ? kfree+0x12e/0x140 [ 281.715017] [<ffffffff80231350>] ? activate_task+0x40/0x70 [ 281.715017] [<ffffffff80239506>] ? try_to_wake_up+0x116/0x250 [ 281.715017] [<ffffffff8041f885>] ? ide_complete_cmd+0x75/0x100 [ 281.715017] [<ffffffff8020c145>] ? invalid_op+0x15/0x20 [ 281.715017] [<ffffffff8041f885>] ? ide_complete_cmd+0x75/0x100 [ 281.715017] [<ffffffff80224b60>] ? native_apic_mem_write+0x0/0x10 [ 281.715017] [<ffffffff802b3d4e>] ? kfree+0x12e/0x140 [ 281.715017] [<ffffffff802b3c9f>] ? kfree+0x7f/0x140 [ 281.715017] [<ffffffff8041f885>] ? ide_complete_cmd+0x75/0x100 [ 281.715017] [<ffffffff80423801>] ? ide_finish_cmd+0x51/0xa0 [ 281.715017] [<ffffffff80424101>] ? task_no_data_intr+0xf1/0x170 [ 281.715017] [<ffffffff80424010>] ? task_no_data_intr+0x0/0x170 [ 281.715017] [<ffffffff8041f4ec>] ? ide_intr+0x1ec/0x250 [ 281.715017] [<ffffffff8026c3dd>] ? handle_IRQ_event+0xad/0x220 [ 281.715017] [<ffffffff8026e081>] ? handle_edge_irq+0xc1/0x160 [ 281.715017] [<ffffffff8020e167>] ? handle_irq+0x17/0x20 [ 281.715017] [<ffffffff8020d8d5>] ? do_IRQ+0x65/0xf0 [ 281.715017] [<ffffffff8020bd53>] ? ret_from_intr+0x0/0xa [ 281.715017] <EOI> [<ffffffff8021321b>] ? default_idle+0x9b/0x150 [ 281.715017] [<ffffffff802590f7>] ? notifier_call_chain+0x37/0x70 [ 281.715017] [<ffffffff8020a64a>] ? cpu_idle+0x5a/0xc0 [ 281.715017] [<ffffffff806ebc55>] ? start_kernel+0x342/0x408 [ 281.715017] [<ffffffff806eb378>] ? x86_64_start_kernel+0xe1/0xf2
--
Andrew Price
Quoted 2 lines · Andrew Price
Using rootfstype=ext4 with today's linux-2.6.git causes a panic on my two amd64 machines (haven't tested it on any others).
Hmm, git commit id, please? The stack traces are in the IDE interrupt handler, so it seems surprising that ext4 would trigger it but ext3 would not. Have you tried ext4 on any earlier kernel?
The main difference I can think of is that ext4 enables barriers by default; maybe that's the case of the IDE breakage? Can you try booting with the boot command option "rootfsflags=barrier=0" as well as "rootfstype=ext4", and see if that helps?
If so, it's a bug in the IDE code in that it's not handling barriers correctly.
- Ted
Quoted 5 lines · Theodore Tso
Quoted 2 lines · Andrew Price
Using rootfstype=ext4 with today's linux-2.6.git causes a panic on my two amd64 machines (haven't tested it on any others).
Hmm, git commit id, please?
commit 3ee8da87ba6151ec91b2b8bbd27633bb248ea0d5
Merge: a2c252e 9dd175f
Author: Linus Torvalds <[email protected]>
Date: Wed Apr 15 09:11:11 2009 -0700
Quoted 3 lines
The stack traces are in the IDE interrupt
handler, so it seems surprising that ext4 would trigger it but ext3 would not. Have you tried ext4 on any earlier kernel?
It happened with linux-2.6.git kernels earlier in the week when I started trying rootfstype=ext4 but I haven't tried properly bisecting it yet.
Quoted 4 lines
The main difference I can think of is that ext4 enables barriers by default; maybe that's the case of the IDE breakage? Can you try booting with the boot command option "rootfsflags=barrier=0" as well as "rootfstype=ext4", and see if that helps?
I added rootflags=barrier=0 ...
(aside: the ext4 docs say the param is "barriers" with an s; it isn't).
... and it doesn't panic.
Quoted 2 lines
If so, it's a bug in the IDE code in that it's not handling barriers correctly.
Bingo.
--
Andrew Price
Quoted 14 lines · Andrew Price
Quoted 3 lines · Theodore Tso
The stack traces are in the IDE interrupt
handler, so it seems surprising that ext4 would trigger it but ext3 would not. Have you tried ext4 on any earlier kernel?It happened with linux-2.6.git kernels earlier in the week when I started trying rootfstype=ext4 but I haven't tried properly bisecting it yet.
Quoted 4 lines
The main difference I can think of is that ext4 enables barriers by default; maybe that's the case of the IDE breakage? Can you try booting with the boot command option "rootfsflags=barrier=0" as well as "rootfstype=ext4", and see if that helps?
Quoted 8 lines
I added rootflags=barrier=0 ...
... and it doesn't panic.
Quoted 2 lines
If so, it's a bug in the IDE code in that it's not handling barriers correctly.
Bingo.
OK, can you confirm that you tried using ext4 with either 2.6.29 or 2.6.28, and it was working previously? That would make it a regression which Rafael would track.
Also, I'd suggest sending Bartolmiej details about what IDE controller you have (the dmesg during a boot under ext3 or ext4 w/ barrier=0 would also be useful), so he can track it down. It might be specific to a particular IDE controller/device, and not be a generalized IDE problem, after all.
Anyway, I'll hand this bug off to Raefael's and/or Bartlomiej's very capable hands, since I suspect you could also reproduce this using rootfstype=ext3 rootfsflags=barrier=1, and that this is purely an IDE driver problem.
Regards,
- Ted
Quoted 1 line
(aside: the ext4 docs say the param is "barriers" with an s; it isn't).
P.S. Where in the documentation file do we have the mount option as "barriers=0". I went looking for it, and I couldn't find it. What I see is:
barrier=<0|1(*)> This enables/disables the use of write barriers in
the jbd code. barrier=0 disables, barrier=1 enables. This also requires an IO stack which can support barriers, and if jbd gets an error on a barrier write, it will disable again with a warning. Write barriers enforce proper on-disk ordering of journal commits, making volatile disk write caches safe to use, at some performance penalty. If your disks are battery-backed in one way or another, disabling barriers may safely improve performance.
Quoted 15 lines · On Thursday 16 April 2009 16:53:57 Theodore Tso wrote:
Quoted 14 lines · Andrew Price
Quoted 3 lines · Theodore Tso
The stack traces are in the IDE interrupt
handler, so it seems surprising that ext4 would trigger it but ext3 would not. Have you tried ext4 on any earlier kernel?It happened with linux-2.6.git kernels earlier in the week when I started trying rootfstype=ext4 but I haven't tried properly bisecting it yet.
Quoted 4 lines
The main difference I can think of is that ext4 enables barriers by default; maybe that's the case of the IDE breakage? Can you try booting with the boot command option "rootfsflags=barrier=0" as well as "rootfstype=ext4", and see if that helps?
Quoted 8 lines
Quoted 8 lines
I added rootflags=barrier=0 ...
... and it doesn't panic.
Quoted 2 lines
If so, it's a bug in the IDE code in that it's not handling barriers correctly.
Bingo.
Freeing non-slab objects is bad.
Andrew, does this patch help?
--- drivers/ide/ide-io.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
drivers/ide/ide-io.c +7 -4
@@ -102,11 +102,14 @@ void ide_complete_cmd(ide_drive_t *drive drive->dev_flags |=IDE_DFLAG_PARKED ; } -if (rq && rq->cmd_type ==REQ_TYPE_ATA_TASKFILE ) -memcpy (rq->special, cmd,sizeof (*cmd)); +if (rq && rq->cmd_type ==REQ_TYPE_ATA_TASKFILE ) { +struct ide_cmd *orig_cmd = rq->special; -if (cmd->tf_flags &IDE_TFLAG_DYN ) -kfree (cmd); +if (cmd->tf_flags &IDE_TFLAG_DYN ) +kfree (orig_cmd); +else +memcpy (orig_cmd, cmd,sizeof (*cmd)); + } }/* obsolete, blk_rq_bytes() should be used instead */
Quoted 3 lines · Bartlomiej Zolnierkiewicz
Freeing non-slab objects is bad.
Andrew, does this patch help?
Yes, that seems to fix it. I tested it with gfs2 and ext4 the usual way and I couldn't reproduce the panic.
Thanks Bartlomiej.
Quoted 18 lines
--- drivers/ide/ide-io.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)Index:
b/drivers/ide/ide-io.c
===================================================================drivers/ide/ide-io.c +2
-2@@ -102,11 +102,14 @@ void ide_complete_cmd(ide_drive_t *drive drive->dev_flags |=IDE_DFLAG_PARKED ; } -if (rq && rq->cmd_type ==REQ_TYPE_ATA_TASKFILE ) -memcpy (rq->special, cmd,sizeof (*cmd)); +if (rq && rq->cmd_type ==REQ_TYPE_ATA_TASKFILE ) { +struct ide_cmd *orig_cmd = rq->special;
drivers/ide/ide-io.c void ide_complete_cmd(ide_drive_t *drive · line 115
Quoted 10 lines
drivers/ide/ide-io.c +5
-2-if (cmd->tf_flags &IDE_TFLAG_DYN ) -kfree (cmd); +if (cmd->tf_flags &IDE_TFLAG_DYN ) +kfree (orig_cmd); +else +memcpy (orig_cmd, cmd,sizeof (*cmd)); + } }/* obsolete, blk_rq_bytes() should be used instead */
--
Andrew Price
Quoted 9 lines · On Thursday 16 April 2009 19:05:56 Andrew Price wrote:
Quoted 3 lines · Bartlomiej Zolnierkiewicz
Freeing non-slab objects is bad.
Andrew, does this patch help?
Yes, that seems to fix it. I tested it with gfs2 and ext4 the usual way and I couldn't reproduce the panic.
Thanks Bartlomiej.
Great, I'll queue the following patch for the next round of IDE fixes.
From: Bartlomiej Zolnierkiewicz <[email protected]>
Subject: [PATCH] ide: fix barriers support
Freeing non-slab objects is bad and results in an oops. Fix it.
--- drivers/ide/ide-io.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
drivers/ide/ide-io.c +7 -4
@@ -102,11 +102,14 @@ void ide_complete_cmd(ide_drive_t *drive drive->dev_flags |=IDE_DFLAG_PARKED ; } -if (rq && rq->cmd_type ==REQ_TYPE_ATA_TASKFILE ) -memcpy (rq->special, cmd,sizeof (*cmd)); +if (rq && rq->cmd_type ==REQ_TYPE_ATA_TASKFILE ) { +struct ide_cmd *orig_cmd = rq->special; -if (cmd->tf_flags &IDE_TFLAG_DYN ) -kfree (cmd); +if (cmd->tf_flags &IDE_TFLAG_DYN ) +kfree (orig_cmd); +else +memcpy (orig_cmd, cmd,sizeof (*cmd)); + } }/* obsolete, blk_rq_bytes() should be used instead */
Quoted 3 lines · Theodore Tso
OK, can you confirm that you tried using ext4 with either 2.6.29 or 2.6.28, and it was working previously? That would make it a regression which Rafael would track.
I've tested this with a local GFS2 mount (which uses barriers too) and the same bug occurs. Apologies for making it seem like an ext4 bug. I'm currently waiting for a kernel to build with Bartlomiej's patch so we'll see how that goes.
Quoted 5 lines
Also, I'd suggest sending Bartolmiej details about what IDE controller you have (the dmesg during a boot under ext3 or ext4 w/ barrier=0 would also be useful), so he can track it down. It might be specific to a particular IDE controller/device, and not be a generalized IDE problem, after all.
00:0f.1 IDE interface [0101]: VIA Technologies, Inc. VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE [1106:0571] (rev 06) (prog-if 8a [Master SecP PriP])
Subsystem: ASUSTeK Computer Inc. A7V600/K8V-X/A8V Deluxe motherboard [1043:80ed]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 32
Interrupt: pin A routed to IRQ 20
Region 0: [virtual] Memory at 000001f0 (32-bit, non-prefetchable) [size=8]
Region 1: [virtual] Memory at 000003f0 (type 3, non-prefetchable) [size=1]
Region 2: [virtual] Memory at 00000170 (32-bit, non-prefetchable) [size=8]
Region 3: [virtual] Memory at 00000370 (type 3, non-prefetchable) [size=1]
Region 4: I/O ports at fc00 [size=16]
Capabilities: [c0] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Kernel driver in use: VIA_IDE
Quoted 4 lines
Anyway, I'll hand this bug off to Raefael's and/or Bartlomiej's very capable hands, since I suspect you could also reproduce this using rootfstype=ext3 rootfsflags=barrier=1, and that this is purely an IDE driver problem.
Thanks Ted.
Quoted 4 lines
P.S. Where in the documentation file do we have the mount option as "barriers=0". I went looking for it, and I couldn't find it. What I see is:
<snip>
Further up in that file in the last paragraph of the 'Quick usage instructions' it says:
[...]
'-o barriers=[0|1]' mount option for both ext3 and ext4 filesystems [...]
(I searched the file for barrier and it was the first chunk of text which matched.)
--
Andrew Price
Quoted 9 lines · Andrew Price
Quoted 3 lines · Theodore Tso
OK, can you confirm that you tried using ext4 with either 2.6.29 or 2.6.28, and it was working previously? That would make it a regression which Rafael would track.
I've tested this with a local GFS2 mount (which uses barriers too) and the same bug occurs. Apologies for making it seem like an ext4 bug. I'm currently waiting for a kernel to build with Bartlomiej's patch so we'll see how that goes.
No, no, that's fine. That sort of thing we get all the time; it's part of trying to find the root cause of the bug. What I was more interested in was trying to figure out when (at which kernel version) the bug had introduced; if it's a recent regression then Rafael would be interested in tracking it as such.
Quoted 7 lines
Further up in that file in the last paragraph of the 'Quick usage instructions' it says:
[...]
'-o barriers=[0|1]' mount option for both ext3 and ext4 filesystems [...]
OK, got it. Thanks. I'll queue up a patch to fix that up.
- Ted