sysfs Kernel BUG when RAID bitmap file has IO errors
Subsystems (2)
I have a RAID-5 bitmap file on external media:
# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdb1[0] sdf1[4] sde1[3] sdd1[2] sdc1[1]
1562834944 blocks level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
bitmap: 0/373 pages [0KB], 512KB chunk, file: /root/raid/bitmapunused devices: <none>
Unfortunately, that external media is flash which slowly dies:
# md5sum /root/raid/bitmap md5sum: /root/raid/bitmap: Input/output error
Let's access "/sys/block/md0/md/dev-sdd1/super":
# cat /sys/block/md0/md/dev-sdd1/super
# dmesg -c ------------[ cut here ]------------ Kernel BUG at 78178626 [verbose debug info unavailable] invalid opcode: 0000 [#8] Modules linked in: iscsi_scst crc32c libcrc32c scst_vdisk scst iptable_filter ip_tables x_tables as_iosched deadline_iosched nfs nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs dm_snapshot dm_mirror bonding e1000 sata_mv
Pid: 4901, comm: cat Tainted: G D (2.6.24.2-1 #1)
EIP: 0060:[<78178626>] EFLAGS: 00010212 CPU: 0
EIP is at sysfs_read_file+0x88/0xd4
EAX: 00000001 EBX: 91611780 ECX: 00000000 EDX: 976fecc0
ESI: 00001000 EDI: 976fed20 EBP: 9745b344 ESP: 8353df54
DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
Process cat (pid: 4901, ti=8353c000 task=7d4caa40 task.ti=8353c000)
Stack: 93511080 0804b4d8 00001000 0804e000 91611794 783515f0 847e3380 0804e000
7817859e 00001000 7814ad7c 8353dfa0 00001000 847e3380 fffffff7
0804e000
8353c000 7814b0aa 8353dfa0 00000000 00000000 00000000 00000003
0804e000
Call Trace:
[<7817859e>] sysfs_read_file+0x0/0xd4
[<7814ad7c>] vfs_read+0x88/0x10a
[<7814b0aa>] sys_read+0x41/0x67
[<78103bca>] syscall_call+0x7/0xb
=======================
Code: c0 74 61 8b 47 18 8b 4b 0c 8b 40 04 89 43 24 89 e8 8b 74 24 14 8b 57 14 ff 16 89 c6 89 f8 e8 18 0b 00 00 81 fe ff 0f 00 00 7e 04 <0f> 0b eb fe 85 f6 78 31 c7 43 20 00 00 00 00 89 33 eb 07 be f4
EIP: [<78178626>] sysfs_read_file+0x88/0xd4 SS:ESP 0068:8353df54
---[ end trace 7e6d8636f39e737a ]---Signature · 2 lines
Tomasz Chmielewski http://wpkg.org
Tomasz Chmielewski schrieb:
(...)
Quoted 7 lines
Let's access "
/sys/block/md0/md/dev-sdd1/super":# cat /sys/block/md0/md/dev-sdd1/super# dmesg -c ------------[ cut here ]------------ Kernel BUG at 78178626 [verbose debug info unavailable]
It turns out a broken RAID bitmap file has nothing to do with it - the same happens on a different machine without a bitmap file:
------------[ cut here ]------------ Kernel BUG at 7817736a [verbose debug info unavailable] invalid opcode: 0000 [#1] Modules linked in: as_iosched nfs lockd nfs_acl sunrpc bonding dm_mirror dm_snapshot e1000 sata_mv
Pid: 2494, comm: cat Not tainted (2.6.24.3-1 #1)
EIP: 0060:[<7817736a>] EFLAGS: 00010212 CPU: 0
EIP is at sysfs_read_file+0x88/0xd4
EAX: 00000001 EBX: 961b5880 ECX: 00000000 EDX: 964ef360
ESI: 00001000 EDI: 964ef3c0 EBP: 9705bd04 ESP: 971f1f54
DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
Process cat (pid: 2494, ti=971f0000 task=970ad9a0 task.ti=971f0000)
Stack: 96443080 0804b4d8 00001000 0804e000 961b5894 7835f6f0 96193400 0804e000
781772e2 00001000 78149bd5 971f1fa0 00001000 96193400 fffffff7
0804e000
971f0000 78149f03 971f1fa0 00000000 00000000 00000000 00000003
0804e000
Call Trace:
[<781772e2>] sysfs_read_file+0x0/0xd4
[<78149bd5>] vfs_read+0x88/0x10a
[<78149f03>] sys_read+0x41/0x67
[<78103bba>] syscall_call+0x7/0xb
=======================
Code: c0 74 61 8b 47 18 8b 4b 0c 8b 40 04 89 43 24 89 e8 8b 74 24 14 8b 57 14 ff 16 89 c6 89 f8 e8 18 0b 00 00 81 fe ff 0f 00 00 7e 04 <0f> 0b eb fe 85 f6 78 31 c7 43 20 00 00 00 00 89 33 eb 07 be f4
EIP: [<7817736a>] sysfs_read_file+0x88/0xd4 SS:ESP 0068:971f1f54
---[ end trace e39b88900c26a529 ]---Certainly, it doesn't happen with 2.6.18 kernels.
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org
Quoted 12 lines · On Wed, 12 Mar 2008 10:51:38 +0100 Tomasz Chmielewski <[email protected]> wrote:
Tomasz Chmielewski schrieb:
(...)
Quoted 7 lines
Let's access "
/sys/block/md0/md/dev-sdd1/super":# cat /sys/block/md0/md/dev-sdd1/super# dmesg -c ------------[ cut here ]------------ Kernel BUG at 78178626 [verbose debug info unavailable]
Quoted 5 lines
It turns out a broken RAID bitmap file has nothing to do with it - the same happens on a different machine without a bitmap file:
------------[ cut here ]------------ Kernel BUG at 7817736a [verbose debug info unavailable]
argh. Please do enable CONFIG_DEBUG_BUGVERBOSE.
Quoted 24 lines
invalid opcode: 0000 [#1]
Modules linked in: as_iosched nfs lockd nfs_acl sunrpc bonding dm_mirror
dm_snapshote1000sata_mvPid: 2494, comm: cat Not tainted (2.6.24.3-1 #1)
EIP: 0060:[<7817736a>] EFLAGS: 00010212 CPU: 0
EIP is atsysfs_read_file+0x88/0xd4
EAX: 00000001 EBX:961b5880ECX: 00000000 EDX:964ef360
ESI: 00001000 EDI:964ef3c0EBP:9705bd04ESP:971f1f54
DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
Process cat (pid: 2494, ti=971f0000 task=970ad9a0 task.ti=971f0000)Stack: 96443080 0804b4d8 00001000 0804e000 961b5894 7835f6f0 96193400 0804e000 781772e2 00001000 78149bd5 971f1fa0 00001000 96193400 fffffff7 0804e000 971f0000 78149f03 971f1fa0 00000000 00000000 00000000 00000003 0804e000 Call Trace: [<781772e2>] sysfs_read_file+0x0/0xd4 [<78149bd5>] vfs_read+0x88/0x10a [<78149f03>] sys_read+0x41/0x67 [<78103bba>] syscall_call+0x7/0xb ======================= Code: c0 74 61 8b 47 18 8b 4b 0c 8b 40 04 89 43 24 89 e8 8b 74 24 14 8b
Quoted 3 lines
57 14 ff 16 89 c6 89 f8 e8 18 0b 00 00 81 fe ff 0f 00 00 7e 04 <0f> 0b eb fe 85 f6 78 31 c7 43 20 00 00 00 00 89 33 eb 07 be f4
EIP: [<7817736a>]sysfs_read_file+0x88/0xd4SS:ESP 0068:971f1f54
I assume this is the BUG_ON(count >= (ssize_t)PAGE_SIZE) in fill_read_buffer().
This was reported recently and we prepared a debug patch but the reporter was unable to trigger the bug again.
Please add the below and retest?
From: Andrew Morton <[email protected]>
Try to find the culprit who caused
http://bugzilla.kernel.org/show_bug.cgi?id=10150
---
drivers/base/core.c | 5 +++++ fs/sysfs/file.c | 8 +++++++- 2 files changed, 12 insertions(+), 1 deletion(-)
diff -puN fs/Kconfig~driver-core-debug-for-bad-dev_attr_show-return-value fs/Kconfig
fs/sysfs/file.c renamed from fs/sysfs/file.c~driver-core-debug-for-bad-dev_attr_show-return-value +7 -1
@@ -12,6 +12,7 @@#include <linux/module.h> #include <linux/kobject.h> +#include <linux/kallsyms.h> #include <linux/namei.h> #include <linux/poll.h> #include <linux/list.h>
@@ -94,7 +95,12 @@ static int fill_read_buffer(struct dentr* The code works fine with PAGE_SIZE return but it's likely to * indicate truncated result or overflow in normal use cases. */ -BUG_ON (count >= (ssize_t )PAGE_SIZE ); +if (count >= (ssize_t )PAGE_SIZE ) { +print_symbol ("fill_read_buffer: %s returned bad count\n" , + (unsigned long )ops->show); +/* Try to struggle along */ + count =PAGE_SIZE -1 ; + }if (count >=0 ) { buffer->needs_read_fill =0 ; buffer->count = count;
drivers/base/core.c renamed from drivers/base/core.c~driver-core-debug-for-bad-dev_attr_show-return-value +5 -0
@@ -19,6 +19,7 @@#include <linux/kdev_t.h> #include <linux/notifier.h> #include <linux/genhd.h> +#include <linux/kallsyms.h> #include <asm/semaphore.h> #include "base.h"
@@ -68,6 +69,10 @@ static ssize_t dev_attr_show(struct kobjif (dev_attr->show) ret = dev_attr->show (dev, dev_attr, buf); +if (ret >= (ssize_t )PAGE_SIZE ) { +print_symbol ("dev_attr_show: %s returned bad count\n" , + (unsigned long )dev_attr->show); + }return ret; }
_
Quoted 7 lines · On Wed, 12 Mar 2008 15:36:18 -0700 Andrew Morton <[email protected]>…
I assume this is the
BUG_ON(count >= (ssize_t)PAGE_SIZE)infill_read_buffer().This was reported recently and we prepared a debug patch but the reporter was unable to trigger the bug again.
Please add the below and retest?
err, don't bother.
rdev->sb_size = MD_SB_BYTES;
...
#define MD_SB_BYTES 4096
...
and
rdev->sb_size = le32_to_cpu(sb->max_dev) * 2 + 256; bmask = queue_hardsect_size(rdev->bdev->bd_disk->queue)-1; if (rdev->sb_size & bmask) rdev->sb_size = (rdev->sb_size | bmask) + 1;
I'm assuming that either the above sequence of code never ran at all, or it produced sb_size >= PAGE_SIZE.
then this:
static ssize_tsuper_show(mdk_rdev_t *rdev, char *page)
{
if (rdev->sb_loaded && rdev->sb_size) {
memcpy(page, page_address(rdev->sb_page), rdev->sb_size);
return rdev->sb_size;
} else
return 0;
}returns something >= PAGE_SIZE then this:
static int fill_read_buffer(struct dentry * dentry, struct sysfs_buffer * buffer)
{
struct sysfs_dirent *attr_sd = dentry->d_fsdata;
struct kobject *kobj = attr_sd->s_parent->s_dir.kobj;
struct sysfs_ops * ops = buffer->ops;
int ret = 0;
ssize_t count;
if (!buffer->page)
buffer->page = (char *) get_zeroed_page(GFP_KERNEL);
if (!buffer->page)
return -ENOMEM;
/* need attr_sd for attr and ops, its parent for kobj */
if (!sysfs_get_active_two(attr_sd))
return -ENODEV;
buffer->event = atomic_read(&attr_sd->s_attr.open->event);
count = ops->show(kobj, attr_sd->s_attr.attr, buffer->page);
sysfs_put_active_two(attr_sd);
/*
* The code works fine with PAGE_SIZE return but it's likely to
* indicate truncated result or overflow in normal use cases.
*/
BUG_ON(count >= (ssize_t)PAGE_SIZE);declares game over.
Neil, pls fix?
Quoted 5 lines · On Wednesday March 12, [email protected] wrote:
Quoted 2 lines · On Wed, 12 Mar 2008 15:36:18 -0700 Andrew Morton <[email protected]>…
I assume this is the
BUG_ON(count >= (ssize_t)PAGE_SIZE)infill_read_buffer().
That's a new "BUG_ON".
Before commit 8118a859dc7abd873193986c77a8d9bdb877adc8 it was
BUG_ON(count > (ssize_t)PAGE_SIZE);
But I've been looking for an excuse to remove the "super" file from sysfs for a while.
It doesn't really fit the sysfs model, and now clearly breaks it. And it isn't really all that useful (you can just read from the raw device if you want to see the superblock.
It was a "seems like a good idea at the time" thing which I now regret.
So: I'll just remove it. (with apologies to anyone who actually uses it, but I suspect there are only two of you :-)
Patch to follow
NeilBrown