Kernel Linux 2.6.23.16 hangs when run updatedb
Subsystems (9)
cc'd: Thomas Gleixner
cc'd: Ingo Molnar
About hrtimer comment by Alan Jenkins
Quoted 45 lines · Alan Jenkins
Quoted 21 lines · Renato S. Yamane
Quoted 20 lines · [email protected] wrote:
Quoted 6 lines · Renato S. Yamane
when cron start updatedb daemon, my machine hangs. None key work, so I need reset (press power button to restart).
The only way to get log is runing updatedb from terminal (CTRL + ALT + F1) and take a picture from screen:
Quoted 8 lines
Up screen:
<http://img137.imageshack.us/img137/7128/01pg1.jpg>Down screen:
<http://img100.imageshack.us/img100/1456/02kx1.jpg>Call trace detail:
<http://img100.imageshack.us/img100/8233/calltracedetailsj7.jpg>looks like reiserfs issue.
maybe corrupt filesystem and improper handling of the reiserfs-driver?Quoted 7 lines
Quoted 1 line
could you try some "find /" ?
"find /" don't hangs my machine.
The problem is only with updatedb.How can I fix this? Is safe run reiserfsck?
I think he's wrong.
Looking at the call trace, the BUG happens during an interrupt. It could be a coincidence that the interrupt happened during this particular system call.
It looks like a timer callback has been corrupted / set to an invalid value. The BUG is due to accessing the invalid address
61060fe0withinenqueue_hrtimer, and the EIP (instruction pointer) is also equal to61060fe0. This would be consistent with the source code ofenqueue_hrtimer. It's not an obvious reiserfs issue.I don't know how to find out where this corruption is happening, but it's worth asking the hrtimers people.
Quoted 9 lines · Renato S. Yamane
Quoted 9 lines
Quoted 1 line
How can I fix this? Is safe run reiserfsck?
I think he's wrong.
Looking at the call trace, the BUG happens during an interrupt. It could be a coincidence that the interrupt happened during this particular system call.
Quoted 5 lines
Quoted 5 lines
It looks like a timer callback has been corrupted / set to an invalid value. The BUG is due to accessing the invalid address
61060fe0withinenqueue_hrtimer, and the EIP (instruction pointer) is also equal to61060fe0. This would be consistent with the source code ofenqueue_hrtimer. It's not an obvious reiserfs issue.
It's pretty inconsistent with the source code of enqueue_hrtimer().
The only possibility to have a callback from enqueue_hrtimer() is in hrtimer_enqueue_reprogram() in the HRTIMER_CB_IRQSAFE_NO_RESTART case. Such timers can not be requeued in interrupt context, hence the name HRTIMER_CB_IRQSAFE_NO_RESTART :)
Also in hrtimer_interrupt context, hrtimer_enqueue() is called with reprogram = 0, which ensures that we do not call hrtimer_enqueue_reprogram().
Quoted 2 lines
Quoted 2 lines
I don't know how to find out where this corruption is happening, but it's worth asking the hrtimers people.
Let's gather some more information.
Renato, some questions:
1) is this fully reproducible with updatedb ?
2) are you sure that this is the first stacktrace you captured, there might be some BUG before that which scrolled out of sight. Any chance to use a serial console ?
3) Can you please recompile the kernel with CONFIF_DEBUG_INFO set and then run the following addresses from the backtrace through addr2line with the new vmlinux:
# addr2line -e vmlinux 0xc013dad9 0xc0107c3b
Please provide the output.
4) Looking at your .config it seems you have some more patches applied aside of the .16 stable. Can you please upload a full patch queue somewhere ?
Thanks,
tglx
Hi Thomas,
Quoted 2 lines · Thomas Gleixner
Renato, some questions:
1) is this fully reproducible with updatedb ?
Yes, this crash is fully reproducible.
All time wich I turn-on my laptopt I need kill find daemon in my Debian Etch. If I don't do that, my laptop hangs.
Quoted 3 lines
2) are you sure that this is the first stacktrace you captured, there might be some BUG before that which scrolled out of sight. Any chance to use a serial console?
I can't use scroll, because none key/mouse work after crash. And, sorry I can't use a serial console. Any other idea?
Quoted 4 lines
3) Can you please recompile the kernel with
CONFIF_DEBUG_INFOset and then run the following addresses from the backtrace through addr2line with the new vmlinux:# addr2line -e vmlinux 0xc013dad9 0xc0107c3b
Yes, I try compile 2.6.24.3 without any patch.
Quoted 3 lines
4) Looking at your .config it seems you have some more patches applied aside of the .16 stable. Can you please upload a full patch queue somewhere ?
I attached all patchs used in my 2.6.23.16.
Regards,
Renato S. Yamane
================
arch/i386/Kconfig.debug +1 -0
@@ -59,6 +59,7 @@ config 4KSTACKS bool "Use 4Kb for kernel stacks instead of 8Kb" depends on DEBUG_KERNEL + default y help If you say Y here the kernel will use a 4Kb stacksize for the kernel stack attached to each process/thread. This facilitates
Genetic I/O CFQ scheduler support.
block/cfq-iosched.c +398 -11
473 lines not on this page · raw message
block/Kconfig.iosched +9 -0
@@ -77,6 +77,15 @@ anticipatory scheduler autonomically and will adapt tunables depending on the present workload. +config GENETIC_IOSCHED_CFQ + bool "Genetic CFQ I/O scheduler (EXPERIMENTAL)" + depends on IOSCHED_CFQ && GENETIC_LIB && EXPERIMENTAL + default n + ---help--- + This will use a genetic algorithm to tweak the tunables of the + CFQ scheduler autonomically and will adapt tunables + depending on the present workload. + endmenu endif
block/genhd.c +18 -6
@@ -30,6 +30,8 @@ static struct blk_major_name {char name[16 ]; } *major_names[BLKDEV_MAJOR_HASH_SIZE ]; +LIST_HEAD (gendisks); +/* index in the above - for now: assume no multimajor ranges */ static inline int major_to_index (int major) {
@@ -395,19 +397,22 @@ static ssize_t disk_stats_read(struct gejiffies_to_msecs (disk_stat_read (disk, io_ticks)), jiffies_to_msecs(disk_stat_read(disk, time_in_queue))); } + +#ifdef CONFIG_FINGERPRINTING static ssize_t disk_fp_read (struct gendisk * disk,char *page) { -return sprintf (page,"reads: %llx\n" -"writes: %llx\n" -"head_pos: %llx\n" -"avg_dist: %llx\n" -"avg_size: %llx\n" , +return sprintf (page,"reads: %lld\n" +"writes: %lld\n" +"head_pos: %lld\n" +"avg_dist: %lld\n" +"avg_size: %lld\n" , (unsigned long long )disk->fp_ss->reads, (unsigned long long )disk->fp_ss->writes, (unsigned long long )disk->fp_ss->head_pos, (unsigned long long )disk->fp_ss->avg_dist, (unsigned long long )disk->fp_ss->avg_size); } +#endif static struct disk_attribute disk_attr_uevent = { .attr = {.name ="uevent" , .mode =S_IWUSR },
@@ -433,10 +438,13 @@ static struct disk_attribute disk_attr_s .attr = {.name = "stat", .mode = S_IRUGO }, .show = disk_stats_read }; + +#ifdef CONFIG_FINGERPRINTING static struct disk_attribute disk_attr_fp = { .attr = {.name ="fp" , .mode =S_IRUGO }, .show = disk_fp_read }; +#endif #ifdef CONFIG_FAIL_MAKE_REQUEST
@@ -476,7 +484,9 @@ static struct attribute * default_attrs[#ifdef CONFIG_FAIL_MAKE_REQUEST &disk_attr_fail.attr,#endif - &disk_attr_fp.attr, +#ifdef CONFIG_FINGERPRINTING + &disk_attr_fp.attr, +#endif NULL, };
@@ -485,6 +495,7 @@ static void disk_release(struct kobjectstruct gendisk *disk =to_disk (kobj);kfree (disk->random);kfree (disk->part); +list_del (&disk->gendisks);free_disk_stats (disk);kfree (disk); }
@@ -685,8 +696,9 @@ struct gendisk *alloc_disk_node(int minokobj_set_kset_s (disk,block_subsys);kobject_init (&disk->kobj);rand_initialize_disk (disk); +list_add_tail (&disk->gendisks, &gendisks);INIT_WORK (&disk->async_notify, media_change_notify_thread); } disk->fp_ss =kmalloc (sizeof (struct fp_snapshot ),GFP_KERNEL );
block/ll_rw_blk.c +138 -0
150 lines not on this page · raw message
include/linux/genhd.h +1 -0
@@ -134,6 +134,7 @@ struct gendisk {atomic_t sync_io;/* RAID */ unsigned long stamp;int in_flight; +struct list_head gendisks;#ifdef CONFIG_SMP struct disk_stats *dkstats;#else
include/linux/blkdev.h +14 -0
@@ -861,6 +861,20 @@ void kblockd_flush_work(struct work_stru MODULE_ALIAS("block-major-" __stringify(major) "-*") +#if defined(CONFIG_GENETIC_IOSCHED_AS) || \ +defined (CONFIG_GENETIC_IOSCHED_DEADLINE ) || \ + defined(CONFIG_GENETIC_IOSCHED_CFQ) + +struct disk_stats_snapshot +{ +unsigned long reads; +unsigned long writes; +unsigned long read_sectors; +unsigned long write_sectors; +unsigned long time_in_queue; +}; +#endif /* CONFIG_GENETIC_IOSCHED_AS || CONFIG_GENETIC_IOSCHED_CFQ */ +#else /* CONFIG_BLOCK */ /* * stubs for when the block layer is configured out
include/linux/genetic.h new file +285 -0
285 lines not on this page · raw message
lib/Kconfig +6 -0
@@ -63,6 +63,12 @@ config AUDIT_GENERIC depends on AUDIT && !AUDIT_ARCH default y +config GENETIC_LIB + bool "Genetic Library" + help + This option will build in a genetic library that will tweak + kernel parameters autonomically to improve performance. + # # compression support is select'ed if needed #
lib/Makefile +1 -0
@@ -47,6 +47,7 @@ obj-$(CONFIG_CRC32) += crc32.o obj-$(CONFIG_CRC7) += crc7.o obj-$(CONFIG_LIBCRC32C) += libcrc32c.o obj-$(CONFIG_GENERIC_ALLOCATOR) += genalloc.o +obj-$(CONFIG_GENETIC_LIB) += genetic.o obj-$(CONFIG_ZLIB_INFLATE) += zlib_inflate/ obj-$(CONFIG_ZLIB_DEFLATE) += zlib_deflate/
lib/genetic.c new file +892 -0
892 lines not on this page · raw message
lib/genetic-debug.c new file +153 -0
153 lines not on this page · raw message
lib/fingerprinting.c new file +291 -0
291 lines not on this page · raw message
include/linux/fingerprinting.h new file +125 -0
125 lines not on this page · raw message
block/fingerprinting.c new file +205 -0
205 lines not on this page · raw message
block/ll_rw_blk.c +4 -0
@@ -30,6 +30,7 @@#include <linux/cpu.h> #include <linux/blktrace_api.h> #include <linux/fault-inject.h> +#include <linux/fingerprinting.h> /* * for max sense size
@@ -2893,6 +2894,9 @@ static void init_request_from_bio(struct { req->cmd_type =REQ_TYPE_FS ; +#ifdef CONFIG_FINGERPRINTING +update_fp_snapshot (bio); +#endif /* * inherit FAILFAST from bio (for read-ahead, and explicit FAILFAST) */
include/linux/genhd.h +2 -0
@@ -67,6 +67,7 @@ struct partition {#include <linux/string.h> #include <linux/fs.h> #include <linux/workqueue.h> +#include <linux/fingerprinting.h> struct partition {unsigned char boot_ind;/* 0x80 - active */
@@ -91,6 +92,7 @@ struct gendisk {struct disk_stats dkstats;#endif struct work_struct async_notify; +struct fp_snapshot * fp_ss; };/* Structure for sysfs attributes on block devices */
block/genhd.c +23 -0
@@ -445,6 +445,20 @@ static ssize_t disk_stats_read(struct gejiffies_to_msecs (disk_stat_read (disk, io_ticks)), jiffies_to_msecs(disk_stat_read(disk, time_in_queue))); } +static ssize_t disk_fp_read (struct gendisk * disk,char *page) +{ +return sprintf (page,"reads: %llx\n" +"writes: %llx\n" +"head_pos: %llx\n" +"avg_dist: %llx\n" +"avg_size: %llx\n" , + (unsigned long long )disk->fp_ss->reads, + (unsigned long long )disk->fp_ss->writes, + (unsigned long long )disk->fp_ss->head_pos, + (unsigned long long )disk->fp_ss->avg_dist, + (unsigned long long )disk->fp_ss->avg_size); +} +static struct disk_attribute disk_attr_uevent = { .attr = {.name ="uevent" , .mode =S_IWUSR }, .store = disk_uevent_store
@@ -473,6 +487,10 @@ static struct disk_attribute disk_attr_s .attr = {.name = "stat", .mode = S_IRUGO }, .show = disk_stats_read }; +static struct disk_attribute disk_attr_fp = { + .attr = {.name ="fp" , .mode =S_IRUGO }, + .show = disk_fp_read +};#ifdef CONFIG_FAIL_MAKE_REQUEST
@@ -513,6 +531,7 @@ static struct attribute * default_attrs[#ifdef CONFIG_FAIL_MAKE_REQUEST &disk_attr_fail.attr, #endif + &disk_attr_fp.attr, NULL, };
@@ -745,6 +764,10 @@ struct gendisk *alloc_disk_node(int minoINIT_WORK (&disk->async_notify, media_change_notify_thread); } + + disk->fp_ss =kmalloc (sizeof (struct fp_snapshot ),GFP_KERNEL ); +memset (disk->fp_ss,0 ,sizeof (struct fp_snapshot )); +return disk; }
block/Kconfig +6 -0
@@ -65,3 +65,9 @@ config BLK_DEV_BSG endif # BLOCK source block/Kconfig.iosched + +config FINGERPRINTING + bool "I/O Workload Fingerprinting" + help + Say Y here if you want workload data to be classified and + used to tune the I/O schedulers. Otherwise say N.
block/Makefile +3 -0
@@ -11,3 +11,6 @@ obj-$(CONFIG_IOSCHED_DEADLINE) += deadli obj-$(CONFIG_IOSCHED_CFQ) += cfq-iosched.o obj-$(CONFIG_BLK_DEV_IO_TRACE) += blktrace.o + + +obj-$(CONFIG_FINGERPRINTING) += fingerprinting.o
Subject: [patch] [patch] implement smarter atime updates support
From: Ingo Molnar <[email protected]>
change relatime updates to be performed once per day. This makes relatime a compatible solution for HSM, mailer-notification and tmpwatch applications too.
also add the CONFIG_DEFAULT_RELATIME kernel option, which makes "norelatime" the default for all mounts without an extra kernel boot option.
add the "default_relatime=0" boot option to turn this off.
also add the /proc/sys/fs/default_relatime flag which can be changed runtime to modify the behavior of subsequent new mounts.
tested by moving the date forward:
# date Sun Aug 5 22:55:14 CEST 2007 # date -s "Tue Aug 7 22:55:14 CEST 2007" Tue Aug 7 22:55:14 CEST 2007
access to a file did not generate disk IO before the date was set, and it generated exactly one IO after the date was set.
--- Documentation/kernel-parameters.txt | 8 +++++ fs/Kconfig | 22 ++++++++++++++ fs/inode.c | 53 +++++++++++++++++++++++++++--------- fs/namespace.c | 24 ++++++++++++++++ include/linux/mount.h | 3 ++ kernel/sysctl.c | 17 +++++++++++ 6 files changed, 114 insertions(+), 13 deletions(-)
Documentation/kernel-parameters.txt +8 -0
@@ -525,6 +525,10 @@ and is between 256 and 4096 characters. This is a 16-member array composed of values ranging from 0-255. + default_relatime= + [FS] mount all filesystems with relative atime + updates by default. + default_utf8= [VT] Format=<0|1> Set system-wide default UTF-8 mode for all tty's.
@@ -1468,6 +1472,10 @@ and is between 256 and 4096 characters. Format: <reboot_mode>[,<reboot_mode2>[,...]] See arch/*/kernel/reboot.c or arch/*/kernel/process.c + relatime_interval= + [FS] relative atime update frequency, in seconds. + (default: 1 day: 86400 seconds) + reserve= [KNL,BUGS] Force the kernel to ignore some iomem area reservetop= [X86-32]
fs/Kconfig +22 -0
@@ -2060,6 +2060,28 @@ config 9P_FS endmenu +config DEFAULT_RELATIME + bool "Mount all filesystems with relatime by default" + default y + help + If you say Y here, all your filesystems will be mounted + with the "relatime" mount option. This eliminates many atime + ('file last accessed' timestamp) updates (which otherwise + is performed on every file access and generates a write + IO to the inode) and thus speeds up IO. Atime is still updated, + but only once per day. + + The mtime ('file last modified') and ctime ('file created') + timestamp are unaffected by this change. + + Use the "norelatime" kernel boot option to turn off this + feature. + +config DEFAULT_RELATIME_VAL + int + default "1" if DEFAULT_RELATIME + default "0" + if BLOCK menu "Partition Types"
fs/inode.c +40 -13
68 lines not on this page · raw message
fs/namespace.c +24 -0
@@ -1107,6 +1107,7 @@ int do_add_mount(struct vfsmount *newmnt goto unlock; newmnt->mnt_flags = mnt_flags; +if ((err =graft_tree (newmnt, nd))) goto unlock;
@@ -1362,6 +1363,24 @@ int copy_mount_options(const void __user }/* +* Allow users to disable (or enable) atime updates via a .config +* option or via the boot line, or via /proc/sys/fs/default_relatime: +*/ +int default_relatime__read_mostly =CONFIG_DEFAULT_RELATIME_VAL ; + +static int __init set_default_relatime (char *str) +{ +get_option (&str, &default_relatime); + +printk (KERN_INFO "Mount all filesystems with" +"default relative atime updates: %s.\n" , + default_relatime ?"enabled" :"disabled" ); + +return 1 ; +} +__setup ("default_relatime=" , set_default_relatime); + +/* * Flags is a 32-bit value that allows up to 31 non-fs dependent flags to * be given to the mount() call (ie: read-only, no-dev, no-suid etc). *
@@ -1409,6 +1428,11 @@ long do_mount(char *dev_name, char *dir_ mnt_flags |=MNT_NODIRATIME ;if (flags &MS_RELATIME ) mnt_flags |=MNT_RELATIME ; +else if (default_relatime && + !(flags & (MNT_NOATIME |MNT_NODIRATIME ))) { + mnt_flags |=MNT_RELATIME ; + flags |=MS_RELATIME ; + } flags &= ~(MS_NOSUID |MS_NOEXEC |MS_NODEV |MS_ACTIVE |MS_NOATIME |MS_NODIRATIME |MS_RELATIME );
include/linux/mount.h +3 -0
@@ -103,5 +103,8 @@ extern void shrink_submounts(struct vfsmextern spinlock_t vfsmount_lock;extern dev_t name_to_dev_t (char *name); +extern int default_relatime; +extern int relatime_interval; +#endif #endif /* _LINUX_MOUNT_H */
kernel/sysctl.c +17 -0
@@ -30,6 +30,7 @@#include <linux/capability.h> #include <linux/smp_lock.h> #include <linux/fs.h> +#include <linux/mount.h> #include <linux/init.h> #include <linux/kernel.h> #include <linux/kobject.h>
@@ -1206,6 +1207,22 @@ static ctl_table fs_table[] = { .mode = 0644, .proc_handler = &proc_dointvec, }, + { + .ctl_name = CTL_UNNUMBERED, + .procname = "default_relatime", + .data = &default_relatime, + .maxlen = sizeof(int), + .mode = 0644, + .proc_handler = &proc_dointvec, + }, + { + .ctl_name = CTL_UNNUMBERED, + .procname = "relatime_interval", + .data = &relatime_interval, + .maxlen = sizeof(int), + .mode = 0644, + .proc_handler = &proc_dointvec,
Lines of this file are not shown. View the raw message.
The rest of this diff is not shown. View the raw message.
--- arch/i386/kernel/ioport.c | 17 ++++++++++++++--- arch/x86_64/kernel/ioport.c | 12 ++++++++++-- drivers/block/loop.c | 5 ++++- include/linux/sched.h | 7 +++++++ kernel/Kconfig.preempt | 17 +++++++++++++++++ kernel/sched.c | 40 ++++++++++++++++++++++++++++++++++++++++ kernel/workqueue.c | 2 +- mm/oom_kill.c | 4 +++- 8 files changed, 96 insertions(+), 8 deletions(-)
The rest of this diff is not shown. View the raw message.
set_user_nice(current, -20);
+
+ /*
+ * The loop thread is important enough to be given a boost:
+ */
+ sched_privileged_task(current);
while (!kthread_should_stop() || lo->lo_bio) {
kernel/sched.c +47 -0
47 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
init/Kconfig.cfs +130 -0
130 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
+ default CUSTOM_SCHED_BATCH_WAKEUP_GRANULARITY if CUSTOM_SCHED_SETUP
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
Quoted 6 lines · Renato S. Yamane
Quoted 2 lines
Renato, some questions:
1) is this fully reproducible with updatedb ?Yes, this crash is fully reproducible.
All time wich I turn-on my laptopt I need kill find daemon in my Debian Etch. If I don't do that, my laptop hangs.
Good. Not for you, but for isolating the bug. :)
Quoted 6 lines
Quoted 3 lines
2) are you sure that this is the first stacktrace you captured, there might be some BUG before that which scrolled out of sight. Any chance to use a serial console?
I can't use scroll, because none key/mouse work after crash. And, sorry I can't use a serial console. Any other idea?
netconsole might work. Documentation/networking/netconsole.txt
Quoted 6 lines
Quoted 4 lines
3) Can you please recompile the kernel with
CONFIF_DEBUG_INFOset and then run the following addresses from the backtrace through addr2line with the new vmlinux:# addr2line -e vmlinux 0xc013dad9 0xc0107c3bYes, I try compile 2.6.24.3 without any patch.
Ok. If the problem persists, you might try 2.6.25-rc5 as well. I have timer debugging infrastructure for the latter, which I want to avoid to backport to .23/24 :)
Does it happen with straight 2.6.23.16 as well ?
Quoted 5 lines
Quoted 3 lines
4) Looking at your .config it seems you have some more patches applied aside of the .16 stable. Can you please upload a full patch queue somewhere ?
I attached all patchs used in my 2.6.23.16.
Hmm, I can't see an obvious candidate.
Thanks,
tglx