Skip to contentSkip to conversation

XFS crash on linux raid

DISCUSSION Emmanuel Florac 7 participants 34 messages May 3, 2007 – Nov 21, 2007 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

Hello,
Apparently quite a lot of people do encounter the same problem from time to time, but I couldn't find any solution.

When writing quite a lot to the filesystem (heavy load on the fileserver), the filesystem crashes when filled at 2.5~3TB (varies from time to time). The filesystems tested where always running on a software raid 0, with disabled barriers. I tend to think that disabled write barriers are causing the crash but I'll do some more tests to get sure.

I've met this problem for the first time on 12/23 (yup... merry christmas :) when a 13 TB filesystem went belly up :

Dec 23 01:38:10 storiq1 -- MARK --
Dec 23 01:58:10 storiq1 -- MARK --
Dec 23 02:10:29 storiq1 kernel: xfs_iunlink_remove: xfs_itobp()
returned an error 990 on md0. Returning error.
Dec 23 02:10:29 storiq1 kernel: xfs_inactive:^Ixfs_ifree() returned an
error = 990 on md0
Dec 23 02:10:29 storiq1 kernel: xfs_force_shutdown(md0,0x1) called from
line 1763 of file fs/xfs/xfs_vnodeops.c. Return address = 0xc027f78b
Dec 23 02:38:11 storiq1 -- MARK --
Dec 23 02:58:11 storiq1 -- MARK -- 

When mounting, it did that :

Filesystem "md0": Disabling barriers, not supported by the underlying device XFS mounting filesystem md0
Starting XFS recovery on filesystem: md0 (logdev: internal) Filesystem "md0": xfs_inode_recover: Bad inode magic number, dino ptr = 0xf7196600, dino bp = 0xf718e980, ino = 119318 Filesystem "md0": XFS internal error xlog_recover_do_inode_trans(1) at line 2352 of file fs/xfs/xfs_log_recover.c. Caller 0xc025d180 <c025cb9d>

xlog_recover_do_inode_trans+0x93d/0xa00 <c025d180>
xlog_recover_do_trans+0x140/0x160 <c0277afb>
xfs_buf_delwri_queue+0x2b/0xb0 <c025d180>
xlog_recover_do_trans+0x140/0x160 <c027385f> kmem_zalloc+0x1f/0x50
<c025d27f> xlog_recover_commit_trans+0x3f/0x50 <c025d39a>
xlog_recover_process_data+0xea/0x240 <c025e48a>
xlog_do_recovery_pass+0x39a/0xb70 <c013d189>
hrtimer_run_queues+0x29/0x110 <c025ecf6>
xlog_do_log_recovery+0x96/0xd0 <c025ed6b> xlog_do_recover+0x3b/0x170
<c025ef7d> xlog_recover+0xdd/0xf0 <c0255da1> xfs_log_mount+0xa1/0x110
<c02607e5> xfs_mountfs+0x825/0xf30 <c02420c7> xfs_fs_cmn_err+0x27/0x30
<c0251137> xfs_ioinit+0x27/0x50 <c02693ef> xfs_mount+0x2ff/0x520
<c027f3f3> vfs_mount+0x43/0x50 <c027f20a> xfs_fs_fill_super+0x9a/0x200
<c013e42d> debug_mutex_add_waiter+0x3d/0xd0 <c029eb07>
snprintf+0x27/0x30 <c01a9204> disk_name+0xb4/0xc0 <c01727bf>
sb_set_blocksize+0x1f/0x50 <c01721a6> get_sb_bdev+0x106/0x150
<c027f3a0> xfs_fs_get_sb+0x30/0x40 <c027f170>
xfs_fs_fill_super+0x0/0x200 <c01723ff> do_kern_mount+0x5f/0xe0
<c0189e57> do_new_mount+0x77/0xc0 <c018a4ad> do_mount+0x18d/0x1f0
<c014007b> take_cpu_down+0xb/0x20 <c018a2c3>
copy_mount_options+0x63/0xc0 <c018a85f> sys_mount+0x9f/0xe0 <c01031d7>
syscall_call+0x7/0xb XFS: log mount/recovery failed: error 990 XFS: log

mount failed

XFS_repair (too old a version...) hosed the filesystem and destroyed most of the 2.6TB of data. Yes, there were no backup, I wrote a recovery tool to restore the video data from the raw device but the is a different story.

The system was running vanilla 2.6.17.9, and md0 was made of 3 striped RAID-5 on 3 3Ware-9550 cards, each hardware RAID-5 made of 8 750 GB drives.

On a similar hardware with 2 3Ware-9550 16x750GB striped together, but running 2.6.17.13, I had a similar fs crash last week. Unfortunately I don't have the logs at hand, but we where able to reproduce several times the crash at home :

Filesystem "md0": XFS internal error xfs_btree_check_sblock at line 336 of file fs/xfs/xfs_btree.c. Caller 0xc01fb282 <c0214568>

xfs_btree_check_sblock+0x58/0xe0  <c01fb282>
xfs_alloc_lookup+0x142/0x400 <c01fb282> xfs_alloc_lookup+0x142/0x400
<c0254e59> kmem_zone_alloc+0x59/0xd0 <c02149e3>
xfs_btree_init_cursor+0x23/0x190  <c01f7834>
xfs_alloc_ag_vextent_near+0x54/0x9e0 <c0204893>
xfs_bmap_add_extent+0x383/0x430  <c020a9e6>
xfs_bmap_search_multi_extents+0x76/0xf0 <c01f7619>
xfs_alloc_ag_vextent+0x119/0x120  <c01f9b1b>
xfs_alloc_vextent+0x3db/0x4f0 <c0208e4e> xfs_bmap_btalloc+0x3ee/0x890
<c020cf96> xfs_bmapi+0x1216/0x1690 <c021b2e6>
xfs_dir2_grow_inode+0xf6/0x400  <c015f636>
cache_alloc_refill+0xb6/0x1e0 <c023263b> xfs_idata_realloc+0x3b/0x130
<c021cebc> xfs_dir2_sf_to_block+0xac/0x5d0 <c021ad69>
xfs_dir2_lookup+0x129/0x130  <c0223147> xfs_dir2_sf_addname+0x97/0x110
<c021ac34> xfs_dir2_createname+0x144/0x150  <c0249f9b>
xfs_trans_ijoin+0x2b/0x80 <c0245b74> xfs_rename+0x354/0x9f0  <c024ec6f>
xfs_access+0x3f/0x50 <c025c278> xfs_vn_rename+0x48/0xa0  <c017146c>
__link_path_walk+0xc7c/0xc90 <c024dbcf> xfs_getattr+0x23f/0x2f0
<c017e72b> mntput_no_expire+0x1b/0x80 <c015f636>
cache_alloc_refill+0xb6/0x1e0  <c0173946> vfs_rename_other+0x96/0xd0
<c0173bd8> vfs_rename+0x258/0x2d0  <c0173dc1> do_rename+0x171/0x1a0
<c015f52b> cache_grow+0x10b/0x160  <c015f636>
cache_alloc_refill+0xb6/0x1e0 <c017000b> do_getname+0x4b/0x80
<c0173e37> sys_renameat+0x47/0x80 <c0173e98> sys_rename+0x28/0x30
<c0103037> syscall_call+0x7/0xb Filesystem "md0": XFS internal error
xfs_trans_cancel at line 1150 of file fs/xfs/xfs_trans.c.  Caller
0xc0245ec7 <c0248aa0> xfs_trans_cancel+0xd0/0x100  <c0245ec7>
xfs_rename+0x6a7/0x9f0 <c0245ec7> xfs_rename+0x6a7/0x9f0  <c024ec6f>
xfs_access+0x3f/0x50 <c025c278> xfs_vn_rename+0x48/0xa0  <c017146c>
__link_path_walk+0xc7c/0xc90 <c024dbcf> xfs_getattr+0x23f/0x2f0
<c017e72b> mntput_no_expire+0x1b/0x80 <c015f636>
cache_alloc_refill+0xb6/0x1e0  <c0173946> vfs_rename_other+0x96/0xd0
<c0173bd8> vfs_rename+0x258/0x2d0  <c0173dc1> do_rename+0x171/0x1a0
<c015f52b> cache_grow+0x10b/0x160  <c015f636>
cache_alloc_refill+0xb6/0x1e0 <c017000b> do_getname+0x4b/0x80
<c0173e37> sys_renameat+0x47/0x80 <c0173e98> sys_rename+0x28/0x30
<c0103037> syscall_call+0x7/0xb xfs_force_shutdown(md0,0x8) called from

line 1151 of file fs/xfs/xfs_trans.c. Return address = 0xc025f7b9 Filesystem "md0": Corruption of in-memory data detected. Shutting down
filesystem: md0 Please umount the filesystem, and rectify the problem(s) xfs_force_shutdown(md0,0x1) called from line 338 of file fs/xfs/xfs_rw.c. Return address = 0xc025f7b9 xfs_force_shutdown(md0,0x1) called from line 338 of file fs/xfs/xfs_rw.c. Return address = 0xc025f7b9

After xfs_repair, the fs is fine. However, it crashes again when writing again a couple of GBs of data. It crashes again under 2.6.17.13, 2.6.17.13 SMP, 2.6.18.8, 2.6.16.36...

Out of curiosity, I've tried to use reiserfs (just to see how it compares regarding this). Reiserfs crashed before even writing 100MB!

So I tend to believe this is a "write barrier" problem and it looks really nasty!!!

To sort this out I've started a test on a single 3Ware raid, without software raid.

Any idea on how to circumvent the problem to make software RAID/LVM usable?

Signature · 3 lines
----------------------------------------
Emmanuel Florac     |   Intellique
----------------------------------------
#1 lore raw
Quoted 3 lines · Emmanuel Florac

After xfs_repair, the fs is fine. However, it crashes again when writing again a couple of GBs of data. It crashes again under 2.6.17.13, 2.6.17.13 SMP, 2.6.18.8, 2.6.16.36...

4K stacks?

Quoted 2 lines

So I tend to believe this is a "write barrier" problem and it looks really nasty!!!

You could try "mount -o nobarrier ...."

#2 lore raw
In reply to Emmanuel Florac
Quoted 14 lines · Emmanuel Florac

Hello,
Apparently quite a lot of people do encounter the same problem from time to time, but I couldn't find any solution.

When writing quite a lot to the filesystem (heavy load on the fileserver), the filesystem crashes when filled at 2.5~3TB (varies from time to time). The filesystems tested where always running on a software raid 0, with disabled barriers. I tend to think that disabled write barriers are causing the crash but I'll do some more tests to get sure.

I've met this problem for the first time on 12/23 (yup... merry christmas :) when a 13 TB filesystem went belly up :

Quoted 10 lines
Dec 23 01:38:10 storiq1 -- MARK --
Dec 23 01:58:10 storiq1 -- MARK --
Dec 23 02:10:29 storiq1 kernel: xfs_iunlink_remove: xfs_itobp()
returned an error 990 on md0. Returning error.
Dec 23 02:10:29 storiq1 kernel: xfs_inactive:^Ixfs_ifree() returned an
error = 990 on md0
Dec 23 02:10:29 storiq1 kernel: xfs_force_shutdown(md0,0x1) called from
line 1763 of file fs/xfs/xfs_vnodeops.c. Return address = 0xc027f78b
Dec 23 02:38:11 storiq1 -- MARK --
Dec 23 02:58:11 storiq1 -- MARK -- 

So, trying to remove an inode there was a corruption found on disk and it shut the filesystem down.

Where there any I/o errors reported before the shutdown?

Quoted 7 lines

When mounting, it did that :

Filesystem "md0": Disabling barriers, not supported by the underlying device XFS mounting filesystem md0
Starting XFS recovery on filesystem: md0 (logdev: internal) Filesystem "md0": xfs_inode_recover: Bad inode magic number, dino ptr = 0xf7196600, dino bp = 0xf718e980, ino = 119318 Filesystem "md0": XFS

Which was found again during log recovery.

Quoted 8 lines

The system was running vanilla 2.6.17.9, and md0 was made of 3 striped RAID-5 on 3 3Ware-9550 cards, each hardware RAID-5 made of 8 750 GB drives.

On a similar hardware with 2 3Ware-9550 16x750GB striped together, but running 2.6.17.13, I had a similar fs crash last week. Unfortunately I don't have the logs at hand, but we where able to reproduce several times the crash at home :

Hmm - 750GB drives are brand new. i wouldn't rule out media issues at this point...

Quoted 2 lines

Filesystem "md0": XFS internal error xfs_btree_check_sblock at line 336 of file fs/xfs/xfs_btree.c. Caller 0xc01fb282 <c0214568>

Memory corruption?

Quoted 8 lines

line 1151 of file fs/xfs/xfs_trans.c. Return address = 0xc025f7b9 Filesystem "md0": Corruption of in-memory data detected. Shutting down
filesystem: md0 Please umount the filesystem, and rectify the problem(s) xfs_force_shutdown(md0,0x1) called from line 338 of file fs/xfs/xfs_rw.c. Return address = 0xc025f7b9 xfs_force_shutdown(md0,0x1) called from line 338 of file fs/xfs/xfs_rw.c. Return address = 0xc025f7b9

Quoted 6 lines

After xfs_repair, the fs is fine. However, it crashes again when writing again a couple of GBs of data. It crashes again under 2.6.17.13, 2.6.17.13 SMP, 2.6.18.8, 2.6.16.36...

Out of curiosity, I've tried to use reiserfs (just to see how it compares regarding this). Reiserfs crashed before even writing 100MB!

That indicates there's something wrong other than the filesystem. I'd suggest making sure your raid arrays, memory, etc are all functioning correctly first.

What platform are you running on? Are you running ia32 with 4k stacks?

Cheers,

Dave.

Signature · 3 lines
Dave Chinner
Principal Engineer
SGI Australian Software Group
#3 lore raw

Le Fri, 4 May 2007 10:59:22 +1000 vous écriviez:

Quoted 2 lines

Where there any I/o errors reported before the shutdown?

Nope. To make it clear : the problem can be reproduce on several different systems, different motherboards, different drives, different RAID controllers... This isn't a hardware problem.

Quoted 7 lines
Quoted 4 lines

On a similar hardware with 2 3Ware-9550 16x750GB striped together, but running 2.6.17.13, I had a similar fs crash last week. Unfortunately I don't have the logs at hand, but we where able to reproduce several times the crash at home :

Hmm - 750GB drives are brand new. i wouldn't rule out media issues at this point...

The problem is quite easily reproduced with 500GB drives too.

Quoted 4 lines
Quoted 2 lines

Filesystem "md0": XFS internal error xfs_btree_check_sblock at line 336 of file fs/xfs/xfs_btree.c. Caller 0xc01fb282 <c0214568>

Memory corruption?

Tried with different RAMs, and the problem occurs on ECC RAM too.

Quoted 8 lines
Quoted 4 lines

Out of curiosity, I've tried to use reiserfs (just to see how it compares regarding this). Reiserfs crashed before even writing 100MB!

That indicates there's something wrong other than the filesystem. I'd suggest making sure your raid arrays, memory, etc are all functioning correctly first.

They are. I've tested 5 different machines so far (Supermicro or Tyan mobos, kingston RAM, Intel or AMD cpus, hitachi and seagate drives...)

Quoted 1 line

What platform are you running on? Are you running ia32 with 4k stacks?

Yes. I'll try this week 2.6.18.8 thoroughly and 2.6.20.11 too. Then jfs, just to be sure.

Signature · 3 lines
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------
#4 lore raw
Quoted 5 lines · Emmanuel Florac

Le Fri, 4 May 2007 10:59:22 +1000 vous écriviez:

Quoted 1 line

What platform are you running on? Are you running ia32 with 4k stacks?

Yes. I'll try this week 2.6.18.8 thoroughly and 2.6.20.11 too. Then jfs, just to be sure.

Well, there's your problem. Stack overflows. IMO, if you use a filesystem, you shouldn't use 4k stacks. ;)

If you remake you kernel with 8k stacks then your problems will most likely go away.

Cheers,

Dave.

Signature · 3 lines
Dave Chinner
Principal Engineer
SGI Australian Software Group
#5 lore raw

Le Fri, 4 May 2007 17:33:44 +1000
David Chinner <[email protected]> écrivait:

Quoted 5 lines

Well, there's your problem. Stack overflows. IMO, if you use a filesystem, you shouldn't use 4k stacks. ;)

If you remake you kernel with 8k stacks then your problems will most likely go away.

Well, I've double-checked the asm-i386/module.h, and it actually looks like 4K stacks is NOT the default, so I must be using 8K, isn't it?

I've ran the same test on the same machine but WITHOUT software raid-0 (so write barriers are in use), and all went well, more than 3TB written without a glitch. I still think there's something related to the write barriers here. I'll try with another RAID controller, Adaptec for instance, to get sure the 3ware driver isn't involved. I'll also try again with an amd64 kernel.

I'd really like to sort this out...

Signature · 3 lines
----------------------------------------
Emmanuel Florac     |   Intellique
----------------------------------------
#6 lore raw
Quoted 11 lines · Emmanuel Florac

Le Fri, 4 May 2007 17:33:44 +1000
David Chinner <[email protected]> écrivait:

Quoted 5 lines

Well, there's your problem. Stack overflows. IMO, if you use a filesystem, you shouldn't use 4k stacks. ;)

If you remake you kernel with 8k stacks then your problems will most likely go away.

Well, I've double-checked the asm-i386/module.h, and it actually looks like 4K stacks is NOT the default, so I must be using 8K, isn't it?

Depends on how you config'd it, just look at the .config you built with, and search for CONFIG_4KSTACKS

On Fedora at least (and I can't remember - I don't think this is a fedora-ism...) you can do "modinfo" on some module, and see:

vermagic: 2.6.21 SMP mod_unload 686 4KSTACKS

-Eric

#7 lore raw

Le Fri, 04 May 2007 09:55:30 -0500
Eric Sandeen <[email protected]> écrivait:

Quoted 10 lines
Quoted 9 lines · Emmanuel Florac

Le Fri, 4 May 2007 17:33:44 +1000
David Chinner <[email protected]> écrivait:

Quoted 5 lines

Well, there's your problem. Stack overflows. IMO, if you use a filesystem, you shouldn't use 4k stacks. ;)

If you remake you kernel with 8k stacks then your problems will most likely go away.

Quoted 6 lines
Quoted 3 lines

Well, I've double-checked the asm-i386/module.h, and it actually looks like 4K stacks is NOT the default, so I must be using 8K, isn't it?

Depends on how you config'd it, just look at the .config you built with, and search for CONFIG_4KSTACKS

config-2.6.17.13:

# CONFIG_4KSTACKS is not set

So the problem lies elsewhere...

Signature · 3 lines
----------------------------------------
Emmanuel Florac     |   Intellique
----------------------------------------
#8 lore raw
Quoted 3 lines · Emmanuel Florac

# CONFIG_4KSTACKS is not set

So the problem lies elsewhere...

CONFIG_4KSTACKS is badly named.

It means you have 4K process + 4K interrupt stacks. Without this set you have just a single 8K stack for processes and interrupts.

One argument for 4K+4K stacks is that 8K+0K isn't really safer in many cases --- it just appears that way becasue the problems are harder to hit.

Almost three years ago I posted patches to split the CONFIG_4KSTACKS option into two options. I quickly just ported that to 2.6.21 just now (very quickly, I might have goofed fixing up the rejects).

You could if you have time try this and enable CONFIG_I386_IRQSTACKS but don't enable CONFIG_I386_4KSTACKS and see if that helps...

arch/i386/Kconfig.debug +10 -3
index 458bc16..f32fbec 100644
@@ -56,15 +56,22 @@ config DEBUG_RODATA
 	  portion of the kernel code won't be covered by a 2MB TLB anymore.
 	  If in doubt, say "N".
 
-config 4KSTACKS
+config I386_4KSTACKS
 	bool "Use 4Kb for kernel stacks instead of 8Kb"
 	depends on DEBUG_KERNEL
 	help
 	  If you say Y here the kernel will use a 4Kb stacksize for the
 	  kernel stack attached to each process/thread. This facilitates
 	  running more threads on a system and also reduces the pressure
-	  on the VM subsystem for higher order allocations. This option
-	  will also use IRQ stacks to compensate for the reduced stackspace.
+	  on the VM subsystem for higher order allocations.
+
+config I386_IRQSTACKS
+	bool "Allocate separate IRQ stacks"
+	depends on DEBUG_KERNEL
+	default y
+	help
+	  If you say Y here the kernel will allocate and use separate
+	  stacks for interrupts.
 
 config X86_FIND_SMP_CONFIG
 	bool
arch/i386/defconfig +0 -0
arch/i386/kernel/irq.c +7 -7
index 8db8d51..f6224fd 100644
@@ -47,7 +47,7 @@ void ack_bad_irq(unsigned int irq)
 #endif
 }
 
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_IRQSTACKS
 /*
  * per-CPU IRQ handling contexts (thread information and stack)
  */
@@ -58,7 +58,7 @@ union irq_ctx {
 
 static union irq_ctx *hardirq_ctx[NR_CPUS] __read_mostly;
 static union irq_ctx *softirq_ctx[NR_CPUS] __read_mostly;
-#endif
+#endif /* CONFIG_I386_IRQSTACKS */
 
 /*
  * do_IRQ handles all normal device IRQ's (the special
@@ -71,7 +71,7 @@ fastcall unsigned int do_IRQ(struct pt_regs *regs)
 	/* high bit used in ret_from_ code */
 	int irq = ~regs->orig_eax;
 	struct irq_desc *desc = irq_desc + irq;
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_IRQSTACKS
 	union irq_ctx *curctx, *irqctx;
 	u32 *isp;
 #endif
@@ -99,7 +99,7 @@ fastcall unsigned int do_IRQ(struct pt_regs *regs)
 	}
 #endif
 
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_IRQSTACKS
 
 	curctx = (union irq_ctx *) current_thread_info();
 	irqctx = hardirq_ctx[smp_processor_id()];
@@ -136,7 +136,7 @@ fastcall unsigned int do_IRQ(struct pt_regs *regs)
 			: "memory", "cc"
 		);
 	} else
-#endif
+#endif /* CONFIG_I386_IRQSTACKS */
 		desc->handle_irq(irq, desc);
 
 	irq_exit();
@@ -144,7 +144,7 @@ fastcall unsigned int do_IRQ(struct pt_regs *regs)
 	return 1;
 }
 
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_IRQSTACKS
 
 /*
  * These should really be __section__(".bss.page_aligned") as well, but
@@ -234,7 +234,7 @@ asmlinkage void do_softirq(void)
 }
 
 EXPORT_SYMBOL(do_softirq);
-#endif
+#endif /* CONFIG_I386_IRQSTACKS */
 
 /*
  * Interrupt statistics:
include/asm-i386/irq.h +3 -3
index 11761cd..7db95e1 100644
@@ -24,14 +24,14 @@ static __inline__ int irq_canonicalize(int irq)
 # define ARCH_HAS_NMI_WATCHDOG		/* See include/linux/nmi.h */
 #endif
 
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_IRQSTACKS
   extern void irq_ctx_init(int cpu);
   extern void irq_ctx_exit(int cpu);
 # define __ARCH_HAS_DO_SOFTIRQ
-#else
+#else /* !CONFIG_I386_IRQSTACKS */
 # define irq_ctx_init(cpu) do { } while (0)
 # define irq_ctx_exit(cpu) do { } while (0)
-#endif
+#endif /* CONFIG_I386_IRQSTACKS */
 
 #ifdef CONFIG_IRQBALANCE
 extern int irqbalance_disable(char *str);
include/asm-i386/module.h +3 -3
index 02f8f54..7d5d2df 100644
@@ -62,11 +62,11 @@ struct mod_arch_specific
 #error unknown processor family
 #endif
 
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_4KSTACKS
 #define MODULE_STACKSIZE "4KSTACKS "
-#else
+#else /* not using CONFIG_I386_4KSTACKS */
 #define MODULE_STACKSIZE ""
-#endif
+#endif /* CONFIG_I386_4KSTACKS */
 
 #define MODULE_ARCH_VERMAGIC MODULE_PROC_FAMILY MODULE_STACKSIZE
 
include/asm-i386/thread_info.h +1 -1
index 4b187bb..f5268e0 100644
@@ -53,7 +53,7 @@ struct thread_info {
 #endif
 
 #define PREEMPT_ACTIVE		0x10000000
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_4KSTACKS
 #define THREAD_SIZE            (4096)
 #else
 #define THREAD_SIZE		(8192)
#9 lore raw

Le Fri, 4 May 2007 16:20:28 -0700 vous écriviez:

Quoted 2 lines

You could if you have time try this and enable CONFIG_I386_IRQSTACKS but don't enable CONFIG_I386_4KSTACKS and see if that helps...

That sounds very interesting, I'll give it a try monday.

Signature · 3 lines
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------
#10 lore raw
Quoted 7 lines · Emmanuel Florac

Le Fri, 4 May 2007 16:20:28 -0700 vous écriviez:

Quoted 2 lines

You could if you have time try this and enable CONFIG_I386_IRQSTACKS but don't enable CONFIG_I386_4KSTACKS and see if that helps...

That sounds very interesting, I'll give it a try monday.

There are also stack debugging config options; one that will warn if you are about to overflow (CONFIG_DEBUG_STACKOVERFLOW) and one that will print max stack depth in sysrq-t output (CONFIG_DEBUG_STACK_USAGE).

-Eric

#11 lore raw
Quoted 4 lines · Eric Sandeen

There are also stack debugging config options; one that will warn if you are about to overflow (CONFIG_DEBUG_STACKOVERFLOW) and one that will print max stack depth in sysrq-t output (CONFIG_DEBUG_STACK_USAGE).

I was in such a hurry I don't think I tweaked that sanely. I'll go over the patch checking that and test it later today.

Is there some preferred kernel version people would like?

#12 lore raw

Le Sat, 5 May 2007 13:35:25 -0700 vous écriviez:

Quoted 2 lines

Is there some preferred kernel version people would like?

Well I prefer staying away from the very latest bleeding edge, so I stick to 2.6.20.11 for now.

Signature · 3 lines
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------
#13 lore raw
Quoted 2 lines · Emmanuel Florac

Well I prefer staying away from the very latest bleeding edge, so I stick to 2.6.20.11 for now.

arch/i386/Kconfig.debug +10 -3
index f68cc6f..908b755 100644
@@ -56,15 +56,22 @@ config DEBUG_RODATA
 	  portion of the kernel code won't be covered by a 2MB TLB anymore.
 	  If in doubt, say "N".
 
-config 4KSTACKS
+config I386_4KSTACKS
 	bool "Use 4Kb for kernel stacks instead of 8Kb"
 	depends on DEBUG_KERNEL
 	help
 	  If you say Y here the kernel will use a 4Kb stacksize for the
 	  kernel stack attached to each process/thread. This facilitates
 	  running more threads on a system and also reduces the pressure
-	  on the VM subsystem for higher order allocations. This option
-	  will also use IRQ stacks to compensate for the reduced stackspace.
+	  on the VM subsystem for higher order allocations.
+
+config I386_IRQSTACKS
+	bool "Allocate separate IRQ stacks"
+	depends on DEBUG_KERNEL
+	default y
+	help
+	  If you say Y here the kernel will allocate and use separate
+	  stacks for interrupts.
 
 config X86_FIND_SMP_CONFIG
 	bool
arch/i386/kernel/irq.c +7 -7
index 3201d42..0da8251 100644
@@ -33,7 +33,7 @@ void ack_bad_irq(unsigned int irq)
 }
 #endif
 
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_IRQSTACKS
 /*
  * per-CPU IRQ handling contexts (thread information and stack)
  */
@@ -44,7 +44,7 @@ union irq_ctx {
 
 static union irq_ctx *hardirq_ctx[NR_CPUS] __read_mostly;
 static union irq_ctx *softirq_ctx[NR_CPUS] __read_mostly;
-#endif
+#endif /* CONFIG_I386_IRQSTACKS */
 
 /*
  * do_IRQ handles all normal device IRQ's (the special
@@ -57,7 +57,7 @@ fastcall unsigned int do_IRQ(struct pt_regs *regs)
 	/* high bit used in ret_from_ code */
 	int irq = ~regs->orig_eax;
 	struct irq_desc *desc = irq_desc + irq;
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_IRQSTACKS
 	union irq_ctx *curctx, *irqctx;
 	u32 *isp;
 #endif
@@ -85,7 +85,7 @@ fastcall unsigned int do_IRQ(struct pt_regs *regs)
 	}
 #endif
 
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_IRQSTACKS
 
 	curctx = (union irq_ctx *) current_thread_info();
 	irqctx = hardirq_ctx[smp_processor_id()];
@@ -122,7 +122,7 @@ fastcall unsigned int do_IRQ(struct pt_regs *regs)
 			: "memory", "cc"
 		);
 	} else
-#endif
+#endif /* CONFIG_I386_IRQSTACKS */
 		desc->handle_irq(irq, desc);
 
 	irq_exit();
@@ -130,7 +130,7 @@ fastcall unsigned int do_IRQ(struct pt_regs *regs)
 	return 1;
 }
 
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_IRQSTACKS
 
 /*
  * These should really be __section__(".bss.page_aligned") as well, but
@@ -220,7 +220,7 @@ asmlinkage void do_softirq(void)
 }
 
 EXPORT_SYMBOL(do_softirq);
-#endif
+#endif /* CONFIG_I386_IRQSTACKS */
 
 /*
  * Interrupt statistics:
include/asm-i386/irq.h +3 -3
index 11761cd..7db95e1 100644
@@ -24,14 +24,14 @@ static __inline__ int irq_canonicalize(int irq)
 # define ARCH_HAS_NMI_WATCHDOG		/* See include/linux/nmi.h */
 #endif
 
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_IRQSTACKS
   extern void irq_ctx_init(int cpu);
   extern void irq_ctx_exit(int cpu);
 # define __ARCH_HAS_DO_SOFTIRQ
-#else
+#else /* !CONFIG_I386_IRQSTACKS */
 # define irq_ctx_init(cpu) do { } while (0)
 # define irq_ctx_exit(cpu) do { } while (0)
-#endif
+#endif /* CONFIG_I386_IRQSTACKS */
 
 #ifdef CONFIG_IRQBALANCE
 extern int irqbalance_disable(char *str);
include/asm-i386/module.h +3 -3
index 02f8f54..7d5d2df 100644
@@ -62,11 +62,11 @@ struct mod_arch_specific
 #error unknown processor family
 #endif
 
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_4KSTACKS
 #define MODULE_STACKSIZE "4KSTACKS "
-#else
+#else /* not using CONFIG_I386_4KSTACKS */
 #define MODULE_STACKSIZE ""
-#endif
+#endif /* CONFIG_I386_4KSTACKS */
 
 #define MODULE_ARCH_VERMAGIC MODULE_PROC_FAMILY MODULE_STACKSIZE
 
include/asm-i386/thread_info.h +1 -1
index 4b187bb..f5268e0 100644
@@ -53,7 +53,7 @@ struct thread_info {
 #endif
 
 #define PREEMPT_ACTIVE		0x10000000
-#ifdef CONFIG_4KSTACKS
+#ifdef CONFIG_I386_4KSTACKS
 #define THREAD_SIZE            (4096)
 #else
 #define THREAD_SIZE		(8192)
#14 lore raw

Le Sat, 5 May 2007 15:12:50 -0700 vous écriviez:

arch/i386/Kconfig.debug

Quoted 1 line
arch/i386/Kconfig.debug +0 -0

Thanks!

Signature · 3 lines
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------
#15 lore raw
In reply to Eric Sandeen

Le Sat, 05 May 2007 11:50:12 -0500 vous écriviez:

Quoted 4 lines

There are also stack debugging config options; one that will warn if you are about to overflow (CONFIG_DEBUG_STACKOVERFLOW) and one that will print max stack depth in sysrq-t output (CONFIG_DEBUG_STACK_USAGE).

Fine, I'll try that.

Signature · 3 lines
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------
#16 lore raw
In reply to Chris Wedgwood

Hi!

Several months ago, you posted a patch for 8k stacks together with irqstacks:

Quoted 3 lines · Chris Wedgwood

Almost three years ago I posted patches to split the CONFIG_4KSTACKS option into two options. I quickly just ported that to 2.6.21 just now (very quickly, I might have goofed fixing up the rejects).

Do you have a working version of your patch for 2.6.23? I've been using a similar patch (attached) for several years now but since 2.6.23, it produces oopses like below. The patch applies cleanly but there seems to be some major change between 2.6.22 and 2.6.23 that isn't covered correctly.

Thanks,
--leo

P.S.: The attached patch is part of ATrpms' 8k kernel:
http://atrpms.net/dist/f8/kernel-tuxonice/
2.6.23.1-49_0.99.cubbi_tuxonice_8k and 2.6.23.1-42_0.99.cubbi_tuxonice_8k both show the problem, the corresponding versions without _8k work without any problem. 2.6.22.1-41_0.99.cubbi_tuxonice_8k on Fedora 7 worked fine too.

-------------------- 8< --------------------

BUG: unable to handle kernel NULL pointer dereference at virtual address
00000050
printing eip: 00000050 *pde = 3ed4c067
Oops: 0000 [#1] SMP
Modules linked in: ipv6 ext2 mbcache loop ahci sr_mod cdrom ata_generic

firewire_ohci firewire_core pata_pdc2027x crc_itu_t sata_sil iTCO_wdt i2c_i801 button parport_pc parport iTCO_vendor_support i2c_core ata_piix pcspkr intel_agp sky2 floppy sg dm_snapshot dm_zero dm_mirror dm_mod pata_it821x libata sd_mod scsi_mod raid456 async_xor async_memcpy async_tx xor raid1 xfs uhci_hcd ohci_hcd ehci_hcd

CPU:    0
EIP:    0060:[<00000050>]    Not tainted VLI
EFLAGS: 00210082   (2.6.23.1-49_0.99.cubbi_tuxonice_8k.fc8 #1)

EIP is at 0x50
eax: 00000001 ebx: f8947796 ecx: 01073000 edx: c0799200
esi: f8897310 edi: f79d8000 ebp: f79d8a2c esp: c07defa0
ds: 007b es: 007b fs: 00d8 gs: 0033 ss: 0068
Process rklogd (pid: 1787, ti=c07de000 task=f74fb840 task.ti=f744e000)

Stack: 00200292 f894472d 00000000 000000dc f7f5a6a0 f7f8f2b4 00000002
00000001
       00200292 00000001 00000012 00000012 c041e022 c0465fcb c0741700
c0741700
       00000012 00000000 c04672e5 c07def7c 00000012 c0467249 c0741700
c04074cb
Call Trace:
 [<f894472d>] ata_interrupt+0x1ab/0x1be [libata]
 [<c041e022>] ack_ioapic_quirk_irq+0x34/0x86
 [<c0465fcb>] handle_IRQ_event+0x23/0x51
 [<c04672e5>] handle_fasteoi_irq+0x9c/0xa6
 [<c0467249>] handle_fasteoi_irq+0x0/0xa6
 [<c04074cb>] do_IRQ+0x8c/0xb9
 =======================
Code:  Bad EIP value.
EIP: [<00000050>] 0x50 SS:ESP 0068:c07defa0
BUG: unable to handle kernel paging request at virtual address 010b2fc8
printing eip: c07dee7c *pde = 00000000
Oops: 0002 [#2] SMP
Modules linked in: ipv6 ext2 mbcache loop ahci sr_mod cdrom ata_generic

firewire_ohci firewire_core pata_pdc2027x crc_itu_t sata_sil iTCO_wdt i2c_i801 button parport_pc parport iTCO_vendor_support i2c_core ata_piix pcspkr intel_agp sky2 floppy sg dm_snapshot dm_zero dm_mirror dm_mod pata_it821x libata sd_mod scsi_mod raid456 async_xor async_memcpy async_tx xor raid1 xfs uhci_hcd ohci_hcd ehci_hcd

CPU:    0
EIP:    0060:[<c07dee7c>]    Tainted: G      D VLI
EFLAGS: 00210086   (2.6.23.1-49_0.99.cubbi_tuxonice_8k.fc8 #1)

EIP is at hardirq_stack+0x1e7c/0x40000
eax: 00200046 ebx: f74fb88c ecx: 00200286 edx: 01073000
esi: 00051b65 edi: 001e8480 ebp: 00200006 esp: c07dee58
ds: 007b es: 007b fs: 00d8 gs: 0033 ss: 0068
Process rklogd (pid: 1787, ti=c07de000 task=f74fb840 task.ti=f744e000)

Stack: c0425763 153fdb8a 00000001 13125d1e 00000001 f7bb5840 f74fb840
f7bb5840
       c07dee90 c042ab0f 001e8480 00000000 00000001 00000000 00200082
c0427c1d
       ffffff10 c04300ad 00000000 0000000f f7bb5840 00000000 c180c200
00000000
Call Trace:
 [<c0425763>] __check_preempt_curr_fair+0x55/0x86
 [<c042ab0f>] check_preempt_curr_fair+0x6b/0x71
 [<c0427c1d>] try_to_wake_up+0x2ef/0x2f9
 [<c04300ad>] do_exit+0x11b/0x6fc
 [<c0434d58>] del_timer+0x48/0x4e
 [<f894564f>] ata_scsi_qc_complete+0x3bd/0x3cb [libata]
 [<c062cdeb>] do_page_fault+0x521/0x5ef
 [<f893feea>] __ata_qc_complete+0x8c/0x92 [libata]
 [<f8940dd9>] ata_hsm_move+0x6d1/0x70c [libata]
 [<f8897310>] it821x_passthru_bmdma_stop+0x17/0x36 [pata_it821x]
 [<c062c8ca>] do_page_fault+0x0/0x5ef
 [<c062b5b2>] error_code+0x72/0x78
 [<f8947796>] ata_altstatus+0x1c/0x20 [libata]
 [<f894796c>] ata_bmdma_stop+0x1a/0x23 [libata]
 [<f8947796>] ata_altstatus+0x1c/0x20 [libata]
 [<f8897310>] it821x_passthru_bmdma_stop+0x17/0x36 [pata_it821x]
 [<f894472d>] ata_interrupt+0x1ab/0x1be [libata]
 [<c041e022>] ack_ioapic_quirk_irq+0x34/0x86
 [<c0465fcb>] handle_IRQ_event+0x23/0x51
 [<c04672e5>] handle_fasteoi_irq+0x9c/0xa6
 [<c0467249>] handle_fasteoi_irq+0x0/0xa6
 [<c04074cb>] do_IRQ+0x8c/0xb9
 =======================
Code: 00 00 00 86 00 21 00 63 57 42 c0 8a db 3f 15 01 00 00 00 1e 5d 12

13 01 00 00 00 40 58 bb f7 40 b8 4f f7 40 58 bb f7 90 ee 7d c0 <0f> ab 42 c0 80 84 1e 00 00 00 00 00 01 00 00 00 00 00 00 00 82
EIP: [<c07dee7c>] hardirq_stack+0x1e7c/0x40000 SS:ESP 0068:c07dee58 Fixing recursive fault but reboot is needed! -------------------- 8< --------------------

Signature · 3 lines
e-mail   ::: Alexander.Bergolth (at) wu-wien.ac.at
fax      ::: +43-1-31336-906050
location ::: Computer Center | Vienna University of Economics | Austria
arch/i386/Kconfig.debug renamed from arch/i386/Kconfig.debug.orig +5 -0
@@ -85,4 +85,9 @@
           option saves about 4k and might cause you much additional grey
           hair.
 
+ config IRQSTACKS
+ 	bool "use IRQ stacks"
+ 	depends on !4KSTACKS
+ 	default n
+ 
 endmenu
arch/i386/kernel/irq.c renamed from arch/i386/kernel/irq.c.orig +4 -4
@@ -50,7 +50,7 @@
 #endif
 }
 
-#ifdef CONFIG_4KSTACKS
+#if defined(CONFIG_4KSTACKS) || defined(CONFIG_IRQSTACKS)
 /*
  * per-CPU IRQ handling contexts (thread information and stack)
  */
@@ -74,7 +74,7 @@
 	/* high bit used in ret_from_ code */
 	int irq = ~regs->orig_eax;
 	struct irq_desc *desc = irq_desc + irq;
-#ifdef CONFIG_4KSTACKS
+#if defined(CONFIG_4KSTACKS) || defined(CONFIG_IRQSTACKS)
 	union irq_ctx *curctx, *irqctx;
 	u32 *isp;
 #endif
@@ -102,7 +102,7 @@
 	}
 #endif
 
-#ifdef CONFIG_4KSTACKS
+#if defined(CONFIG_4KSTACKS) || defined(CONFIG_IRQSTACKS)
 
 	curctx = (union irq_ctx *) current_thread_info();
 	irqctx = hardirq_ctx[smp_processor_id()];
@@ -147,7 +147,7 @@
 	return 1;
 }
 
-#ifdef CONFIG_4KSTACKS
+#if defined(CONFIG_4KSTACKS) || defined(CONFIG_IRQSTACKS)
 
 static char softirq_stack[NR_CPUS * THREAD_SIZE]
 		__attribute__((__section__(".bss.page_aligned")));
include/asm-i386/irq.h renamed from include/asm-i386/irq.h.orig +1 -1
@@ -24,7 +24,7 @@
 # define ARCH_HAS_NMI_WATCHDOG		/* See include/linux/nmi.h */
 #endif
 
-#ifdef CONFIG_4KSTACKS
+#if defined(CONFIG_4KSTACKS) || defined(CONFIG_IRQSTACKS)
   extern void irq_ctx_init(int cpu);
   extern void irq_ctx_exit(int cpu);
 # define __ARCH_HAS_DO_SOFTIRQ
#17 lore raw
Quoted 2 lines · Alexander Bergolth

Several months ago, you posted a patch for 8k stacks together with irqstacks:

I've posted it off and on for two or three years now --- quite a few people are using it or the equivalent of it, but thus far people resist merging this.

The other idea (also nixed) that I posed was to dosable XFS if people are using 4KSTACKS

Quoted 1 line

Do you have a working version of your patch for 2.6.23?

No. I can go digg out a 32-bit machine and retest though.

Quoted 4 lines

I've been using a similar patch (attached) for several years now but since 2.6.23, it produces oopses like below. The patch applies cleanly but there seems to be some major change between 2.6.22 and 2.6.23 that isn't covered correctly.

I'll take a look. Can you make sure it works with 8K stacks (more or less works).

#18 lore raw
Quoted 6 lines · Chris Wedgwood
Quoted 2 lines · Alexander Bergolth

Several months ago, you posted a patch for 8k stacks together with irqstacks:

I'll take a look. Can you make sure it works with 8K stacks (more or less works).

It works (more or less) fine without the patch that splits irqstacks from 4k stacks. (I tried it both with CONFIG_4KSTACKS set and unset.)

Thanks,
--leo

Signature · 3 lines
e-mail   ::: Alexander.Bergolth (at) wu-wien.ac.at
fax      ::: +43-1-31336-906050
location ::: Computer Center | Vienna University of Economics | Austria
#19 lore raw
In reply to Emmanuel Florac

Am Freitag 04 Mai 2007 schrieb Emmanuel Florac:

Quoted 6 lines

I've ran the same test on the same machine but WITHOUT software raid-0 (so write barriers are in use), and all went well, more than 3TB written without a glitch. I still think there's something related to the write barriers here. I'll try with another RAID controller, Adaptec for instance, to get sure the 3ware driver isn't involved. I'll also try again with an amd64 kernel.

Hello Emmanuel!

When you can't use write barriers as XFS tell you in the logs, you better switch of write caching for the harddisks / raid controller, unless you happen to have NVRAM or safe power supply.

But then using write cache without barrier should not make any difference unless you actually have a crash or power failure during write operation.

Did you test with ext3 as well? You wrote it crashes with ReiserFS (version 3) even faster. When it crashes with several filesystems its unlikely to be a filesystem issue.

Regards,

Signature · 2 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7
#20 lore raw

Le Fri, 4 May 2007 17:58:21 +0200 vous écriviez:

Quoted 3 lines

Did you test with ext3 as well? You wrote it crashes with ReiserFS (version 3) even faster. When it crashes with several filesystems its unlikely to be a filesystem issue.

Unfortunately ext3 doesn't support volumes bigger than 8TB, so that's useless to me. I plan to test jfs, however.
I think it's more a dm/md issue, but I'm not sure...

Signature · 3 lines
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------
#21 lore raw
Quoted 8 lines · Emmanuel Florac

Le Fri, 4 May 2007 17:58:21 +0200 vous écriviez:

Quoted 1 line

Did you test with ext3 as well? You wrote it crashes with ReiserFS (version 3) even faster. When it crashes with several filesystems its unlikely to be a filesystem issue.

Unfortunately ext3 doesn't support volumes bigger than 8TB, so that's useless to me. I plan to test jfs, however.
I think it's more a dm/md issue, but I'm not sure...

Most recent kernels (2.6.19 or so IIRC) & cvs e2fsprogs (or that from rhel5/centos5) can do up to 16T ext3 filesystems, so you should be able to test that if you like.

-Eric

#22 lore raw

Le Fri, 04 May 2007 23:49:28 -0500 vous écriviez:

Quoted 4 lines

Most recent kernels (2.6.19 or so IIRC) & cvs e2fsprogs (or that from rhel5/centos5) can do up to 16T ext3 filesystems, so you should be able to test that if you like.

Thanks, I'll try that too. Though it won't cover all my needs (I plan to set up 50 and 150TB systems really soon).

Signature · 3 lines
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------
#23 lore raw
Quoted 8 lines · Emmanuel Florac

Le Fri, 04 May 2007 23:49:28 -0500 vous écriviez:

Quoted 2 lines

Most recent kernels (2.6.19 or so IIRC) & cvs e2fsprogs (or that from rhel5/centos5) can do up to 16T ext3 filesystems, so you should be able to test that if you like.

Thanks, I'll try that too. Though it won't cover all my needs (I plan to set up 50 and 150TB systems really soon).

Sure, I understand - it may be helpful in figuring out what the problem is, though. I'll be curious to see how it goes...

Oh, btw, you'll need the -F (force) flag for mkfs.ext3

-Eric

#24 lore raw

Le Sat, 05 May 2007 11:47:58 -0500 vous écriviez:

Quoted 2 lines

Sure, I understand - it may be helpful in figuring out what the problem is, though. I'll be curious to see how it goes...

Sure, stay tuned!

Quoted 1 line

Oh, btw, you'll need the -F (force) flag for mkfs.ext3

Thanks!

Signature · 3 lines
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------

A message that is not in the archive (sent off-list or not yet received): [email protected]

#26 lore raw

Le Sat, 5 May 2007 14:00:02 -0700 vous écriviez:

Quoted 5 lines

A 50TB filesystem might suck horrible on a 32-bit platform. I'm not sure there is *ANY* way you coiuld fsck that should you need in some cases.

Is that what you're planning to do?

Nope, I'll use an x86_64 system running an x86_64 kernel :)

Signature · 3 lines
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------
#27 lore raw
Quoted 1 line · Emmanuel Florac

Nope, I'll use an x86_64 system running an x86_64 kernel :)

How much RAM? I think you'll want 10s of GBs possibly (well, it depends very much on what you're storing but you can fit a lot of small files in 150TB...)

#28 lore raw

Le Sun, 6 May 2007 10:26:06 -0700 vous écriviez:

Quoted 3 lines

How much RAM? I think you'll want 10s of GBs possibly (well, it depends very much on what you're storing but you can fit a lot of small files in 150TB...)

It will be video storage, big to huge file mainly. But I'll remember to stick as much RAM as I can :)

Signature · 3 lines
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------
#29 lore raw
In reply to Emmanuel Florac
Quoted 2 lines · Emmanuel Florac

Unfortunately ext3 doesn't support volumes bigger than 8TB, so that's useless to me. I plan to test jfs, however.

Is jfs supported by anyone right now?

#30 lore raw

Le Sat, 5 May 2007 13:56:17 -0700 vous écriviez:

Quoted 1 line

Is jfs supported by anyone right now?

Huh, IBM I hope :)

Signature · 3 lines
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------
#31 lore raw
In reply to Chris Wedgwood

Am Samstag 05 Mai 2007 schrieb Chris Wedgwood:

Quoted 5 lines · Emmanuel Florac
Quoted 2 lines

Unfortunately ext3 doesn't support volumes bigger than 8TB, so that's useless to me. I plan to test jfs, however.

Is jfs supported by anyone right now?

David 'Dave' Kleikamp was still taking care of JFS as I asked him some questions about write barrier support back in July 2007. He concentrated on bug fixes tough, not on new features.

Regards,

Signature · 2 lines
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7
#32 lore raw
In reply to Emmanuel Florac
Quoted 11 lines · Emmanuel Florac

Le Fri, 4 May 2007 17:33:44 +1000
David Chinner <[email protected]> écrivait:

Quoted 5 lines

Well, there's your problem. Stack overflows. IMO, if you use a filesystem, you shouldn't use 4k stacks. ;)

If you remake you kernel with 8k stacks then your problems will most likely go away.

Well, I've double-checked the asm-i386/module.h, and it actually looks like 4K stacks is NOT the default, so I must be using 8K, isn't it?

Yes.

Quoted 6 lines

I've ran the same test on the same machine but WITHOUT software raid-0 (so write barriers are in use), and all went well, more than 3TB written without a glitch. I still think there's something related to the write barriers here. I'll try with another RAID controller, Adaptec for instance, to get sure the 3ware driver isn't involved. I'll also try again with an amd64 kernel.

So you use software raid and you get corruptions, right? I doubt this has anything to do with write barriers - if it does thats an indication of broken drivers or hardware.....

Can you run with "-o nobarrier" and no software raid and see if you still have a problem?

Cheers,

Dave.

Signature · 3 lines
Dave Chinner
Principal Engineer
SGI Australian Software Group
#33 lore raw

Le Mon, 7 May 2007 12:11:22 +1000 vous écriviez:

Quoted 6 lines

So you use software raid and you get corruptions, right? I doubt this has anything to do with write barriers - if it does thats an indication of broken drivers or hardware.....

Can you run with "-o nobarrier" and no software raid and see if you still have a problem?

I tried on the same machine without software RAID and barriers, and i worked OK. I'll try today with nobarrier. Stay tuned :)

Signature · 3 lines
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------
#34 lore raw
Quoted 7 lines · Emmanuel Florac

Le Mon, 7 May 2007 12:11:22 +1000 vous écriviez:

Quoted 4 lines

So you use software raid and you get corruptions, right? I doubt this has anything to do with write barriers - if it does thats an indication of broken drivers or hardware.....

Quoted 11 lines
Quoted 2 lines

Can you run with "-o nobarrier" and no software raid and see if you still have a problem?

I tried on the same machine without software RAID and barriers, and i worked OK. I'll try today with nobarrier. Stay tuned :)

Signature · 3 lines
--------------------------------------------------
Emmanuel Florac               www.intellique.com   
--------------------------------------------------

Can you update us on the status? Not to thread hijack, but I am having the same problem on my fileserver. Searched all over and can't seem to find a definitive answer.

Signature · 2 lines
View this message in context: http://www.nabble.com/XFS-crash-on-linux-raid-tf3687076.html#a11857832
Sent from the Xfs - General mailing list archive at Nabble.com.