Skip to contentSkip to conversation

Reiserfs 3.6 + quota enabled, crash on delete (or maybe truncate)

DISCUSSION Guillaume Pelat 4 participants 4 messages Aug 4, 2005 – Aug 10, 2005

lore
Share & tools
lore
#0 lore raw

Hi,

Quoted 9 lines
Quoted 8 lines

I'm having a crash with reiserfs 3.6 + user quota enabled, on 2.6.11.10 kernel (no smp), apparently when deleting files (or maybe
during a runcate operation). The problem seems to happen under high load.
When the error occurs, all the processes accessing the reiserfs partition seems to hang. This problem happened several times on different servers (having the same hardware configuration) during last weeks.

Quoted 3 lines

The attached patch should help to get rid of clm-2100 and to avoid crash.
Also, I think you should reiserfsck sda3.

Thanks for your answer. I tried the patch attached to your mail, but it seems that it doesnt solve my problem :)
The error also occured after i did a reiserfsck on sda3, I also tested 2.6.13-rc4 with your patch applied, without success (the error also occured on serveral servers having the same hardware configuration).

Here are the error logs:

------------[ cut here ]------------
kernel BUG at fs/reiserfs/prints.c:362!
invalid operand: 0000 [#1]
CPU:    0
EIP:    0060:[<c019ae2f>]    Not tainted VLI
EFLAGS: 00010296   (2.6.13-rc4-endy)

EIP is at reiserfs_panic+0x4f/0x80
eax: 00000053 ebx: c02b8fde ecx: 00000000 edx: c02dfdac
esi: 00000000 edi: 00000140 ebp: e75b383c esp: e75b3824
ds: 007b es: 007b ss: 0068
Process pure-ftpd (pid: 12771, threadinfo=e75b2000 task=f091d530)

Stack: c02bd610 c02b8fde c03acdc0 00000fa0 c0971154 00000002 e75b3864 c01ac75f
       00000000 c02bf89c 00000fa0 00000002 00020000 00000000 c097101c 
00000000
       e75b38b8 c01939d3 c097101c 00000fd0 00000000 00000000 00000000 
00000000
Call Trace:
 [<c0102e5f>] show_stack+0x7f/0xa0
 [<c0103002>] show_registers+0x152/0x1c0
 [<c01031f8>] die+0xc8/0x140
 [<c0103325>] do_trap+0xb5/0xc0
 [<c010366c>] do_invalid_op+0xbc/0xd0
 [<c0102aa3>] error_code+0x4f/0x54
 [<c01ac75f>] direntry_check_left+0x8f/0x90
 [<c01939d3>] get_num_ver+0x303/0x350
 [<c01949ac>] ip_check_balance+0x3dc/0xbc0
 [<c0195948>] check_balance+0x58/0x70
 [<c019623b>] fix_nodes+0x15b/0x420
 [<c01a2daf>] reiserfs_cut_from_item+0x10f/0x570
 [<c01a359b>] reiserfs_do_truncate+0x2db/0x5e0
 [<c01a282f>] reiserfs_delete_object+0x3f/0x80
 [<c0189baf>] reiserfs_delete_inode+0xaf/0x150
 [<c0161835>] generic_delete_inode+0x95/0x130
 [<c0161a18>] generic_drop_inode+0x18/0x30
 [<c0161a86>] iput+0x56/0x80
 [<c018d07d>] reiserfs_new_inode+0x16d/0x7e0
 [<c0187d31>] reiserfs_create+0xc1/0x1f0
 [<c0156a4f>] vfs_create+0x9f/0x120
 [<c015732c>] open_namei+0x5cc/0x620
 [<c0146eac>] filp_open+0x3c/0x60
 [<c01471c5>] sys_open+0x55/0x90
 [<c0102889>] syscall_call+0x7/0xb
Code: 01 00 00 89 04 24 e8 31 fd ff ff c7 04 24 10 d6 2b c0 85 f6 89 d8 0f 45 c7

ba c0 cd 3a c0 89
54 24 08 89 44 24 04 e8 21 80 f7 ff <0f> 0b 6a 01 2f 95 2b c0 c7 04 24 34 d6 2b c0 85 f6 be c0 cd 3
a
Badness in do_exit at kernel/exit.c:787

 [<c0102e9e>] dump_stack+0x1e/0x30
 [<c0114fdc>] do_exit+0x2ec/0x300
 [<c010326f>] die+0x13f/0x140
 [<c0103325>] do_trap+0xb5/0xc0
 [<c010366c>] do_invalid_op+0xbc/0xd0
 [<c0102aa3>] error_code+0x4f/0x54
 [<c01ac75f>] direntry_check_left+0x8f/0x90
 [<c01939d3>] get_num_ver+0x303/0x350
 [<c01949ac>] ip_check_balance+0x3dc/0xbc0
 [<c0195948>] check_balance+0x58/0x70
 [<c019623b>] fix_nodes+0x15b/0x420
 [<c01a2daf>] reiserfs_cut_from_item+0x10f/0x570
 [<c01a359b>] reiserfs_do_truncate+0x2db/0x5e0
 [<c01a282f>] reiserfs_delete_object+0x3f/0x80
 [<c0189baf>] reiserfs_delete_inode+0xaf/0x150
 [<c0161835>] generic_delete_inode+0x95/0x130
 [<c0161a18>] generic_drop_inode+0x18/0x30
 [<c0161a86>] iput+0x56/0x80
 [<c018d07d>] reiserfs_new_inode+0x16d/0x7e0
 [<c0187d31>] reiserfs_create+0xc1/0x1f0
 [<c0156a4f>] vfs_create+0x9f/0x120
 [<c015732c>] open_namei+0x5cc/0x620
 [<c0146eac>] filp_open+0x3c/0x60
 [<c01471c5>] sys_open+0x55/0x90
 [<c0102889>] syscall_call+0x7/0xb

Best Regards,

Guillaume Pelat

#1 lore raw

Hello

Quoted 11 lines · Guillaume Pelat

Hi,

Quoted 8 lines
Quoted 8 lines

I'm having a crash with reiserfs 3.6 + user quota enabled, on 2.6.11.10 kernel (no smp), apparently when deleting files (or maybe
during a runcate operation). The problem seems to happen under high load.
When the error occurs, all the processes accessing the reiserfs partition seems to hang. This problem happened several times on different servers (having the same hardware configuration) during last weeks.

Quoted 7 lines
Quoted 3 lines

The attached patch should help to get rid of clm-2100 and to avoid crash.
Also, I think you should reiserfsck sda3.

Thanks for your answer. I tried the patch attached to your mail, but it seems that it doesnt solve my problem :)
The error also occured after i did a reiserfsck on sda3, I also tested 2.6.13-rc4 with your patch applied, without success (the error also occured on serveral servers having the same hardware configuration).

Would you, please, try to reproduce the problem having reiserfs check mode on. (it is File systems->Reiserfs support->Enable reiserfs debug mode in kernel configuration) and with attached patch.

Quoted 2 lines

Here are the error logs:

Was there anything about reiserfs in the logs before this dump?

Quoted 21 lines
------------[ cut here ]------------
kernel BUG at fs/reiserfs/prints.c:362!
invalid operand: 0000 [#1]
CPU:    0
EIP:    0060:[<c019ae2f>]    Not tainted VLI
EFLAGS: 00010296   (2.6.13-rc4-endy)

EIP is at reiserfs_panic+0x4f/0x80
eax: 00000053 ebx: c02b8fde ecx: 00000000 edx: c02dfdac
esi: 00000000 edi: 00000140 ebp: e75b383c esp: e75b3824
ds: 007b es: 007b ss: 0068
Process pure-ftpd (pid: 12771, threadinfo=e75b2000 task=f091d530)

Stack: c02bd610 c02b8fde c03acdc0 00000fa0 c0971154 00000002 e75b3864 c01ac75f
       00000000 c02bf89c 00000fa0 00000002 00020000 00000000 c097101c 
00000000
       e75b38b8 c01939d3 c097101c 00000fd0 00000000 00000000 00000000 
00000000
Call Trace:
 [<c0102e5f>] show_stack+0x7f/0xa0
 [<c0103002>] show_registers+0x152/0x1c0
 [<c01031f8>] die+0xc8/0x140
 [<c0103325>] do_trap+0xb5/0xc0
Quoted 8 lines
 [<c010366c>] do_invalid_op+0xbc/0xd0
 [<c0102aa3>] error_code+0x4f/0x54
 [<c01ac75f>] direntry_check_left+0x8f/0x90
 [<c01939d3>] get_num_ver+0x303/0x350
 [<c01949ac>] ip_check_balance+0x3dc/0xbc0
 [<c0195948>] check_balance+0x58/0x70
 [<c019623b>] fix_nodes+0x15b/0x420
 [<c01a2daf>] reiserfs_cut_from_item+0x10f/0x570
 fs/reiserfs/prints.c |    3 ---
 1 files changed, 3 deletions(-)
fs/reiserfs/prints.c renamed from fs/reiserfs/prints.c~reiserfs-panic-fix +0 -3
diff -puN fs/reiserfs/prints.c~reiserfs-panic-fix fs/reiserfs/prints.c
@@ -359,9 +359,6 @@ void reiserfs_panic(struct super_block *
 	do_reiserfs_warning(fmt);
 	printk(KERN_EMERG "REISERFS: panic (device %s): %s\n",
 	       reiserfs_bdevname(sb), error_buf);
-	BUG();
-
-	/* this is not actually called, but makes reiserfs_panic() "noreturn" */
 	panic("REISERFS: panic (device %s): %s\n",
 	      reiserfs_bdevname(sb), error_buf);
 }

_

#2 lore raw

Hi,

Quoted 3 lines

Would you, please, try to reproduce the problem having reiserfs check mode on. (it is File systems->Reiserfs support->Enable reiserfs debug mode in kernel configuration) and with attached patch.

Here is the error log with reiserfs check mode on + patch applied :

ReiserFS: sda3: found reiserfs format "3.6" with standard journal
ReiserFS: sda3: warning: CONFIG_REISERFS_CHECK is set ON
ReiserFS: sda3: warning: - it is slow mode for debugging.
ReiserFS: sda3: using ordered data mode
ReiserFS: sda3: journal params: device sda3, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30
ReiserFS: sda3: checking transaction log (sda3)
ReiserFS: sda3: journal-1153: found in header: first_unflushed_offset 4607, last_flushed_trans_id 401988
ReiserFS: sda3: journal-1206: Starting replay from offset 1726529608356351, trans_id 0
ReiserFS: sda3: journal-1299: Setting newest_mount_id to 23
ReiserFS: sda3: Using r5 hash to sort names
ReiserFS: sda3: warning: vs-8301: reiserfs_kmalloc: allocated memory 202992
[..a few days later..]
REISERFS: panic (device Null superblock): vs-8025: set_entry_sizes: (mode==c, insert_size==-4958), invalid length of directory item

Kernel panic - not syncing: REISERFS: panic (device Null superblock): vs-8025: set_entry_sizes: (mode==c, insert_size==-4958

), invalid length of directory item

The partition had just been checked with reiserfsck (2 days before) and it was ok. I didnt reboot between the reiserfsck and the crash.

Here was the result of reiserfsck before the crash:
Checking internal tree..finished
Comparing bitmaps..finished
Checking Semantic tree:
finished
No corruptions found
There are on the filesystem:
Leaves 423085
Internal nodes 2932
Directories 1046685
Other files 8739829
Data block pointers 75038187 (0 of them are zero) Safe links 0
###########

Btw, i forgot to mention the mount options:
noatime,notail,usrquota

I just applied the patch submitted by Jan Kara:
http://bugzilla.kernel.org/show_bug.cgi?id=4771#c3
I dont know yet if it solves the problem :)

Best Regards,

Guillaume Pelat

#3 lore raw

Hi,

Quoted 3 lines

I just applied the patch submitted by Jan Kara:
http://bugzilla.kernel.org/show_bug.cgi?id=4771#c3
I dont know yet if it solves the problem :)

I actually should cure your problem but can have some unexpected
sideffects. Please try an attached patch (it's the new one I posted to bugzilla) - that should be a better fix.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SuSE CR Labs

Initialize key object ID in inode so that we don't try to remove the inode when we fail on some checks even before we manage to allocate something.

fs/reiserfs/namei.c +3 -0
diff -rupX /home/jack/.kerndiffexclude linux-2.6.13-rc6/fs/reiserfs/namei.c linux-2.6.13-rc6-reiser_create_fix/fs/reiserfs/namei.c
@@ -593,6 +593,9 @@ static int new_inode_init(struct inode *
 	 */
 	inode->i_uid = current->fsuid;
 	inode->i_mode = mode;
+	/* Make inode invalid - just in case we are going to drop it before
+	 * the initialization happens */
+	INODE_PKEY(inode)->k_objectid = 0;
 
 	if (dir->i_mode & S_ISGID) {
 		inode->i_gid = dir->i_gid;