Skip to contentSkip to conversation

BUG: assertion failure in fs/jbd/checkpoint.c persists in 2.6.11.12

DISCUSSION David Wilk 2 participants 2 messages Jun 16, 2005

lore
Share & tools
lore

David Wilk

author
#0 lore raw

Howdy all,

We've been plagued buy this ext3 bug since 2.6.10, and it only happens on heavily loaded postgres systems. We run our postgres DB on ext3 data=journal on a dmcrypt partition. Our kernel is also patched with grsec, but that doesn't appear to play any role.

After upgrading to 2.6.11.12 (specifically for the ext3 checkpoint.c fix) we noticed two things. The assertion failure persists, and now we get a condition where a postgres process will spin in state 'D' forever and hog 100% of a CPU (in system, not user).

I've attached the trace in plain text so the formatting doesn't get screwed.

Let me know if anyone would like more information. I'm no programmer, but I'd like to help in any way that I can.

Jun 14 17:21:24 qacluster4 kernel: Assertion failure in __journal_drop_transaction() at fs/jbd/checkpoint.c:627: "transaction->t_forget == NULL"
Jun 14 17:21:24 qacluster4 kernel: ------------[ cut here ]------------
Jun 14 17:21:26 qacluster4 kernel: kernel BUG at fs/jbd/checkpoint.c:627!
Jun 14 17:21:26 qacluster4 kernel: invalid operand: 0000 [#1]
Jun 14 17:21:26 qacluster4 kernel: SMP 
Jun 14 17:21:26 qacluster4 kernel: Modules linked in: memstat
Jun 14 17:21:26 qacluster4 kernel: CPU:    0
Jun 14 17:21:26 qacluster4 kernel: EIP:    0060:[<c0266520>]    Tainted: P      VLI
Jun 14 17:21:26 qacluster4 kernel: EFLAGS: 00010292   (2.6.11.12-grsec) 
Jun 14 17:21:26 qacluster4 kernel: EIP is at __journal_drop_transaction+0x2d0/0x384
Jun 14 17:21:26 qacluster4 kernel: eax: 00000071   ebx: f7f7b500   ecx: c05cdff0   edx: 00000286
Jun 14 17:21:26 qacluster4 kernel: esi: f7101200   edi: c5ab532c   ebp: f762a000   esp: f762ad68
Jun 14 17:21:26 qacluster4 kernel: ds: 007b   es: 007b   ss: 0068
Jun 14 17:21:26 qacluster4 kernel: Process kjournald (pid: 6953, threadinfo=f762a000 task=f7628020)
Jun 14 17:21:26 qacluster4 kernel: Stack: c0529ac0 c05016f5 c051ddd3 00000273 c051de23 f7f7b500 f7101200 c026611a 
Jun 14 17:21:26 qacluster4 kernel:        f7101200 f7f7b500 f24b668c f7f7b500 c026584a c5ab532c c5ab532c c02660c8 
Jun 14 17:21:26 qacluster4 kernel:        c5ab532c c5ab532c 00000001 da2cf780 da2cf780 e5911d4c f764c480 00000000 
Jun 14 17:21:26 qacluster4 kernel: Call Trace:
Jun 14 17:21:26 qacluster4 kernel:  [<c026611a>] __journal_remove_checkpoint+0x4a/0xa0
Jun 14 17:21:26 qacluster4 kernel:  [<c026584a>] __try_to_free_cp_buf+0x5a/0x90
Jun 14 17:21:26 qacluster4 kernel:  [<c02660c8>] __journal_clean_checkpoint_list+0x98/0xa0
Jun 14 17:21:26 qacluster4 kernel:  [<c0263ee9>] journal_commit_transaction+0x1d9/0x11d0
Jun 14 17:21:26 qacluster4 kernel:  [<c01bcc50>] autoremove_wake_function+0x0/0x60
Jun 14 17:21:26 qacluster4 kernel:  [<c01bcc50>] autoremove_wake_function+0x0/0x60
Jun 14 17:21:26 qacluster4 kernel:  [<c01a3b93>] scheduler_tick+0x63/0x320
Jun 14 17:21:26 qacluster4 kernel:  [<c01a30c4>] find_busiest_group+0xd4/0x300
Jun 14 17:21:26 qacluster4 kernel:  [<c01a337c>] find_busiest_queue+0x8c/0xb0
Jun 14 17:21:26 qacluster4 kernel:  [<c01a35ab>] load_balance_newidle+0x8b/0xa0
Jun 14 17:21:26 qacluster4 kernel:  [<c01a275c>] finish_task_switch+0x3c/0x90
Jun 14 17:21:26 qacluster4 kernel:  [<c04f3142>] schedule+0x3e2/0xc90
Jun 14 17:21:26 qacluster4 kernel:  [<c01b0b3c>] del_timer_sync+0x9c/0xe0
Jun 14 17:21:26 qacluster4 kernel:  [<c02672d5>] kjournald+0xe5/0x250
Jun 14 17:21:26 qacluster4 kernel:  [<c01bcc50>] autoremove_wake_function+0x0/0x60
Jun 14 17:21:26 qacluster4 kernel:  [<c01bcc50>] autoremove_wake_function+0x0/0x60
Jun 14 17:21:26 qacluster4 kernel:  [<c018a076>] ret_from_fork+0x6/0x20
Jun 14 17:21:26 qacluster4 kernel:  [<c02671d0>] commit_timeout+0x0/0x10
Jun 14 17:21:26 qacluster4 kernel:  [<c02671f0>] kjournald+0x0/0x250
Jun 14 17:21:26 qacluster4 kernel:  [<c0188375>] kernel_thread_helper+0x5/0x10
Jun 14 17:21:26 qacluster4 kernel: Code: 52 c0 b8 23 de 51 c0 89 44 24 10 b8 73 02 00 00 89 44 24 0c b8 d3 dd 51 c0 89 44 24 08 b8 f5 16 50 c0 89 44 24 04 e8 60 19 f4 ff <0f> 0b 73 02 d3 dd 51 c0 e9 ba fd ff ff 8d 76 00 c7 04 24 c0 9a 
Jun 15 12:21:43 qacluster4 syslogd 1.4.1: restart.
Jun 15 12:21:43 qacluster4 syslog: syslogd startup succeeded
Jun 15 12:21:43 qacluster4 kernel: klogd 1.4.1, log source = /proc/kmsg started.
Jun 15 12:21:43 qacluster4 kernel: Linux version 2.6.11.12-grsec ([email protected]) (gcc version 3.3.5 (Gentoo Linux 3.3.5-r1, ssp-3.3.2-3, pie-8.7.7.1)) #1 SMP Mon Jun 13 11:17:39 MDT 2005
#1 lore raw
Quoted 4 lines · * David Wilk ([email protected]) wrote:

We've been plagued buy this ext3 bug since 2.6.10, and it only happens on heavily loaded postgres systems. We run our postgres DB on ext3 data=journal on a dmcrypt partition. Our kernel is also patched with grsec, but that doesn't appear to play any role.

Can you recreate w/out grsec, and w/out dm-crypt? IOW, just ext3 plus your load?

Quoted 9 lines

After upgrading to 2.6.11.12 (specifically for the ext3 checkpoint.c fix) we noticed two things. The assertion failure persists, and now we get a condition where a postgres process will spin in state 'D' forever and hog 100% of a CPU (in system, not user).

I've attached the trace in plain text so the formatting doesn't get screwed.

Let me know if anyone would like more information. I'm no programmer, but I'd like to help in any way that I can.

Would you mind trying this patch:

From: Jan Kara <[email protected]>

On one path, cond_resched_lock() fails to return true if it dropped the lock. We think this might be causing the crashes in JBD's log_do_checkpoint().

(chrisw: backport to 2.6.11.12)
---

 kernel/sched.c |    7 +++++--
 1 files changed, 5 insertions(+), 2 deletions(-)
kernel/sched.c +5 -2
Index: release-2.6.11/kernel/sched.c
===================================================================
@@ -3788,11 +3788,14 @@ EXPORT_SYMBOL(cond_resched);
  */
 int cond_resched_lock(spinlock_t * lock)
 {
+	int ret = 0;
+
 #if defined(CONFIG_SMP) && defined(CONFIG_PREEMPT)
 	if (lock->break_lock) {
 		lock->break_lock = 0;
 		spin_unlock(lock);
 		cpu_relax();
+		ret = 1;
 		spin_lock(lock);
 	}
 #endif
@@ -3800,10 +3803,10 @@ int cond_resched_lock(spinlock_t * lock)
 		_raw_spin_unlock(lock);
 		preempt_enable_no_resched();
 		__cond_resched();
+		ret = 1;
 		spin_lock(lock);
-		return 1;
 	}
-	return 0;
+	return ret;
 }
 
 EXPORT_SYMBOL(cond_resched_lock);