Skip to contentSkip to conversation

BUG: soft lockup detected on CPU#1!

DISCUSSION [email protected] 5 participants 15 messages Feb 11, 2009 – Feb 19, 2009

lore
Share & tools
lore
#0 lore raw

Hello,

I am running the 2.6.28 based xfs kernel driver on a custom kernel with following kernel config enabled.

CONFIG_PREEMPT
CONFIG_DETECT_SOFTLOCKUP

Running the following xfsqa causes a soft lockup. The configuration is a x86 with Hyperthreading, 4GB RAM and a AHCI connected JBOD. Its 100% reproducible.

Any suggestions/inputs on where to start debugging the problem would be much appreciated.

#! /bin/sh
# FS QA Test No. 008
#

# randholes test
#
BUG: soft lockup detected on CPU#1!
 [<4013d525>] softlockup_tick+0x9c/0xaf
 [<40123246>] update_process_times+0x3d/0x60
 [<401100ab>] smp_apic_timer_interrupt+0x52/0x58
 [<40103633>] apic_timer_interrupt+0x1f/0x24
 [<402a1557>] _spin_lock_irqsave+0x48/0x61
 [<f8b8fe30>] xfs_iflush_cluster+0x16d/0x31c [xfs]
 [<f8b9018b>] xfs_iflush+0x1ac/0x271 [xfs]
 [<f8ba49a1>] xfs_inode_flush+0xd6/0xfa [xfs]
 [<f8bb13c8>] xfs_fs_write_inode+0x27/0x40 [xfs]
 [<401789d9>] __writeback_single_inode+0x1b0/0x2ff
 [<40101ad5>] __switch_to+0x23/0x1f9
 [<40178f87>] sync_sb_inodes+0x196/0x261
 [<4017920a>] writeback_inodes+0x67/0xb1
 [<401465df>] wb_kupdate+0x7b/0xe0
 [<40146bc3>] pdflush+0x0/0x1b5
 [<40146ce1>] pdflush+0x11e/0x1b5
 [<40146564>] wb_kupdate+0x0/0xe0
 [<4012be6d>] kthread+0xc1/0xec
 [<4012bdac>] kthread+0x0/0xec
 [<401038b3>] kernel_thread_helper+0x7/0x10
 =======================

Thanks,
Rakesh

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw
Quoted 51 lines · On Tue, 10 Feb 2009, [email protected] wrote:

Hello,

I am running the 2.6.28 based xfs kernel driver on a custom kernel with following kernel config enabled.

CONFIG_PREEMPT
CONFIG_DETECT_SOFTLOCKUP

Running the following xfsqa causes a soft lockup. The configuration is a x86 with Hyperthreading, 4GB RAM and a AHCI connected JBOD. Its 100% reproducible.

Any suggestions/inputs on where to start debugging the problem would be much appreciated.

#! /bin/sh
# FS QA Test No. 008
#

# randholes test
#
BUG: soft lockup detected on CPU#1!
[<4013d525>] softlockup_tick+0x9c/0xaf
[<40123246>] update_process_times+0x3d/0x60
[<401100ab>] smp_apic_timer_interrupt+0x52/0x58
[<40103633>] apic_timer_interrupt+0x1f/0x24
[<402a1557>] _spin_lock_irqsave+0x48/0x61
[<f8b8fe30>] xfs_iflush_cluster+0x16d/0x31c [xfs]
[<f8b9018b>] xfs_iflush+0x1ac/0x271 [xfs]
[<f8ba49a1>] xfs_inode_flush+0xd6/0xfa [xfs]
[<f8bb13c8>] xfs_fs_write_inode+0x27/0x40 [xfs]
[<401789d9>] __writeback_single_inode+0x1b0/0x2ff
[<40101ad5>] __switch_to+0x23/0x1f9
[<40178f87>] sync_sb_inodes+0x196/0x261
[<4017920a>] writeback_inodes+0x67/0xb1
[<401465df>] wb_kupdate+0x7b/0xe0
[<40146bc3>] pdflush+0x0/0x1b5
[<40146ce1>] pdflush+0x11e/0x1b5
[<40146564>] wb_kupdate+0x0/0xe0
[<4012be6d>] kthread+0xc1/0xec
[<4012bdac>] kthread+0x0/0xec
[<401038b3>] kernel_thread_helper+0x7/0x10
=======================

Thanks,
Rakesh

Quoted 5 lines

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

There were some pretty nasty bugs in 2.6.28 for XFS, can you reproduce it on 2.6.28.4?

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#2 lore raw

Hi Justin,

Thank you for the pointer. Well as suggested I did and now the xfsqa goes upto test 011

#! /bin/sh
# FS QA Test No. 011
#

# dirstress

but dies with a oops. Any suggestions?

Here is the oops trace -

BUG: unable to handle kernel NULL pointer dereference

at virtual address 00000000
printing eip:
f8bd02c2
*pde = e7167067

Oops: 0000 [#1]
PREEMPT SMP 
last sysfs file:
/devices/pci0000:00/0000:00:1f.3/i2c-0/0-002e/temp1_input
Modules linked in: xfs sg sunrpc m24c02 pca9554
pca9555 mcp23016 lm85 hwmon_vid i2c_i801 i2c_core
midplane uhci_hcd sk98lin tg3 e1000 mv_sata sd_mod
ahci libata
CPU:    1
EIP:    0060:[<f8bd02c2>]    Not tainted VLI
EFLAGS: 00010286   (2.6.18.rhel5 #2) 

EIP is at xfs_iget_core+0x4d6/0x5e9 [xfs]
eax: 00000000 ebx: e8ae7850 ecx: e79b2000 edx:
00000000
esi: e8ae7920 edi: ea44d2d0 ebp: ea44d298 esp:
e79b3ce4
ds: 007b es: 007b ss: 0068
Process dirstress (pid: 9927, ti=e79b2000
task=e790faa0 task.ti=e79b2000)

Stack: ea7b0034 e9072c00 ea2a6380 0003b84a e8ae7850
ea2a6380 02c9f600 00000004 
       e9072c00 f8bd0478 1003b84a 00000000 00000001
00000004 e79b3d5c 00000000 
       00000000 ea7b0034 e9072c00 ea7b0034 1003b84a
00000000 f8bebbf1 1003b84a 
Call Trace:
 [<f8bd0478>] xfs_iget+0xa3/0x12f [xfs]
 [<f8bebbf1>] xfs_trans_iget+0x1bd/0x249 [xfs]
 [<f8bd4ebe>] xfs_ialloc+0xb9/0x5a9 [xfs]
 [<f8bdc777>] xlog_grant_push_ail+0x105/0x12c [xfs]
 [<f8bec673>] xfs_dir_ialloc+0x7e/0x26b [xfs]
 [<f8be8c81>] xfs_trans_reserve+0x15c/0x240 [xfs]
 [<f8bf12c8>] xfs_symlink+0x34a/0x862 [xfs]
 [<401a62dd>] avc_has_perm_noaudit+0x38f/0x42d
 [<401a6e9d>] avc_has_perm+0x3b/0x46
 [<f8bf9d8e>] xfs_vn_symlink+0x6a/0xad [xfs]
 [<4016a06d>] vfs_symlink+0xb2/0x11a
 [<4016a149>] sys_symlinkat+0x74/0xab
 [<402a14c2>] _spin_lock+0xd/0x5a
 [<402a1593>] _spin_unlock+0xd/0x21
 [<4016f416>] dput+0x83/0x11c
 [<4015c89d>] __fput+0x152/0x175
 [<4016a18f>] sys_symlink+0xf/0x13
 [<40102b73>] syscall_call+0x7/0xb
 =======================
Code: 8b 40 08 a8 08 74 05 e8 6d ff 6c 47 8b 44 24 04

05 38 01 00 00 e8 24 02 6d 47 8b 44 24 04 8b 98 2c 01 00 00 85 db 74 3d 8b 43 04 <39> 18 74 14 b9 14 01 00 00 ba 20 8a c0 f8 b8 d8 8c c0 f8 e8 32
EIP: [<f8bd02c2>] xfs_iget_core+0x4d6/0x5e9 [xfs] SS:ESP 0068:e79b3ce4
<0>Kernel panic - not syncing: Fatal exception

Thanks,
Rakesh

Quoted 65 lines · --- Justin Piszcz
Quoted 4 lines · On Tue, 10 Feb 2009, [email protected] wrote:

Hello,

I am running the 2.6.28 based xfs kernel driver on

a

Quoted 1 line

custom kernel with following kernel config

enabled.

Quoted 5 lines

CONFIG_PREEMPT
CONFIG_DETECT_SOFTLOCKUP

Running the following xfsqa causes a soft lockup.

The

Quoted 1 line

configuration is a x86 with Hyperthreading, 4GB

RAM

Quoted 3 lines

and a AHCI connected JBOD. Its 100% reproducible.

Any suggestions/inputs on where to start debugging

the

Quoted 37 lines

problem would be much appreciated.

#! /bin/sh
# FS QA Test No. 008
#

# randholes test
#
BUG: soft lockup detected on CPU#1!
[<4013d525>] softlockup_tick+0x9c/0xaf
[<40123246>] update_process_times+0x3d/0x60
[<401100ab>] smp_apic_timer_interrupt+0x52/0x58
[<40103633>] apic_timer_interrupt+0x1f/0x24
[<402a1557>] _spin_lock_irqsave+0x48/0x61
[<f8b8fe30>] xfs_iflush_cluster+0x16d/0x31c [xfs]
[<f8b9018b>] xfs_iflush+0x1ac/0x271 [xfs]
[<f8ba49a1>] xfs_inode_flush+0xd6/0xfa [xfs]
[<f8bb13c8>] xfs_fs_write_inode+0x27/0x40 [xfs]
[<401789d9>] __writeback_single_inode+0x1b0/0x2ff
[<40101ad5>] __switch_to+0x23/0x1f9
[<40178f87>] sync_sb_inodes+0x196/0x261
[<4017920a>] writeback_inodes+0x67/0xb1
[<401465df>] wb_kupdate+0x7b/0xe0
[<40146bc3>] pdflush+0x0/0x1b5
[<40146ce1>] pdflush+0x11e/0x1b5
[<40146564>] wb_kupdate+0x0/0xe0
[<4012be6d>] kthread+0xc1/0xec
[<4012bdac>] kthread+0x0/0xec
[<401038b3>] kernel_thread_helper+0x7/0x10
=======================

Thanks,
Rakesh

Quoted 5 lines

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Quoted 4 lines

There were some pretty nasty bugs in 2.6.28 for XFS, can you reproduce it on
2.6.28.4?

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#12 lore raw
In reply to Justin Piszcz

With debug enabled it fails with this -

BUG: unable to handle kernel NULL pointer dereference

at virtual address 00000000
printing eip:
f8bd02c2
*pde = d658e067
Assertion failed: atomic_read(&ip->i_pincount) > 0,
file: fs/xfs/xfs_inode.c, line: 2703

------------[ cut here ]------------
Kernel BUG at [verbose debug info unavailable]

invalid opcode: 0000 [#1]
PREEMPT SMP
last sysfs file:
/devices/pci0000:00/0000:00:1f.3/i2c-0/0-002e/temp2_input

Modules linked in: sg xfs sunrpc m24c02 pca9554
pca9555 mcp23016 lm85 hwmon_vid i2c_i801 i2c_core
midplane uhci_hcd sk98lin tg3 e1000 mv_sata sd_mod
ahci libata
CPU:    0
EIP:    0060:[<f8bfde19>]    Not tainted VLI
EFLAGS: 00010296   (2.6.18.rhel5 #2) 

EIP is at assfail+0xd/0x13 [xfs]
eax: 0000005c ebx: e1fb4980 ecx: e4f46000 edx:
00000000
esi: 00000007 edi: e1fb0a58 ebp: 00000007 esp:
e4f47eb8
ds: 007b es: 007b ss: 0068
Process xfslogd/0 (pid: 4370, ti=e4f46000 task=75118aa0 task.ti=e4f46000)

Stack: f8c10ccb f8c090ae f8c08d97 00000a8f f8bd0dca
00000526 f8be8e28 00000526 
       00000007 e9db1d80 e861c008 e9db1da0 e861c000
00000003 e9db1d80 e9db1ca4 
       e9db1ca0 00000000 f8be8f56 00000000 00000000
e9db1ca4 ea449880 ea449800 
Call Trace:
 [<f8bd0dca>] xfs_iunpin+0x21/0x49 [xfs]
 [<f8be8e28>] xfs_trans_chunk_committed+0xc3/0xe6
[xfs]
 [<f8be8f56>] xfs_trans_committed+0x38/0xd1 [xfs]
 [<f8bdba5c>] xlog_state_do_callback+0x1b7/0x329 [xfs]
 [<f8bf66b7>] xfs_buf_iodone_work+0x41/0x63 [xfs]
 [<401294c5>] run_workqueue+0x71/0xae
 [<f8bf6676>] xfs_buf_iodone_work+0x0/0x63 [xfs]
 [<40129666>] worker_thread+0xd9/0x10a
 [<40116ca2>] default_wake_function+0x0/0xc
 [<4012958d>] worker_thread+0x0/0x10a
 [<4012be6d>] kthread+0xc1/0xec
 [<4012bdac>] kthread+0x0/0xec
 [<401038b3>] kernel_thread_helper+0x7/0x10
 =======================
Code: d2 df 51 47 89 ea b8 94 59 c2 f8 e8 a9 37 6a 47

83 c4 0c 85 ff 75 02 0f 0b 5b 5e 5f 5d c3 51 52 50 68 cb 0c c1 f8 e8 ab df 51 47 <0f> 0b 83 c4 10 c3 55 57 56 53 83 ec 14 89 44 24 04 89 d7 89 cd
EIP: [<f8bfde19>] assfail+0xd/0x13 [xfs] SS:ESP 0068:e4f47eb8
<0>Kernel panic - not syncing: Fatal exception

Quoted 65 lines · --- Justin Piszcz
Quoted 4 lines · On Tue, 10 Feb 2009, [email protected] wrote:

Hello,

I am running the 2.6.28 based xfs kernel driver on

a

Quoted 1 line

custom kernel with following kernel config

enabled.

Quoted 5 lines

CONFIG_PREEMPT
CONFIG_DETECT_SOFTLOCKUP

Running the following xfsqa causes a soft lockup.

The

Quoted 1 line

configuration is a x86 with Hyperthreading, 4GB

RAM

Quoted 3 lines

and a AHCI connected JBOD. Its 100% reproducible.

Any suggestions/inputs on where to start debugging

the

Quoted 37 lines

problem would be much appreciated.

#! /bin/sh
# FS QA Test No. 008
#

# randholes test
#
BUG: soft lockup detected on CPU#1!
[<4013d525>] softlockup_tick+0x9c/0xaf
[<40123246>] update_process_times+0x3d/0x60
[<401100ab>] smp_apic_timer_interrupt+0x52/0x58
[<40103633>] apic_timer_interrupt+0x1f/0x24
[<402a1557>] _spin_lock_irqsave+0x48/0x61
[<f8b8fe30>] xfs_iflush_cluster+0x16d/0x31c [xfs]
[<f8b9018b>] xfs_iflush+0x1ac/0x271 [xfs]
[<f8ba49a1>] xfs_inode_flush+0xd6/0xfa [xfs]
[<f8bb13c8>] xfs_fs_write_inode+0x27/0x40 [xfs]
[<401789d9>] __writeback_single_inode+0x1b0/0x2ff
[<40101ad5>] __switch_to+0x23/0x1f9
[<40178f87>] sync_sb_inodes+0x196/0x261
[<4017920a>] writeback_inodes+0x67/0xb1
[<401465df>] wb_kupdate+0x7b/0xe0
[<40146bc3>] pdflush+0x0/0x1b5
[<40146ce1>] pdflush+0x11e/0x1b5
[<40146564>] wb_kupdate+0x0/0xe0
[<4012be6d>] kthread+0xc1/0xec
[<4012bdac>] kthread+0x0/0xec
[<401038b3>] kernel_thread_helper+0x7/0x10
=======================

Thanks,
Rakesh

Quoted 5 lines

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Quoted 4 lines

There were some pretty nasty bugs in 2.6.28 for XFS, can you reproduce it on
2.6.28.4?

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#3 lore raw
In reply to [email protected]
Quoted 31 lines · On Wed, 11 Feb 2009, [email protected] wrote:

Hi Justin,

Thank you for the pointer. Well as suggested I did and now the xfsqa goes upto test 011

#! /bin/sh
# FS QA Test No. 011
#

# dirstress

but dies with a oops. Any suggestions?

Here is the oops trace -

BUG: unable to handle kernel NULL pointer dereference

at virtual address 00000000
printing eip:
f8bd02c2
*pde = e7167067

Oops: 0000 [#1]
PREEMPT SMP
last sysfs file:
/devices/pci0000:00/0000:00:1f.3/i2c-0/0-002e/temp1_input
Modules linked in: xfs sg sunrpc m24c02 pca9554
pca9555 mcp23016 lm85 hwmon_vid i2c_i801 i2c_core
midplane uhci_hcd sk98lin tg3 e1000 mv_sata sd_mod
ahci libata
CPU:    1
EIP:    0060:[<f8bd02c2>]    Not tainted VLI
EFLAGS: 00010286   (2.6.18.rhel5 #2)
Quoted 1 line

EIP is at xfs_iget_core+0x4d6/0x5e9 [xfs]

Can you show uname -a output?

EFLAGS: 00010286 (2.6.18.rhel5 #2)

                    ^^^^^^^^^^^^

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#4 lore raw

Hi Justin,

Yes it is a 2.6.18 rhel5 based custom kernel but the XFS driver is a back port from the 2.6.28.4

Thanks,
Rakesh

Quoted 15 lines · --- Justin Piszcz
Quoted 4 lines · On Wed, 11 Feb 2009, [email protected] wrote:

Hi Justin,

Thank you for the pointer. Well as suggested I did

and

Quoted 7 lines

now the xfsqa goes upto test 011

#! /bin/sh
# FS QA Test No. 011
#

# dirstress
Quoted 15 lines
Quoted 5 lines

but dies with a oops. Any suggestions?

Here is the oops trace -

BUG: unable to handle kernel NULL pointer

dereference

Quoted 8 lines

at virtual address 00000000
printing eip:
f8bd02c2
*pde = e7167067

Oops: 0000 [#1]

PREEMPT SMP
last sysfs file:

/devices/pci0000:00/0000:00:1f.3/i2c-0/0-002e/temp1_input

Quoted 9 lines
Quoted 8 lines
Modules linked in: xfs sg sunrpc m24c02 pca9554
pca9555 mcp23016 lm85 hwmon_vid i2c_i801 i2c_core
midplane uhci_hcd sk98lin tg3 e1000 mv_sata sd_mod
ahci libata
CPU:    1
EIP:    0060:[<f8bd02c2>]    Not tainted VLI
EFLAGS: 00010286   (2.6.18.rhel5 #2)

EIP is at xfs_iget_core+0x4d6/0x5e9 [xfs]

Quoted 5 lines

Can you show uname -a output?

EFLAGS: 00010286 (2.6.18.rhel5 #2)

                     ^^^^^^^^^^^^

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#13 lore raw
In reply to [email protected]
Quoted 6 lines · On Tue, Feb 10, 2009 at 11:16:25PM -0800, [email protected] wrote:

Hello,

I am running the 2.6.28 based xfs kernel driver on a custom kernel with following kernel config enabled.

Quoted 6 lines

CONFIG_PREEMPT
CONFIG_DETECT_SOFTLOCKUP

Running the following xfsqa causes a soft lockup. The configuration is a x86 with Hyperthreading, 4GB RAM and a AHCI connected JBOD. Its 100% reproducible.

Is the system making progress, or has it hung? i.e. does the test complete?

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#5 lore raw
In reply to [email protected]
Quoted 5 lines · On Thu, Feb 12, 2009 at 01:22:16AM -0800, [email protected] wrote:

Hi Justin,

Yes it is a 2.6.18 rhel5 based custom kernel but the XFS driver is a back port from the 2.6.28.4

Then you get to keep all the broken bits to yourself. If you want to throw random versions of XFS at random versions of kernels then we can't help you - we don't have the time or resources to support random backports of XFS to older kernels (and non-vanilla kernels at that).

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#14 lore raw
In reply to Dave Chinner

Hi Dave,

No it does not. Just sits there any access to mount point (ll) also blocks forever.

I must point out that I brought the changes from 2.6.28.4 xfs into my private space and that seems to show improvement on test 008 however now test 011 dies with kernel oops.

Please see my reply posts.

Thanks,
Rakesh

Quoted 19 lines · --- Dave Chinner
Quoted 4 lines · On Tue, Feb 10, 2009 at 11:16:25PM -0800, [email protected] wrote:

Hello,

I am running the 2.6.28 based xfs kernel driver on

a

Quoted 1 line

custom kernel with following kernel config

enabled.

Quoted 5 lines

CONFIG_PREEMPT
CONFIG_DETECT_SOFTLOCKUP

Running the following xfsqa causes a soft lockup.

The

Quoted 1 line

configuration is a x86 with Hyperthreading, 4GB

RAM

Quoted 1 line

and a AHCI connected JBOD. Its 100% reproducible.

Quoted 11 lines

Is the system making progress, or has it hung? i.e. does the test
complete?

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#6 lore raw
In reply to Dave Chinner

Well the problem is the older kernel XFS driver is buggy to such a large extent that there is data loss even for data on rest should a power loss occur.

With a newer version back port I can preserve the kernel version change since it becomes far more reaching to the other kernel components and they have to move, to which ..... there is strong reservation.

Hope this gives you the perspective.

Thanks,
Rakesh

Quoted 19 lines · --- Dave Chinner
Quoted 4 lines · On Thu, Feb 12, 2009 at 01:22:16AM -0800, [email protected] wrote:

Hi Justin,

Yes it is a 2.6.18 rhel5 based custom kernel but

the

Quoted 1 line

XFS driver is a back port from the 2.6.28.4

Then you get to keep all the broken bits to yourself. If you want
to throw random versions of XFS at random versions of kernels then
we can't help you - we don't have the time or resources to support
random backports of XFS to older kernels (and non-vanilla kernels
at that).

Quoted 7 lines

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#7 lore raw
Quoted 10 lines · [email protected] wrote:

Well the problem is the older kernel XFS driver is buggy to such a large extent that there is data loss even for data on rest should a power loss occur.

With a newer version back port I can preserve the kernel version change since it becomes far more reaching to the other kernel components and they have to move, to which ..... there is strong reservation.

Hope this gives you the perspective.

It's totally understandable why you might want to do it.

It's also totally understandable why upstream developers can't spend a lot of time on your custom codebase.

What you need, of course, is a distribution with good support for xfs, so you can make it Someone Else's Problem. :)

-Eric

Quoted 2 lines

Thanks,
Rakesh

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#8 lore raw

Guys,

Thank you for taking the time to write. Having said where I stand and we are kind of on the same page. Is there something I can expect which would put me on a track of nailing down the problem. It maybe a wild goose chase but something that I can start with would be much appreciated.

Unfortunately there is no distro which gets closer to where mainline lives today. Reading the changelog there are several problems that I have already come across and has convincingly driven me to take on this task.

Thanks,
Rakesh

Quoted 22 lines · --- Eric Sandeen
Quoted 2 lines · [email protected] wrote:

Well the problem is the older kernel XFS driver is buggy to such a large extent that there is data

loss

Quoted 5 lines

even for data on rest should a power loss occur.

With a newer version back port I can preserve the kernel version change since it becomes far more reaching to the other kernel components and they

have

Quoted 1 line

to move, to which ..... there is strong

reservation.

Quoted 2 lines

Hope this gives you the perspective.

It's totally understandable why you might want to do it.

It's also totally understandable why upstream
developers can't spend a
lot of time on your custom codebase.

Quoted 12 lines

What you need, of course, is a distribution with
good support for xfs,
so you can make it Someone Else's Problem. :)

-Eric

Quoted 2 lines

Thanks,
Rakesh

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#9 lore raw
Quoted 8 lines · [email protected] wrote:

Guys,

Thank you for taking the time to write. Having said where I stand and we are kind of on the same page. Is there something I can expect which would put me on a track of nailing down the problem. It maybe a wild goose chase but something that I can start with would be much appreciated.

Just random debugging thoughts...

Try stock 2.6.28.4, to see if you have the same problem. If so, and esp. if you also see it on 2.6.29, then you'll get a lot more attention here. :)

If not, then it's something with your backport most likely. Figure out what you had to backport and see if it's possibly causing the error(s).

If it's locked up, try sysrq-w (echo w > /proc/sysrq-trigger) and look at dmesg to see if other threads are locked against it. Figure out why.

On the oops try memory debugging etc, see if you're referencing freed memory, using corrupt lists, etc.

Look for other errors in the logs prior to this.

See if your filesystem is corrupted.

Bug Red Hat for XFS support, assuming you're actually buying RHEL5 support from them. :)

Quoted 5 lines

Unfortunately there is no distro which gets closer to where mainline lives today. Reading the changelog there are several problems that I have already come across and has convincingly driven me to take on this task.

well certainly there are distros with kernels newer than 2.6.18, but it depends on your needs & goals I guess.

Good luck,
-Eric

-Eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#11 lore raw
In reply to [email protected]
Quoted 2 lines · On Donnerstag 12 Februar 2009 [email protected] wrote:

Unfortunately there is no distro which gets closer to where mainline lives today.

If I understand correctly, I would say use openSUSE 11.1, which has kernel 2.6.27.7-9 as of today (and full XFS support).

And you can still download a newer vanilla kernel and use it on your distro. You do not need to use the distro kernel, you just need to install kernel updates on your own then.

mfg zmi

Signature · 10 lines
// Michael Monnerie, Ing.BSc    -----      http://it-management.at
// Tel: 0660 / 415 65 31                      .network.your.ideas.
// PGP Key:         "curl -s http://zmi.at/zmi.asc | gpg --import"
// Fingerprint: AC19 F9D5 36ED CD8A EF38  500E CE14 91F7 1C12 09B4
// Keyserver: wwwkeys.eu.pgp.net                  Key-ID: 1C1209B4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#10 lore raw
In reply to Eric Sandeen

I think I am getting closer now. Can you suggest what to look for from this oops -

Stack traceback for pid 192
0xad12f030 192 11 1 0 R 0xad12f1d0
*xfsdatad/0
esp eip Function (args)
0xaff37eec 0x7815007b map_vm_area+0xc3
0xaff37ef0 0x7814007b find_get_pages_contig+0x3d
0xaff37ef8 0x78156c04 free_block+0x41
0xaff37f20 0x78156d9b cache_flusharray+0x63
0xaff37f3c 0x78156b4e kmem_cache_free+0x52
0xaff37f4c 0x7814296f mempool_free_slab+0xb
0xaff37f50 0x78142954 mempool_free+0x60
0xaff37f60 0x781f429f xfs_destroy_ioend+0x4e
0xaff37f6c 0x781f43d0 xfs_end_bio_read+0x5
0xaff37f70 0x78128498 run_workqueue+0x71
0xaff37f74 0x781f43cb xfs_end_bio_read
0xaff37f8c 0x78128634 worker_thread+0xd9
0xaff37fac 0x781164c0 default_wake_function
0xaff37fc8 0x7812855b worker_thread
0xaff37fcc 0x7812ad7a kthread+0xc1
0xaff37fd8 0x7812acb9 kthread
0xaff37fe4 0x781036df kernel_thread_helper+0x7

Thanks,
Rakesh

Quoted 59 lines · --- Eric Sandeen
Quoted 3 lines · [email protected] wrote:

Guys,

Thank you for taking the time to write. Having

said

Quoted 1 line

where I stand and we are kind of on the same page.

Is

Quoted 1 line

there something I can expect which would put me on

a

Quoted 2 lines

track of nailing down the problem. It maybe a wild goose chase but something that I can start with

would

Quoted 1 line

be much appreciated.

Just random debugging thoughts...

Try stock 2.6.28.4, to see if you have the same problem. If so, and
esp. if you also see it on 2.6.29, then you'll get a lot more attention
here. :)

If not, then it's something with your backport most likely. Figure out
what you had to backport and see if it's possibly causing the error(s).

If it's locked up, try sysrq-w (echo w > /proc/sysrq-trigger) and look
at dmesg to see if other threads are locked against it. Figure out why.

On the oops try memory debugging etc, see if you're referencing freed
memory, using corrupt lists, etc.

Look for other errors in the logs prior to this.

See if your filesystem is corrupted.

Bug Red Hat for XFS support, assuming you're actually buying RHEL5
support from them. :)

Quoted 1 line

Unfortunately there is no distro which gets closer

to

Quoted 2 lines

where mainline lives today. Reading the changelog
there are several problems that I have already

come

Quoted 1 line

across and has convincingly driven me to take on

this

Quoted 1 line

task.

well certainly there are distros with kernels newer than 2.6.18, but it
depends on your needs & goals I guess.

Good luck,

Review

Quoted 3 lines
(quoted diff) +0 -2
-Eric
 
-Eric

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs