kernels 3.4 slower due to allocation workqueue

Hello,
last week we received new machines (DELL R720xd) for an extension of our ceph cluster.
(64 Gb ram, 2x Xeon E5-2650, PERC H710P (really LSI MEGARAID), and 12x3 TB disks + 2SSD (not used as cachecade))

I was doing test on the raid card with kernel 3.4.38 to try to find what I can get of this beast with RAID5, when I noticed an unusual slow values on compilebench. The difference is very visible on the initial create tests (can detail more if needed).

I finally observed that ONLY 3.4 kernels exhibit that behaviour ; 3.3.xxx and before are OK, 3.5.xxx and later are back to good values.

I bisected the problem to this commit

c999a223c2f0d31c64ef7379814cea1378b2b800 is the first bad commit commit c999a223c2f0d31c64ef7379814cea1378b2b800
Author: Dave Chinner <[email protected]>
Date: Thu Mar 22 05:15:07 2012 +0000

xfs: introduce an allocation workqueue

I understand this regression is not a bug, and probably just a corner case of the new code, that was certainly corrected after during 3.5 development (didn't tried to bisect this one, maybe dave know what is the corrective patch ?)

The problem is that 3.4 is the last long-term kernel for the moment, and it's unfortunate it shows this regression.

Maybe a backport of the fix (if this backport is possible AND not very intrusive) could be a good idea ?

Cheers,

Here are the allocation worker changes.

The biggest performance commit should be aa292847, which limits the callers to the worker.

commit 3b876c8f2a361ceeed3fed894980c69066f903a0

Author: Jeff Liu <[email protected]>
Date:   Thu Jun 7 15:44:32 2012 +0800

xfs: fix debug_object WARN at xfs_alloc_vextent()

commit aa292847b9fc6e187547110de833a7d3131bbddf

Author: Dave Chinner <[email protected]>
Date:   Thu Jul 12 07:40:43 2012 +1000

xfs: don't defer metadata allocation to the workqueue

commit 2455881c0b52f87be539c4c7deab1afff4d8a560

Author: Dave Chinner <[email protected]>
Date:   Fri Oct 5 11:06:58 2012 +1000

xfs: introduce XFS_BMAPI_STACK_SWITCH

commit e04426b9202bccd4cfcbc70b2fa2aeca1c86d8f5

Author: Dave Chinner <[email protected]>
Date:   Fri Oct 5 11:06:59 2012 +1000

xfs: move allocation stack switch up to xfs_bmapi_allocate

commit 9e96fe6df44425b69ed89f6ac20352cec1f127d7

Author: Brian Foster <[email protected]>
Date:   Thu Jan 17 13:11:29 2013 -0500

xfs: pull up stack_switch check into xfs_bmapi_write

The last 3 patches address an AGF buffer hang with the allocation worker.

--Mark.

Quoted 3 lines · Mark Tinguely

Here are the allocation worker changes.

Hello mark, thanks for you answer

Quoted 2 lines

The biggest performance commit should be aa292847, which limits the callers to the worker.

Quoted 3 lines · Mark Tinguely

Here are the allocation worker changes.

Hello mark, thanks for you answer

Quoted 7 lines

The biggest performance commit should be aa292847, which limits the callers to the worker.

commit 3b876c8f2a361ceeed3fed894980c69066f903a0
Author: Jeff Liu <[email protected]>
Date: Thu Jun 7 15:44:32 2012 +0800

Quoted 8 lines

xfs: fix debug_object WARN at xfs_alloc_vextent()

commit aa292847b9fc6e187547110de833a7d3131bbddf
Author: Dave Chinner <[email protected]>
Date: Thu Jul 12 07:40:43 2012 +1000

xfs: don't defer metadata allocation to the workqueue

Only These 2 commits are candidates, the others are post 3.5. I'll try to patch a 3.4 with each patch.

Quoted 19 lines

commit 2455881c0b52f87be539c4c7deab1afff4d8a560
Author: Dave Chinner <[email protected]>
Date: Fri Oct 5 11:06:58 2012 +1000

xfs: introduce XFS_BMAPI_STACK_SWITCH

commit e04426b9202bccd4cfcbc70b2fa2aeca1c86d8f5
Author: Dave Chinner <[email protected]>
Date: Fri Oct 5 11:06:59 2012 +1000

xfs: move allocation stack switch up to xfs_bmapi_allocate

commit 9e96fe6df44425b69ed89f6ac20352cec1f127d7
Author: Brian Foster <[email protected]>
Date: Thu Jan 17 13:11:29 2013 -0500

xfs: pull up stack_switch check into xfs_bmapi_write

Quoted 3 lines

The last 3 patches address an AGF buffer hang with the allocation worker.

--Mark.

As 3.4 Kernels don't have thoses patches, is there a risk associated with 3.4 kernels ?

Cheers,

The filesystem can hang but only if the OS cannot allocate another worker when doing certain calls.

--Mark.

Quoted 5 lines · Mark Tinguely

The filesystem can hang but only if the OS cannot allocate another worker when doing certain calls.

--Mark.

Ok, any chance to see these fixes backported to 3.4 ?

Cheers,

Quoted 5 lines · Yann Dupont

Quoted 4 lines · Mark Tinguely

The filesystem can hang but only if the OS cannot allocate another worker when doing certain calls.

Quoted 6 lines

Quoted 1 line

--Mark.

Ok, any chance to see these fixes backported to 3.4 ?

Cheers,

All of the patches apply to Linux 3.4 without modification.

It does not sound like the patches will be pushed back to stable at this time.